Defects that appear on a leather surface may be the result of natural variations or poor handling during the manufacturing process. Visual inspection in the factory is one of the essential steps in the process of quality assurance. It should be done before the finished products are being dispatched to the customer. Thus far, the detection of the leather defects is still carried out manually, which is labour intensive, tedious, and might be liable to human error. Therefore, in this paper, we propose an automatic leather defect localization and detection system by employing a series of digital image processing methods based on deep learning. Succinctly, a convolutional neural network (CNN) is utilized to perform the detection task, that is to determine the presence of the defect on a leather patch. Then, the detected defective leather patch is processed to the localization operation, which is to identify the boundary coordinates in pixel level. For the detection task, the result achieved using AlexNet as the feature descriptor and SVM as the classifier is 100%. For the localization stage, we have demonstrated that the instance segmentation technique, Faster R-CNN outperforms the YOLOv2 by obtaining the Intersection over Union (IoU) of 73%. In addition, extensive experiments and comparisons of the state-of-the-art approaches are presented to verify the effectiveness of the proposed algorithms.