Defect classification is an important part of automated surface defect detection systems. Generally, the types of surface defects of industrial products are complex and diverse, and the correct defect type classification can help to subsequently extract the characteristic of defects. Moreover, defect classification can help to count the number of defect modes automatically and provide data support for the precise maintenance of product production. Due to the large number of surface defects and the small difference between defect types, traditional classification methods are difficult to classify defect accurately. Therefore, in order to improve the accuracy of defect image classification, this paper proposes a defect image classification method based on transfer learning and sparse coding. Firstly, a deep CNN feature extraction algorithm for defect images is proposed in combination with transfer learning. Then, the deep CNN features of the defect image are dimension-reduced and sparsely optimized using the sparse coding techniques, and the sparse CNN features are obtained. Finally, the sparse CNN features are classified to realize the defect type determination using the linear SVM. The accuracy of the proposed method is verified by using a steel surface defect image benchmark database, and the effectiveness of the proposed method is proved.