Insulation defects that occur in gas-insulated switchgear (GIS), which is one of the most important types of equipment in the power grid, can lead to serious accidents. The ultrasonic detection method is commonly used to detect partial discharge (PD) signals in power equipment to discover defects. However, the traditional method to diagnose defects in GIS with ultrasonic PD signals is still based on the experience of testers. In this study, a classification system was proposed to identify insulation defects of GIS, based on voiceprint recognition technology. Twelve coefficients from mel frequency cepstral coefficient (MFCC) and 24 delta MFCC features were extracted as the acoustic features of the system. A support vector machine (SVM) multi-classifier was constructed to perform the classification and the sequential minimal optimization (SMO) algorithm was used to optimize the computational efficiency of the SVM. The experiments were conducted on a 110 kV GIS with different kinds of insulation defects. The results verified that the classification system with SMO-SVM achieved better identification accuracy and efficiency than the system with SVM. Therefore, it reveals the feasibility of the system to realize identification of insulation defects in GIS automatically and accurately.