According to the similarity of speech data of the same type and characteristic that data of different types has different geometrical distance, the paper proposed a speaker recognition system based on SVM-GMM. The system combines advantages of GMM and SVM, solves problems that GMM cannot distinguish differences between the voice data while the data is small, and recognition rate of SVM drops while handling large amounts of data. The improved K-Means algorithm is used for initialization of model parameters to improve accuracy. The experiment results show that speaker recognition system based on SVM-GMM has better recognition rate and robustness than the system using GMM or SVM alone.