The universal background model represents the speaker independent distribution of features, so it can be used to construct a reference space for speaker recognition. In the anchor models, one speaker utterance can be located at one point in the anchor space. We construct the reference space using the Gaussian distributions in the universal background model instead of the virtual speakers in the anchor models, and one speaker's all utterances mainly locate in a small portion of the whole space. On the other hand, we use the support vector machine to separate this speaker dependent portion from the whole space while the Euclidean distance measure is used in the anchor models in general. The experiments on the YOHO database show that our method can get better performance comparing with the decision fashion based on the Euclidean distance which is widely used in the anchor models.