Multiple Sclerosis is one of the predominant occurring brain diseases during the last decade. Many automatic segmentation models have been widely investigated with objective to identify the disease at the early stage and thereby helping the patients. In this research, a Multivariate Asymmetric Gaussian Mixture Model is considered for segmenting the brain MRI images and to identify the Multiple Sclerosis disease from the T1 weighted, T2 weighted and photon density images. The earlier works on GMM failed to converge to arrive at optimal value and thus becomes time consuming and also could not able to achieve the expected results. To overcome these disadvantages, this research study aims at proposing a model based on Multivariate Asymmetric Gaussian Mixture Model. The images are compressed by using Non Negative factorization method and the decomposed images are considered for further evaluation. The results of the proposed methods are tested on brain web images and in most of the cases; the recognition accuracy is underlined at above 85%. This shows that the performance of the proposed method is far better than the existing methods and also this methodology is very much time efficient.