In this paper we suggest a new variational Bayesian approach. Variational Expectation-Maximization (VEM) algorithm is proposed in order to estimate a set of hyperparameters modelling distributions of parameters characterizing mixtures of Gaussians. We consider maximum log-likelihood (ML) estimation for the initialization of the hyperparameters. The ML estimation is employed on distributions of parameters obtained from successive runs of the EM algorithm on the same data set. The proposed algorithm is used for unsupervised detection of quadrature amplitude and phase-shift-key modulated signals.