Nowadays novelty detection methods based on one-class classification are widely used for many important applications associated with computer and information security. In these areas, there is a need to detect anomalies in complex high-dimensional data. An effective approach for analyzing such data uses kernels that map the input feature space into a reproducing kernel Hilbert space (RKHS) for further outlier detection. The most popular methods of this type are support vector clustering (SVC) and kernel principle component analysis (KPCA). However, they have some drawbacks related to the shape and the position of contours they build in the RKHS. To overcome the disadvantages a new algorithm based on fuzzy clustering with Mahalanobis distance in the RKHS is proposed in this paper. Unlike SVC and KPCA it simultaneously builds elliptic contours and finds optimal center in the RKHS. The proposed method outperforms SVC and KPCA in such important security related problems as user authentication based on keystroke dynamics and detecting online extremist information on web forums.