The Least Mean Kurtosis (LMK) algorithm was initially proposed as an adaptive algorithm that is robust to the observation noise distribution. Good performances of this algorithm have been shown for non-Gaussian additive measurement noise. However, the complexity of the algorithm imposes difficulties for the development of a reasonably complete theoretical stochastic model for its behavior. The purpose of this paper is to contribute to the development of such a model. We study the stochastic behavior of Least Mean Kurtosis (LMK) algorithm for Gaussian inputs and for additive noises with even probability density functions. Deterministic recursions are derived for the adaptive weight error covariance matrix in a very novel manner, leading to a recursive model for the excess mean square error (EMSE) behavior that is shown to be accurate for Gaussian, uniform and binary noise distributions. The analysis results are then used to compare the performances of LMK with the least mean squares (LMS) and least mean fourth (LMF) algorithms under different circumstances.