This article studies the mean and mean-square behaviors of the M-estimate based normalized subband adaptive filter algorithm (M-NSAF) with robustness against impulsive noise. Based on the contaminated-Gaussian noise model, the stability condition, transient and steady-state results of the algorithm are formulated analytically. These analysis results help us to better understand the M-NSAF performance in impulsive noise. To further obtain fast convergence and low steady-state estimation error, we derive a variable step size (VSS) M-NSAF algorithm. This VSS scheme is also generalized to the proportionate M-NSAF variant for sparse systems. Computer simulations on the system identification in impulsive noise and the acoustic echo cancellation with double-talk are performed to demonstrate our theoretical analysis and the effectiveness of the proposed algorithms.