Sampling from the lattice Gaussian distribution is emerging as an important problem in coding and cryptography. In this paper, the conventional Gibbs sampling algorithm is demonstrated to be geometrically ergodic in tackling with lattice Gaussian sampling, which means its induced Markov chain converges exponentially fast to the stationary distribution. Moreover, as the exponential convergence rate is dominated by the spectral radius of the forward operator of the Markov chain, a comprehensive analysis is given and we show that the convergence performance can be further enhanced by usages of blocked sampling strategy and choices of selection probabilities.