In order to solve the defects of ViBe moving object detection algorithm, an improved algorithm based on spatial-temporal gradient has been proposed. According to the complexity and the changes of the background, parameters of discrimination mechanism have been adjusted adaptively. The mean of spatial gradients has been calculated to adjust the detection radius, which is capable of improving the detection accuracy. According to the illumination invariant of the spatial gradient, the light information has been removed, which is able to reduce the disturbance of shadow. The sum of bivariate temporal gradients has been counted so as to adjust the threshold, which can suppress the fallout ratio caused by the changes of background. Based on the temporal gradient of consecutive frames, the background changes have been detected, which achieved the rapid elimination of “ghost”. Through the experiment, the improved algorithm and other algorithms were compared and analyzed in three indicators(FPR, FNR and PWC). The experimental results show that the improved algorithm has a good effect in removing the shadow and eliminating the “ghost”. Meanwhile, it has a better overall performance on several indicators compared with other algorithms.