The traditional background difference method is based on gray image. Some information is lost when color image is transformed into gray image. So it is difficult to discriminate different colors with similar gray values and easily disturbed by noise and shadows. In this paper, the background difference is based on RGB color model. It is proposed to use the average value of each pixel of the color image sequences to extract the background, and then use the three-dimensional color values of the current frame and background image to compute the difference to detect the moving objects. The proposed approach is simple and easy to implement. The experimental results show that it is more sensitive to colors and has higher accuracy and robustness than the traditional background difference method. Besides, it is more resistant to shadows.