An algorithm was proposed to remove the moving shadow in video surveillance. This method gets the moving objects from motion detection in first, in which removal shadow on invariable features of colors and local texture, then gets rid of the change suddenly pixels in edge of shadow. Experiments show that having a strong ability to adapt to different scenes, the algorithm retains the integrity of the foreground., by combining invariable color feature and local change value in pixels.