In this paper, we present a method using pixel-level information, local region-level information and global-level information to remove shadow. At the pixel-level, we employ GMM to model the behavior of cast shadow for every pixel in the HSV color space, as it can deal with complex illumination conditions. However, unlike the GMM for background which can obtain sample every frame, this model for shadow needs more frames to get the same number of sample, because shadow may not appear at the same pixel for each frame. Therefore, it will take a long time to converge. To overcome this drawback, we use the local region-level information to get more samples and global-level information to improve a preclassifier and then, by using it, we get samples which are more likely to be shadow. Also, at the local region-level, we use Markov random fields to represent dependencies between the label of single pixel and labels of its neighborhood. Moreover, to make global level information more robust, tracking information is used. Experimental results show that the proposed method is efficient and robust.