Background subtraction is a commonly used method to detect moving objects from videos captured by static cameras. However, shadows and reflections significantly affect the output of background subtraction algorithms, and distort the shape of the objects obtained as a result. Thus, shadow detection and removal is a crucial post-processing step to perform accurate object tracking required by different applications. We present a lightweight method to detect and remove shadows as well as reflection effects in indoor and outdoor environments by using spatial and spectral features. This method incorporates an adaptive way to set thresholds to avoid preset numbers. We present a comparison of the outputs we obtained with those of several other methods. The experimental results demonstrate the success of the proposed algorithm.