This paper presents a novel approach to automatic shadow identification and removal from video input. Based on the observation that the length and position of a shadow changes linearly over a relatively long period in outdoor environments, due to the relative movement of the sun, we can distinguish a shadow from other dark regions in an input video. Subsequently, we can identify the Reference Shadow as that with the highest confidence of the aforementioned linear changes. This Reference Shadow is used to fit the shadow-free invariant model, with which the shadow-free invariant images can be computed for all frames in the input video. Our method does not require camera calibration and shadows from stationary objects, as moving objects are detected automatically.