We present a novel object tracking framework that can efficiently track such 2-D affine motions of the object image. A coarse-to-fine tracking strategy is explored in the tracking framework. Firstly, the object image region is selected from the first frame, the scale-invariant feature (SIFT) extracted from the object image region is utilized as the object model. Secondly, the highest likelihood object region is preliminarily located by the particle filter (PF) algorithm in the following video frame, and the SIFT feature points are also extracted from the highest likelihood object region, matching point set between object model and the highest likelihood region is attained by the scale-invariant feature transform, then the corresponding homography matrix is robustly estimated through the random sample consensus (RANSAC) algorithm. Finally, the affine parameters which determine the location and pose of the object are obtained. Simulation results demonstrate the efficiency of our approach.