Humans have the capability to quickly prioritize external visual stimuli and localize their most interest in a scene. Inspired by this mechanism, we propose a robust object tracking algorithm based on visual attention. We fuse motion feature and color feature to estimate the target state under the guidance of saliency map. Principal Component Analysis method is used to compute saliency feature based on the dense appearance model generated from the background templates. Motion feature is extracted by using the method which is a Bayesian decision rule for classification of background and foreground. Numerous experiments demonstrate the proposed method performs well against state-of-the-art tracking methods when dealing with illumination change, pose variation, occlusion, and background clutter situations.