Although many successful algorithms have been proposed for visual tracking, it is still a challenging task due to occlusion, scale variation, fast motion, and deformation. To handle these challenges, we propose a collaborative model and focus on three key factors: 1) an effective representation to consider appearance variations; 2) an effective application of the keypoints; and 3) an incorporation of contextual information. In this paper, we propose a novel algorithm that takes into account the three key factors based on complex cells and keypoints. The complex cells can effectively explore the contextual information at multiple scales. Meanwhile, a keypoint is an ideal local representation. Keypoints-based tracking method is used to make coarse tracking. A precise tracking-by-detection whose samples come from keypointsbased tracking is followed by considering the scale information. In addition, measurement of appearance variation is measured by matching the current inner cell with template's individualistically. In the basis of the measurement, an adaptive learning rate parameter is estimated for updating the object appearance model to avoid noises. Experimental results demonstrate that our tracker is able to handle appearance variations and recover from drifts. In conjunction with tracking acceleration modules, the proposed method performs in real time and outperforms favorably many state-of-the-art algorithms for object tracking.