This paper proposes a novel real-time robust hand tracking algorithm, integrating multi-cues, and a limb?s degree of freedom. For this purpose, we construct a limb model and maintain the model obtained from KLT-AR methods with respect to second-order auto-regression model and Kanade-Lucas-Tomasi(KLT) features, respectively. Furthermore, this method provides directivity of a target, enabling us to predict the next motion. Thus, we can develop a method of hand tracking for gesture and behavior recognition techniques frequently used in conjunction with Human-Robot Interaction (HRI) components. The experimental results show that the proposed method yields a good performance in the intelligent service robots, so called Wever developed in ETRI.