This paper proposes a novel real-time hand tracking algorithm in the presence of occlusion. For this purpose, we construct a limb model and maintain the model obtained from ARKLT methods with respect to second-order auto-regression model and Kanade-Lucas-Tomasi(KLT) features, respectively. Furthermore, this method do not require to categorize types of superimposed hand motion based on directivity obtained by the slope’s direction of KLT regression. Thus, we can develop a method of hand tracking for gesture and activity recognition techniques frequently used in conjunction with Human-Robot Interaction (HRI) components.