We propose a novel approach to program a robot by demonstrating the task multiple number of times in front of a binocular vision system. We track artificially-induced features appearing in the image plane due to nonimpedimental color stickers attached at different fingertips and wrist joint, in a simultaneous feature detection and tracking framework. A Kalman filter does the tracking by recursively predicting the tentative feature location and a higher order statistics (HOS)-based data clustering algorithm extracts the feature. A fast and efficient algorithm for the vision system thus developed processes a binocular video sequence to obtain the trajectories and the orientation information of the end effector from the images of a human hand. The concept of trajectory bundle is introduced to avoid singularities and to obtain an optimal path.