Automatic human lip tracking is one of the key components tomany facial image analysis tasks, such as, lip-reading and emotion from lips. It has been a classical hard image analysis problem over decades. In this paper, we propose an indirect lip tracking strategy: `lipless tracking'. It is based on the observation that many of us don't have clear lips and some even don't have visible lips. The strategy is to select and localize stable lip features around the mouth for tracking. For this purpose deformable contour-segments are modelled based on lip features and tracking is done using dynamic programming and Viterbi Algorithm. The strength of proposed algorithm is demonstrated in emotion estimation domain. Finally, real-time video experiments performed on private and publicity available data set (MMI face database) have shown the robustness of our proposed lipless tracking technique.