Markerless vision-based 3D hand motion tracking is a key and popular component for interaction studies in many domains such as virtual reality and natural human-computer interfaces. While this research field has been well studied in the last decades, most approaches have considered the human hand in isolation and not in action or in interaction with the environment or the other articulated human body parts. Employing contextual information about the surrounding environment (e.g. the shape, the texture, and the posture of the object in the hand) can remarkably constrain the tracking problem. The goal of this survey is to develop an up-to-date taxonomy of existing vision-based hand tracking methods with a new classification scheme: hand-object constraints. This taxonomy allows us to examine the strengths and weaknesses of the current state of the art and to highlight future trends in the domain.