Visual tracking of multiple objects with continuous interactions is important for many applications such as surveillance and effective user interfaces. Complex interactions between objects results in both temporally and spatially significant occlusions, making multi-object tracking a challenging problem. We present an approach for tracking varying number of objects through significant occlusions. Our approach builds on the idea of object permanence. That is, partially or fully occluded objects, even though not observable, still exist in the close proximity of their occluders. To demonstrate the viability of this approach, we have concentrated on tracking hands and objects as a person interacts with multiple objects on a desktop. We experimented with 5 indoor video sequences and present the results of two representative ones: lego sequence and shell game.