Tracking by individual features, such as color or motion, is the main reason why most tracking algorithms are not as robust as expected. In order to better describe the object, multi-feature fusion is very necessary. In this paper we introduce a graph grammar based method to fuse the low level features and apply them to object tracking. Our tracking algorithm consists of two phases: key point tracking and tracking by graph grammar rules. The key points are computed using salient level set components. All key points, as well as the colors and the tangent directions, are fed to a Kalman filter for object tracking. Then the graph grammar rules are used to dynamically examine and adjust the tracking procedure to make it robust.