In this paper we introduce a graph grammar based method to fuse the low level features and apply them to object detecting and tracking. In our algorithm, the graph grammar rules are used to detect the object in the beginning of the video sequence and then dynamically adjust the tracking procedure. Our tracking algorithm consists of two phases: key points tracking and tracking by graph grammar rules. The key points are computed by using salient level set components. All key points, as well as the colors and the tangent directions, are fed to a Kalman filter for object tracking. Then the graph grammar rules are used to dynamically examine and adjust the tracking procedure to make it robust. The effectiveness of the algorithm has been demonstrated by experiments.