We introduce a robust tracking procedure for a regular pattern marked on a flexible moving surface such as cloth. A video of an actor performing a range of motions is processed with our algorithm to yield a dynamic geometric representation. The system is capable of maintaining the tracked grid structure for long periods of time without quality deterioration, and requires minimal user interaction. It has been tested on videos of an actor dressed in a specially marked T-shirt and behaves favorably with the presence of self-occlusions, self-shadowing and folding of the cloth. The focus of this paper is on single camera video sequence processing, even though 3D shape reconstruction with multiple cameras is the motivating goal.