Understanding activities arising out of the interactions of a configuration of moving objects is an important problem in video understanding, with applications in surveillance and monitoring. A special situation is when the objects are small enough to be represented as points on a 2D plane. In this paper, we introduce a novel method of representing the activity by the deformations of the point configuration in a properly defined shape space. Instead of inferring about the activity directly from the motion tracks of the individual points, we propose to model an activity by the polygonal shape formed by joining the locations of these point masses at any time t, and its deformation as the activity unfolds. Given the locations of the 2D points over a sequence of frames in the video, the factorization theorem for matrices is used to obtain a set of basis shapes for each activity. An unknown activity can now be recognized by projecting onto these basis shapes. Also, once a specific activity is recognized, the deviations from it can be modeled by the deformations from the basis shape. This is used to identify an abnormal activity. We demonstrate the applicability of our algorithm using real-life video sequences in an airport surveillance environment. We are able to identify the major activities that take place in that setting and detect abnormal ones.