This paper proposes to define moving objects in video scene in terms of MPEG like motion vectors. Local Areas in a video scene where large magnitude of motion vectors is detected are regarded to contain a moving object(s). Macro blocks indicating a large motion vector are segmented from the frame of video image. In order to identify the moving object with information relevant to its geometric shape, and to track the moving object, the segmented parts of video image are further processed to find salient corners with the SUSAN edge/corner detector. Local minima of the output from the SUSAN algorithm are the representative feature points of the moving object. For the motion tracking applicable to video surveillance, correspondence between the frame to frame change of those feature points is calculated by the Scott and Longuet-Higgins algorithm, which provides a direct way of associating features of two arbitrary patterns consisting of feature points.