Human action in the image sequence can be seen as the relation of the movement of body parts. Since, human has an articulated body, each body part cannot move freely. In each action, the specific directions of body parts arrangement cause a change in posture and movement over time. In this paper, the image oriented gradient and histogram of motion direction are used to describe the posture and movement information in a bounding box. We use distances between regions in a bounding box which contains the whole body to find the relations of the oriented gradient and motion direction over time. The cosine distance is used to measure the similarity of direction histograms. These features are combined and concatenated with the previous frames to construct a feature vector. Then, K-nearest neighbor is used to classify actions in frame by frame. We test the system performance with a human action dataset and a dataset from a surveillance camera. The system can classify action in human action dataset for each frame and can detect the anomaly from the surveillance dataset.