Activity recognition and tracking of animals in video sequences is the main objective of this work. Each video is divided into 200 frames. Histogram of Oriented Optical flow (HOOF) features of size 32 bins are calculated for each of these frames and merged for a particular action .K-means clustering technique is being applied on these merged feature of size 2000x32 to find out 100 cluster centers each of size 32. Distance calculation of HOOF feature for each video sequence of 200 frames with cluster centers is being carried out to find the closest cluster centre for a particular action. Feature vector of size 200 is generated for each video sequence based on the minimum total distance of all the frames of a video with the cluster centers. This feature vector is used to train Support vector machine and K-Nearest neighbors algorithm for recognition of four different actions Running, walking, jumping and resting for a particular animal and the result of various algorithms involved here are compared based on their performance.