Abstract: Detecting the behavior of crowd in real life scenario is still a challenging task for researchers. In the last two decades, several methods have been reported by researchers to overcome this problem. Most of them are based on only low semantic features like gray value, motion estimation, and gradient information. These features are not sufficient to provide discriminative information of the crowded scenes. In this paper, we proposed a framework based on optical flow approach, which incorporates heat-map generation, the point of interest extraction segmentation and feature extraction. It follows Bag of Feature (BoF) classification model to catch abnormalities in crowded scenes. This frame work is applied on three scenario of UMN (University of Minnesota) dataset which outperforms other state-of-the-art methods in terms of classification accuracy. The average classification accuracy got from the experiment is 99.6%. KEYWORDS: Motion estimation, Heat-map, optical flow, points of interest extraction, Bag of feature.