Security is a fundamental word/issue in today's world. Nowadays, many surveillances, CCTV cameras, and other types of monitoring cameras are used in various major and minor ways. Visual tracking by human beings is becoming a complex job as there are many ways to simultaneously inflow data from various sources. In this project, we are trying to eliminate that constraint by using an intelligent security surveillance system using computer vision. Here, we are going to detect the activity going on in a suspected video and going to predict the action involved in it. The suspect video is selected when an object's movement is detected. We are using the UCF-50 dataset. It contains different categories of videos which were extracted from YouTube. There are 50 categories of videos and within each video category there are 25 groups. If two videos are in same group then they have similar view point of camera and some other common features like same background and same person, etc. We trained this model using this UCF-50 dataset. Here the model works by breaking the video into each frame, then classifying it, and then predicting the activity of the video by classifying each frame by CNN and then predicting using LSTM.