Video surveillance systems are a powerful tool applied in various systems. Traditional systems based on human vision are to be avoided due to human errors. An automated surveillance system based on suspicious behavior presents a great challenge to developers. Such detection is a rather complex procedure and also a rather time-consuming one. An abnormal behavior could be identified by: actions, faces, route, etc. The definition of the characteristics of an abnormal behavior still present a big problem. This paper proposes a specific architecture for a surveillance system. The aim is to accelerate the system and obtain a reliable and accelerated suspicious behavior recognition. Finally, the experiment section illustrates the results with comparison of some of the most recent approaches.