In this work we present a new approach to learn, detect and predict unusual and abnormal behaviors of people, groups and vehicles in real-time. The proposed OBSERVER video surveillance system acquires images from a stationary color video camera and applies state-of-the-art algorithms to segment and track moving objects. The segmentation is based in a background subtraction algorithm with cast shadows, highlights and ghost’s detection and removal. To robustly track objects in the scene, a technique based on appearance models was used. The OBSERVER is capable of identifying three types of behaviors (normal, unusual and abnormal actions). This achievement was possible due to the novel N-ary tree classifier, which was successfully tested on synthetic data.