With the increasing focus on safety and security in public areas, anomaly detection in video surveillance systems has become increasingly more important. In this paper, we describe a method that models the temporal behavior and detects behavioral anomalies in the scene using probabilistic graphical models. The Coupled Hidden Markov Model (CHMM) method that we use shows that sparse features obtained via feature detection and description algorithms are suitable for modeling the temporal behavior patterns and performing global anomaly detection. We model the scene using these features, perform perspective independent velocity analysis for anomaly detection purposes and demonstrate the results obtained on UCSD pedestrian walkway dataset. The training is unsupervised and does not require any data having anomaly. This eliminates the need to obtain anomaly data and to define anomalies in advance.