The objective of Video based Emotion Recognition System is to recognize the emotions such as normal, happy and anger from the facial expression images in the video. The face and mouth regions are detected using Viola Jones algorithm for each frame in the video. Gray Level Co-occurrence Matrix (GLCM) and Local Binary Pattern (LBP) features are extracted from the mouth region and it is used to train the Support Vector Machine. The Support Vector Machine is used to classify the emotion in the test video. Experimental results show that GLCM and LBP based emotion recognition system archives an accuracy of % and %, respectively.