Video offers solutions to many of the traditional problems with coach, trainer, commenter, umpires and other security issues of modern team games. This paper presents a novel framework to perform player identification and tracking technique for the sports (Kabaddi) with extending the implementation towards the event handling process which expands the game analysis of the third umpire assessment. In the proposed methodology, video preprocessing has done with Kalman Filtering (KF) technique. Extended Gaussian Mixture Model (EGMM) implemented to detect the object occlusions and player labeling. Morphological operations have given the more genuine results on player detection on the spatial domain by applying the silhouette spot model. Team localization and player tracking has done with Robust Color Table (RCT) model generation to classify each team members. Hough Grid Transformation (HGT) and Region of Interest (RoI) method has applied for background annotation process. Through which each court line tracing and labeling in the half of the court with respect to their state-of-art for foremost event handling process is performed. Extensive experiments have been conducted on real time video samples to meet out the all the challenging aspects. Proposed algorithm tested on both Self Developed Video (SDV) data and Real Time Video (RTV) with dynamic background for the greater tracking accuracy and performance measures in the different state of video samples.