In this paper a framework for an airport ground traffic surveillance system based on visual information is presented. The framework is designed to improve state-of-the-art Advanced Surface Movement Guidance and Control Systems (A-SMGCS) adding an image processing module able to act as a “gap-filler” for airport zones that traditional systems fail to reliably cover due to reflections and blind spots. Results of the approach based on a set of videos is presented demonstrating that the proposed framework is a flexible and cost-effective solution for improving existing systems' performance.