As maritime smuggling is being combatted more effectively, the criminal “modus operandi” consists more frequently of using small aircraft and drones for drug transport. To address this issue, we report our efforts to develop a system capable of accurately tracking suspicious flying objects and identifying them on video streams. Our solution consists in coupling classical computer vision with deep learning to perform tracking and object detection. A discrete Kalman filter is used to predict the location of each object being tracked while the Hungarian algorithm is used to match objects between successive frames. Whenever a potential target is considered suspicious the input images are zoomed and fed into a deep learning pipeline that separates images into the classes aircraft, drones, birds or clouds. A literature survey indicates that this problem with important applications is yet to be fully explored.