This paper presents a new image based target tracking design. The state of the target is described in a hybrid state space where the dynamic of the target is indexed by a discrete variable mode or regime, each mode corresponding to a maneuver hypothesis. The novelty here is the use of the imaging capacity of electro-optical sensor to improve the tracking performance by a better regime estimation than with the classical methods. This is achieved by a new image based mode estimator that fully exploit the information contained in the pixels. The coupling of this image based mode estimator to an adaptive EKF and to a multiple model tracker akin to the interacting multiple model filter is described. The results are illustrated with realistic simulations of an air defense scenario. The efficiency and robustness of this new mode estimator lead to a significant improvement of the tracking performance in both cases.