In this paper, we deal with the problem of real-time detection, recognition and tracking of moving objects in open and unknown environments using an infrared (IR) and visible vision system. A thermo-camera and two stereo visible-cameras synchronized are used to acquire multi-source information: three-dimensional data about target geometry and its thermal information are combined to improve the robustness of the tracking procedure. Firstly, target detection is performed by extracting its characteristic features from the images and then by storing the computed parameters on a specific database; secondly, the tracking task is carried on using two different computational approaches. A Hierarchical Artificial Neural Network (HANN) is used during active tracking for the recognition of the actual target, while, when partial occlusions or masking occur, a database retrieval method is used to support the search of the correct target followed. A prototype has been tested on case studies regarding the identification and tracking of animals moving at night in an open environment, and the surveillance of known scenes for unauthorized access control.