In this paper, a conceptual solution and preliminary results regarding the integrated multi-sensory on-board obstacle detection (OD) system are presented. The presented system is under development within the project “SMART-SMart Automation of Rail Transport”, which is funded under the H2020-Shift2Rail-RIA funding scheme. The proposed SMART conceptual solution will provide prototype hardware and software algorithms for OD. The system will combine different vision technologies: thermal camera, night vision sensor (camera augmented with image intensifier), multi stereo-vision system and laser scanner in order to create a sensor fusion system for mid (up to 200 m) and long range (up to 1000 m) obstacle detection, which is independent of light and weather conditions. The overall requirements for the integrated multi-sensory object recognition system are described, and preliminary results of processing of individual sensor data are given.