Ahstract- For autonomous localization and navigation, a robot's ego-motion estimation is fundamental. RGB camera-based visual odometry (VO) has proven to be a robust technique used to determine a robot's motion. In situations when direct sunlight, the absence of light or presence of dust as well as smoke make vision difficult, RGB cameras may not provide a sufficient number of RGB features for an accurate and robust visual odometry. In contrast to the visual spectrum of light, imaging modalities like thermal cameras can still be used to identify a stable but small number of image features in the described situations. Unfortunately, the smaller number of image features results in a less accurate VO. In this paper, we present an approach to monocular visual odometry using multimodal image features of different imaging modalities as RGB, thermal and hyperspectral images. By using the strengths of various imaging modalities, the robustness and accuracy of VO can be drastically increased compared to traditional unimodal approaches. The presented method merges different motion hypotheses based on various imaging modalities to create a more accurate motion estimation as well as a map of multimodal image features. The uni- and multimodal motion estimations are evaluated regarding the absolute and relative trajectory errors. The results show that our multimodal approach works robustly in presence of partial sensor failures still creating a multimodal map containing image features of all modalities.