In this paper, we present an image-based robot incremental localization algorithm which uses a panoramic image-based map enhanced with depth from a laser range finder. The image-based map (model) contains both intensity information as well as sparse 3D geometric features. By assuming motion continuity, a robot can use the depth information in the image-model to project the relevant 3D model features, specifically vertical lines, of the environment to its camera coordinate frame. To determine its location, the robot first acquires an intensity image and then matches the 2D geometric features in the image with the projected model features. The first contribution of this research is that we avoid the difficult problem of full 3D reconstruction from images by employing a range sensor registered with respect to the intensity image sensor; secondly, we provide an algorithm that performs incremental robot localization using only 2D images. Experimental results in indoor map building and localization demonstrate the feasibility of our approach and evaluate the performance of the algorithm.