A stereo mapping system developed for a prototype lunar rover is presented in the paper. All the components, including camera calibration, image rectification, stereo matching and reconstruction, are implemented to build a model of the natural terrain. For camera calibration, an iterative calibration algorithm that accounts for both radial and tangential distortion is presented. In each step a closed- form solution is arrived at and the reconstruction results validate its property in improving calibration precision. After image rectification, we utilized a geometry constraint in a multi-stage stereo matching algorithm, which enables the system to process stereo image pairs of various types of natural terrain successfully. Finally we are able to construct the quantitative models of the terrain around the rover, which will be used to detect obstacles and plan path. We have tested the system in laboratory and the preliminary experimental results indicate that the precision and robustness of the system is acceptable to date.