In this paper we propose a new segment-based stereo matching algorithm using scene hierarchical structure. In particular, we highlight a previously overlooked geometric fact: the most foreground objects can be easily detected by intensity-based cost function and the farer objects can be matched using local occlusion model constructed by former recognized objects. Then the scene structure is achieved from foreground to background. Two occlusion relations are proposed to establish occlusion model and to update cost function. Image segmentation technique is adopted to increase algorithm efficiency and to decrease discontinuity of disparity map. Experiments demonstrate that the performance of our algorithm is among the state of the art stereo algorithms on various data sets.