In this paper, we present an image-based method to recover a geometric proxy and generate novel views. We use an integrated modeling and rendering approach to deal with the difficulty of modeling, and reduce the sampling rate. Our system is based on two novel techniques. First, we propose the Adaptive Mesh Segmentation (AMS) technique for recovering geometric proxy of a scene environment. Second, we propose the Trifocal Morphing technique for efficient rendering with the geometric proxy, which can handle non-matched regions of the scene. Our method allows images to be sparsely captured and thus highly reduces the manual image acquisition effort as well as the data size.