Previous works on image-based rendering (IBR) suggests that there is a tradeoff between the number of images and the amount of geometry required for anti-aliased rendering. However, to reconstruct the necessary geometry information from images is not trivial. Even for latest computer vision techniques, the quality is still not good enough for image based rendering. One major problem is that they lack the ability to properly recognize the occlusion boundaries and group the object at different depths. In this thesis, we introduce an interactive image based modeling framework, rendering driven image based modeling, which leverages the vision algorithms with human's interaction. It reconstructs the geometry in an iterative way. The criterion for the reconstruction is to improve the rendering quality. Through a well designed user interface, the user can supervise the reconstruction process and specify certain information to computer, such as grouping of pixels, errors, or confirmation of correctness. One of our essential observations is that the alpha matting along occluding boundary is very important for anti-aliasing rendering, while the geometry error inside opaque object is not as important. Therefore, we introduce the coherent layer representation to rectify the boundary location and extraction. To correctly separate the foreground colors and transparencies for the coherent layers, we also designed coherent matting, an enhanced Bayesian matting algorithm. This system, also known as pop-up light field, is a practical interactive system for sparse lightfield modeling and rendering. In order to save user's interaction time, we also designed an interactive image segmentation tool, called Lazy Snapping. It separates the segmentation process into two steps, from coarse scale to fine detail. We deliberately designed two user interfaces for the two tasks. One is a coarse but quick marking process, and another is a polygon editing UI for boundary refinement. Moreover, Lazy Snapping provides instant visual feedback due to ...