Depth information is an important ingredient in many multiview applications including image-based rendering (IBR). With the advent of electronics, low-cost and high-speed depth cameras, such as the Microsoft Kinect, are getting increasingly popular. In this paper, we propose a superpixel-based joint color–depth restoration approach for Kinect depth camera and study its application to view synthesis in IBR systems. Thus, an edge-based matching method is proposed to reduce the color–depth registration errors. Then the Kinect depth map is restored based on probabilistic color–depth superpixels, probabilistic local polynomial regression and joint color–depth matting. The proposed restoration algorithm does not only inpaint the missing data, but also correct and refine the depth map to provide better color–depth consistency. Last but not the least, a dynamic background modeling scheme is proposed to address the disocclusion problem in the view synthesis for dynamic environment. The experimental results show the effectiveness of the proposed algorithm and system.