In free viewpoint video (FVV) and 3DTV, the depth image-based rendering method has been put forward for rendering virtual view video based on multi-view video plus depth (MVD) format. However, the projection with slightly different perspective turns the covered background regions into hole regions in the rendered video. This paper presents a depth enhanced image summarization generation model for the hole-filling via exploiting the texture fidelity and the geometry consistency between the hole and the remaining nearby regions. The texture fidelity and the geometry consistency are enhanced by drawing texture details and pixel-wise depth information into the energy cost of similarity measure correspondingly. The proposed approach offers significant improvement in terms of 0.2dB PSNR gain, 0.06 SSIM gain and subjective quality enhancement for the hole-filling images in virtual viewpoint video.