While the depth maps of 3D video are represented as luminance images, they are used to aid rendering of novel views and are not viewed by an end user. Therefore, metrics that measure the quality of images that are for end user viewing does not necessarily reflect the quality of depth maps in terms of its ability to render views. This paper investigates the relationship between the quality of the rendered views and different quality measures of the depth map. A novel depth map quality metric is proposed based on a distortion model that approximates rendering errors due to pixel errors in the depth map. The proposed depth map quality metric correlates very well with the quality of the rendered views, as compared to the PSNR and SSIM of the depth map. The application of the proposed depth map quality metric is further illustrated by incorporating the metric at the encoding mode selection stage of a video encoder. Experimental results suggest that with the proposed encoding mode selection scheme bit rate savings of up to 30% can be achieved compared to traditional encoding mode selection scheme based on sum of squared errors.