Video quality estimation is considered a means of monitoring quality of service in broadcasting or IPTV services. In this paper, a no-reference peak signal-to-noise ratio (PSNR) estimation method is first presented for a quadtree-based motion estimation or compensation and transform coding scheme such as HEVC test model (HM), which is expected to be popularly used due to its highly enhanced coding efficiency, in 2-D and 3-D high resolution videos. The proposed no-reference PSNR estimation method is based on a Laplacian mixture distribution, which takes into account the distribution characteristics of residual transform coefficients in different quadtree depths and coding types of coding units (CUs). In order to predict the model parameters of the Laplacian mixture distribution for all zero quantized coefficients case, an exponential regression scheme is employed over quadtree depth levels of CUs. The proposed no-reference PSNR estimation method yields fairly accurate results from 0.970 to 0.983 in correlation and from 0.530 to 0.890 in RMSE between the actual and the estimated PSNR values for HM encoded bitstreams, outperforming single PDF based models.