High Efficiency Video Coding (HEVC) is a new video compression standard for high resolution video content, which only needs 50% bit rate of the H.264/AVC at the same perceptual quality. However, high computational complexity increases dramatically for adopting quad-tree structured Coding Unit (CU). In this paper, a fast CU size decision and Prediction Unit (PU) mode decision algorithm is presented for HEVC intra coding. It exploits the depth information of neighboring CUs to make an early CU split decision or CU pruning decision. Moreover, there are correlations between the higher layer prediction mode and current layer mode in PUs. By using these correlations, we can terminate some prediction modes which are rarely selected as the optimal mode. Experimental results illustrate that the proposed method can save 37.91% computational complexity on average as compared with the current HM with only 0.66% increase in BDBR and 0.03 dB loss in BDPSNR.