High Efficiency Video Coding(HEVC) obtains high compression ratio by applying recursive quad-tree structured coding unit(CU). However, this recursive quad-tree structure brings very high computational complexity to HEVC encoder. In this paper, we present fast CU decision algorithm in recursive quad-tree structure. The proposed algorithm estimates initial CU size before CTU encoding and checks the proposed condition using Coded Block Flag(CBF) and Rate-distortion cost to achieve the fast encoding time saving. And, intra mode estimation is also possible to be skipped using the CBF values acquired during the inter PU mode estimations. Experiment results shows that the proposed algorithm saved about 49.91% and 37.97% of encoding time according to the weighting condition.