This paper proposes the texture-based coding unit (CU) size decision (TBCUD) algorithm that includes texture-based block partitioning. While the existing algorithm calculates every CU size, the proposed algorithm only considers the determined CU sizes that are decided by the texture of the image. Because the prediction is performed by the prediction unit (PU) derived from CU, the proposed algorithm can reduce the encoding time by skipping the unnecessary computation. Experimental results show that the proposed algorithm can reduce the average encoding time by 47.21 % at a cost of 2.55 % bit rate increase compared to the high efficiency video coding (HEVC) Test Model (HM) 14.0.