In-loop filtering is an important part of high efficiency video coding (HEVC), which consists of deblocking filter and sample adaptive offset (SAO) filter. It can not only improve the compression efficiency of HEVC, but also improve the visual quality of the reconstructed videos significantly. However, the high computational complexity hampers its applications for real-time encoding scenarios. In this paper, we propose a parallel strategy for in-loop filtering in HEVC encoder on graphics processing unit (GPU). In the proposed strategy, the pipeline structure for HEVC encoding by parallel processing deblocking filter and SAO on GPU is described first. Then, the joint optimization for deblocking filter and SAO on GPU is detailed by parallel processing of deblocking filter and parallel processing of SAO separately. The joint optimization can improve the degree of parallelism and ease the computational burden of the CPU. Experimental results demonstrate that the proposed method can achieve about 47% (up to 67%) time saving on average for test sequences.