The depth image obtained by Kinect usually exists cavities at the edge of the object and occlusions. For this problem, many scholars have put forward many solutions. The existing cavity filling algorithms can repair the holes effectively, but these algorithms have a long execution time and are difficult to apply in practical applications. In order to solve the problems above, a hole filling algorithm for Kinect depth image is proposed. Firstly, the color and depth images acquired by Kinect are aligned. Secondly, small cavities will be filled with a pixel filtering method. Thirdly, an improved fast joint bilateral filtering algorithm is designed to fill the missing data in the large cavities. The experimental results show that the method can well repair the hole in the original depth image, and it has high real-time performance.