We propose an efficient algorithm using layered holographic stereogram for three-dimensional (3D) computer-generated holograms. The hologram is spatially partitioned into multiple holographic elements (hogels) to provide the occlusion effect and motion parallax by use of multiple viewpoint rendering. Each hogel is calculated with inverse Fresnel diffraction by slicing the viewing frustum according to the depth image. The sliced layers can provide accurate depth cues for reconstruction since the geometric information of the 3D scene is faithfully matched. The algorithm is compatible with computer graphics rendering techniques and robust for holograms with different parameters. When the hogel size equals 1 mm, the signal-to-noise ratio of the diffraction calculation is above 39 dB with a propagation distance longer than 10 mm. Numerical simulations and optical experiments have demonstrated that the proposed method can reconstruct quality 3D images with reduced computational load.