A layered depth image (LDI) is a new popular representation and rendering method for objects with complex geometries. Similar to a two-dimensional (2-D) image, the LDI consists of an array of pixels. However, unlike the 2-D image, an LDI pixel has depth information, and there are multiple layers at a pixel location. We develop a novel LDI compression algorithm that handles the multiple layers and the depth coding. The algorithm records the number of LDI layers at each pixel location, and compresses LDI color and depth components separately. For LDI layer with sparse pixels, the data is aggregated and then encoded. An empirical rate-distortion model is used to optimally allocate bits among different components. Compared with the benchmark compression tools such as JPEG-2000 and MPEG-4, our scheme improves the compression performance significantly.