In this paper, we describe a new method for shape from focus using a set of the color segmentation and the bilateral filtering. The shape from focus estimates the geometrical shape of objects from the defocus level of images taken with different focal state. However, the conventional methods tend to fail in the shape estimation around the edge parts of objects and thin texture areas. To address the problems, we use the color segmentation in order to merge such areas by color, and then we interpolate the shape of lost part at each segment. The bilateral smoothing filter is used as the noise suppressor to improve the segmentation accuracy and also this bilateral filter is used to obtain the detail of textures, in other words, high frequency components of the focus measure. In addition, we also acquire the All-in-Focus image by this algorithm and the depth estimation is complemented by this image. By using this method, the lossless and noiseless shape of the object is obtained