In this paper, we present a novel generalized Segment-Forest Model (SFM) to segment an object as well as label all the object's semantic parts simultaneously. Segment-Forest is composed by various generated segment trees that act directly on super pixels. Unlike recent works, SFM does not need the prior information like skeleton to capture the core structure of an object, but actively learns the structure from semantic parts during the learning stage. The prediction is an Inference-On-Tree-Voting-On-Forest process, which precludes the expensive computation of the multi-objective optimization. Both training and inference in SFM are extremely efficient, especially compared with traditional multi-objective optimization approaches using in segmentation. We demonstrate superior performance particularly in object articulation, non-rigid deformation on two standard datasets over current state-of-the-art methods. Through further occlusion experiments, we show that SFM can work robustly in the real world.