Abstract Automatic face analysis, including head pose estimation, gender recognition, and expression classification, strongly benefits from an accurate segmentation of the human face. In this paper we present a multi-feature framework which first segments a face image into six parts, and then performs classification tasks on head pose, gender, and expression. Segmentation is achieved by training a discriminative model on a manually labeled face database, namely FASSEG , which we extend from previous versions, and which we publicly share. Three kinds of features accounting for location, shape, and color are extracted from uniformly sampled square image patches. Facial images are then pixel-wise segmented into six semantic classes – hair, skin, nose, eyes, mouth, and background, – using a Random Forest classifier (RF). Then a linear Support Vector Machine (SVM) is trained for each face analysis task i.e., head pose estimation, gender recognition, and expression classification by using the probability maps obtained during the segmentation step. Performance of the proposed framework is evaluated on four face databases, namely Pointing’04, FEI, FERET, and MPI, with results which outperform the current state-of-the-art.