This paper presents a fusion of the active appearance model (AAM) and the Riemannian elasticity framework which yields a non-linear shape model and a linear texture model ‐ the active elastic appearance model (EAM). The non-linear elasticity shape model is more flexible than the usual linear subspace model, and it is therefore able to capture more complex shape variations. Local rotation and translation invariance are the primary explanation for the additional flexibility. In addition, we introduce global scale invariance into the Riemannian elasticity framework which together with the local translation and rotation invariances eliminate the need for separate pose estimation. The new approach was tested against AAM in three experiments; face labeling, face labeling with poor initialization and corpus callosum segmentation. In all the examples the EAM performed significantly better than AAM. Our Matlab implementation can be downloaded through svn from https://svn.imm.dtu.dk/AAMLab/svn/AAMLab/trunk/