Since the goal of Active Appearance Model (AAM) is to minimize the residual error between the model appearance and the input image, it often fails to converge accurately to the landmark points of the input image. To alleviate this weakness, we have combined Active Shape Model (ASM) into AAM, where ASM tries to find correct landmark points using the local profile model. Because the original objective function and search scheme of the ASM is not appropriate for combining these methods, we modified the objective function of the ASM and proposed a new objective function that combining that of two methods. The proposed objective function can be optimized using a gradient based algorithm as in the AAM. Experimental results show that the proposed method reduces the average fitting error when compared with existing fitting methods such as ASM, AAM, and Texture Constrained-ASM (TC-ASM).