The face pattern is described by pairs of template-based histogram and Fisher projection orientation under the paradigm of AdaBoost learning in this paper. We assume that a set of templates are available first. To avoid making strong assumptions about distributional structure while still retaining good properties for estimation, the classical statistical model, histogram, is used to summarize the response of each template. By introducing a novel “integral histogram image”, we can compute histograms rapidly. Then we turn to Fisher linear discriminant for each template to project histograms from d–dimensional to one-dimensional subspace. Best features, used to describe face pattern, are selected by AdaBoost learning. The results of preliminary experiments demonstrate that the selected features are much more powerful to represent the face pattern than the simple rectangle features used by Viola and Jones and some variants.