Face recognition is a difficult problem, whether using still images or video. A robust solution is still elusive after 30 years of research. The main reason postulated for this is that two people look more alike than images of the same person under different viewing conditions, i.e. the inter-class variability is smaller than the intra-class variability. In this paper, we propose a way to investigate this, and other, phenomenon more quantitatively. This is done by exploring the space of face images. We first synthesize images under different illumination and pose, and then estimate the probability density function (pdf) for each person. The pdfs are then analyzed for their separability, and for where they overlap. Class regions, regions where the Bayes' classifier would correctly classify each person, are also determined. These class regions are subjected to k-means clustering. By examining cluster boundaries, we can determine lighting and pose conditions that make face recognition difficult. Similarly, the cluster centers tell us the viewing conditions most suited for discriminating between the persons. Our paper makes three key contributions: (1) we show how face space may be modeled and explored; (2) we show that the traditional inter-class/intra-class variability is not a good measure of the separability of two classes, and instead propose the use of the Bhattacharyya distance, and (3) we determine the viewing conditions that are best (or worst) for face recognition.