Neighborhood Preserving Embedding (NPE) effectively preserves the geometry of high dimensional data. But, it fails to discover the variation of the values among nearby data, which characterizes the most important modes of variability of patterns. In this paper, we introduce a linear approach, called joint geometry and variability analysis (JGVA), which explicitly considers the geometry and modes of variability of patterns. To be specific, we model the geometrical structure and variability of the local neighborhoods by constructing two adjacency graphs over the training data, and then incorporate the geometry and variability into the objective function of dimensionality reduction. Experiments on four real-world image databases show the effectiveness of the proposed approach.