In this paper, we propose a unified optimization framework for feature extraction that lets us simultaneously take into account image data and semantic knowledge: We model objects using a language that specifies both photometric and geometric constraints and defines an information-theoretic objective function that measures the fit of the models to the data. We then treat the problem of finding objects as one of generating the optimal description of the image in terms of this language.