This chapter deals with the problem of processing and analyzing digital images of ancient or degraded documents to increase the possibilities of inferring their structures. Classification and recognition are needed to infer structure but, when dealing with degraded documents, they are particularly difficult to apply directly to unprocessed images. This is why an intermediate step is needed that extracts automatically the "perceptual components" of the documents from their appearance. By "appearance" of a document, we mean the "raw" data set, containing the "sensorial components" of the object under study. Ancient documents of historical importance pose specific problems that are now being solved with the help of information technology. As much information as possible should be drawn from the physical documents and should be structured so as to permit specialized searches to be performed in large databases. The tools we use to treat unstructured, low-level information are both mathematical and descriptive. Under a mathematical point of view, we model our appearance as a function of all the perceptual components, or patterns we want to identify. Once the model has been established, its parameters can be learned from the data available and from reasonable assumptions on both the model itself and the patterns. Our descriptive tools form a specialized metadata schema that can help both the storage and the indexing of all the digital objects produced to represent the original document, and provides a complete description of all the processing performed. Suitable links fully interconnect the various descriptions in order to relate the different representations of the physical object and to trace the history of all the processing performed. Inferring structure is much easier by analyzing the patterns and their mutual relationships than by analyzing the appearance.