Histograms have been widely used for feature representation in image and video content analysis. However, due to the orderless nature of the summarization process, histograms generally lack spatial information. This may degrade their discrimination capability in visual classification tasks. Although there have been several research attempts to encode spatial context into histograms, how to extend the encodings to higher order spatial context is still an open problem. In this paper,we propose a general histogram contextualization method to encode efficiently higher order spatial context. The method is based on the cooccurrence of local visual homogeneity patterns and hence is able to generate more discriminative histogram representations while remaining compact and robust. Moreover, we also investigate how to extend the histogram contextualization to multiple modalities of context. It is shown that the proposed method can be naturally extended to combine both temporal and spatial context and facilitate video content analysis. In addition, a method to combine cross-feature context with spatial context via the technique of random forest is also introduced in this paper. Comprehensive experiments on face image classification and human activity recognition tasks demonstrate the superiority of the proposed histogram contextualization method compared with the existing encoding methods.