In this paper we propose a new methodology to improve the performance of classifiers on relatively difficult classification problems with complex boundaries between classes, overlapping classes, and a lack of sufficient number of samples for some classes. We investigate the use of contextual information to overcome such problems, especially in the case of class overlapping, high number of classes and high dimensional feature spaces. The proposed methodology assumes that contextual information is available and that it can be used to disambiguate overlapping classes. In fact, the contextual information is used to reduce the number of classes as well as at the design of the classifier. This new methodology was applied to the problem of unconstrained handwritten character recognition where we have up to 52 different classes (A-Z, a-z). Experimental results on a 100,000-character database show that it is possible to reduce the number of classes and the complexity of the classifier and, at the same time, to improve the recognition accuracy in more than 17%.