The automatic classification of images is now widely used in a range of applications. These include the diagnosis of arthritis from joint images, the classification of environmental noise from spectrograms and automatic text analysis. However, satisfactory performance is difficult to achieve in uncontrolled environments, as images are often contaminated by high levels of noise, outliers and global contamination due to illumination changes and environmental effects. We address these issues using a semi-parametric modelling strategy and a novel robust Bayesian classifier. This model is driven by additive Gaussian noise with non-uniform variance to describe outliers and uses the parametric and non-parametric components to describe contamination of different types. We assess the performance of our approach in two experiments based on real and simulated data. These show that our approach can significantly outperform a number of competitors in uncontrolled environments.