The detection of clinical mastitis in dairy cow is still a major hurdle for many farms around the world. The classical way of detection by visual inspection of the milk often comes with additional time-consuming work for the farmer. To tackle this and add a reliable method in finding potential mastitis cases, automation of these processes is essential. In this study, we present an image classification algorithm using simple image processing tools and machine learning to score images of a visual trap mastitis detector (Ambic Vision 2000) that was installed in the tubes of the milking system. This algorithm offers an objective classification using visual indicators independent of the human eye and helps with future evaluation of detectors. Furthermore, it shows the potential for automation of the mastitis detection process with a visual trap detector as part of typical milking systems.