Automatic image annotation methods automatically assign labels to images in order to facilitate tasks such as image retrieval, search, organizing and management. Incorrect labels may negatively influence the search results so image annotation should be as accurate as possible. Labels pertaining to objects or to whole scenes are commonly used for image annotation, and precision is especially important in case when scene labels are inferred from objects, as errors in the object labels may propagate to the scene level. One way to improve the annotation precision is by detecting and discarding the automatically assigned object labels that do not fit the context of other detected objects. This procedure is referred to as annotation refinement. Here, an approach to detection of likely incorrect labels based on the context of other labels and prior knowledge about mutual occurrence of various objects in images is tested.