In content based image retrieval, many researchers have worked to improve image retrieval results. Many papers where written to address that problem in many ways. Most papers have worked with how to represent the image in order to find a match to it. Some worked in representing the image using Scale-invariant feature transform (SIFT) descriptors other using Speeded up Robust Features (SURF) or representing it as a set of visual words or many other representations. But all these methods relied purely on representing the image visually without any reference to the semantic of the image. Our object was to introduce the image semantics to the retrieval by combining textual annotation with the visual representation to give a refined representation of the image that will improve retrieval results. This report is divided mainly into two parts. In the first part we introduce a literature review about content based image retrieval and annotation based image retrieval. In the second part, we introduced our own methodology and backed it up with tests and results. Our approach is evaluated over 14 categories with each containing example image(s) and annotation statements.