In real-world image understanding and retrieval applications, there exists a large number of images containing "verb-object" semantic. The most existing image annotation approaches which mainly focus on annotating images with "object" concepts may not well describe the image semantics. In this paper, we propose a novel image annotation approach by learning "verb-object" concepts. The "verb-object" concept learning method is developed based on the assumption that the classifiers of the "verb-object" concepts which contain the same object usually share a common structure. We formulate each "verb-object" concept classifier as a combination of a private part and a common part shared by all the "verb-object" concepts containing the same object. These classifiers are learned simultaneously through a joint optimization process. Experiments on a Web image data set containing 22,812 images with 28 concepts demonstrate that the proposed approach achieved promising performance compared to the baseline method.