In this article relevant developments in relevance feedback based image annotation and retrieval are reported. A new approach to infer semantic concepts representing meaningful objects in images is also described. The proposed technique combines user relevance feedback and underlying low-level properties of elementary building blocks making up semantic objects in images. Images are regarded as mosaics made of small building blocks featuring good representations of colour, texture and edgeness. The approach is based on accurate classification of these building blocks. Once this has been achieved, a signature for the object of concern is built. It is expected that this signature features a high discrimination power and consequently it becomes very suitable to find other images containing the same semantic object. The model combines fuzzy clustering and relevance feedback in the training stage, and uses fuzzy support vector machines in the generalization stage.