A novel automatic polyp recognition scheme called Annular Spatial Pyramid Matching (ASPM) with Locality-Constrained Linear Coding (LLC) is proposed by considering the annular structure of the intestinal images at multilevel. Firstly, detailed texture features extracted from the samples including normal and polyp images are calculated and then LLC method is employed on these features to obtain a sparse representation. Secondly, a strategy of annular region segmentation based on Spatial Pyramid Matching is proposed to improve the effectiveness of processing for intestinal images. Then, the final representation for each image is obtained by max-pooling the codes of features. Finally, SVM classifier is developed to carry out polyp images classification tasks. The experimental results indicate that the proposed algorithm outperforms the analysed state-of-the-art methods on the polyps recognition.