There are much more researches on the recognition of the cracks on the distress pavement, but the research on the potholes is relatively less. In this paper, Texture measure based on the histogram is extracted as the features of the image region, and the non-linear support vector machine is built up to identify whether a target region is a pothole. Based on this, an algorithm for recognizing the potholes of the pavement is proposed. The experimental results show that the algorithm can achieve a high recognition rate.