The retreat of coastal cliffs is a natural process that occurs due to the interaction of different forcings that can be marine and atmospheric, and conditioned by the lithological properties of the rock material. Several attempts have been done at different scales to quantify and rank the various parameters that influence erosion rates, most of them agreeing that cliff retreat is governed by the lithological properties of the cliffs. Although, due to the large number of parameters involved there is not a clear consensus. The present study aims to characterize the cliffs along the Cantabrian coast by using an unsupervised classification of their physical and lithological characteristics, and by analyzing their retreat behavior. The proposed methodology is scalable to larger coastal areas. The study found that Cantabrian coastal cliffs have a low mean retreat rate of 0.042 m/year, with a maximum retreat rate of up to 0.4 m/year in two locations. Nine distinct groups of cliff behavior have been found, with only two of them presenting high erosional records, which are controlled by lithological features. Cliffs with the highest erosion rates are composed of alternating lithologies and more erodible materials. The results suggest that the factors most influencing erosive retreat in cliffs are the type of lithology and the alternation of different lithologies.