Olive anthracnose caused by Colletotrichum species causes dramatic losses of fruit yield and oil quality worldwide. A total of 185 Colletotrichum isolates obtained from olives and other hosts showing anthracnose symptoms in Spain and other olive-growing countries over the world were characterized. Colony and conidial morphology, benomyl-sensitive, and casein-hydrolysis activity were recorded. Multilocus alignments of ITS, TUB2, ACT, CHS-1, HIS3, and/or GAPDH were conducted for their molecular identification. The pathogenicity of the most representative Colletotrichum species was tested to olive fruits and to other hosts, such as almonds, apples, oleander, sweet oranges, and strawberries. In general, the phenotypic characters recorded were not useful to identify all species, although they allowed the separation of some species or species complexes. ITS and TUB2 were enough to infer Colletotrichum species within C. acutatum and C. boninense complexes, whereas ITS, TUB2, ACT, CHS-1, HIS-3, and GADPH regions were necessary to discriminate within the C. gloesporioides complex. Twelve Colletotrichum species belonging to C. acutatum, C. boninense, and C. gloeosporioides complexes were identified, with C. godetiae being dominant in Spain, Italy, Greece, and Tunisia, C. nymphaeae in Portugal, and C. fioriniae in California. The highest diversity with eight Colletotrichum spp. was found in Australia. Significant differences in virulence to olives were observed between isolates depending on the Colletotrichum species and host origin. When other hosts were inoculated, most of the Colletotrichum isolates tested were pathogenic in all the hosts evaluated, except for C. siamense to apple and sweet orange fruits, and C. godetiae to oleander leaves.