Among naturally occurring pigments, carotenoids are importantly involved in the photosynthesis of plants and responsible for the coloration of petals and fruits. Osmanthus fragrans Lour., a famous ornamental plant, has many cultivars with different flower color. Petal coloration in O. fragrans mainly depends on the kinds of carotenoids and their contents. To investigate the mechanism of flower coloration in different cultivars, an analysis of phenotypic classification, phytochemistry, as well as the expression of carotenoid metabolism genes based on different groups was performed in the present study. Two main clusters including the orange-red cluster containing Aurantiacus cultivars and the yellowish-white cluster containing the other three cultivar groups were classified using the CIEL∗a∗b∗ system. No significant differences in flavonoid contents were observed between these two clusters. However, carotenoids, especially α-carotene and β-carotene, were found to have crucial roles in the diversity of floral coloration among the different cultivars. Carotenoid compositions in the petals of cultivars from both clusters consisted of α-carotene, β-carotene, α-cryptoxanthin, β-cryptoxanthin, lutein, and zeaxanthin, but carotenoid accumulation patterns during the flowering process were different. The petals of the yellowish-white cultivars exhibited high contents of β-carotene, lutein and α-carotene, whereas the petals of the orange-red cultivars mainly contained β-carotene and α-carotene. The profound diversity in the total carotenoid concentrations in the two clusters was determined by the transcript levels of OfCCD4. Furthermore, the accumulation of upstream products with orange color in orange-red cultivars was partially due to the low expression of OfCHYB, whereas the relatively higher OfCHYB expression in the petals of the yellowish-white cultivars led to higher proportions of lutein, which is yellow. We also found that downregulation of OfLCYE, which encodes 𝜀-ring cyclase, indicated that the carotenoid flux of most cultivars mainly resulted in more β, β-branched products. Additionally, carotenoid biosynthesis in green tissues and petals was compared, revealing the tissue specificity of carotenoid accumulation in O. fragrans. Therefore, the effects of multiple genes on carotenoid accumulation give rise to the colorful O. fragrans.