Abstract Sweet osmanthus (Osmanthus fragrans) is famous in China for its flowers and contains four groups: Albus, Luteus, Aurantiacus, and Asiaticus. Understanding the relationships among these groups and the genetic mechanisms of flower color and aroma biosynthesis are of tremendous interest. In this study, we sequenced representative varieties from two of the four sweet osmanthus groups. Multiomics and phylogenetic analyses of varieties from each of the four groups showed that Asiaticus split first within the species, followed by Aurantiacus and the sister groups Albus and Luteus. We show that the difference in flower color between Aurantiacus and the other three groups was caused by a 4-bp deletion in the promoter region of carotenoid cleavage dioxygenase 4 (OfCCD4) that leads to expression decrease. In addition, we identified 44 gene pairs exhibiting significant structural differences between the multiseasonal flowering variety “Rixianggui” in the Asiaticus group and other autumn-flowering varieties. Through correlation analysis between intermediate products of aromatic components and gene expression, we identified eight genes associated with the linalool and α- and β-ionone biosynthesis pathways. Overall, our study offers valuable genetic resources for sweet osmanthus, while also providing genetic clues for improving the flower color and multiseasonal flowering of osmanthus and other flowers.