Camellia oleifera is an important woody oil species in China. Its seed oil has been widely used as a cooking oil. Seed size is a crucial factor influencing the yield of seed oil. In this study, the horizontal diameter, vertical diameter and volume of C. oleifera seeds showed a rapid growth tendency from 235 days after pollination (DAP) to 258 DAP but had a slight increase at seed maturity. During seed development, the expression of genes related to cell proliferation and expansion differ greatly. Auxin plays an important role in C. oleifera seeds; YUC4 and IAA17 were significantly downregulated. Weighted gene co-expression network analysis screened 21 hub transcription factors for C. oleifera seed horizontal diameter, vertical diameter and volume. Among them, SPL4 was significantly decreased and associated with all these three traits, while ABI4 and YAB1 were significantly increased and associated with horizontal diameter of C. oleifera seeds. Additionally, KLU significantly decreased (2040-fold). Collectively, our data advances the knowledge of factors related to seed size and provides a theoretical basis for improving the yield of C. oleifera seeds.