Abstract Autism spectrum disorder is an early-onset neurodevelopmental condition. This study aimed to investigate the progressive structural alterations in the autistic brain during early childhood. Structural magnetic resonance imaging scans were examined in a cross-sectional sample of 67 autistic children and 63 demographically matched typically developing (TD) children, aged 2–7 years. Voxel-based morphometry and a general linear model were used to ascertain the effects of diagnosis, age, and a diagnosis-by-age interaction on the gray matter volume. Causal structural covariance network analysis was performed to map the interregional influences of brain structural alterations with increasing age. The autism group showed spatially distributed increases in gray matter volume when controlling for age-related effects, compared with TD children. A significant diagnosis-by-age interaction effect was observed in the fusiform face area (FFA, Fpeak = 13.57) and cerebellum/vermis (Fpeak = 12.73). Compared with TD children, the gray matter development of the FFA in autism displayed altered influences on that of the social brain network regions (false discovery rate corrected, P < 0.05). Our findings indicate the atypical neurodevelopment of the FFA in the autistic brain during early childhood and highlight altered developmental effects of this region on the social brain network.