Objective Childhood brain tumor survivors (CBTS) are at risk to develop hypothalamic–pituitary (HP) dysfunction (HPD). The risk for HPD may vary between different age groups due to maturation of the brain and differences in oncologic treatment protocols. Specific studies on HPD in infant brain tumor survivors (infant-BTS, 0–1 years at diagnosis) or toddler brain tumor survivors (toddler-BTS, ≥1–3 years) have not been performed. Patients and methods A retrospective nationwide cohort study in CBTS was performed. Prevalence and risk factors for HPD were compared between infant-, toddler-, and older-BTS. Subgroup analysis was performed for all non-irradiated CBTS (n = 460). Results In total, 718 CBTS were included, with a median follow-up time of 7.9 years. Overall, despite the less frequent use of radiotherapy (RT) in infants, no differences in the prevalence of HPD were found between the three groups. RT (OR: 16.44; 95% CI: 8.93–30.27), suprasellar tumor location (OR: 44.76; 95% CI: 19.00–105.49), and younger age (OR: 1.11; 95% CI: 1.05–1.18) were associated with HP dysfunction. Infant-BTS and toddler-BTS showed more weight gain (P < 0.0001) and smaller height SDS (P = 0.001) during follow-up. In non-irradiated CBTS, infant-BTS and toddler-BTS were significantly more frequently diagnosed with TSH-, ACTH-, and ADH deficiency, compared to older-BTS. Conclusion Infant and toddler brain tumor survivors seem to be more vulnerable to develop HP dysfunction than older children. These results emphasize the importance of special infant and toddler brain tumor treatment protocols and the need for endocrine surveillance in children treated for a brain tumor at a young age.