Stress during pregnancy can impair biological and behavioral responses in the adult offspring and some of these effects are associated with structural changes in specific brain regions. Furthermore, these outcomes can vary according to strain, gender, and type and duration of the maternal stress. Indeed, early stress can induce sexually dimorphic long-term effects on diverse endocrine axes, including subsequent responses to stress. However, whether hypothalamic structural modifications are associated with these endocrine disruptions has not been reported. Thus, we examined the gender differences in the long-term effects of prenatal and adult immobilization stress on the hypothalamic-pituitary-adrenocortical (HPA) axis and the associated changes in hypothalamic structural proteins. Pregnant Wistar rats were subjected to immobilization stress three times daily (45 min each) during the last week of gestation. One half of the offspring were subjected to the same regimen of stress on 10 consecutive days starting at postnatal day (PND) 90. At sacrifice (PND 180), serum corticosterone levels were significantly higher in females compared to males and increased significantly in females subjected to both stresses with no change in males. Prenatal stress increased pituitary ACTH content in males, with no effect in females. Hypothalamic CRH mRNA levels were significantly increased by prenatal stress in females, but decreased in male rats. In females neither stress affected hypothalamic cell death, as determined by cytoplasmic histone-associated DNA fragment levels or proliferation, determined by proliferating cell nuclear antigen levels (PCNA); however, in males there was a significant decrease in cell death in response to prenatal stress and a decrease in PCNA levels with both prenatal and adult stress. In all groups BrdU immunoreactivity colocalized in glial fibrillary acidic protein (GFAP) positive cells, with few BrdU/NeuN labelled cells found. Furthermore, in males the astrocyte marker S100β increased with prenatal stress and decreased with adult stress, suggesting affectation of astrocytes. Synapsin-1 levels were increased by adult stress in females and by prenatal stress in males, while, PSD95 levels were increased in females and decreased in males by both prenatal and adult stress. In conclusion, hypothalamic structural rearrangement appears to be involved in the long-term endocrine outcomes observed after both chronic prenatal and adult stresses. Furthermore, many of these changes are not only different between males and females, but opposite, which could underlie the gender differences in the long-term sequelae of chronic stress, including subsequent responses to stress.