Alzheimer’s disease is marked by intracellular tau aggregates in the medial temporal lobe (MTL) and extracellular amyloid aggregates in the default network (DN). Here, we examined codependent structural variations between the MTL’s most vulnerable structure, the hippocampus (HC), and the DN at subregion resolution in individuals with Alzheimer’s disease and related dementia (ADRD). By leveraging the power of the approximately 40,000 participants of the UK Biobank cohort, we assessed impacts from the protectiveAPOEɛ2 and the deleteriousAPOEɛ4 Alzheimer’s disease alleles on these structural relationships. We demonstrate ɛ2 and ɛ4 genotype effects on the inter-individual expression of HC-DN co-variation structural patterns at the population level. Across these HC-DN signatures, recurrent deviations in the CA1, CA2/3, molecular layer, fornix’s fimbria, and their cortical partners related to ADRD risk. Analyses of the rich phenotypic profiles in the UK Biobank cohort further revealed male-specific HC-DN associations with air pollution and female-specific associations with cardiovascular traits. We also showed thatAPOEɛ2/2 interacts preferentially with HC-DN co-variation patterns in estimating social lifestyle in males and physical activity in females. Our structural, genetic, and phenotypic analyses in this large epidemiological cohort reinvigorate the often-neglected interplay betweenAPOEɛ2 dosage and sex and linkAPOEalleles to inter-individual brain structural differences indicative of ADRD familial risk.