Abstract Study Objectives Apolipoprotein E ɛ4 (APOE4) is the strongest genetic risk factor for Alzheimer’s disease (AD). In addition, APOE4 carriers may exhibit sleep disturbances, but conflicting results have been reported, such that there is no clear consensus regarding which aspects of sleep are impacted. Our objective was to compare objective sleep architecture between APOE4 carriers and non-carriers, and to investigate the modulating impact of age, sex, cognitive status, and obstructive sleep apnea (OSA). Methods A total of 198 dementia-free participants aged >55 years old (mean age: 68.7 ± 8.08 years old, 40.91% women, 41 APOE4 carriers) were recruited in this cross-sectional study. They underwent polysomnography, APOE4 genotyping, and a neuropsychological evaluation. ANCOVAs assessed the effect of APOE4 status on sleep architecture, controlling for age, sex, cognitive status, and the apnea–hypopnea index. Interaction terms were added between APOE4 status and covariates. Results Rapid eye movement (REM) sleep percentage (F = 9.95, p = .002, ηp2 = 0.049) and duration (F = 9.23, p = .003, ηp2 = 0.047) were lower in APOE4 carriers. The results were replicated in a subsample of 112 participants without moderate-to-severe OSA. There were no significant interactions between APOE4 status and age, sex, cognitive status, and OSA in the whole sample. Conclusions Our results show that APOE4 carriers exhibit lower REM sleep duration, including in cognitively unimpaired individuals, possibly resulting from early neurodegenerative processes in regions involved in REM sleep generation and maintenance.