Background Speech-based features extracted from telephone-based cognitive tasks show promise for detecting cognitive decline in prodromal and manifest dementia. Little is known about the cerebral underpinnings of these speech features. Objective To examine associations between speech features, brain atrophy, and longitudinal cognitive decline in individuals at risk for Alzheimer's disease (AD). Methods Healthy volunteers, individuals with subjective cognitive decline, and those with mild cognitive impairment completed phonebot-guided semantic verbal fluency (SVF) and 15-word verbal learning task (VLT). Speech features were automatically extracted, and a global cognitive score (SB-C score) was computed. We analyzed data from 161 participants for cognitive trajectories, 141 for cross-sectional brain atrophy, and 102 for longitudinal brain changes. Analyses were conducted using multiple linear regressions, mixed-effects models, and voxel-based morphometry. Results The SB-C score was associated with bilateral hippocampal volumes, SVF features were primarily associated with left hemisphere regions, including the inferior frontal, parahippocampal, and superior/middle temporal gyri ( p uncorr < 0.001). SB-C score, SVF correct counts, and VLT delayed recall were associated with atrophy rates in the hippocampal/parahippocampal gyrus and left middle/inferior temporal gyri ( p FDR < 0.05). These features were also associated with cognitive decline assessed via Preclinical Alzheimer's Cognitive Composite 5, SVF, and Wordlist learning delayed recall ( p FDR < 0.01). Word frequency and temporal cluster switches showed varying associations with cognitive trajectories. Other features did not show robust associations. Conclusions In this study, we highlight the potential of digital speech features for identifying brain atrophy and cognitive decline over time in at-risk AD populations.