Abstract Large-scale genomic studies focusing on the genetic contribution to human aging have mostly relied on cross-sectional data. With the release of longitudinally curated aging phenotypes by the UK Biobank (UKBB), it is now possible to study aging over time at genome-wide scale. In this work, we evaluated the suitability of competing models of change in realistic simulation settings, performed genome-wide association scans on simulation-validated measures of age-related deweekcline, and followed up with LD-score regression and Mendelian Randomization (MR) analyses. Focusing on global cognitive and physical function, we observed marked differences between baseline function (θ) and accelerated decline (Δ). Both outcomes showed distinct heritability levels (e.g., 31.38% $${h}_{\theta }^{2}$$ h θ 2 versus 3.15% $${h}_{\Delta }^{2}$$ h Δ 2 for physical function) and different associated loci (e.g., DUSP6 specific to physical Δ). Further, we found little commonalities across the two dimensions of aging—while cognitive decline was largely driven by Alzheimer’s disease liability (standardized MR-effect, γ = 0.17), physical decline was mostly impacted by telomere length (γ = −0.05) and bone mineral density (γ = −0.05). Our work highlights the utility of longitudinal genomic efforts to scrutinize age-dependent genetic and environmental effects on physical and cognitive outcomes. Careful modelling and attention to participation characteristics are, however, crucial for valid inference.