Intramuscular fat may mediate associations between obesity and physical disability. We examined the associations between muscle attenuation, a proxy for intramuscular fat, and physical function. Paraspinous muscle computed tomography attenuation was obtained on a Framingham Heart Study subgroup (n = 1152, 56 % women, mean age 66 years). Regressions modeled cross-sectional associations between muscle attenuation and mobility disability, grip strength, and walking speed with standard covariates; models additionally adjusted for body mass index (BMI) and visceral adipose tissue (VAT). Separate models investigated associations between VAT and subcutaneous adipose tissue (SAT) and physical function. Per 1 standard deviation decrement in muscle attenuation (i.e., more muscle fat), we observed 1.29 (95 % CI = 1.11, 1.50; p = 0.0009) increased odds of walking speed ≤1 m/s in women and men. This persisted after separate BMI and VAT adjustments (p < 0.02). In men, there was a 1.29 kg (95 % CI = 0.57, 2.01; p = 0.0005) decrement in grip strength, which persisted after BMI and VAT adjustments (p ≤ 0.0004). For VAT and SAT, similar associations were not observed. Intramuscular fat is associated with increased odds of walking speed ≤1 m/s in both sexes and lower grip strength in men. There were no similar associations for VAT and SAT, highlighting the specificity of intramuscular fat in association with physical function.