<b><i>Background:</i></b> The positive relationship between cognitive and physical performance has been widely established. The influence of brain structure on both domains has been shown as well. <b><i>Objective:</i></b> We studied whether the relationship between brain structure and physical performance is independent of cognitive performance. <b><i>Methods:</i></b> This was a cross-sectional analysis of 297 middle-aged to older adults (mean age ± SD 65.4 ± 6.8 years). Memory function, executive function and physical performance measured by the Tandem Stance Test, Chair Stand Test, 4-meter walk and 25-meter walk were assessed. Magnetic resonance imaging was available in 237 participants and used to determine the (sub)cortical gray matter, white matter, hippocampal and basal ganglia volumes and the presence of cerebral small-vessel disease, i.e. white matter hyperintensities, cerebral microbleeds (CMBs) and lacunar infarcts (LIs). Regression analysis was used adjusting for age, gender, education and whole-brain volume. A Bonferroni correction was applied considering p values <0.017 as statistically significant. <b><i>Results:</i></b> Poor memory function was associated with a slower 4-meter walking speed (p < 0.01). No association was found between brain structure and cognitive performance. The presence of CMBs and LIs was associated with a slower 25-meter walking speed (p < 0.001). This result did not change after additional adjustment for cognitive performance. <b><i>Conclusions:</i></b> In middle-aged to older adults, CMBs and LIs are associated with walking speed independent of cognitive performance. This emphasizes the clinical relevance of identifying each of the possible underlying mechanisms of physical performance, which is required for the development of timely and targeted therapies.