ABSTRACT Purpose It remains unclear to what extent habitual physical activity and sedentary time (ST) are associated with visceral fat and liver fat. We studied the substitution of ST with time spent physically active and total body fat (TBF), visceral adipose tissue (VAT), and hepatic triglyceride content (HTGC) in middle-age men and women. Design In this cross-sectional analysis of the Netherlands Epidemiology of Obesity study, physical activity was assessed in 228 participants using a combined accelerometer and heart rate monitor. TBF was assessed by the Tanita bioelectrical impedance, VAT by magnetic resonance imaging, and HTGC by proton-MR spectroscopy. Behavioral intensity distribution was categorized as ST, time spent in light physical activity (LPA), and moderate to vigorous physical activity (MVPA). To estimate the effect of replacing 30 min·d−1 of ST with 30 min·d−1 LPA or MVPA, we performed isotemporal substitution analyses, adjusted for sex, age, ethnicity, education, the Dutch Healthy Diet index, and smoking. Results Included participants (41% men) had a mean ± SD age of 56 ± 6 yr and spent 88 ± 56 min in MVPA and 9.0 ± 2.1 h of ST. Replacing 30 min·d−1 of ST with 30 min of MVPA was associated with 1.3% less TBF (95% confidence interval = −2.0 to −0.7), 7.8 cm2 less VAT (−11.6 to −4.0), and 0.89 times HTGC (0.82–0.97). Replacement with LPA was not associated with TBF (−0.03%; −0.5 to 0.4), VAT (−1.7 cm2; −4.4 to 0.9), or HTGC (0.98 times; 0.92–1.04). Conclusions Reallocation of time spent sedentary with time spent in MVPA, but not LPA, was associated with less TBF, visceral fat, and liver fat. These findings contribute to the development of more specified guidelines on ST and physical activity.