ABSTRACT Introduction Given the growing population of colorectal cancer (CRC) survivors, identifying ways to enhance health-related quality of life (HRQoL) and alleviate complaints of fatigue and chemotherapy-induced peripheral neuropathy (CIPN) is essential. Purpose We aimed to assess longitudinal associations of sedentary behavior (SB) and moderate to vigorous physical activity (MVPA) independently, as well as their joint association, with HRQoL, fatigue, and CIPN in CRC survivors. Methods In a prospective cohort among stage I–stage III CRC survivors (n = 396), five repeated home visits from diagnosis up to 24 months posttreatment were executed. SB was measured using triaxial accelerometers, and MVPA, HRQoL, fatigue, and CIPN were measured by validated questionnaires. We applied confounder-adjusted linear mixed models to analyze longitudinal associations from 6 wk until 24 months posttreatment. Results Average time in prolonged SB (accumulated in bouts of duration ≥30 min) was 5.3 ± 2.7 h·d−1, and approximately 82% of survivors were classified as sufficiently active (≥150 min·wk−1 of MVPA) at 6 wk posttreatment. Decreases in SB and increases in MVPA were independently associated with better HRQoL and less fatigue over time. No associations were found for CIPN complaints. A synergistic interaction was observed between prolonged SB and MVPA in affecting functioning scales. Relative to CRC survivors with low prolonged SB and high MVPA, survivors with high prolonged SB and low MVPA reported a stronger decrease in physical functioning and role functioning over time than expected based on the independent associations of prolonged SB and MVPA. Conclusion Our longitudinal results show that less SB and more MVPA are beneficial for CRC survivors’ HRQoL and fatigue levels. Our findings regarding interaction underscore that joint recommendations to avoid prolonged sitting and accumulate MVPA are important.