We investigate the potential of CO rotational lines at redshifts $z\sim 0-6$ being an appreciable source of extragalactic foreground anisotropies in the cosmic microwave background. Motivated by previous investigations, we specifically focus on the frequency bands and small scales probed by ground-based surveys. Using an empirical parameterization for the relation between the infrared luminosity of galaxies and their CO line luminosity, conditioned on sub-mm observations of CO luminosity functions from $J=1$ to $J=7$ at $ν= \{100,250\}$ GHz, we explore how uncertainty in the CO luminosity function translates into uncertainty in the signature of CO emission in the CMB. We find that at $\ell = 3000$ the amplitude of the CO cross-correlation with the CIB could be detectable in an ACT-like experiment with 90, 150 and 220 GHz bands, even in the scenarios with the lowest amplitude consistent with sub-mm data. We also investigate, for the first time, the amplitude of the CO$\times$CIB correlation between different frequency bands and find that our model predicts that this signal could be the second-largest extragalactic foreground at certain wavelengths, behind the CIB cross-frequency spectrum. This implies current observations can potentially be used to constrain the bright end of CO luminosity functions, which are difficult to probe with current sub-mm telescopes due to the small volumes they survey. Our findings corroborate past results and have significant implications in template-based searches for CMB secondaries, such as the kinetic Sunyaev Zel'dovich effect, using the frequency-dependent high-$\ell$ TT power spectrum.

14+3 pages, 8+2 figures. Reflects published version, with an additional post-publication revision which corrects a normalization error in the original work. Results have changed quantitatively