Although it is generally acknowledged that the Cognitive Reflection Test (CRT) captures intelligence and numerical ability, many agree that it cannot be completely reduced to these constructs. Rather, it is presumed that the CRT also assesses some kind of thinking disposition towards reflective and open-minded thinking. In this manuscript, we report the results of a study that tested this assumption by exploring convergent validity of both the numerical and verbal version of the CRT. Using structural equation modelling, we investigated whether intelligence and numerical ability can account for all the variance in the CRT and if not, what is the nature of the unaccounted variance. Our conclusions about the convergent validity differed for the two types of test. For the numerical CRT, we found that the correlation between the latent numerical CRT and numerical ability was so high that the constructs were practically indistinguishable. As for the verbal CRT, the correlations between the latent verbal CRT and intelligence and numerical ability constructs were substantially lower, meaning that these two constructs do not account for all the variance in the test. However, the latent verbal CRT failed to correlate with belief bias and actively open-minded thinking, two closely related constructs, once the variance of intelligence and numerical ability was partialled out. We concluded that, despite its name, the CRT does not seem to assess the construct of cognitive reflection and its correlation with other variables found in the literature might mostly be driven by its overlap with intelligence and numerical ability.