The incidence of many types of cancer arising in organs with non-reproductive functions is significantly higher in male populations than in female populations, with associated differences in survival. Occupational and/or behavioural factors are well-known underlying determinants. However, cellular and molecular differences between the two sexes are also likely to be important. In this Opinion article, we focus on the complex interplay that sex hormones and sex chromosomes can have in intrinsic control of cancer-initiating cell populations, the tumour microenvironment and systemic determinants of cancer development, such as the immune system and metabolism. A better appreciation of these differences between the two sexes could be of substantial value for cancer prevention as well as treatment.