On average, men and women differ in brain structure and behaviour, raising the possibility of a link between sex differences in brain and behaviour. But women and men are also subject to different societal and cultural norms. We navigated this challenge by investigating variability of sex-differentiated brain structure within each sex. Using data from the Queensland Twin IMaging study (N=1,040) and Human Connectome Project (N=1,113), we obtained data-driven measures of individual differences along a male-female dimension for brain and behaviour based on average sex differences in brain structure and behaviour, respectively. We found a weak association between these brain and behavioural differences, driven by brain size. These brain and behavioural differences were moderately heritable. Our findings suggest that behavioural sex differences are to some extent related to sex differences in brain structure, but that this is mainly driven by differences in brain size, and causality should be interpreted cautiously.