Although divergent natural selection is common in nature, the extent to which genetic constraints bias evolutionary trajectories in its presence remains largely unknown. Here we develop a general framework to integrate estimates of divergent selection and genetic constraints to estimate their contributions to phenotypic divergence among natural populations. We apply these methods to estimates of phenotypic selection and genetic covariance from sexually selected traits that have undergone adaptive divergence among nine natural populations of the fly Drosophila serrata. Despite ongoing sexual selection within populations, differences in its direction among them, and genetic variance for all traits in all populations, divergent sexual selection only weakly resembled the observed pattern of divergence. Accounting for the influence of genetic covariance among the traits significantly improved the alignment between observed and predicted divergence. Our results suggest that the direction in which sexual selection generates divergence may depend on the pattern of genetic constraint in individual populations, ultimately restricting how sexually selected traits may diversify. More generally, we show how evolution is likely to proceed in the direction of major axes of genetic variance, rather than the direction of selection itself, when genetic variance-covariance matrices are ill conditioned and genetic variance is low in the direction of selection.