According to the current, widely accepted paradigm, the evolutionary transition from hermaphroditism toward separate sexes occurs in two successive steps: an initial, intermediate step in which unisexual individuals, male or female, sterility mutants coexist with hermaphrodites and a final step that definitively establishes dioecy. Two nonexclusive processes can drive this transition: inbreeding avoidance and reallocation of resources from one sexual function to the other. Here, we report results of controlled crosses between males and hermaphrodites in Phillyrea angustifolia, an androdioecious species with two mutually intercompatible, but intraincompatible groups of hermaphrodites. We observed different segregation patterns that can be explained by: (1) epistatic interactions between two unlinked diallelic loci, determining sex and mating compatibility, and (2) a mutation with pleiotropic effects: female sterility, full compatibility of males with both hermaphrodite incompatibility groups, and complete male-biased sex-ratio distortion in one of the two groups. Modeling shows that these mechanisms can explain the high frequency of males in populations of P. angustifolia and can promote the maintenance of androdioecy without requiring inbreeding depression or resource reallocation. We thus argue that segregation distortion establishes the right conditions for the evolution of cryptic dioecy and potentially initiates the evolution toward separate sexes.