Many organisms spend a significant portion of their life cycle as haploids and as diploids (a haploid-diploid life cycle). However, the evolutionary processes that could maintain this sort of life cycle are unclear. Most previous models of ploidy evolution have assumed that the fitness effects of new mutations are equal in haploids and homozygous diploids, however, this equivalency is not supported by empirical data. With different mutational effects, the overall (intrinsic) fitness of a haploid would not be equal to that of a diploid after a series of substitution events. Intrinsic fitness differences between haploids and diploids can also arise directly, for example because diploids tend to have larger cell sizes than haploids. Here, we incorporate intrinsic fitness differences into genetic models for the evolution of time spent in the haploid versus diploid phases, in which ploidy affects whether new mutations are masked. Life-cycle evolution can be affected by intrinsic fitness differences between phases, the masking of mutations, or a combination of both. We find parameter ranges where these two selective forces act and show that the balance between them can favor convergence on a haploid-diploid life cycle, which is not observed in the absence of intrinsic fitness differences.