Assumptions about the probability distribution of gene tree topologies provides a basis for phylogenetic reconstruction and introgression detection. Initial evidence has suggested that in the presence of purifying selection, population size differences can affect the probability distribution of gene tree topologies. Nevertheless, the impact of this phenomenon on phylogenetic reconstruction and introgression detection remains to be explored. Additionally, a theoretical understanding of this phenomenon remains absent. Here, using the population genetic simulator SLiM, we provide evidence that in the presence of purifying selection, population size differences can cause biases in phylogenetic inference. We also provide evidence that in the presence of purifying selection, population size differences can cause statistics used for introgression detection to exhibit patterns resembling those caused by introgression. Additionally, a theoretical analysis is presented to show that the biological basis underlying the formation of gene trees is different under neutral evolution and under purifying selection, and the population size dependency in gene tree distributions can be deduced from the inherent nature of purifying selection. This work underscores the importance of considering the potential confounding impact of purifying selection on phylogenetic inference and introgression detection.

Funding provided by: National Natural Science Foundation of ChinaROR ID: https://ror.org/01h0zpd94Award Number: 31771456 Funding provided by: National Natural Science Foundation of ChinaROR ID: https://ror.org/01h0zpd94Award Number: 32370009 Funding provided by: Natural Science Foundation of Guangdong ProvinceCrossref Funder Registry ID: Award Number: 2022A1515010464