Human betaherpesviruses 6A and 6B (HHV-6A and HHV-6B) are highly prevalent in human populations. The genomes of these viruses can be stably integrated at the telomeres of human chromosomes and be vertically transmitted (inherited chromosomally integrated HHV-6, iciHHV6). We reconstructed the population structure of HHV-6 and we show that HHV-6A genomes diverged less than HHV-6B genomes from the ancestral common HHV-6A/B population. Analysis of ancestry proportions indicated that HHV-6A exogenous viruses and iciHHV-6A derived most of their genomes from distinct ancestral sources. Conversely, exogenous viral and iciHHV-6B populations were similar in terms of ancestry components, with no evident geographic structuring. Most HHV-6B genomes sampled to date derive from viral populations that experienced considerable drift. However, a population of HHV-6 exogenous viruses, currently classified as HHV-6B and sampled in New York state, formed a separate cluster (NY cluster) and harbored a considerable portion of HHV-6A-like ancestry. Recombination detection methods identified these viruses as interspecies recombinants, but phylogenetic reconstruction indicated that the recombination signals are due to shared ancestry. In analogy to iciHHV-6A, NY cluster viruses have high nucleotide diversity and constant population size. We propose that HHV-6A sequences and the NY cluster population diverged from an ancestral HHV-6A-like population. A relatively recent bottleneck of the NY (or a related) population with subsequent expansion originated most HHV-6B genomes currently sampled. Our findings indicate that the distinction between HHV-6A and -6B is not as clear-cut as previously thought. More generally, epidemiological and clinical surveys would benefit from taking HHV-6 genetic diversity into account.