During meiotic prophase, the meiosis-specific telomere-binding protein TERB1 regulates chromosome movement required for homologous pairing and recombination by interacting with the telomeric shelterin subunit TRF1. Here, we report the crystal structure of the TRF1-binding motif of human TERB1 in complex with the TRFH domain of TRF1. Notably, specific disruption of the TERB1-TRF1 interaction by a point mutation in the mouse Terb1 gene results in infertility only in males. We find that this mutation causes an arrest in the zygotene-early pachytene stage and mild telomere abnormalities of autosomes but unpaired X and Y chromosomes in pachytene, leading to massive spermatocyte apoptosis. We propose that the loss of telomere structure mediated by the TERB1-TRF1 interaction significantly affects homologous pairing of the telomere-adjacent pseudoautosomal region (PAR) of the X and Y chromosomes in mouse spermatocytes. Our findings uncover a specific mechanism of telomeres that surmounts the unique challenges of mammalian X-Y pairing in meiosis.