Genetic variation and 3D chromatin structure have major roles in gene regulation. Due to challenges in mapping chromatin conformation with haplotype-specific resolution, the effects of genetic sequence variation on 3D genome structure and gene expression imbalance remain understudied. Here, we applied Genome Architecture Mapping (GAM) to a hybrid mouse embryonic stem cell (mESC) line with high density of single nucleotide polymorphisms (SNPs). GAM resolved haplotype-specific 3D genome structures with high sensitivity, revealing extensive allelic differences in chromatin compartments, topologically associating domains (TADs), long-range enhancer-promoter contacts, and CTCF loops. Architectural differences often coincide with allele-specific differences in gene expression, mediated by Polycomb repression. We show that histone genes are expressed with allelic imbalance in mESCs, are involved in haplotype-specific chromatin contact marked by H3K27me3, and are targets of Polycomb repression through conditional knockouts of Ezh2 or Ring1b. Our work reveals highly distinct 3D folding structures between homologous chromosomes, and highlights their intricate connections with allelic gene expression.