Humans exhibit broad heterogeneity in affiliative social behavior. Twin and family studies show that individual differences in core dimensions of social behavior are heritable, yet there are knowledge gaps in understanding the underlying genetic and neurobiological mechanisms. Animal genetic reference panels (GRPs) provide a tractable strategy for examining the behavioral and genetic architecture of complex traits. Here, using males from 50 mouse strains from the BXD GRP, 4 domains of affiliative social behavior—social approach, social recognition, direct social interaction (DSI) (partner sniffing) and vocal communication—were examined in 2 widely used behavioral tasks—the 3‐chamber and DSI tasks. There was continuous and broad variation in social and nonsocial traits, with moderate to high heritability of social approach sniff preference (0.31), ultrasonic vocalization (USV) count (0.39), partner sniffing (0.51), locomotor activity (0.54‐0.66) and anxiety‐like behavior (0.36). Principal component analysis shows that variation in social and nonsocial traits are attributable to 5 independent factors. Genome‐wide mapping identified significant quantitative trait loci for USV count on chromosome (Chr) 18 and locomotor activity on Chr X, with suggestive loci and candidate quantitative trait genes identified for all traits with one notable exception—partner sniffing in the DSI task. The results show heritable variation in sociability, which is independent of variation in activity and anxiety‐like traits. In addition, a highly heritable and ethological domain of affiliative sociability—partner sniffing—appears highly polygenic. These findings establish a basis for identifying functional natural variants, leading to a new understanding typical and atypical sociability.