Primates undoubtedly have impressive abilities in perceiving, recognizing, manipulating, and predicting other individuals, but only great apes seem to recognize the cognitive basis of manipulative and cooperative tactics or the concept of self. None of these abilities is unique to primates. We distinguish (1) a package of quantitative advantages in social sophistication, perhaps based on more efficient memory, in which neocortical enlargement is associated with the challenge of social living; from (2) a qualitative difference in understanding, whose taxonomic distribution--including several distantly related species, including birds--does not point to an evolutionary origin in social challenges and may instead relate to a need to acquire novel ways of dealing with the physical world. The ability of great apes to learn new manual routines by parsing action components may have driven their qualitatively greater social skill, suggesting that strict partition of physical and social cognition is likely to be misleading.