Grounded theories of cognition claim that concept representation relies on the systems for perception and action. The sensory-motor grounding of abstract concepts presents a challenge for these theories. Some accounts propose that abstract concepts are indirectly grounded via image schemas or situations. Recent research, however, indicates that the role of sensory-motor processing for concrete concepts may be limited, providing evidence against the idea that abstract concepts are grounded via concrete concepts. Hybrid models that combine language and sensory-motor experience may provide a more viable account of abstract and concrete representations. We propose that sensory-motor grounding is important during acquisition and provides structure to concepts. Later activation of concepts relies on this structure but does not necessarily involve sensory-motor processing. Language is needed to create coherent concepts from diverse sensory-motor experiences. This article is part of the theme issue ‘Varieties of abstract concepts: development, use and representation in the brain’.