Data-driven models of concepts are gaining popularity in Psychology and Cognitive Science. Distributional semantic models represent word meanings as abstract word co-occurrence patterns, and excel at capturing human meaning intuitions about conceptual relationships; however, they lack the explicit links to the physical world that humans acquire through perception. Computer vision neural networks, on the other hand, can produce representations of visually-grounded concepts, but they do not support the extraction of information about the relationships between objects. To bridge the gap between distributional semantic models and computer vision networks, we introduce SemanticScape, a model of semantic concepts grounded in the visual relationships between objects in natural images. The model captures the latent statistics of the spatial organization of objects in the visual environment. Its implementation is based on the calculation of the summed Euclidean distances between all object pairs in visual scenes, which are then abstracted by means of dimensionality reduction. We validate our model against human explicit intuitions on semantic and visual similarity, relatedness, analogical reasoning, and several semantic and visual implicit processing measurements. Our results show that SemanticScape explains variance in human responses in the semantic tasks above and beyond what can be accounted for by standard distributional semantic models and convolutional neural networks; however, it is not predictive of human performance in implicit perceptual tasks. Our findings highlight that implicit information about the objects’ spatial distribution in the environment has a specific impact on semantic processing, demonstrating the importance of this often neglected experiential source.