The field of numerical cognition studies the sociocultural, developmental, cognitive, and biological correlates of numerical abilities. Historically, numerical cognition emerged as a subbranch of cognitive science and inherited the computational theory of mind—which defines cognition as a form of computation, with brain structures realizing distinct computational functions—as its cognitive ontology. Recent cognitive neuroscience research shows that the brain is not neatly organized with distinct functional modules, but instead cognitive functions are supported by distributed, transient, and dynamic networks. In addition, research on the bodily and evolutionary foundations of cognition show that cognitive abilities make use of capacities that originally evolved to support action, and that cognition can be best understood in connection with these perceptuomotor capacities. And yet, numerical cognition models are still inherently based on the computational theory of mind, rely on concepts that fail to capture the relation between cognition and action, and suggest unitary functions for distinct brain regions. Here, it is argued that a new cognitive ontology is possible in numerical cognition, by using data-driven informatics approaches with neuroimaging data to identify cognitive building block processes and interpreting results by considering the evolutionary and bodily foundations of numerical abilities. Among other mathematical skills, numerical magnitude processing is presented as a case study to substantiate the arguments made; with a discussion on how systems evolved not for physical magnitude estimation, but for graspability (affordance) is used in the service of number representation.