This review highlights the key role of modularity and the additive factors method in functional neuroimaging. Our focus is on structure-function mappings in the human brain and how these are disclosed by brain mapping. We describe how modularity of processing (and possibly processes) was a key point of reference for establishing functional segregation as a principle of brain organization. Furthermore, modularity plays a crucial role when trying to characterize distributed brain responses in terms of functional integration or coupling among brain areas. We consider additive factors logic and how it helped to shape the design and interpretation of studies at the inception of brain mapping, with a special focus on factorial designs. We look at factorial designs in activation experiments and in the context of lesion-deficit mapping. In both cases, the presence or absence of interactions among various experimental factors has proven essential in understanding the context-sensitive nature of distributed but modular processing and discerning the nature of (potentially degenerate) structure-function relationships in cognitive neuroscience.