This paper presents a comprehensive theory describing the influence of aggregate on the compressive strength of concrete. A first distinction is made between topological and mechanical aspects. The former, called confining effect, includes the effect of the volume and the maximum size of the aggregate, which are best described by means of a single physical parameter, the maximum paste thickness (MPT). MPT is defined as the mean distance between two adjacent coarse aggregates. Equations are given to calculate MPT and its effect on compressive strength. The second type of effect concerns the bond between past and aggregate (bond effect), and a limitation of the strength that originates in the intrinsic strength of the rock (ceiling effect). Experiments have been performed on 13 mixtures made up with five sources of aggregate, in order to compare the strength development on pure paste versus composites (mortar and concrete). The quantitative assessment of the three effects of aggregate on compressive strength gives an accuracy close to 2.2 MPa on the studied mixes. Such a model is suitable to be incorporated into software for computer-aided mixture-proportioning and quality-control of structural concrete, up to the high-performance concrete range.