The distribution of market capitalization in the U.S. is highly concentrated. We investigate how this phenomenon impacts the difference in returns between small and large firms (i.e., the size premium). If the stock market is sufficiently concentrated (i.e., granular), large firms may carry a risk premium because their idiosyncratic risk is not diversified in the market portfolio. At the same time, prior work has shown that small firms may be allocated too little capital in concentrated stock markets, which could increase their expected returns. We find that the expected size premium increases by 13.33 percentage points per annum during periods of higher concentration, indicating that the capital allocation effect dominates. Evidence from a variety of tests on investor attention, equity financing, fundamental volatility, and capital intensity support this conclusion. Nonetheless, we also find evidence of an active granular diversification effect, as the size premium weakens following idiosyncratic increases in granularity.