Despite important recent progress in our understanding of brain evolution, controversy remains regarding the evolutionary forces that have driven its enormous diversification in size. Here, we report that in passerine birds, migratory species tend to have brains that are substantially smaller (relative to body size) than those of resident species, confirming and generalizing previous studies. Phylogenetic reconstructions based on Bayesian Markov chain methods suggest an evolutionary scenario in which some large brained tropical passerines that invaded more seasonal regions evolved migratory behavior and migration itself selected for smaller brain size. Selection for smaller brains in migratory birds may arise from the energetic and developmental costs associated with a highly mobile life cycle, a possibility that is supported by a path analysis. Nevertheless, an important fraction (over 68%) of the correlation between brain mass and migratory distance comes from a direct effect of migration on brain size, perhaps reflecting costs associated with cognitive functions that have become less necessary in migratory species. Overall, our results highlight the importance of retrospective analyses in identifying selective pressures that have shaped brain evolution, and indicate that when it comes to the brain, larger is not always better.