Background Birth order is associated with outcomes such as birth weight and adult socioeconomic position (SEP), but little is known about the association with adult height. This potential birth order-height association is important because height predicts health, and because the association may help explain population-level height trends. We studied the birth order-height association and whether it varies by family characteristics or birth cohort. Methods We used the Swedish Military Conscription Register to analyse adult height among 652 518 men born in 1951–1983 using fixed effects regression models that compare brothers and account for genetic and social factors shared by brothers. We stratified the analysis by family size, parental SEP and birth cohort. We compared models with and without birth weight and birth length controls. Results Unadjusted analyses showed no differences between the first two birth orders but in the fixed effects regression, birth orders 2, 3 and 4 were associated with 0.4, 0.7 and 0.8 cm (p<0.001 for each) shorter height than birth order 1, respectively. The associations were similar in large and small and high-SEP and low-SEP families, but were attenuated in recent cohorts. Birth characteristics did not explain these associations. Conclusions Birth order is an important determinant of height. The height difference between birth orders 3 and 1 is larger than the population-level height increase achieved over 10 years. The attenuation of the effect over cohorts may reflect improvements in living standards. Decreases in family size may explain some of the secular-height increases in countries with decreasing fertility.