This paper uses stop-level passenger count data in four cities to understand the nation-wide bus ridership decline between 2012 and 2018. The local characteristics associated with ridership change are evaluated in Portland, Miami, Minneapolis/St-Paul, and Atlanta. Poisson models explain ridership as a cross-section and the change thereof as a panel. While controlling for the change in frequency, jobs, and population, the correlation with local socio-demographic characteristics are investigated using data from the American Community Survey. The effect of changing neighborhood demographics on bus ridership are modeled using Longitudinal Employer-Household Dynamics data. At a point in time, neighborhoods with high proportions of non-white, carless, and most significantly, high-school-educated residents are the most likely to have high ridership. Over time, white neighborhoods are losing the most ridership across all four cities. In Miami and Atlanta, places with high concentrations of residents with college education and without access to a car also lose ridership at a faster rate. In Minneapolis/St-Paul, the proportion of college-educated residents is linked to ridership gain. The sign and significance of these results remain consistent even when controlling for intra-urban migration. Although bus ridership is declining across neighborhood characteristics, these results suggest that the underlying cause of bus ridership decline must be primarily affecting the travel behavior of white bus riders.