Producing sufficient food to meet rising demand is a precondition to food security. While governance, trade, and social justice are critical dimensions of food access and thus food security, change in caloric sufficiency - adequate caloric supply to meet total demand - is a critical concern for resilience of the global food system in the face of climate and societal changes. Here we show, across five scenarios and assuming crop mix adaptation to new climate conditions, that global caloric sufficiency is likely to decrease, despite increased production, because those gains are outweighed by population growth. At a national scale, caloric sufficiency decreases consistently for most countries, and most countries facing hunger today remain vulnerable. Our results suggest that adapting crop mixes to new climate conditions will likely be insufficient to cope with global changes by 2050.