Theories of adaptation typically ignore the effect of environmental change on population size. But some environmental challenges--challenges to which populations must adapt--may depress absolute fitness below 1, causing populations to decline. Under this scenario, adaptation is a race; beneficial alleles that adapt a population to the new environment must sweep to high frequency before the population becomes extinct. We derive simple, though approximate, solutions to the probability of successful adaptation (population survival) when adaptation involves new mutations, the standing genetic variation, or a mixture of the two. Our results show that adaptation to such environmental challenges can be difficult when relying on new mutations at one or a few loci, and populations will often decline to extinction.