We present simulation results based on a phase-field model that describes the local migration of recrystallization boundaries into varying deformation energy fields. An important finding from the simulations is that the overall migration rate of the recrystallization front can be considerably affected by the variations in the deformed microstructure, resulting in two regimes. For variations with low amplitude, the overall boundary velocity scales with the average stored deformation energy density. This behavior is in agreement with generally accepted theories of recrystallization. For larger amplitudes, however, the velocity scales with the maximum of the deformation energy density along the variation, resulting in a considerably larger velocity than that obtained from standard recrystallization models. The shape of the migrating grain boundary greatly depends on the local characteristics of the varying stored deformation energy field. For different deformation energy fields, the simulation results are in good qualitative agreement with experiments and add information which cannot be directly derived from experiments.