We examine recrystallization and grain growth during processing of accumulative roll-bonded (ARB) Cu-Nb and Zr-Nb composites. Throughout the ARB process, from initial millimeter thick layers down to nanometer thick layers, the mechanism for recrystallization and grain growth is the motion of high-angle grain boundaries (HAGBs). However, the driving forces for these phenomena change as the densities of different types of defects evolve during the process. The creation and redistribution of dislocations, grain boundaries, and phase boundaries has significant effects on recrystallization and grain growth and, thus, on microstructural evolution. Both Cu-Nb and Zr-Nb exhibit a distinct transition in recrystallization and growth behavior at around 500-nm average layer thicknesses. For the thicker layered materials, the microstructure evolution during recrystallization and growth is determined by the density and distribution of dislocations and HAGBs. For layers less than 500 nm, the layers are largely one-grain thick and the grains are nearly dislocation free; coarsening of grains within layers at the nanoscale is due to reduction in phase boundary energy.