The success and continuing progress of the three operating FELs based on energy recovery linacs (ERLs), the Jefferson Lab IR FEL upgrade, the Japan atomic energy agency (JAEA) FEL, and the Novosibirsk high power THz FEL, have inspired multiple future applications of ERLs, which include higher power FELs, synchrotron radiation sources, electron cooling devices, and high luminosity electron-ion colliders. The benefits of using ERLs for these applications are presented. The key accelerator physics and technology challenges of realizing future ERL designs, and recent developments towards resolving these challenges are reviewed.