AbstractAnticipatory thinking is necessary for managing risk in the safety‐ and mission‐critical domains where AI systems are being deployed. We analyze the intersection of anticipatory thinking, the optimization paradigm, and metaforesight to advance our understanding of AI systems and their adaptive capabilities when encountering low‐likelihood/high‐impact risks. We describe this intersection as the anticipatory paradigm. We detail these challenges in concrete examples and propose new types of anticipatory thinking, towards a paradigm shift in how AI systems are evaluated.