Vision and robotics has the well‐defined goal of meeting or exceeding human‐level capabilities in perception, locomotion, and manipulation. Not surprisingly, that is perhaps easier said than done. Beginning in the 1970s, the Defense Advanced Research Projects Agency started the ambitious Imaging Understanding program that would continue for more than 20 years. The Imaging Understanding program began with fundamental research and slowly evolved into a host of more applied efforts with specific systems goals. Robotics programs followed a similar arc as the early research‐oriented programs generated capabilities from which practical systems could be built. A culmination of the vision and robotics research was the Defense Advanced Research Projects Agency Grand Challenge, which turned the impossibility of a self‐driving car into an imminent reality. This article tells the story of how some of the modern‐day technologies we enjoy today trace their evolution from research sponsored by the Defense Advanced Research Projects Agency over the last 40 years.