Much of the early visual system is devoted to sifting the visual scene for the few bits of behaviorally relevant information. In the visual cortex of mammals, a hierarchical system of brain areas leads eventually to the selective encoding of important features, like faces and objects. Here, we report that a similar process occurs in the other major visual pathway, the superior colliculus. We investigate the visual response properties of collicular neurons in the awake mouse with large-scale electrophysiology. Compared to the superficial collicular layers, neuronal responses in the deeper layers become more selective for behaviorally relevant stimuli; more invariant to location of stimuli in the visual field; and more suppressed by repeated occurrence of a stimulus in the same location. The memory of familiar stimuli persists in complete absence of the visual cortex. Models of these neural computations lead to specific predictions for neural circuitry in the superior colliculus.