The ability to select the most salient stimulus among competing ones is essential for animal behavior and operates regardless of the spatial locations that stimuli occupy. Here, we reveal that the brain employs a combinatorially optimized strategy to solve such location-invariant stimulus selection. With experiments in a key inhibitory nucleus in the vertebrate midbrain selection network, called isthmi pars magnocellularis (Imc) in owls, we discovered that the central element is a ‘multilobe’ neuron, which encodes visual locations with multiple firing fields. This multilobed coding of space is necessitated by scarcity of Imc neurons. Although distributed seemingly randomly in space, the locations of these lobes are optimized across the high firing Imc neurons, allowing them to cooperatively suppress stimuli throughout 2D visual space while minimizing metabolic and circuit wiring costs. Our work suggests that combinatorial coding of space by sparse inhibitory neurons may be a general functional module for spatial selection.