The visual marking mechanism (Watson & Humphreys, 1997) allows new objects to be prioritized by applying top-down inhibition to a set of previewed distractors, increasing the efficiency of future visual search. However, if this inhibition results in little or no search facilitation, do people continue to apply it or do they strategically withhold it? Here we present 6 experiments in which we examined how participants control this inhibitory mechanism. Experiments 1 to 3 showed that in difficult search contexts, participants did not modulate the extent to which they applied inhibition based on the proportion of trials in which inhibition would have been useful. This was the case, even when explicitly cued before each trial as to the utility of applying inhibition (Experiment 4). In contrast, when search was conducted in predominantly easy search contexts, there was some evidence that inhibition was applied strategically (Experiments 5 and 6); however, the extent of this control was relatively modest. The findings are discussed in terms of the mechanisms of top-down attentional control and implications for failures of attention in real-world contexts.