The congruency effect in distractor-interference tasks is typically smaller after incongruent trials than after congruent trials. Current views posit that this congruency sequence effect (CSE) reflects control processes that come into play when an irrelevant distractor cues a different response than a relevant target. However, the CSE is counterintuitively larger in the prime-probe task when the prime is occasionally a second target than when the prime is more frequently a distractor. In the present study, we investigated whether this effect occurs because the appearance of an occasional prime target (a) constitutes a rare, unexpected event that triggers heightened control or (b) allows participants to use the same task set (i.e., stimulus-response mapping) for the prime and probe in each trial. Consistent with the latter hypothesis, we observed this effect in Experiment 1 even when the critical trial types appeared equally often. Further, in Experiment 2, we extended this finding while ruling out perceptual differences between conditions as an alternative account. These findings provide novel support for the task set hypothesis and reveal that the CSE reflects control processes that do more than minimize distraction from irrelevant stimuli.