The congruency effect in distracter interference tasks is often reduced after incongruent as compared to congruent trials. Here, we investigated whether this congruency sequence effect (CSE) is triggered by (a) attentional adaptation resulting from perceptual conflict or (b) contingent attentional capture arising from distracters that possess target-defining perceptual features. To distinguish between these hypotheses, we varied the perceptual format in which a distracter (word or arrow) and a subsequent target (word or arrow) appeared in a prime-probe task. In Experiment 1, we varied these formats across four blocks of a factorial design, such that targets always appeared in a single perceptual format. Consistent with both hypotheses, we observed a CSE only when the distracter appeared in the same perceptual format as the target. In Experiment 2, we varied these formats randomly across trials within each block, such that targets appeared randomly in either format. Consistent with the attentional capture account but inconsistent with the perceptual conflict account, we observed equivalent CSEs in the same and different perceptual format conditions. These findings show for the first time that contingent attentional capture plays an important role in triggering the CSE.