Studies on the temporal dynamics of attention have shown that the report of a masked target (T2) is severely impaired when the target is presented with a delay (stimulus onset asynchrony) of less than 500 ms after a spatially separate masked target (T1). This is known as the attentional dwell time. Recently, we have proposed a computational model of this effect building on the idea that a stimulus retained in visual short-term memory (VSTM) takes up visual processing resources that otherwise could have been used to encode subsequent stimuli into VSTM. The resources are locked until the stimulus in VSTM has been recoded, which explains the long dwell time. Challenges for this model and others are findings by Moore, Egeth, Berglan, and Luck (1996) suggesting that the dwell time is substantially reduced when the mask of T1 is removed. Here we suggest that the mask of T1 modulates performance not by noticeably affecting the dwell time but instead by acting as a distractor drawing processing resources away from T2. This is consistent with our proposed model in which targets and masks compete for attentional resources and attention dwells on both. We tested the model by replicating the study by Moore et al., including a new condition in which T1 is omitted but the mask of T1 is retained. Results from this and the original study by Moore et al. are modeled with great precision.