Participants memorized lists of visually presented digits in silence or while ignoring distractor sounds that either came from the front and thus from the direction in which participants' attention was oriented, or from behind. Distractor sounds impaired recall performance, but the largest impairment was observed when the sound source was directionally close to the frontal visual target display. The results are consistent with the assumption of cross-modal attentional links in models of attention, and they are problematic for explanations of the irrelevant-sound effect within working memory models that do not specify an explicit role of attention in the maintenance of information for immediate serial recall.