Multimodal sensory integration is a ubiquitous neural process that can be modeled as optimal cue combination, incorporating both top-down, attention-like signals and bottom-up evidence that impact the precision of response variables. Accordingly, reducing attention or adding noise to one modality is expected to decrease proportionally its contribution while increasing that of the other modality. We tested this prediction using a gender-comparison task employing stimuli for which the face and voice were independently morphed between average male and female exemplars. Top-down influences were manipulated by having observers judge the stimuli with respect to either one or both modalities. Bottom-up influences were manipulated by introducing independent and varying amounts of visual and auditory noise. The contributions of each modality were estimated by maximum likelihood within a signal detection model of the decision process. As expected, if the attended modality was degraded by noise, the contribution of the unattended modality increased in compensation. Contrary to prediction, however, noise in the unattended modality had no impact on the attended modality. The results signal a departure from an optimal cue combination rule and are relevant to theories of predictive processes and observations in bimodal learning in modality-specific agnosia (prosopagnosia, phonagnosia).