Studying how the timing and amplitude of visual evoked responses (VERs) vary between visual areas is important for understanding visual processing but is complicated by difficulties in reliably estimating VERs in individual visual areas using noninvasive brain measurements. Retinotopy constrained source estimation (RCSE) addresses this challenge by using multiple, retinotopically mapped stimulus locations to simultaneously constrain estimates of VERs in visual areas V1, V2, and V3, taking advantage of the spatial precision of fMRI retinotopy and the temporal resolution of magnetoencephalography (MEG) or electroencephalography (EEG). Nonlinear optimization of dipole locations, guided by a group‐constrained RCSE solution as a prior, improved the robustness of RCSE. This approach facilitated the analysis of differences in timing and amplitude of VERs between V1, V2, and V3, elicited by stimuli with varying luminance contrast in a sample of eight adult humans. The V1 peak response was 37% larger than that of V2 and 74% larger than that of V3, and also ∼10–20 ms earlier. Normalized contrast response functions were nearly identical for the three areas. Results without dipole optimization, or with other nonlinear methods not constrained by prior estimates were similar but suffered from greater between‐subject variability. The increased reliability of estimates offered by this approach may be particularly valuable when using a smaller number of stimulus locations, enabling a greater variety of stimulus and task manipulations.Hum Brain Mapp 35:1815–1833, 2014. © 2013 Wiley Periodicals, Inc.