The subjective experience of color by synesthetes when viewing achromatic letters and numbers supposedly relates to real color experience, as exemplified by the recruitment of the V4 color center observed in some brain imaging studies. Phenomenological reports and psychophysics tests indicate, however, that both experiences are different. Using functional magnetic resonance imaging, we tried to precise the degree of coactivation by real and synesthetic colors, by evaluating each color center individually, and applying adaptation protocols across real and synesthetic colors. We also looked for structural differences between synesthetes and nonsynesthetes. In 10 synesthetes, we found that color areas and retinotopic areas were not activated by synesthetic colors, whatever the strength of synesthetic associations measured objectively for each subject. Voxel-based morphometry revealed no white matter (WM) or gray matter difference in those regions when compared with 25 control subjects. But synesthetes had more WM in the retrosplenial cortex bilaterally. The joint coding of real and synesthetic colors, if it exists, must therefore be distributed rather than localized in the visual cortex. Alternatively, the key to synesthetic color experience might not lie in the color system.