Flavor is perhaps the most multi-modal of all of our sensory experiences. Here flavor is defined as a perception that includes gustatory, oral-somatosensory, and retronasal olfactory signals that arise from the mouth as foods and beverages are consumed. Although the sights, sounds and smells of foods that occur just before, or in the absence of eating, can impact flavor perception, it is argued that these sensory signals exert their influence by creating expectations based upon prior associations. The primary aim of the paper is to review anatomical and neurophysiological data towards an understanding of how the core sensory signals combine in the central nervous system of humans. Based upon the extant literature it is proposed that taste, oral-somatosensory and olfactory inputs are first integrated in the anterior ventral insula. The core flavor percept is then conveyed to upstream regions in the brainstem and thalamus, as well as downstream regions in the amygdala, orbitofrontal cortex and anterior cingulate cortex to produce the rich flavorful experiences that guide our feeding behavior.