Caffeine has a significant effect on cerebrovascular systems, and the dual action of caffeine on both neural and vascular responses leads to concerns for the interpretation of blood oxygenation level‐dependent (BOLD) functional MRI. However, potential differences in the brain response to caffeine with regard to consumption habits have not been fully elucidated, as BOLD responses may vary with the dietary caffeine consumption history. The main aim of this study was to characterize the acute effect of caffeine on cerebral hemodynamic responses in participants with different patterns of caffeine consumption habits. Fifteen non‐habitual and 11 habitual volunteers were included in this study. The cerebral blood flow (CBF) and cerebrovascular reactivity (CVR) to the breath‐hold challenge were measured before and after 200 mg caffeine administration. The non‐habitual individuals exhibited a pattern of progressive reduction in CBF with time. The CVR was diminished in the caffeinated condition (P < 0.05). In the habitual group, the pattern of CBF decrease was smaller and homogeneous across the brain, and reached steady state rapidly. The CVR was not affected in the presence of caffeine (P > 0.05). Our results demonstrated that the cerebral hemodynamic response to caffeine was subject to the habitual consumption patterns of the participants. The compromised CVR following caffeine administration in the non‐habitual group may partially explain the suppressed BOLD response to a visual stimulation in low‐caffeine‐level users.