Olfactory modulation of vision is not well understood whereas visual modulation of olfaction has been more fully investigated. This study aimed to reveal in a simple manner whether there is olfactory modulation of colour working memory using an odour that induces a citrus-like smell and is associated with orange colours. We assumed that the odour would have modulatory effects on the colour information stored in working memory. To clarify whether these effects are supportive or disruptive, during the colour working memory task we measured an event-related potential component, P3, which is involved in attentional processes of working memory. The results indicated that odour presentation mediated a decline in the rate of correct guesses for orange colours. Furthermore, the odour suppressed P3 during reddish-colour retrieval, including orange. These results suggest that colour working memory in orange can be disrupted via olfactory modulation with citrus-like odours.