Previous research has shown that multitasking can have a positive or a negative influence on driving performance. The aim of this study was to determine how the interaction between driving circumstances and cognitive requirements of secondary tasks affect a driver's ability to control a car. We created a driving simulator paradigm where participants had to perform one of two scenarios: one with no traffic in the driver's lane, and one with substantial traffic in both lanes, some of which had to be overtaken. Four different secondary task conditions were combined with these driving scenarios. In both driving scenarios, using a tablet resulted in the worst, most dangerous, performance, while passively listening to the radio or answering questions for a radio quiz led to the best driving performance. Interestingly, driving as a single task did not produce better performance than driving in combination with one of the radio tasks, and even tended to be slightly worse. These results suggest that drivers switch to internally focused secondary tasks when nothing else is available during monotonous or repetitive driving environments. This mind wandering potentially has a stronger interference effect with driving than non-visual secondary tasks.