AbstractAdaptive cruise control (ACC) is the core building block of future full autonomous driving. Numerous recent research demonstrated that Autonomous Vehicles (AVs) adopting shorter headways generally increase road capacity and may relieve congestion at bottlenecks for moderate demand scenarios. However, with high demand scenarios, bottlenecks can still be activated causing capacity breakdown. Therefore, extra control measures as dynamic traffic control near bottlenecks is necessary. The challenge is harder on urban freeways with consecutive bottlenecks which affect each other. This paper aims to improve the performance of ACC systems in a high demand scenario. A multi‐bottleneck dynamic headway control strategy based on deep reinforcement learning (DRL) that adapts headways to optimize traffic flow and minimize delay is proposed. The controller dynamically assigns an optimal headway for each controlled section, based on state measurement representing the current traffic conditions. The case study is a freeway stretch with three consecutive bottlenecks which is then extended to include eight bottlenecks. Three different RL agent configurations are presented and compared. It is quantitatively demonstrated that the proposed control strategy improves traffic and enhances the system delay by up to 22.30%, and 18.87% compared to shortest headway setting for the three‐bottleneck and the eight‐bottleneck networks, respectively.