Abstract Following the successful development of advanced driver assistance systems (ADAS), the current research directions focus on highely automated vehicles aiming at reducing human driving tasks, and extending the operational design domain, while maintaining a higher level of safety. Currently, there are high research demands in academia and industry to predict driver intention and understating driver readiness, e.g. in response to a “take‐over request” when a transition from automated driving mode to human mode is needed. A driver intention prediction system can assess the driver's readiness for a safe takeover transition. In this study, a novel deep neural network framework is developed by adopting and adapting the DenseNet, long short‐term memory, attention, FlowNet2, and RAFT models to anticipate the diver maneuver intention. Using the public “Brain4Cars” dataset, the driver maneuver intention will be predicted up to 4 s in advance, before the commencement of the driver's action. The driver intention prediction is assessed based on 1) in‐cabin 2) out‐cabin (road) and 3) both in‐out cabin video data. Utilizing K ‐fold cross‐validation, the performance of the model is evaluated using accuracy, precision, recall, and F1‐score metrics. The experiments show the proposed DIPNet model outperforms the state‐of‐the‐art in the majority of the driving scenarios.