Driving automation has been developing rapidly during the latest decade. However, all current technologies of driving automation still require human drivers' monitoring and intervention. This means that during driving automation, the control by human driver and by the driving automation system are blended. In this case, if the human driver loses the sense of agency over the vehicle, he/she may not be able to actively engage in driving, and may excessively rely on the driving automation system. This review focuses on the subjective feeling of agency of the human driver over the vehicle in such situations. We address the possible measures of agency in driving automation, and discuss the insights from literatures on the sense of agency in joint control, robotics, automation, and driving assistance. We suggest that maintaining the sense of agency for human driver is important for ethical and safety reasons. We further propose a number of avenues for further research, which may help to better design an optimized driving automation considering human sense of agency.