Recently, advanced control methods like machine leaning are increasingly applied to autonomous vehicle. This paper focuses on velocity control in a right-turn traffic scenario. A Markov Decision Processes(MDPs) is modeled and the actor-critic reinforcement learning architecture is employed. Then the kernel-based least squares policy iteration algorithm(KLSPI) is applied. Simulation results show that the proposed method can perform different policy in different cases, which preliminarily verify the rationality.