Abstract: This paper proposes a Morkov state transition model for an isolated intersection in urban traffic and formulates the traffic signal control problem as a Markov Decision Process(MDP). In order to reduce computational burden, a sensitivity-based policy iteration(PI) algorithm is introduced to solve the MDP. The proposed model is stage-varying according to traffic flow variation around the intersection, and the state transition matrices and cost matrices are updated so that a new optimal policy can be searched by the PI algorithm. The proposed model also can be easily extended from an isolated intersection to a traffic network based on the space-time distribution characteristics of traffic flow, so as the PI algorithm. The numerical experiments of a small traffic network show that this approach is capable of reducing the number of vehicles substantially compared with the fixed-time control particularly for high traffic demand, while being computationally efficient.