In order to accurately model the game behaviors of market participants with bounded rationality in power market, this paper proposes the Cournot game dynamic model with bounded rationality considering transmission constraints, which is represented by the discrete difference equations involving the maximization problem of the social benefit of market. Then, by means of the smooth nonlinear complementarity function, the complex difference dynamic system is transformed into a set of familiar difference-algebraic equations. Based on the triopoly Cournot game model, the market dynamic behaviors are numerically simulated in different market parameters, in which the periodic or chaotic dynamic behaviors are focused when the market are out of the stability region.