Many high-end computing systems use an extremely large number of power-hungry commercial components to achieve high performance. Power reduction and energy conservation are important in these systems for the reason of minimizing operating cost. Two main mechanisms are commonly applied to power reduction in these systems: Dynamic Voltage/ Frequency Scaling (DV/FS) and server number controlling: Vary-On Vary-Off (VOVF). Many previous work addressed the power management issue as an optimization problem and utilized these two mechanisms separately or together to reduce power consumption. In this paper, we construct a Constrained Markov Decision Process (CMDP) model for power management in web server clusters. Unlike previous optimization model which obtains the optimal power management solution on-line, our model provides an off-line power control policy that can greatly reduce online computation time. It also takes advantages of both DV/FS and VOVF mechanisms together for power reduction. The constructed CMDP model provides controllable and predictable quantitative control of power consumption with theoretically guaranteed service performance. It is further evaluated via extensive simulations and justified by the real workload data trace. The results prove the effectiveness and efficiency of the proposed model.