Due to the time-varying nature of wireless channels, deterministic quality of service (QoS) is hard to guarantee in wireless networks. In this paper, by integrating information theory with the principle of effective capacity, we formulate an energy efficiency optimization problem with statistical QoS guarantee in the uplink of two-tier femtocell networks. To solve the problem, we introduce a Q-learning mechanism based on Stackelberg game framework, in which macro-user acts as a leader, and knows all femto-users' transmit power strategy; while femto-users are followers, and only communicate with macrocell base station (MBS) not with other femtocell base stations (FBS). In Stackelberg game studying procedure, macro-user selects transmit power level firstly based on the best responses of femto-users, femto-users interact with environment directly, and find their best responses. At last, a distributed Q-learning algorithm is proposed. Simulation results show the proposed algorithm has a better performance in terms of convergence speed while providing delay QoS provisioning.