Bit allocation is an essential issue for the rate-control performance of an H.264 scalable video encoder. In this paper, we propose an efficient bit allocation algorithm at the frame level for H.264 temporal scalable video coding. The bit budget for temporal layers is based on the target bit rate, the hierarchical level, the buffer constraints and the predicted mean absolute difference (MAD) of the current frame. This bit allocation algorithm is also extended to the spatial enhancement layers by considering the inter-layer MAD prediction and the relationship between the inter-layer target bit rates. Experimental results show that the proposed algorithm efficiently prevents the buffer overflow and underflow, achieves the acceptable accurate bitrates (with DBR less than 2%), and better visual quality, as compared to the state-of-the-art approaches in the literature.