Recently, video encryption has been widely investigated to enhance the protection for video data, yet the encryption efficiency may not be considered sufficiently so that some encryption schemes are unsuitable for the real-time applications. In this paper, we propose a novel chaotic selective encryption scheme (CSES) based on Context-Adaptive Binary Arithmetic Coding (CABAC) of H.264/Advanced Video Coding (H.264/AVC) for the practical applications. To satisfy the secure and real-time requirement, we select the important and sensitive encryption objects on the basis of the analysis of syntax elements in CABAC. According to the characteristics of chosen syntax elements, two encryption methods combined with two designed chaos-based key stream generators (KSGs) are presented in CSES to implement the video encryption with the reasonable and acceptable compression and encryption performance. The experimental results and security analysis demonstrate that the proposed CSES with the format-compliance has the good confidentiality and can resist some common security attacks, such as the brute force attack, histogram attack, information entropy attack, replacement attack and some other common attacks. It can be found from the comparison experiments about the encryption efficiency that our CSES has the higher real-time performance. The comparison analysis with other video encryption methods illustrates that the proposed CSES is more suitable for the practical applications.