Achieving a stable video quality is an important task in video compression. In this paper, we present a multi-layer quantization control method to compress video to a certain target video quality based on the new hierarchical partition structure of H.265/HEVC. We first model the rate-quality characteristics of different video sequences using ν-support vector regression with a Gaussian radial basis function as the kernel function. Then, we propose a Kalman filter-based multi-layer quantization control method for the quality-constrained video coding. The advantage of our proposed approach in H.265/HEVC is demonstrated through experimental results. Compared with the other H.265/HEVC rate control algorithms and the fixed-QP scheme, the proposed method achieves a more stable video quality.