A new video coding algorithm called the first-order-residual/second-order-residual (FOR/SOR) codec is proposed for high definition (HD) video coding in this work. Several advanced coding techniques are adopted in the proposed FOR/SOR codec. For the FOR codec, the well known block-based motion compensated predictive codec is used to exploit temporal and spatial correlations in input image frames. However, it is observed that there still exists structured residual signal after the FOR coding, and a SOR coder is developed to encode residual image frames efficiently. To improve the coding performance furthermore, we consider bit allocation between the FOR and SOR coders at the same block and determine their optimal quantization parameters systematically. It is shown by experimental results that the proposed FOR/SOR codec outperforms H.264/AVC significantly in HD video coding.