AVS2 is a next-generation audio and video coding standard currently under development by the Audio Video Coding Standard Workgroup of China. In this paper, a coefficient-group based transform coefficient coding design for AVS2 video coding standard is presented, which includes two main coding tools, namely, two-level coefficient coding and intra-mode based context design. The two-level coefficient coding scheme allows accurate coefficient position information to be used in the context model design and improves the coding efficiency. It also helps increase the entropy coding throughput and facilitate parallel implementation. The intra-mode based context design further improves coding performance by utilizing the intra-prediction mode information in the context model. The two coding tools combined provide consistent rate-distortion performance gains under standard test conditions. Both tools were adopted into the AVS2 working draft. Furthermore, an improved rate-distortion optimized quantization algorithm is designed based on the proposed scheme, which significantly reduces the encoder complexity.