Nowadays, the network transmission and video encryption areas have urgently needed high-speed encryption systems for SM4 algorithm. To speed up the SM4 system in small area, three aspects in existing system is analyzed and optimized. Firstly, aiming at the prior encryption must wait long until completing all 32 rounds key expansion, a method is proposed for outputting round key in each key expansion to accelerate encryption response. Secondly, considering most user passwords are unchanged, we adopt additional memory for comparing old and new keys, so that key expansion can be cancelled sometimes. Thirdly, the paper analyses the relationship between key expansion and encryption/decryption algorithm in SM4. Using module reused technology, the designed key expansion module can also encrypt and decrypt, and the designed another two modules can both encrypt and decrypt. Therefore, the system can achieve three tasks’ synchronous encryption/decryption, which greatly improves the system's processing speed. This paper presents a hardware design scheme for the high-speed system. At last, the optimized design is realized in FPGA. The experimental results show that the design is feasible, and the SM4 encryption speed can increase fourfold.