This paper presents the investigation of a mirror system of a quasi-optical (QO) mode converter for a high-power coaxial-cavity gyrotron. The mirror system consists of three mirrors. The first mirror is a quasi-elliptical one. Based on the Katsenelenbaum-Semenov Algorithm (KSA), the second and the third mirrors are iteratively optimized as adapted phase-correcting mirrors to transform the outgoing wave beam into a fundamental Gaussian structure. The investigation shows that the focal length of the quasi-elliptical mirror has a great influence on the optimized conversion efficiency, and, hence, it should be chosen to match the asymptotic beam growth (ABG) angle well in order to obtain high conversion efficiency. The design of a mirror system has been performed for a 2 MW, continuous wave (CW), 170 GHz, and TE34,19 -mode coaxial-cavity gyrotron, which is under development at Forschungszentrum Karlsruhe, Germany. Taking into account the size of the mirrors and the conversion efficiency, a mirror system has been designed with a conversion efficiency of 98.3%