Optical structure stability is an important issue for far-infrared (FIR) phase measurements. To ensure good signal quality, influence of vibration should be minimized. Mechanical amelioration and optical optimization can be taken in turn to decrease vibration's influence and ensure acceptable measurement. J-TEXT (Joint Texal Experiment Tokamak, formerly TEXT-U) has two FIR diagnostic systems: a HCN interferometer system for electron density measurement and a three-wave polarimeter-interferometer system (POLARIS) for electron density and Faraday effect measurements. All use phase detection techniques. HCN interferometer system has almost eliminated the influence of vibration after mechanical amelioration and optical optimization. POLARIS also obtained first experimental results after mechanical stability improvements and is expected to further reduce vibration's influence on Faraday angle to 0.1° after optical optimization.