In the existing ellipse fitting algorithms, the Lissajous figure is used to solve the demodulation error caused by the non-ideal 3×3 couplers. However, the influence of circuit noise and phase noise on Lissajous figure are not fully considered in the studies. In this work, an ellipse fitting demodulation method based on local outlier factor (LOF) algorithm is proposed, which can effectively eliminate the outlier points in Lissajous figure caused by noises. The proposed method proves to achieve the accurate demodulation of the signals by numerical simulations. In addition, the Monte Carlo analysis is used to obtain the comprehensive error rate of about 0.13%, which verifies the stability of the proposed method. Compared with the traditional least square fitting method, this method improves the stability and accuracy of demodulated signals.