In this paper, a numerical investigation on the nonlinear distortion of OFDM (Orthogonal Frequency Division Multiplexed) signals caused by RSOA (Reflective Semiconductor Optical Amplifier) as an intensity modulator in OFDM-PON system is undertaken. Based on curve fitting approach a fourth order polynomial is, for the first time, used to present the nonlinear gain of RSOA. The expression of in-band and adjacent-band noise due to nonlinear distortion is then derived. Besides, the EVM of system versus input signal power is simulated by VPItransmissionmaker7.6, the results indicate the nonlinear effects-induced degradation of transmission performance depends highly on the third order term of nonlinear distortion power, and the optimum input signal power is provided to improve the transmission performance. Finally, it's worth mentioning that the quantified analysis method is applicable for the nonlinearity analysis of any RSOA-Based OFDM-PON system.