Abstract The performance of mode-division multiplexing (MDM) system is normally limited to mode dependent loss (MDL) and differential mode group delay (DMGD). Where MDL seriously deteriorates the orthogonality of the transmission matrix, resulting the traditional equalization algorithm failure. DMGD causes large dimension of transmission matrix in time domain, leading to equalization difficulties. In this work, we propose a linear detector method based on complex lattice reduction (CLR) algorithm for MDM orthogonal frequency-division multiplexing (OFDM) system with MDL and DMGD. CLR algorithm as an effective mathematical tool which can improve the orthogonality of the transmission matrix. The OFDM modulation solves the problem of large transmission matrix dimension caused by DMGD. The proposed method nearly reaches the performance of maximum likelihood (ML) but with lower computational complexity. The simulation result shows that the optical signal to noise ratio (OSNR ) penalty of the proposed algorithm is 3dB lower than that of zero-forcing (ZF) and minimum mean square error (MMSE) equalizers at a transmission distance of 1200 km with G o f f s e t is 2dB.