Visible light communication (VLC) technology is considered to be an attractive solution for future wireless indoor multimedia communication. To reduce the peak-to-average power ratio, single-carrier frequency division multiple access (SC-FDMA) with time domain asymmetrical clipping is applied for the downlink multi-user VLC system. An improved frequency-domain decision feedback equalizer (FD-DFE) structure is proposed for the SC-FDMA -based VLC system in this letter. Different kinds of subcarrier distributions are considered in the analyses and simulations. Thanks to the clipping noise information on the even subcarriers, which can reduce the decision error, the proposed algorithm provides a lower bit error rate and better transmission performance compared with traditional schemes.