As the demand for higher data rates increases steadily, there will always be a need to develop more efficient wireless communication systems. Adaptive channel equalizers need to be used to correct the disturbing effects of the channel resulting from the time-varying mobile communication channel. One of the most popular channel equalization algorithms is the recursive least squares (RLS) algorithm. However, the performance of the fixed forgetting factor RLS algorithm is falling in the time-varying channel environment. In this paper, a new adaptive variable forgetting factor RLS (AVFF-RLS) algorithm based on the autocorrelation of the error signal has been proposed to increase the convergence speed and performance of the RLS algorithm. In order to test the performance of the proposed method, computer simulations are performed on frequency selective Rayleigh fading channels. It is observed that the proposed AVFF-RLS algorithm has better mean square error (MSE), bit error rate (BER) and channel tracking performances than the conventional RLS, conventional LMS, modified LMS (M-LMS) and normalized LMS (N-LMS) algorithms from the obtained simulation results.