A new technique for combining the LMS and LMF cost functions is proposed. The resulting stochastic gradient adaptive algorithm uses a time varying mixing parameter to optimise a combination of the above cost functions, taking into consideration the noise statistics. Furthermore, the behaviour of the proposed algorithm is analysed and convergence conditions are established. Simulation results verify the ability of the algorithm to adapt itself to the noise characteristics, illustrate its enhanced performance and support very well the theoretic analysis. The continuous adaptation of the mixing parameter adds flexibility and enables rapid response of the algorithm to non-stationarities.