Abstract In this paper, we review the derivation of the Gauss–Levenberg–Marquardt (GLM) algorithm and its extension to ensemble parameter estimation. We explore the use of graphical methods to provide insights into how the algorithm works in practice and discuss the implications of both algorithm tuning parameters and objective function construction in performance. Some insights include understanding the control of both parameter trajectory and step size for GLM as a function of tuning parameters. Furthermore, for the iterative Ensemble Smoother (iES), we discuss the importance of noise on observations and show how iES can cope with non‐unique outcomes based on objective function construction. These insights are valuable for modelers using PEST, PEST++, or similar parameter estimation tools.