In the fields of continuous casting and the roll stepped cooling, the heat transfer coefficient is piecewise linear. However, few papers discuss the solution of the backward heat conduction problem in this situation. Therefore, the aim of this paper is to solve the backward heat conduction problem, which has the piecewise linear heat transfer coefficient. Firstly, the ill-posed of this problem is discussed and the truncated regularized optimization scheme is introduced to solve this problem. Secondly, because the regularization parameter is the key factor for the regularization method, this paper presents an improved method for choosing the regularization parameter to reduce the iterative number and proves the fourth-order convergence of this method. Furthermore, the numerical simulation experiments show that, compared with other methods, the improved method of fourth-order convergence effectively reduces the iterative number. Finally, the truncated regularized optimization scheme is used to estimate the initial temperature, and the results of numerical simulation experiments illustrate that the inverse values match the exact values very well.