Cutting force coefficients are the key factors for efficient and accurate prediction of instantaneous milling force. To calibrate the coefficients, this paper presents an instantaneous milling force model including runout and cutter deformation. Also, forming of surface error is analyzed, and a surface error model considering runout is proposed. Using surface errors of two experiments completed with the same cutting conditions but different axial depth only, cutter deformation is obtained. Then, a new approach for the determination of instantaneous cutting force coefficients is provided. The method can eliminate influences of the other factors except cutter deformation and runout. A series of experiments are designed, and the results are used to identify the parameters. With the evaluated coefficients and runout parameters, the instantaneous milling force and surface error are predicted. A good agreement between predicted results and experimental results is achieved, which shows that the method is efficient, and effect of runout on surface error is not negligible.