The paper is devoted to the description of a new technique (numerical and experimental) identification of the dependences between cutting forces and instantaneous chip thickness. It is required to measure only the cutting forces versus cutting conditions. Experimentally, for the given pair the processed material – the tool, coefficients of the cutting forces model are calculated by means of the optimization method (Nelder-Mead algorithm). The mathematical model of the milling process developed by authors on each step of the Nelder-Mead method is used for the process numerical simulation under given coefficients of the cutting force model. The elaborated numerical modeling algorithm allows investigating the dynamics and the kinematics of the milling process. The dynamic model of the tool, the algorithm of geometrical modeling of the instantaneous chip thickness, the finite element model of the detail are embedded into the whole model of the milling process.