Purpose To address the issue that current dynamic models of drill strings typically simplify the mathematical modeling of the interaction between the drill bit and the rock, thus failing to accurately reflect the actual behavior of the drill bit, this study aims to investigate the dynamic behavior of the horizontal well drill string system equipped with a cone bit under different drilling parameters. Methods In this work, a dynamic mathematical model of the coupled nonlinear system of drillstring-cone bit-rock under the wellbore trajectory of a three-dimensional curved well is established. The drillstring is spatially discretized into multiple Euler–Bernoulli beam elements using the finite element method. The cone bit model is established, which fully accounts for state-dependent delay effects, and multiple cutters interactions. The interaction forces between the bit and the rock were introduced as boundary conditions for the drillstring model, while the displacement of the drillstring's terminal node is used as an input to the bit model, achieving a nonlinear coupling between the bit and the drillstring. The coupled nonlinear dynamic system is solved using the Newmark method combined with an improved Newton–Raphson iteration scheme. Results The numerical results indicate that increasing the rotational speed and the weight on bit (WOB) will intensify the vibration of the drill string, but it will improve the drilling efficiency. A higher rotational speed helps to suppress the sliding vibration of stuck pipe, while a larger WOB can effectively alleviate the axial stuck pipe sliding phenomenon. An increase in the local curvature of the wellbore trajectory will lead to an increase in the vibration intensity of the drill string at that location. An increase in the strength of the formation rock will cause an increase in the vibration intensity of the drill bit and a greater increase in the axial strength than the torsional strength. Conclusion These findings establish a solid theoretical foundation for controlling drillstring vibrations and offer guidance for optimizing drilling parameters and designing suitable bottom-hole assembly configurations.