Flax, an important oil and fiber crop, is widely cultivated in temperate and sub-frigid regions worldwide. China is one of the major producers of flax, with Gansu Province predominantly practicing cultivation in hilly areas. However, common issues such as feeding difficulties, stem entanglement, and low threshing efficiency significantly restrict the improvement of planting efficiency. This study addresses the key technical challenges in flax combine harvesting in hilly regions by developing a discrete element model of the flax plant and utilizing DEM-FEA co-simulation technology. The performance of two threshing drum models (T1 and T2) was analyzed, focusing on motion trajectory, stress distribution, and threshing effects. The simulation results show that the T2 model, with its combination of rib and rod tooth design, significantly improves threshing and separation efficiency. The loss rate was reduced from 5.6% in the T1 model to 1.78% in the T2 model, while the maximum stress and deformation were significantly lower, indicating higher structural stability and durability. Field validation results revealed that the T1 model had a total loss rate of 3.32%, an impurity rate of 3.57%, and an efficiency of 0.09 hm2/h. In contrast, the T2 model achieved a total loss rate of 2.29%, an impurity rate of 3.39%, and an efficiency of 0.22 hm2/h, representing a 144.4% improvement in working efficiency. These findings indicate that the T2 model has a higher potential for flax harvesting in hilly and mountainous regions, especially in improving threshing efficiency and operational stability, providing an important theoretical basis for optimizing threshing equipment design.