Abstract Anti-backlash roller enveloping hourglass worm gear drives, as a new precision transmission system, has been extensively applied in robot joint design, and CNC (computer numerical control) machine tools. However, tooth profiles of the roller enveloping hourglass worms are spatial complex surfaces and very difficult to machine. The meshing and transmission properties such type of worm gear drive are especially sensitive to the errors in machining the tooth profiles of such worms. Thus, in order to improve the accuracy in machining their tooth profile surfaces and reduce the surface roughness, barriers to achieve precision machining of these complex surfaces have to be overcome. In the present study, an equation of grinding contact line for the roller enveloping hourglass worm gear drives with seven critical error parameters was for the first time established based on modern theory of gear meshing. Influence of each error parameter on the grinding contact line was decided through numerical analysis. Also, in order to find the best relationship between the grinding speed and the grinding feed rage, real contact patterns between the grinding rod and the machined worm tooth surface at different grinding speeds were observed using a metallurgical microscope and compared with the theoretical contact pattern calculated from the developed line of contact equation. Based on the obtained results, an innovative grinding method that can offset the loss of radius of the grinding rod was developed and a grinding process under the optimal rotational speed was presented. The proposed grinding process was then applied to generate the complex helical surface of a high precision roller enveloping hourglass worm. Outcomes of this study formed a profound theoretical and practical background for manufacturing and machining of the roller enveloping hourglass worm gear drives.