Roller preload is a critical factor influencing the transmission performance of zero-backlash high-precision roller enveloping reducers (ZHPRER). Due to the unique transmission structure of the ZHPRER and the self-spinning behavior of the worm wheel rollers, it is challenging to precisely regulate and measure the roller preload, as well as to clearly elucidate the mechanisms by which roller preload affects the reducer’s transmission performance; this significantly hinders the optimization of the ZHPRER’s transmission capabilities. To address this issue, this paper proposes a method for regulating roller preload using an eccentric bushing and introduces a performance testing methodology for the ZHPRER that explicitly accounts for roller preload. First, a dynamic simulation model of the ZHPRER, incorporating both roller preload and roller bearing clearance, is established; the validity of this model is subsequently verified by comparing its results with theoretical kinematic parameters. Building upon this foundation, a quantitative analysis is conducted to investigate how roller preload, worm speed, and worm wheel load influence the transmission error of the ZHPRER. Furthermore, a scheme for selecting the optimal roller preload is proposed. Finally, transmission performance tests for the ZHPRER are conducted using a dedicated test platform, and the experimental results are compared with the simulation data to provide comprehensive validation. This study offers significant theoretical support for the optimization of the transmission performance of roller-enveloping toroidal worm gear reducers.