Abstract Gear backlash has an import influence on gear dynamics and due it has a strong non-linear character, so its effect on the gear dynamics is very complex. In order to get reasonable and precise dynamic analytic result, the fractal theory is employed here to express the backlash instead of the existed way of setting backlash as a fixed value or a steady random number that meets the normal distribution. Though the simulation and comparison of three methods: fractal method, fixed value and normal distribution, it is clearly concluded that fractal method is better than the two latter ways which both have the weakness of being unable to reveal the influence when backlash is changing in a small scope; also the normal distribution method cannot get the unique answer when average backlash is set. Furthermore, the results also show that it is advantageous to improve gear dynamics with decreasing fractal dimension, reducing time-variant stiffness and increasing damping ratio. The theory in this paper shows a new method to analyze the gear dynamics in respect of the changing backlash and as well as builds an indirect relationship between gear dynamic performance and tooth surface characters, which can be utilized to improve gear dynamics characters through guiding gear surface design and manufacture in the future.