This paper proposes a new nonlinear time-varying transverse-torsional coupled model for a compound planetary gear sets. In this model, the motion dimensionless differential equations of the gear system are established based on Lagrange equation and the equations are solved, from which the result of the nonlinear dynamic response is obtained. The influences of excitation frequency, damping coefficient and backlash on the bifurcation and chaos characteristics of the system are obtained through dynamic bifurcation diagram, time history curve, phase trajectory and Poincare map. The analysis results show that the coupling of time-varying meshing stiffness, transmission errors, gear backlashes and dimension tolerance make the inner nonlinear dynamic behaviors various for system. Excitation frequency and backlash have strong effects on the dynamic behaviors of the system. The motion state of the system becomes complex within the backlash increasing. The region of chaotic motion would be suppressed and the stable periodic motion would be increased by increasing the damping coefficient. Meanwhile, the routes of period doubling and crisis to chaotic motion were observed under different bifurcation parameters.