Through the angular momentum (AM) transport of the magnetized stellar wind, low-mass main sequence stars spin down. Long-time evolution of the stellar AM has been studied by theoretical modeling and numerical simulations. Although analytic models that assume solid body rotation explain observed basic trends, the wind toque inferred from stellar observations (Matt et al. 2015) yields the twice larger value than the measured one in the solar wind (Finley et al.2019) and the modeled spin-down rate is overestimated for stars rotating slower than the Sun (van Saders et al.2016). In order to solve these discrepancies, we consider the effects of the latitudinal differential rotation (DR) in the long-time evolution of stellar rotation by a simple model. Our model calculations show (1) if the sun has equator-fast rapid DR, the discrepancy of the solar-wind torque can be solved; (2) the spin-down rate of slow rotators with polar-fast DR is reduced. These results suggest that the DR is a key to understanding the evolution of stellar AM.