Ultra-low-frequency oscillation with almost constant amplitude is a kind of frequency oscillation in power system. It has occurred in several hydro-dominant islanded power systems. With the nonlinearity of hydraulic turbines, a nonlinear model of power system is built in the paper. Stability and performance of both linear and nonlinear models are analyzed and compared. Particularly stable limit cycle could appear in the nonlinear dynamic model. Within a range of parameters, the equilibrium point of the dynamic system is unstable, while there is a stable limit cycle around it. System trajectories converge to this attractor and oscillate with constant amplitude. Limit cycle in the nonlinear model can explain the ultra-low-frequency oscillation of an island power system with ±800kV HVDC transmission to other grids in China and provides a new mechanism of frequency oscillation in islanded hydro-dominant power systems.