Abstract Based on the concept that the initiation of coal and gas outburst is the failure of coal and rock mass, the relevant stability theory is applied to build a nonlinear dynamics model of outburst initiation, and the failure mechanism of outburst initiation is systematically analyzed. The result show that the outburst initiation model can be described by a nonlinear partial differential equation set, and the outburst initiation failure depends on the solution stability of this equation set, this question can be strictly defined and determined by Liapunov's stability theory. The influencing factors of outburst initiation mainly include the structure forms, material properties and load patterns of the coal-rock mass system. The occurrence of abutment pressure and limit equilibrium zone in front of working face exacerbates the potential of outburst. There are 4 canonical forms of coal wall failure: extrusion, spallation, collapse and breakage. Outburst initiation is an integration of material failure and structure flexion, they abide by specific energy condition accordingly, but outburst initiation condition can’t be described by a simple energy equation.