We investigate the propagation of electromagnetic waves in a cylindrical waveguide with an arbitrary cross section filled with a nonlinear material. The electromagnetic field is expanded in the usual eigenmodes of the waveguide, and the coupling between the modes is quantified. We derive the wave equations governing each mode with special emphasis on the situation with a dominant TE mode. The result is a strictly hyperbolic system of nonlinear partial differential equations for the dominating mode, whereas the minor modes satisfy hyperbolic systems of linear, nonstationary, and partial differential equations. A growth estimate is given for the minor modes.