Abstract Detonation engines are expected to be included in a number of aerospace thrusters in the future. Several types of detonation engines are currently under examination, including the rotating detonation engine (RDE). Although the RDE has been explored experimentally, its rotating detonation propagation mechanism is not well understood. This paper clarifies the detonation mechanism and dynamics of the RDE by 2D and 3D simulation using compressible Euler equations with a full chemical reaction mechanism of H2/O2 and H2/Air, especially from the triple-point and transverse detonation points of view. A total variation diminishing (TVD) scheme is used for the mixture of H2/Air, and an advection upwind splitting method difference vector (AUSMDV) scheme is used for the mixture of H2/O2. The use of an AUSMDV scheme provides a much clearer detonation structure than does the TVD scheme. We focus on the complex interaction mechanism of the detonation front and burned mixture gases. We found out that at this interaction point, an unreacted gas pocket appears and ignites periodically to generate transverse waves at the detonation front and maintain detonation propagation.