Abstract This research evaluates the combustion behaviors of a single-element, swirl-coaxial injector in an atmospheric combustion chamber with gaseous oxygen and gaseous hydrogen (GO2/GH2) as the propellants. A brief simulated flow field schematic comparison between a shear-coaxial injector and the swirl-coaxial injector reveals the distribution characteristics of the temperature field and streamline patterns. Advanced optical diagnostics, i.e., OH planar laser-induced fluorescence and high-speed imaging, are simultaneously employed to determine the OH radical spatial distribution and flame fluctuations, respectively. The present study focuses on the flame structures under varying O/F mixing ratios and center oxygen swirl intensities. The combined use of several image-processing methods aimed at OH instantaneous images, including time-averaged, root-mean-square, and gradient transformation, provides detailed information regarding the distribution of the flow field. The results indicate that the shear layers anchored on the oxygen injector lip are the main zones of chemical heat release and that the O/F mixing ratio significantly affects the flame shape. Furthermore, with high-speed imaging, an intuitionistic ignition process and several consecutive steady-state images reveal that lean conditions make it easy to drive the combustion instabilities and that the center swirl intensity has a moderate influence on the flame oscillation strength. The results of this study provide a visualized analysis for future optimal swirl-coaxial injector designs.