As water pollution intensifies globally, traditional water treatment methods face significant challenges in terms of efficiency and environmental sustainability. Rotational hydrodynamic cavitation technology, due to its eco-friendly and efficient characteristics, has attracted considerable attention in recent years. However, existing cavitation reactors often suffer from insufficient cavitation intensity, which limits their effectiveness. There is a critical need to optimize both structural design and operating parameters to enhance cavitation performance. This study introduces a novel rotating cavitation generator with a grooved-tooth structure. The effects of various structural parameters and operating conditions on cavitation performance are systematically investigated, and the transient unsteady flow field is analyzed. Cavitation intensity and energy efficiency are quantified using multidimensional indicators. Orthogonal design experiments are employed to extract the vapor-phase volume fraction at various time points, followed by range analysis and response surface methodology for parameter optimization. The results reveal the key factors influencing cavitation behavior and provide valuable insights for optimizing the structure and operational parameters of rotational hydrodynamic cavitation reactors, offering significant potential for practical engineering applications.