This study presents an innovative hybrid optimization approach that combines teaching-learning based optimization (TLBO) with the whale optimization algorithm (WOA) for selective harmonic elimination (SHE) technique in a modified reduced switch topology three phase multilevel inverter (MLI). The proposed topology requires fewer switches than a conventional cascaded H-bridge MLI and another reduced switch topology in a single phase MLI. Once applied to an 11-level inverter, this hybrid strategy effectively tackles the issues of harmonic reduction and total harmonic distortion (THD) on the line-to-line voltage, significantly improving the quality of the output power through the optimal determination of switching angles. The study leverages the TLBO and WOA to solve the non-linear set of equations associated with the SHE controls technique, aiming to overcome the limitations of classical methods prone to local optimal solutions and dependent on initial controlling parameters. This method has been performed in two steps, during the first step TLBO has been executed and in the next step the solutions derived from TLBO has been used as an initial guess for WOA which ensures the attainment of the precisely converged solution. By using MATLAB®/Simulink software environment, the performance of the hybrid TLBO with WOA method has been simulated and benchmarked against traditional standalone metaheuristic techniques. The simulation results reveal that proposed hybrid approach becomes advantageous in terms of SHE and output voltage quality across various modulation indices. The experimental results verified that the proposed algorithm has been validated through the implementation of a three-phase 11-level inverter. This study highlights the significant potential of the hybrid optimization method in progressing harmonic minimization techniques within the multilevel inverters.