Abstract Nowadays, power system protection is increasingly important because of the growing number of customers and the pressing need for timely fault resolution and relay operations. This paper addresses the non‐linear nature of the objective function in the optimal coordination of directional overcurrent relays (DOCRs) by employing a quadratic Taylor series expansion around an operating point, converting the problem into a quadratically constrained quadratic programming problem, ensuring a global optimal solution with increased computational efficiency. Additionally, the quadratic constraints are converted into second‐order cone constraints for compatibility with the CPLEX solver. Using the least square method, the operating point values are determined and further fine‐tuned using iterations with the DOCR operation times. The IEEE 3‐bus, IEEE 8‐bus, and IEEE 14‐bus test systems are used to test the method, which shows higher improvement rates in reducing DOCR operation times and enhancing cooperation than conventional and metaheuristic methods. The simulation results verify the numerical superiority of the method in optimizing the protection system's efficiency while obtaining rapid and accurate solutions. The proposed method was tested on IEEE 3‐bus, 8‐bus, and 14‐bus systems, optimizing relay operating times to 0.87, 2.96, and 7.05 s, respectively, demonstrating the method's efficiency over conventional approaches.