Abstract This paper performs optimal coordination of directional overcurrent and distance relays jointly in a mixed protection scheme. This problem is a mixed non-linear programming problem (MNLP) with a high-level mathematical complexity. The mixed scheme variables are time dial setting (TDS), and the zone-2 time set (tZ2) parametrized as continuous and plug setting (PS) as discrete. This model considers all fault locations described to obtain a complete mixed optimal coordination. An extended continuous domain ant colony optimization (x ℝ -ACO) algorithm and an hybrid algorithm (traditional ACO plus linear programming (LP) solver) are implemented to solve this optimization problem. Initial solutions for ACO are randomly generated whether feasible or not. However, during the iterative ACO process, the algorithm converges into a feasible search space. Four transmission systems (IEEE-3, IEEE-6, IEEE-8, and IEEE-30 Bus) accordingly corroborate the accuracy and effectiveness of the proposed methodologies.