This article proposes the optimal coordination problem of protective relays within a hybrid optimisation framework which is presented based on integer coded genetic algorithm (ICGA) and non‐linear programming (NLP). The optimal coordination problem of directional overcurrent relays (DOCRs) is implemented while aimed at finding the optimal plug setting multiplier (PSM) and time dial setting (TDS). In this respect, PSM is a function of the current transformer (CT) size and the tap of the relay which is discrete in nature. TDS is a function of the operating time of the relay for different short‐circuit currents at different locations of the system. Here, the variables of the problem are decomposed into continuous and discrete variables. The first stage of the problem uses the ICGA to determine the size of CTs considering the permitted tap of relays while the second stage utilises the NLP method to evaluate the feasibility and optimality. The presented framework is then simulated on an 8‐bus test system. The obtained results verify the effectiveness and the applicability of the optimisation technique to find the optimal settings of DOCRs.