Hybrid reactive-extractive distillation (RED) has been considered as a promising approach for the separation of ternary azeotropic mixtures that are widely existing in the chemical industry, due to its high energy efficiency. The optimal design of RED is quite challenging, primarily due to the problem’s inherent complexity involving both reactive and extractive distillation processes with recycle streams. Mathematical modeling of RED using rigorous models introduces highly nonlinear and nonconvex terms, which are difficult to optimize. This work proposes a systematic optimization framework to address these challenges by integrating advanced modeling, simulation, and optimization techniques. First, we develop a novel mixed-integer nonlinear programming (MINLP) optimization problem in which the pseudo-transient continuation technique is employed to construct a pseudo-transient continuation reactive distillation model in an equation-oriented environment to represent column sections, condensers, and reboilers. Our previously proposed feasible path-based branch and bound algorithm is then employed to solve this difficult MINLP problem. Given the presence of recycle streams within the process, an open-loop MINLP optimization problem is first solved to achieve an optimal design based on product specifications, followed by a closed-loop nonlinear programming optimization problem for a fixed column structure to reduce the computational effort. The design of triple-column reactive-extractive distillation (TCRED) and double -column reactive-extractive distillation (DCRED) processes in three examples demonstrates that the proposed framework enables efficient RED process design, which can reduce total annualized cost by 3.5% to 25.1% compared to existing studies.