Integrated approaches to the design of separation systems based on computer‐aided molecular and process design (CAMPD) can yield an optimal solvent structure and process conditions. The underlying design problem, however, is a challenging mixed integer nonlinear problem, prone to convergence failure as a result of the strong and nonlinear interactions between solvent and process. To facilitate the solution of this problem, a modified outer‐approximation (OA) algorithm is proposed. Tests that remove infeasible regions from both the process and molecular domains are embedded within the OA framework. Four tests are developed to remove subdomains where constraints on phase behavior that are implicit in process models or explicit process (design) constraints are violated. The algorithm is applied to three case studies relating to the separation of methane and carbon dioxide at high pressure. The process model is highly nonlinear, and includes mass and energy balances as well as phase equilibrium relations and physical property models based on a group‐contribution version of the statistical associating fluid theory (SAFT‐γ Mie) and on the GC+ group contribution method for some pure component properties. A fully automated implementation of the proposed approach is found to converge successfully to a local solution in 30 problem instances. The results highlight the extent to which optimal solvent and process conditions are interrelated and dependent on process specifications and constraints. The robustness of the CAMPD algorithm makes it possible to adopt higher‐fidelity nonlinear models in molecular and process design. © 2016 The Authors AIChE Journal published by Wiley Periodicals, Inc. on behalf of American Institute of Chemical Engineers AIChE J, 62: 3484–3504, 2016