Abstract This paper deals with the development of an efficient hybrid approach based on genetic algorithm and particle swarm optimization for solving mixed integer nonlinear reliability optimization problems in series, series–parallel and bridge systems. This approach maximizes the overall system reliability subject to the nonlinear resource constraints arising on system cost, volume and weight. To meet these purposes, a novel hybrid algorithm with the features of advanced genetic algorithm and particle swarm optimization has been developed for determining the best found solutions. To test the capability and effectiveness of the proposed algorithm, three numerical examples have been solved and the computational results have been compared with the existing ones. From comparison, it is observed that the values of the system reliability are better than the existing results in all three examples. Moreover, the values of average computational time and standard deviation are better than the same of similar studies available in the existing literature. The proposed approach would be very helpful for reliability engineers/practitioners for better understanding about the system reliability and also to reach a better configuration.