Since the initial objective of power system stabiliser (PSS) units and flexible AC transmission system (FACTS) devices is the damping of inter‐area and local oscillations, it is very substantial that the network electromechanical oscillatory modes are identified for tuning the controller of large‐scale interconnected power systems. In this research, a method based on the combination of continuous wavelet transform and Prony algorithm is presented to study low‐frequency electromechanical oscillations of the power system. The coordinated design of PSS and FACTS parameters become a multi‐objective optimisation problem that will be solved using the SPEA II. The residue method is used to improve damping and satisfy the controller constraints of the suggested power systems. Finally, the proposed method of mode estimation and controller design is simulated on the two well‐known test case systems, namely IEEE 2‐area 4‐generator and New England‐New York 68‐bus 16‐generator systems using MATLAB software. The results represent accurate performance and validity of the proposed method to improve oscillation damping, dynamic stability of the power systems.