This paper addresses the issue of power system transient stability assessment and control (TSA&C) based on phase plane representation of the power angle and rotor speed. First, the relationship between transient stability–instability and concavity–convexity of the phase‐plane trajectory is found and proved based on the single machine infinite bus (SMIB) system, and a transient instability criterion is derived for real‐time instability detection. Second, with the aim that the phase‐plane trajectory swings back before the unstable equilibrium point (UEP) once instability is detected, near‐optimal control size of generator tripping to stabilize the system is determined based on the relationship between unbalanced power and the slope of the phase‐plane trajectory. Finally, the instability detection and control technique is extended to a multimachine power system based on SMIB equivalent transform. The proposed method is independent of the network structure and operation mode, which can be applied for real‐time emergency control based on wide area measurement system (WAMS). The simulations in the Institute of Electrical and Electronics Engineers (IEEE) 39‐bus system and the Sanhua Power Grid (in China) validate the accuracy and efficiency of the proposed method. © 2019 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.