Static synchronous compensator (STATCOM) using self-commutated converter is one of the most effective flexible AC transmission systems (FACTS) devices for voltage control, power factor correction, and power system stability enhancement. However, to achieve such functionality, a proper and sufficient control system should be designed for this device. This paper proposes a new control scheme for STATCOM based on nonlinear predictive control system using optimal control approach for enhancing power system stability and voltage regulation. The proposed control scheme is applied to the mathematical model of STATCOM which is derived based on single-machine infinite bus power system installed with STATCOM. To validate the proposed stabilizing STATCOM controller, we perform simulations on single-machine double line infinite bus power system equipped with STATCOM using PSCAD/EMTDC software package under steady-state and dynamic stability conditions. The simulation results are encouraging and show the robustness and superiority of the proposed controller compared to conventional PI controller. The proposed STATCOM controller is effective: in damping low frequency oscillations resulting from small disturbance conditions, in maintaining transient stability and in improving dynamic performance of power system.