When the quark-gluon plasma (QGP) - a system of deconfined quarks and gluons - is in a nonequilibrium state, it is usually unstable with respect to color collective modes. The instabilities, which are expected to strongly influence dynamics of the QGP produced in relativistic heavy-ion collisions, are extensively discussed under the assumption that the plasma is weakly coupled. We begin by presenting the theoretical approaches to study the QGP, which include: field theory methods based on the Keldysh-Schwinger formalism, classical and kinetic theories, and fluid techniques. The dispersion equations, which give the spectrum of plasma collective excitations, are analyzed in detail. Particular attention is paid to a momentum distribution of plasma constituents which is obtained by deforming an isotropic momentum distribution. Mechanisms of chromoelectric and chromomagnetic instabilities are explained in terms of elementary physics. The Nyquist analysis, which allows one to determine the number of solutions of a dispersion equation without explicitly solving it, and stability criteria are also discussed. We then review various numerical approaches - purely classical or quantum - to simulate the temporal evolution of an unstable quark-gluon plasma. The dynamical role of instabilities in the processes of plasma equilibration is analyzed.

104 pages, review article published in Physics Reports 682, 1 (2017)