We study automorphisms of a relatively hyperbolic group G . When G is one-ended, we describe Out (G) using a preferred JSJ tree over subgroups that are virtually cyclic or parabolic. In particular, when G is toral relatively hyperbolic, Out (G) is virtually built out of mapping class groups and subgroups of \mathrm {GL}_n(\mathbb Z) fixing certain basis elements. When more general parabolic groups are allowed, these subgroups of \operatorname{GL}_n(\mathbb Z) have to be replaced by McCool groups: automorphisms of parabolic groups acting trivially (i.e. by conjugation) on certain subgroups. Given a malnormal quasiconvex subgroup P of a hyperbolic group G , we view G as hyperbolic relative to P and we apply the previous analysis to describe the group Out (P\nearrow G) of automorphisms of P that extend to G : it is virtually a McCool group. If Out (P\nearrow G) is infinite, then P is a vertex group in a splitting of G . If P is torsion-free, then Out (P\nearrow G) is of type VF, in particular finitely presented. We also determine when Out (G) is infinite, for G relatively hyperbolic. The interesting case is when G is infinitely-ended and has torsion. When G is hyperbolic, we show that Out (G) is infinite if and only if G splits over a maximal virtually cyclic subgroup with infinite center. In general we show that infiniteness of Out (G) comes from the existence of a splitting with infinitely many twists, or having a vertex group that is maximal parabolic with infinitely many automorphisms acting trivially on incident edge groups.