An automorphism of a group G is said to be a power automorphism if it maps every subgroup of G onto itself. The set PAut G of all power automorphisms of G is an abelian normal subgroup of the full automorphism group Aut G, whose properties were investigated by \textit{C. Cooper} [Math. Z. 107, 335-356 (1968; Zbl 0169.338)]. This paper deals with the structure of the group IAut G of all automorphisms of G leaving every infinite subgroup of G invariant (I-automorphisms). It is shown that, if the group G either is non-periodic or does not involve any infinite simple group, then IAut G is an abelian group. Moreover, when G is a (locally radical)-by-finite group which is not artinian, the groups IAut G and PAut G coincide. In the last part of the paper the structure of the group of I-automorphisms of a nilpotent p-group is investigated.