Ego-motion (self-motion) of the camera is a considerable problem in outdoor rover applications. Stereo tracking of a single moving target is a difficult problem that becomes even more challenging when rough terrain causes significant and high-acceleration motion of the camera in the world. The paper discusses the use of inertial measurements to estimate camera ego-motion and the use of these estimates to augment stereo tracking. Some initial results from an outdoor rover are presented, illustrating the efficacy of the method. The method causes fast but predictable image location transients, but reduces the amplitude of image location transients due to the rough terrain.