This paper addresses a method of dynamically estimating relative pose between two rigid bodies from four corresponding point sets during motion in applications such as visual servoing and metrology for intelligent manufacturing. Instead of tracking each body individually relative to a camera, the underlying problem is how to track two bodies simultaneously and accurately. For this reason, the proposed method directly assigns the relative pose and motion as a state estimate. Further, an observation model is formulated in such a way that the state estimate is mapped to the observed point sets of two bodies. Since this observation model is nonlinear, an analytical expression of linearization is derived and state estimation by iterative extended Kalman filters (IEKF) is adapted to reduce the linearization errors. The proposed approach is implemented in simulations of position-based visual servoing and validated experimentally on a robotic riveting system for aircraft automated assembly.