Comparing with traditional fixed formation for a group of dynamical systems, time-varying formation can produce the following benefits: i) covering the greater part of complex environments; ii) collision avoidance. This paper studies the time-varying formation tracking for multiple manipulator systems (MMSs) under fixed and switching directed graphs with a dynamic leader, whose acceleration cannot change too fast. An explicit mathematical formulation of time-varying formation is developed based on the related practical applications. A class of extended inverse dynamics control algorithms combining with distributed sliding-mode estimators are developed to address the aforementioned problem. By invoking finite-time stability arguments, several novel criteria (including sufficient criteria, necessary and sufficient criteria) for global finite-time stability of MMSs are established. Finally, numerical experiments are presented to verify the effectiveness of the theoretical results.