Object-driven interactions have various applications, especially in team activities and sports like volleyball, football, tennis, etc. We define a multi-party virtual world as an environment which includes multiple avatars and computer agents. Latency is one of the primary issues while building a multi-party setup. Delays due to latency cause inconsistencies and unnecessary halts resulting in unrealistic experience in VR. For interactions using objects, it becomes vital to handle the latencies. We present a time scaling algorithm for latency management and synchronization between multiple avatars, while they interact using objects in the VR environment. We propose two time scaling schemes—source and target based. Both these methodologies are dependent on which user’s virtual view (source or target) needs to be uniform. Incorporating the presented time scaling schemes, we develop a comprehensive multi-party virtual reality platform to simulate applications which involves interactions between agents and avatars using objects. We adopt a client–server architecture for building our multiuser VR platform which induces flexibility of adding multiple avatars and/or agents to the application. The modular nature of our proposed system enables extension to different VR applications that encompass communication using objects. We exemplify our framework by developing a multi-player VR volleyball game which employs the features of our proposed scheme. We demonstrate the significance of time scaling in the object-driven interactive multi-party VR framework through a comparative user study. We also evaluate the immersion experience in our multi-player volleyball game by a user survey.