Multi-agent collaborative perception has gained significant attention due to its ability to overcome the challenges stemming from the limited line-of-sight visibility of individual agents that raised safety concerns for autonomous navigation. Despite notable progress in collaborative perception, several persistent challenges hinder optimal performance, such as the size of data being shared, communication delays, computationally expensive collaboration mechanisms, and spatial misalignment. To address these challenges, we propose DelAwareCol, a versatile collaborative perception framework that tackles the transmission delay between connected agents in real-life autonomous driving. Our framework introduces three key modules designed to balance perception performance with communication bandwidth and delay. Firstly, an intra-agent information aggregation module captures valuable semantic cues within the temporal context to enhance the local representation of each ego agent. Secondly, an inter-agent information aggregation module manages inter-agent interactions and spatial relationships, addressing common vehicle-to-vehicle (V2V) and vehicle-to-everything (V2X) issues, such as spatial misalignment, asynchronous information sharing, and pose errors. Thirdly, an adaptive fusion mechanism integrates multi-source representations based on dynamic contributions from different agents. The proposed framework is validated on large-scale simulated and real-life collaborative perception datasets OPV2V, V2XSet, and V2VReal. Our experimental results demonstrate that DelAwareCol achieved state-of-the-art performance in collaborative object detection, maintaining robust performance in the presence of high latency and localization error.