Group activity recognition (GAR) is a challenging task in computer vision, which needs to comprehensively model the spatiotemporal relations among actors. However, most previous methods tend to only model unitary actor relations and directly aggregate actor features to form group representation at a single scale. To address these issues, we propose a novel GAR approach termed multi-scale cross-distance transformer (MSCD-Former), capable of capturing diverse actor relation contexts in multiple spatiotemporal scales. A cross-distance attentive block (CDA-Block) is designed to decompose the actor relations into local and distant ones, diversifying the relation features in rearranged groups. The multi-scale group descriptors are then enhanced by deploying stacked CDA-Blocks to cascaded stages and tightening the sampling scales accordingly. Moreover, we introduce a multi-scale reconstructive learning measure (MSR-Learning) between adjacent scales of CDA-Blocks. Via the reconstruction of actor relational features from lower scales to upper scales, MSR-Learning can enforce semantic consistency in multiple spatiotemporal scales. Consequently, our MSCD-Former boosts GAR by fusing such discriminative relation features of different scales. We extensively evaluate the proposed approach on the VolleyTactic, Volleyball, Collective Activity, NBA, and JRDB-PAR datasets, and the experimental results demonstrate its superiority.