CausalOrca is a synthetic diagnostic dataset created through controlled simulations. It is designed to provide annotations of ground-truth causal effects and fine-grained agent categories for social interactions in multi-agent scenarios. The dataset is constructed using a modified RVO2 simulator and incorporates the ORCA optimization-based collision avoidance algorithm known for crowd simulation. With full control over scene configurations, the dataset enables the collection of motion behaviors in paired scenes before and after agent removal, generating a large set of counterfactual pairs with annotations of ground-truth causal effects. CausalOrca can serve as a valuable resource for studying and developing causally-aware neural representations of social interactions and trajectory prediction models. Please see the GitHub repository for a more detailed description of the dataset, including dataset statistics and documentation on how to use, visualize, and generate the data.