Increased traffic on social media platforms, such as X (formerly known as Twitter), is associated with disaster events and consequently, fire incidents. Although detecting fires through social media has garnered research interest in recent years, managing the overwhelming volume of daily posts remains challenging. Efficient collection and filtering of fire-related posts are crucial for detecting fires through X. The FireXPosts dataset presented in this article is a collection of posts from Canada and Greece, binarily annotated to aid emergency responders effectively. We train and evaluate uni-modal and bi-modal models for filtering fire-relevant posts and establishing performance baselines on the FireXPosts dataset. Experimental results indicate relatively similar performance between the text modality and the best-performing bi-modal models, suggesting that visual supervision does not positively influence the outcomes of late fusion models.