The FireMan-UAV-RGBT dataset is a collection of UAV-captured RGB and thermal videos aimed at improving wildfire detection methods in boreal forests. Captured during four controlled burns in Finland from 2022 to 2023, the dataset includes 34 RGB videos and 20 RGB-Thermal paired videos. These high-resolution images have been annotated using both manual and semi-automatic methods to ensure accuracy. Using advanced drone technology, including the DJI Matrice 30T and Matrice 300 with Zenmuse H20T and H20N cameras, the dataset provides essential visual and thermal data for early fire detection and management. The annotations, formatted in YOLO, support machine learning applications in wildfire detection. To validate the dataset, state-of-the-art deep learning models were trained, showing the reliability and applicability of the FireMan-UAV-RGBT dataset in real-world scenarios. This dataset is publicly available to support researchers in developing new methods for wildfire management.