Forest fires cause devastating damage to the natural environment, making prompt and precise detection of smoke and fires in forests crucial. When processing forest fire images based on ground and aerial perspectives, current object detection methods still encounter issues, such as inadequate detection precision, elevated false detection and omission rates, as well as difficulties in detecting small targets in complex forest environments. Multi-task learning represents a framework in machine learning where a model can handle detection and segmentation tasks concurrently, enhancing the accuracy and generalization capacity for object detection. Therefore, this study proposes a Multi-Task Learning-based Forest Smoke and Fire Detection model (MTL-FSFDet). Firstly, an improved Bilateral Filtering-Multi-Scale Retinex (BF-MSR) method for enhancing images was proposed, to lessen the effect of lighting on smoke images and improve the quality of the dataset. Secondly, a Hybrid Feature Extraction module, which integrates local and global information, was introduced to distinguish between targets and backgrounds, addressing smoke and fire detection in complex backgrounds. Furthermore, Dysample, a method utilizing point sampling, was designed to capture richer feature information when dealing with small targets. In addition, a feature fusion approach based on Context Gate Aggregation (CGA) was proposed to weightedly fuse low-level and high-level features, boosting the precision in detecting small targets. Finally, multi-task learning improves the capability to detect small targets and tackle complex scenarios by sharing the feature extraction module and leveraging refined supervision of the segmentation task. The findings from the experiments show that, in comparison to the baseline model, MTL-FSFDet improved the mAP@0.5 by 5.3%.