The "Date Fruit Detection Dataset for Computer Vision-Based Automatic Harvesting" is a collection of videos and images showcasing date fruits from four Moroccan varieties, namely Majhoul, Boufagous, Bouisthami, and Khoult. This dataset is specifically designed to detect and classify date fruits, with the primary goal being the automation of the harvesting process. All the images in this dataset were captured in two orchards located in Morocco, with the first orchard situated in the southeast of Errachidia and the second in Tismoumine, Alnif, Tinghir. These images were taken under various natural conditions, encompassing differing lighting, contrast, shadows, and instances where the dates were concealed by bags or hidden beneath palm leaves. The dataset was meticulously compiled over the period spanning from June to September 2022, ensuring comprehensive coverage of all four maturity stages of date fruits, which include immature, khalal, rutab, and tamer. The dataset is intended for both object detection and classification purposes, and it includes a YOLO annotation txt file for each image. These annotations have been tailored to precisely recognize not only the date fruit but also to distinguish the specific variety and its maturity stage.