RaspberrySet contains annotated images of the Raspberries (Rubus idaeus). The images were captured in the four development stages. The 2039 images in the dataset have a resolution of 1773 x 1773 pixels and were taken by iPhone XS. The annotation was carried out with the help of LabelImg software (version 1.8.6) manually be the experts. Annotations are presented in YOLO format. The dataset has five classes, which are Bud, Flower, Unripe Berry, Ripe Berry and Damaged buds. Out of 46659 annotations present in the dataset 11788 was for Buds, 4748 for Flowers, 29156 for Unripe Berries, 463 for Ripe Berries and 504 for Damaged Buds. The images were captured on the site at the Institute of Horticulture in Dobele, Latvia.