The annotated dataset for Japanese quince - QuinceSet - consists of images of Japanese quince (Chaenomeles japonica) fruits taken at two phenological developmental stages and annotated for detection and phenotyping. First, after flowering, when the second fruit fall is over and the fruits have reached 30-50% of their final size, and second, at the ripening stage of quince, just before the fruits are yielded. Both stages of quince images classified as unripe and ripe were annotated using ground truth ROI and presented in YOLO format. The dataset contains 1515 high-resolution RGB .jpg images with the same number of annotated .txt files. Images in the dataset were manually annotated using LabelImg version 1.8.6 software. A total of 17 171 annotations were provided by the experts. The images were acquired on site at the Institute of Horticulture in Dobele, Latvia. Homogenization of the images was performed under different weather conditions, at different times of the day, and from different capturing angles. The dataset contains both fully visible quinces and quinces partially obscured by leaves.