This dataset represents 709 herbarium specimens of Streptanthus tortuosus, a widespread herb in California, which were scored for phenology. Reproductive structures on each sheet were counted by both human observers (manual) and by machine learning (mask R-CNN). These counts were used to calculate a quantitative metic of phenological status (the phenological index or PI). Collection date and relevant climate data (spring maximum temperature, degrees C; winter precipitation, mm) at the location and during the year of specimen collection (YOC) are also included in the dataset.