This dataset contains all pictures used for the publication "Applying deep neural networks to predict incidence and phenology of plant pests and diseases" published in Ecosphere. We included three main folders: The folders "standardized_pictures" and "field_pictures" contain the full dataset with all pictures taken during the sampling for the study. The folder "categorized" contains pictures of the six main categories of damages which were used to train deep neural networks. Leaves for standardized pictures and pictures in the field were sampled weekly between April 15th and August 28th 2019 in three apple orchards in central Switzerland, in Kleinwangen, Gelfingen and Waedenswil (split into Weadenswil and Gottshalde in the dataset). We used cameras of two different smartphones, an IPhone 6 (8 megapixel camera) and a Sony Xperia X (23 megapixel camera). Detailed methods can be found in the publication. Codes can be found under: https://doi.org/10.5281/zenodo.5497319