This data set includes supplementary files to our article \emph{"Deep learning image recognition of cow behavior near an automatic milking robot with an open data set"} (Leonardo Santiago Benitez Pereira, Olli Koskela, Ilpo Pölönen, Ilmo Aronen and Iivari Kunttu, in review, 2020). We acquired continuous video data over two-month period of cows in front of Automatic Milking Station (AMS). Each frame of the video is labeled as a single image belonging to one of classes described below. This data set includes 253 files of video data having in total \num{1526473} labeled frames. The data set used in the article consisted of 280 videos, but for privacy reasons, 27 video files including persons were removed from this data set, but their Java Script Object Notation (JSON) label files are left for, e.g., temporal analyses.