This dataset and software accompany the article "Label-free deep learning based species classification of bacteria imaged by phase-contrast microscopy" for reproducing the results. The study applies deep-learning based image and video models for species classification of the bacteria Enterococcus faecalis, Escherichia coli, Klebsiella pneumoniae, and Pseudomonas aeruginosa growing in microfluidic chip traps.