Purpose: Automatic tracking and detection of motile cells in time-lapse mode during microscopy is an important requirement in many biological applications. Although significant developments have been made in cell tracking algorithms, current datasets are limited in size and diversity, especially for data-dependent generalized deep learning models. In this paper, we introduce a new and larger standard sperm tracking dataset with a useful framework for assessing motile cell tracking algorithms. Acquisition and validation methods: Semen samples were used to obtain image datasets. The sequences of motile sperm data were acquired using a microscope. We recorded motile sperm from different donors under different magnification, to provide a suitable sperm pool and achieve a degree of image heterogeneity. Sperm images were manually annotated and checked by several biologists using semi-automatic software to generate the dataset. Data format and usage notes: We present a new dataset, EVISAN – Expert Visual Sperm Annotation, comprising partially annotated sperm images from different donors at a varying magnification that is readily usable as training data for computer vision applications. With 6,000 images, our collection is an unparalleled heterogeneous dataset for deep learning sperm detection application development. All images are stored in XML and JPEG formats. We also provide a standard evaluation framework for the proposed dataset. Potential applications: Our standard dataset is highly suitable for quantitatively evaluating state-of-the-art, multi-target tracking and detection algorithms. We also foresee the clinical application of these algorithms in assisted reproduction in humans, such as real-time computer-assisted sperm analysis.