This paper considers the problem of tracking the players in handball videos using a single video source. Tracking of handball players in the video is a difficult task as they can frequently leave and re-enter the camera field of view, often change directions quickly and occlude each other. Players wear similar team uniforms, play indoor under artificial illumination, with the background than can vary significantly as the handball court is often painted in various colors. The continually improving accuracy of CNN-based object detectors makes tracking-by-detection methods an attractive choice for tracking players in sports videos as they can perform online and with low computational requirements on top of object detection. Here we consider the use of three tracking-by-detection methods in conjunction with the YOLO object detector, namely the standard Hungarian assignment algorithm, the Simple Online, and Real-time Tracking (SORT) algorithm that adds a motion model, and its extension Deep SORT. The methods are tested on a custom dataset of handball video scenes.