<p>Traffic video analytics has become one of the core components in the evolution of transportation systems. Artificially intelligent traffic management systems apply computer vision techniques to alleviate the monotony of manually monitoring the video feed from surveillance cameras. Locating the objects of interest is the most crucial step in the pipeline of such video analytics systems. An abundance of research has been conducted to find the location of the targets in traffic scenes. This paper presents a comprehensive review of different algorithms used for object detection in traffic surveillance applications in addition to the recent trends and future directions. Based on the approaches used in the related studies, we categorize the object detection methods into motion-based and appearance-based techniques. We further classify each group of techniques into a number of subcategories and analyze the advantages and disadvantages of each method. The major challenges, limitations, and potential solutions are also discussed along with the future scope and directions.</p>