To encourage research in computer vision for the railway, we present Rail-5k: a real-world image dataset for object detection of defects and accessories on the rail, along with methods for shooting, fine-frained category definition, and instance-level annotation. We collected 5,000 high-quality RGB images from high-speed railway and subway across China, where each image with resolution as high as 0.03mm per pixel. We annotate 1100 images with 13 types of defects and accessories that are the most important to rail maintenance such as rail surface, wheel-rail contact band, crack, spalling, corrugation, fastening, screw. The dataset is superior to existing datasets in image quantity, resolution, annotation quality, dense and small objects. It also contains real-world corrupted images with dark, overexposure, blur, other tools, different lens distance, category transition, different screws, which are infeasible for non-experts to annotate and recognize. As a pilot study of rail defect detection, we perform comprehensive experiments using SOTA models. Our experiments demonstrate several challenges Rail-5k posed to both computer vision and railway engineering. Future versions of this dataset will include even more images, segmentation annotations as well as more channels.