Due to the rapid advances in railway industry, the rail surface defect detection task which inspects whether the rail is defective has become an increasingly critical issue. Detecting rails by an automatic and swift approach instead of present manual inspections enables the work more efficient and safe currently. In this paper, we propose a novel two-stage pipeline method for rail defect detection by localizing and classifying rail images. Specifically, in the first stage, we get the cropped images which focus on the rail part instead of the whole-original image by integrating traditional image processing methods. In the second stage, we put the cropped images into a fine-tuned convolution neural network (CNN) and extract part-level features for rail images classification. Especially, in the rail image defect detection scenario, we should take recall into account to some extent, so we propose a novel loss function to leverage both of them in the second stage. The results show that the proposed method has strong robustness and achieves practical performance in defect detection precision.