Detection of railway shape and dangerous obstacles plays a critical role in the auxiliary driving of the train. Speed and accuracy are both of great significance to real-world railway traffic detection, which demands a higher efficiency and effectiveness. The goal of this paper is to design an architecture that achieves the right speed (for effectiveness)/accuracy (for effectiveness) balance for actual railway detection. Driven by this motivation and based on the advantages of some current algorithms, we propose FB-Net (faster better network), a robust end-to-end convolutional neural network. Detectors based on deep learning method are composed of feature extraction, candidate region generation,and classification. Specifically, our framework is focusing on with three embedded modules: 1) To improve efficiency, we replace standard convolutions with depthwise-pointwise convolutions in the feature extraction stage, aiming to red reduce model parameters; 2) To address the effectiveness, a priori module is added for candidate boxes to provide a coarse location for subsequent regressor and to reduce the searching space of objects significantly; 3) Meanwhile, we design a feature fusion module to enhance the semantic context interaction of adjacent feature maps for better detection of small objects. Experiments for railway traffic datasets on both computer device and mobile device demonstrate that FB-Net achieves good results when the input size is 320 pixels × 320 pixels.