Pedestrian Detection (PD) is one of the most studied issues of driver assistance systems. Although a tremendous effort is already given to create datasets and to develop classifiers for cars, studies about railway systems remain very limited. This article shows that direct application of neither existing advanced object detectors (such as AlexNet, VGG, YOLO etc.), nor specifically created systems for PD (such as Caltech/INRIA trained classifiers), can provide enough performance to overcome railway specific challenges. Fortunately, it is also shown that without waiting the collection of a mature dataset for railways as comprehensively diverse and annotated as the existing ones for cars, a Transfer Learning (TL) approach to fine-tune various successful deep models (pre-trained using both extensive image and pedestrian datasets) to railway PD tasks provides an effective solution. To achieve TL, a new RAilWay PEdestrian Dataset (RAWPED) is collected and annotated. Then, a novel three-stage system is designed. At its first stage, a feature-classifier fusion is created to overcome the localization and adaptation limitations of deep models. At the second stage, the complementarity of the transferred models and diversity of their results are exploited by conducted measurements and analyses. Based on the findings, at the third stage, a novel learning strategy is developed to create an ensemble, which conditionally weights the outputs of individual models and performs consistently better than its components. The proposed system is shown to achieve a log average miss rate of 0.34 and average precision of 0.93, which are significantly better than the performance of compared well-established models.