Abstract This paper proposes a novel RPN-FCN based rust detection approach. The RPN-FCN generates region proposals with RPN and performs full convolution for semantic segmentation of rust. The experimental result demonstrate that this approach improves the accuracy of rust detection compared with other neural networks.