Damage detection is essential for the maintenance of infrastructure, yet it is often hindered by the scarcity of specialized data. To address this issue, we introduce a novel sequential fine-tuning strategy designed to enhance knowledge transfer within Deep Neural Networks(DNNs) and overcome data scarcity challenges. In this context, our strategy incorporates an intermediate step involving the use of a ‘bridge’ dataset, which substantially enhances the effectiveness of traditional fine-tuning methods. This approach facilitates better adaptation of the model to the target domain, leveraging the data to mediate the transfer of knowledge between the general and specific datasets. Our study investigates the influence of dataset characteristics on the efficacy of DNN knowledge transfer and highlights the critical importance of domain relevance in the selection of datasets. Extensive experimental evaluation on the benchmark pipeline detection dataset showcases that the proposed training method significantly enhances the state-of-the-art (SOTA) detectors for the damage detection task. We advocate for further research in sequential fine-tuning and its potential implications on model performance, emphasizing the need to explore this approach across different contexts and applications.