Visual target tracking has been studied for decades and still remains a challenging problem. Ship tracking on infrared images has numerous challenges compared to conventional target tracking such as fast changing of appearance. Rapid appearance change caused by the manoeuvring movement of the target of image acquiring platform, confusion and occlusion caused by the active countermeasures employed by the target and disguise by cooling systems causes the target tracking algorithms to have low performance. In this work, a convolutional neural network and correlation filter based algorithm which tracks surface vessels on infrared imagery is proposed. Performance of the proposed algorithm is compared against the distinguished, popular target tracking algorithms from the literature. Performances are evaluated on a specially created infrared ship images dataset.