This paper presents a vision-based system for automatic detection and extraction of overground oil pipelines using an Unmanned Aerial Vehicle (UAV) equipped with an optical sensor. The aim of the project is to develop a highly automated onboard system for detecting the pipelines as this could potentially reduce the cost of monitoring and surveying pipelines when compared to manned aircraft. Several image sets have been analyzed, and two aerial videos were used to test the system. The first video has been captured by an aircraft flying over the Trans-Alaska Pipeline in the United States, while the second video shows a pipeline being constructed in the United Kingdom. The first step of the processing involves enhancing the image to improve the edge detection performance. Following this, the edges are filtered by comparing them with known pipeline segments in the area, allowing fast and accurate pipeline identification. This allows the system to reliably reject false measurements while retaining the correct pipeline detections. This improvement can be seen when analyzing the test data sets. In the Trans-Alaska Pipeline set the detection system locate the pipeline with a success rate of 94% over 7,000 frames. The second video is used to stress-test the system in terms of false detections, which causes the detection rate to drop to 66%. Both of these results were obtained without the geo-referencing filter. To determine the effectiveness of the geo-referencing filter the system was tested indoors in a lab, where the detection rate was 95%.