Abstract To maintain the reliability, availability, and sustainability of electricity supply, electricity companies regularly perform visual inspections on their transmission and distribution networks. These inspections have been typically carried out using foot patrol and/or helicopter-assisted methods to plan for necessary repair or replacement works before any major damage, which may cause power outage. This solution is quite slow, expensive, and potentially dangerous. In recent years, numerous researches have been conducted to automate the visual inspections by using automated helicopters, flying robots, and/or climbing robots. However, due to the high accuracy requirements of the task and its unique challenges, automatic vision-based inspection has not been widely adopted. In this paper, with the aim of providing a good starting point for researchers who are interested in developing a fully automatic autonomous vision-based power line inspection system, we conduct an extensive literature review. First, we examine existing power line inspection methods with special attention paid to highlight their advantages and disadvantages. Next, we summarize well-suited tasks and review potential data sources for automatic vision-based inspection. Then, we survey existing automatic vision-based power line inspection systems. Based on that, we propose a new automatic autonomous vision-based power line inspection concept that uses Unmanned Aerial Vehicle (UAV) inspection as the main inspection method, optical images as the primary data source, and deep learning as the backbone of data analysis and inspection. Then, we present an overview of possibilities and challenges of deep vision (deep learning for computer vision) approaches for both UAV navigation and UAV inspection and discuss possible solutions to the challenges. Finally, we conclude the paper with an outlook for the future of this field and propose potential next steps for implementing the concept.