The economic importance of Bactrocera oleae (Rossi) and the problems associated with insecticides make necessary new management approaches, including deeper biological knowledge and its relationship with landscape structure. Landscape complexity reduces B. oleae abundance in late summer–autumn in areas of high dominance of olive groves, but the effect of landscape structure in spring and in areas less dominated by olive groves has not been studied. It is also unknown whether the insect disperses from olive groves, using other land uses as a refugee in summer. This work evaluates the effect of landscape structure on olive fruit fly abundance and movement in spring and autumn, and infestation in autumn, in central Spain, an area where the olive crop does not dominate the landscape. A cost–distance analysis is used to evaluate the movement of the fly, especially trying to know whether the insects move away from olive groves in summer. The results indicate that B. oleae abundance is consistently lower in complex landscapes with high scrubland area (CAS), patch richness (PR) and Simpson landscape diversity index (SIEI), and low olive grove area (CAO). The cost–distance analysis shows that the fly moves mainly in spring, and amongst olive groves, but there is no evidence that land uses other than olive groves serve as a summer refuge. Olive fly infestation decreased with decreasing CAO and increasing CAS and SIEI, accordingly with the effect of landscape on abundance. Thus, mixing olive groves with other land uses, which are not a source of flies, can help improve control of this important pest.