Extra virgin olive oil (EVOO) is renowned for its numerous minor compounds, particularly phenols, which contribute to its health-promoting properties and commercial quality. The phenolic composition of EVOO largely relies on the initial phenolic content of the olive fruits, which represents, therefore, an important trait to be considered in olive breeding programs. However, only limited studies have been conducted so far to compare the relative influence of genotype and environment in the variability of this trait. For that reason, this study aims to assess the influence of genotype (cultivars ‘Arbequina’, ‘Coratina’, ‘Hojiblanca’, ‘Koroneiki’, ‘Martina’, and ‘Picual’) and environment (harvest date, season, and location) on the fruit phenolic content and composition. A wide general variability was obtained in the whole dataset, with total phenols ranging from 6019 to 38,380 μg/g. A stronger effect of the genotype than the environment was observed for total phenolic content (representing 76–90% of total sums of squares) and the different groups of compounds. Notably, significant variations were found among cultivars not only in the total phenol content but also in the percentage of individual compounds within each main phenolic group. Overall, taking into account the entire dataset available, a clear grouping of samples according to genotypes was observed. The results obtained highlight the prominent influence of the genotype (cultivar) over the environment and genotype–environment interactions on the phenolic content and composition in olive fruits; even when considering very contrasting environments as the Mediterranean and subtropical in the present work. These findings suggest the feasibility of breeding selection of new cultivars with distinctive phenolic content and composition.