Fruit size is an important parameter both for scientific understanding and for commercial purposes. In many species, mature fruit size is often related to floral ovary size, but no literature exists in olive that demonstrates such a relationship. Previous work suggests that olive cultivars with different fruit sizes have similar cell number and size in the ovary transectional area, but ovary and fruit dry weight was not measured. In the present study, ovary dry weight and fruit dry weight during the whole fruit development season until harvest were measured in olive cultivars with different fruit size, over three years. Flower dry weight was also measured. Fruit weight at harvest was strongly correlated to ovary weight at bloom, both in single-year data and when data from three years were pooled. Flower dry weight, excluding the ovary, was also correlated to ovary dry weight. Ovary dry weight was strongly correlated not only to the fruit dry weight at maturity, but also at any date during fruit development. The mature fruit/ovary dry weight ratio ranged between 1000 and 4000 among cultivars, but was not correlated to the fruit dry weight at maturity. These results suggest that, in olive, fruit weight is genetically controlled through the ovary weight at bloom. This knowledge may have implications in the understanding of fruit set and source-sink relationships in olive.