Summary The colors of fleshy fruits are considered to be a signal to seed‐dispersing animals, but their diversity remains poorly understood. Using an avian color space to derive a sensory morphospace for fruit color, we tested four hypotheses of fruit color diversity: fruit colors occupy a limited area of the color space; they are less diverse than flower colors; fruit colors within localities are similar to each other; and fruit color diversity reflects phylogeny. The global fruit color diversity of 948 primarily bird‐dispersed plant species and the color diversity of localities were compared with null models of random, unconstrained evolution of fruit color. Fruit color diversity was further compared with the diversity of 1300 flower colors. Tests of phylogenetic effects on fruit color were used to assess the degree of correspondence with phylogeny. Global and local fruit color diversity was limited compared with null models and fruits have achieved only half the color diversity of flowers. Interestingly, we found little indication of phylogenetic conservatism. Constraints resulting from the chemical properties of pigments probably limit global fruit and flower color diversity. Different types of selection on fruits and flowers may further explain the smaller color diversity of fruits.