This paper offers a conceptual clarification of the phenomenon commonly referred to as categorical perception of color, both in adults and in infants. First, I argue against the common notion of categorical perception as involving a distortion of the perceptual color space. The effects observed in the categorical perception research concern categorical discrimination performance and the underlying processing; they need not directly reflect the relations of color similarity and difference. Moreover, the methodology of the research actually presupposes that the relations of similarity and difference do not vary with languages. The observed categorical perception effects should be conceived independently of the perceptual color space. Second, I challenge the usual opinion that the existing evidence on infant “categorical perception” allows us to conclude that infants perceptually categorize color, and in particular, that they have perceptual categories that resemble the basic color categories of English. Such conclusions rest on an unjustified interpretation of the infant “categorical perception” findings in terms of adult linguistic categorical boundaries. Based on the suggested new understanding, I propose that the phenomenon, as present in infants, should be conceived and examined as a possible explanatory factor with respect to the existing patterns of color naming in languages of the world.