Categorical perception provides a potential link between color perception and the linguistic categories that correspond to the basic color terms. We examined whether the sensory information of the second-stage chromatic mechanisms is further processed so that sensitivity for color differences yields categorical perception. In this case, sensitivity for color differences should be higher across than within category boundaries. We measured discrimination thresholds (JNDs) and color categories around an isoluminant hue circle in Derrington-Krauskopf-Lennie (DKL) color space at three levels of lightness. At isoluminant lightness, the global pattern of JNDs coarsely followed an ellipse. Deviations from the ellipse coincided with the orange-pink and the blue-green category borders, but these minima were also aligned with the second-stage cone-opponent mechanisms. No evidence for categorical perception of color was found for any other category borders. At lower lightness, categories changed substantially, but JNDs did not change accordingly. Our results point to a loose relationship between color categorization and discrimination. However, the coincidence of some boundaries with JND minima is not a general property of color categorical boundaries. Hence, our basic ability to discriminate colors cannot fully explain why we use the particular set of categories to communicate about colors. Moreover, these findings seriously challenge the idea that color naming forms the basis for the categorical perception of colors. With respect to previous studies that concentrated on the green-blue boundary, our results highlight the importance of controlling perceptual distances and examining the full set of categories when investigating category effects on color perception.