This paper presents a framework for using high-level visual information to enhance the performance of automatic color constancy algorithms. The approach is based on recognizing special visual object categories, called here as memory color categories, which have a relatively constant color (e.g. the sky). If such category is found from image, the initial white balance provided by a low-level color constancy algorithm can be adjusted so that the observed color of the category moves toward the desired color. The magnitude and direction of the adjustment is controlled by the learned characteristics of the particular category in the chromaticity space. The object categorization is performed using bag-of-features method and raw camera data with reduced preprocessing and resolution. The proposed approach is demonstrated in experiments involving the standard gray-world and the state-of-the-art gray-edge color constancy methods. In both cases the introduced approach improves the performance of the original methods.