In everyday scenes, from perceived colors of objects and terrains, observers can simultaneously identify objects across illuminants and identify the nature of the light, e.g., as sunlight or cloudy. As a formal problem, identifying objects and illuminants from the color information provided by sensor responses is underdetermined. It is shown how the problem can be simplified considerably by the empirical result that chromaticities of sets of objects under one illuminant are approximately affine transformations of the chromaticities under spectrally different illuminants. Algorithms that use the affine nature of the correlation as a heuristic can identify objects of identical spectral reflectance across scenes lit simultaneously or successively by different illuminants. The relative chromaticities of the illuminants are estimated as part of the computation. Because information about objects and illuminants is useful in many different tasks, it would be more advantageous for the visual system to use such algorithms to extract both sorts of information from retinal signals than to discount either automatically at an early neural stage.