We propose a new method for classification of photometric factors, such as diffuse reflection, specular reflection, attached shadow, and cast shadow. For analyzing real images, we utilize the photometric linearization method which was originally proposed for image synthesis. First, we show that each pixel can be photometrically classified by the simple comparison of the pixel intensity. Our classification algorithm requires neither 3D shape information nor color information of the scene. Then, we show that the accuracy of the photometric linearization can be improved by introducing a new classification-based criterion to the linearization process. Experimental results show that photometric factors can be correctly classified without any special device.