The collinear mirage technique is widely used to measure the thermal diffusivity of semi-transparent materials. However, in a recent paper [A. Salazar, M. Gateshki and A. Sanchez-Lavega: Appl. Phys. Lett. 76, 2665 (2000)], it was shown that for isotropic materials, because of the influence of photoelastic effect, the method was sensitive to the polarization state of the probe beam. The present paper extends the previous work to include anisotropic materials. In particular, we focus on the experimental conditions under which the thermal diffusivity of each crystal system can be measured using the phase method. Our theoretical model indicates that while the thermal diffusivity of isotropic materials can be measured using an unpolarized probe beam, for anisotropic materials, even the use of an unpolarized probe beam does not guarantee the validity of the method in all crystal systems. Experimental measurements performed on cubic, hexagonal and monoclinic crystals confirm the validity of the model.