The optical anisotropy of a semiconductor surface can have different origins, such as, local-field effects, the electro-optics effect, reconstruction, surface dislocations, and surface roughness. A comprehensive, quantitative picture on how these effects are related to surface optical anisotropies (SOA) can now be obtained thanks to the progress in the modeling of optical properties of surfaces. Linear optical spectra can now be calculated very accurately even for large and complex surface structures. This allows a much better understanding of the origin of specific SOA. In this paper we make a review focused on the microscopic origin of SOA and the present state of the art in the study of reflectance anisotropy spectroscopy of semiconductor surfaces.