The natural linewidth of lasers is shown to be enhanced with respect to the value predicted by the Schawlow-Townes formula when the gain is high and inhomogeneous in transverse or longitudinal directions. A general formula for the linewidth enhancement is derived from first principles: Maxwell's equations and the fluctuation-dissipation theorem, for media with arbitrary bi-anisotropy and dispersion. The result is expressed in terms of an integral of the resonating field over the cavity volume. For isotropic media (scalar e and μ), this formula generalizes previous results by Petermann for gain-guided lasers, by Ujihara for lasers with low reflectivity mirrors, and results obtained by other authors for lasers coupled to long external cavities. The role of non-reciprocity is discussed.