The critical behavior of large-$N$ matrix models defined by means of integrals over Lie algebras is shown to be universal. All such models give rise to the same theory of orientable random surfaces. By matching to the perturbative expansion, matrix models over symplectic groups are found to exhibit critical behavior distinct from that of unitary matrices, and from that of orthogonal groups. Symplectic and orthogonal groups have contributions to the specific heat from (odd Euler characteristic) unoriented surfaces.