In recent years, a growing zoo of compact stellar systems (CSSs) have been found whose physical properties (mass, size, velocity dispersion) place them between classical globular clusters (GCs) and true galaxies, leading to debates about their nature. Here we present results using a so far underutilized discriminant, their stellar population properties. Based on new spectroscopy from 8-10m telescopes, we derive ages, metallicities, and [alpha/Fe] of 29 CSSs. These range from GCs with sizes of merely a few parsec to compact ellipticals (cEs) larger than M32. Together with a literature compilation, this provides a panoramic view of the stellar population characteristics of early-type systems. We find that the CSSs are predominantly more metal rich than typical galaxies at the same stellar mass. At high mass, the cEs depart from the mass-metallicity relation of massive early-type galaxies, which forms a continuous sequence with dwarf galaxies. At lower mass, the metallicity distribution of ultracompact dwarfs (UCDs) changes at a few times 10(7) M-circle dot, which roughly coincides with the mass where luminosity function arguments previously suggested the GC population ends. The highest metallicities in CSSs are paralleled only by those of dwarf galaxy nuclei and the central parts of massive early types. These findings can be interpreted as CSSs previously being more massive and undergoing tidal interactions to obtain their current mass and compact size. Such an interpretation is supported by CSSs with direct evidence for tidal stripping, and by an examination of the CSS internal escape velocities.