Stellar atmosphere models for M dwarfs frequently produce inaccurate spectra due to poorly constrained opacities and incomplete physical treatments. These deficiencies limit our ability to interpret high-resolution spectral observations and introduce systematics into downstream analyses, including exoplanet mass measurements from radial velocities (RVs).We present postellar, a data-driven framework in which the stellar spectrum is treated as a variable inferred directly from the data, rather than as a fixed prediction from an atmosphere model. Postellar combines a machine-learning prior trained on PHOENIX M-dwarf spectra, which captures the common structure of these spectra and the atmospheric assumptions they typically encode, with a likelihood that anchors the inference to observations. This approach yields physically informed and empirically constrained spectral modeling. Postellar further provides direct uncertainties on spectral features through the spread of posterior samples. We validate postellar using synthetic SPIRou observations based on empirical spectra of Barnard’s Star and Proxima Centauri. Postellar recovers the underlying spectrum more accurately than both PHOENIX models and empirical templates, which are constructed from stacked observations, improving RV accuracy by up to a factor of three. The framework performs particularly well in low signal-to-noise regimes relevant for faint targets, which can enable improved stellar compositional analyses. In addition, postellar naturally highlights spectral regions where model predictions are least reliable through increased posterior uncertainty, providing empirical guidance for future atmosphere model development.By explicitly combining information from physical models and observations, postellar provides a more precise way to interpret cool star spectra and reduce stellar-driven systematics, while informing theoretical advances in atmosphere modeling.