Spectroscopic characterization of imaged exoplanets and brown dwarfs is essential for understanding their atmospheres, formation, and evolution, but such work is challenged by the unavoidably simplified model atmospheres needed to interpret spectra. While most previous work has focused on single or at most a few objects, comparing a large collection of spectra to models can uncover trends in data-model inconsistencies needed to improve model predictions, thereby leading to robust properties from exoplanet and brown dwarf spectra. Therefore, we are conducting a systematic analysis of a valuable but underutilized resource: the numerous high-quality spectra of (directly imaged and free-floating) exoplanets and brown dwarfs already accumulated by the community. Focusing on the cool-temperature end, we have constructed a Bayesian modeling framework using the new Sonora-Bobcat model atmospheres and have applied it to study near-infrared low-resolution spectra of >50 late-T imaged planets and brown dwarfs (≈600-1200K, ≈10-70 MJup) and infer their physical properties (effective temperature, surface gravity, metallicity, radii, mass). By virtue of having such a large sample of high-quality spectra, our analysis identifies the systematic offsets between observed and model spectra as a function of wavelength and physical properties to pinpoint specific shortcomings in model predictions. We have also found that the spectroscopically inferred metallicities, ages, and masses of our sample all considerably deviate from expectations, suggesting the physical and chemical assumptions made within these models need to be improved to fully interpret data. Our work has established a systematic validation of cloudless model atmospheres to date and we discuss extending such analysis to wider temperature and wavelength (e.g., JWST) ranges, as well as finding new planetary-mass and brown dwarf benchmarks, in order to validate ultracool model atmospheres over larger parameter space.