Abstract Perception of vocalizations and other behaviorally relevant sounds requires integrating acoustic information over hundreds of milliseconds, but the latency of sound-evoked activity in auditory cortex typically has much shorter latency. It has been observed that the acoustic context, i.e., sound history, can modulate sound evoked activity. Contextual effects are attributed to modulatory phenomena, such as stimulus-specific adaption and contrast gain control. However, an encoding model that links context to natural sound processing has yet to be established. We tested whether a model in which spectrally tuned inputs undergo adaptation mimicking short-term synaptic plasticity can account for contextual effects during natural sound processing. Single-unit activity was recorded from primary auditory cortex of awake ferrets during presentation of noise with natural temporal dynamics and fully natural sounds. Encoding properties were characterized by a standard linear-nonlinear spectro-temporal receptive field model (LN STRF) and STRF variants that incorporated STP-like adaptation. In two models, STP was applied either globally across all spectral channels or locally to subsets of channels. For most neurons, STRFs incorporating locally tuned STP predicted neural activity as well or better than the LN and global STP STRF. The strength of nonlinear adaptation varied across neurons. Within neurons, adaptation was generally stronger for activation with excitatory than inhibitory gain. Neurons showing improved STP model performance also tended to undergo stimulus-specific adaptation, suggesting a common mechanism for these phenomena. When STP STRFs were compared between passive and active behavior conditions, response gain often changed, but average STP parameters were stable. Thus, spectrally and temporally heterogeneous adaptation, subserved by a mechanism with STP-like dynamics, may support representation of the diverse spectro-temporal patterns that comprise natural sounds. Author summary Successfully discriminating between behaviorally relevant sounds such as vocalizations and environmental noise requires processing how acoustic information changes over many tens to hundreds of milliseconds. The sound-evoked activity measured for most auditory cortical neurons is relatively short (< 50 ms), so it is not clear how the auditory cortex encodes sound information over longer periods. In this study, we propose that nonlinear adaptation, mimicking the effects of short-term synaptic plasticity (STP), enables auditory neurons to encode longer and more complex spectro-temporal patterns. A model in which sound history is stored in the latent state of plastic synapses is able to describe responses of single cortical neurons to natural sounds better than a standard encoding model that does not include nonlinear adaptation. Moreover, STP-like adaptation can account for contextual effects on sound evoked activity that cannot be accounted for by standard encoding models.