We introduce a new estimation framework which extends the Generalized Method of Moments (GMM) to settings where a subset of the parameters vary over time with unknown dynamics. To filter out the dynamic path of the time-varying parameter, we approximate the dynamics by an autoregressive process driven by the score of the local GMM criterion function. Our approach is completely observation driven, rendering estimation and inference straightforward. It provides a unified framework for modeling parameter instability in a context where the model and its parameters are only specified through (conditional) moment conditions, thus generalizing approaches built on fully specified parametric models. We provide examples of increasing complexity to highlight the advantages of our method.