Sparse coding has become a popular way to learn feature representation from the data itself. However, temporal context, when present, can provide useful information and alleviate instability in sparse representation. Here we show that when sparse coding is used in conjunction with a dynamical system, the extracted features can provide better descriptors for time-varying observations. We show a marked improvement in classification performance on COIL-100 and animal datasets using our model. We also propose a simple extension to our model to learn invariant representations.