We propose to model the dynamics of the entire implied volatility surface (IVS) multi-step ahead by letting the parameters of a stochastic volatility model with an explicit expression for the smile be dynamically evolved. We assume that these model parameters are stochastic processes driven by some explanatory variables and use deep learning to infer their dynamics. For simplicity, we focus on the SVI model, let each model parameter have a term-structure, and learn to predict the future values of these parameters. The explanatory variables are the time series of the fitted model parameters and the time series of the forward prices. To capture the spatiotemporal relations of the IVS, we stack multiple convolutional LSTM (ConvLSTM) layers and form an encoding-forecasting structure, getting a network model capable of understanding the spatiotemporal relationships between strikes and time-to-maturities. However, this model is very sensitive to the term-structure of the model parameters and requires a very fine grid of volatility to converge. Thus, we simplify the model by considering a kernel of size one. The future smiles are reconstructed by using the parametric smile representation, where each parameter is replaced by its estimated value. We can then use the forecasted volatility surface for pricing and hedging options, performing risk analysis, as well as for volatility trading. We explore the performance of our model against a naive strategy by forecasting the volatility surface on the S&P 500 option prices several steps ahead, and computing some measures of accuracy. On average, our model systematically outperforms the naive approach at predicting long term forecasts for short to mid-range maturities. This shows that the dynamics of the IVS are dominated by trend and mean reversion, hence predictable.