In this paper, a new deep learning framework named encoding convolution long short-term memory (encoding ConvLSTM) is proposed to build a surrogate structural model with spatiotemporal evolution of structure, estimate the structural spatiotemporal state and predict the dynamic response under similar future dynamic load conditions. The main work of this study includes: (a) The spatiotemporal response tensor database is developed using discrete-time history data of structural dynamic response. (b) As an extension of LSTM, convolution operation is combined with LSTM network to construct structural surrogate model from the spatiotemporal evolution structural performance. (c) To enhance the anti-interference ability of structural surrogate models, a new three-layer encoding layer is added for denoising autoencoders of the hybrid network. The influence of building types and input noise on the accuracy and antinoise performance of the surrogate models are analyzed through the dynamic response prediction of a frame-shear wall, a cylindrical, and a spherical reticulated shell structure. As a testbed for the proposed network, a case study is performed on a laboratory stadium structure. The results demonstrate that the developed surrogate model can predict the structural dynamic response precisely with more under 30% noise interference.