Thanks to its prominent applications in science, medicine, industry and finance, time series forecasting has been a key area of research. Several recent solutions for time series forecasting are based on convolutional neural networks. While convolutional layers act as local pattern detectors, we point out that they match local patterns in a rigid manner in the sense that they do not account for local shifts and elongations. In this paper, we address this issue and propose distortion-aware convolution. We discuss how to train neural networks with distortion-aware convolution in a multi-phase training process. Our experimental results on publicly available real-world datasets from various domains show that replacing conventional convolution by distortion-aware convolutions leads to more accurate models for time series forecasting both in terms of mean squared error and mean absolute error. Furthermore, distortion-aware convolution may serve as an essential building block of future neural networks. In order to support reproduction and follow-up works, we made our prototypical implementation publicly available at https://github.com/kr7/dcnn-forecast.