Historical records of climate fields are often sparse because of missing measurements, especially before the introduction of large-scale satellite missions. Several statistical and model-based methods have been introduced to fill gaps and reconstruct historical records. Here, we use a recently introduced deep learning approach based on Fourier convolutions, trained on numerical climate model output, to reconstruct historical climate fields. Using this approach, we are able to realistically reconstruct large and irregular areas of missing data and to reproduce known historical events, such as strong El Niño or La Niña events, with very little given information. Our method outperforms the widely used statistical kriging method, as well as other recent machine learning approaches. The model generalizes to higher resolutions than the ones it was trained on and can be used on a variety of climate fields. Moreover, it allows inpainting of masks never seen before during the model training.