A systematic synthesis of sky image enhancement techniques for ground-based solar irradiance forecasting Data files and deep learning models supporting a study on image enhancement methods for improving ground-based solar irradiance forecasting using sky images and pyranometer readings. Code repository The source code repository used to study the effects of image enhancement methods and tricks on solar irradiance forecasting, and to generate the models below, is available at https://github.com/PUTvision/solar-irradiance. Additionally, Zenodo contains the latest version of the compressed source code in ZIP format. The README.md file in the source code repository describes dependencies, requirements and installation steps. Models Pretrained models checkpoints for a 15-minute forecasting horizon with sky image enhancement pipeline applied for solar irradiance forecasting using ground-based sky images. Model Forecast Skill (%) PyTorch (Lightning) checkpoint Smart Persistence Model - - ResNet10 21.71 resnet10t.ckpt ResNet18 18.80 resnet18.ckpt ResNet34 21.09 resnet34.ckpt ResNet50 21.71 resnet50.ckpt SE ResNet50 20.31 seresnet50.ckpt ResNeXt-50 32x4d 20.05 resnext50_32x4d.ckpt MobileNetV4-M 15.78 mobilenetv4_conv_medium.ckpt MixNet-XL 18.71 mixnet_xl.ckpt EfficientNetB0 19.71 efficientnet_b0.ckpt EfficientNetB2 18.60 efficientnet_b2.ckpt EfficientNetB4 19.81 efficientnet_b4.ckpt RegNetY-64 19.90 regnety_064.ckpt RegNetZ-D8 18.66 regnetz_d8.ckpt ConvNeXt-T 18.72 convnext_tiny.ckpt ConvNeXtV2-T 16.96 convnextv2_tiny.ckpt ConvFormer-S18 19.67 convformer_s18.ckpt EfficientViT-B2 20.98 efficientvit_b2.ckpt MViTv2-T 20.00 mvitv2_tiny.ckpt MambaOut-T 18.38 mambaout_tiny.ckpt Irradiance forecasting models available in the literature. Model Forecast Skill (%) PyTorch (Lightning) checkpoint Smart Persistence Model - - MLP Regressor 8.93 - Wen et al. [1] 11.70 wen_resnet18.ckpt Papatheofanous et al. [2] (ResNet50 + Sun mask) 16.10 papatheofanous_resnet50.ckpt Venitourakis et al. [3] (Xception-based + Sun mask) 17.32 venitourakis_xception.ckpt ResNet50 + BoT 21.71 resnet50.ckpt [1] H. Wen, Y. Du, X. Chen, E. Lim, H. Wen, L. Jiang, W. Xiang, Deep learning based multistep solar forecasting for pv ramp-rate control using sky im-ages, IEEE Transactions on Industrial Informatics 17 (2) (2021) 1397–1406. doi:10.1109/TII.2020.2987916.[2] E. A. Papatheofanous, V. Kalekis, G. Venitourakis, F. Tziolos, D. Reisis, Deep learning-based image regression for short-term solar irradiance forecasting on the edge, Electronics 11 (22) (2022). doi:10.3390/electronics11223794.[3] G. Venitourakis, C. Vasilakis, A. Tsagkaropoulos, T. Amrou, G. Konstantoulakis, P. Golemis, D. Reisis, Neural network-based solar irradiance forecast for edge computing devices, Information 14 (11) (2023). doi:10.3390/info14110617. Dataset The dataset used in this study is based on the Folsom dataset. To make the data preparation and follow-up steps easier to reproduce, these steps were described as a directed acyclic graph (DAG) using the DVC package. In the source code repository, this pipeline is stored in a dvc.yaml file. Data files cleaned_irradiance.csv - cleaned version of the original Folsom_irradiance.csv files with timestamps converted to US/Pacific and added column with corresponding sky image, which is available in the Folsom dataset and is not corrupted. periods_{FORECASTING HORIZON}m.pickle - data (irradiance readings and corresponding image filenames) grouped into periods of historical measurements at times (t-15, t-10, t-5, t), and the corresponding target irradiance at time t+H (H represents the forecasting horizon). solar-irradiance-{RELEASE TAG}.zip - release of the source code related to this study. clear_sky_ineichen_folsom.csv - clear-sky irradiance GHI values computed with Ineichen and Perez clear sky model. Forecasting horizons Dataset splits used in the study for different forecasting horizons. File name Forecasting horizon (H) PyTorch (Lightning) checkpoint periods_5m.pickle 5 minutes resnet50_5m.ckpt periods_10m.pickle 10 minutes resnet50_10m.ckpt periods_15m.pickle 15 minutes resnet50.ckpt periods_30m.pickle 30 minutes resnet50_30m.ckpt Data splits The training, validation and test splits were generated using the following code sample, which is defined in the solar_irradiance/datamodules/forecasting.py file int the source code repository. from pathlib import Path import numpy as np import pandas as pd periods_path = Path("./periods_15m.pickle") with periods_path.open("rb") as f: periods = pd.read_pickle(f) test_periods = list(filter(lambda p: p["history"][-1]["image_name"].startswith("2014"), periods)) train_val_periods = list(filter(lambda p: not p["history"][-1]["image_name"].startswith("2014"), periods)) size = 170 # number of days for validatation dataset to get 80-20 ratio of train-val datasets np.random.seed(42) val_dates = [ str(y) + str(m).zfill(2) + str(d).zfill(2) for y, m, d in zip( np.random.randint(2015, 2017, size=size), np.random.randint(1, 13, size=size), np.random.randint(1, 29, size=size), strict=False, ) ] val_periods = list(filter(lambda p: p["history"][-1]["image_name"][:8] in val_dates, train_val_periods)) train_periods = list(filter(lambda p: p["history"][-1]["image_name"][:8] not in val_dates, train_val_periods)) Cross-section generalization - Poznan dataset poznan_images_19_26_Oct_2024.zip - small image dataset separated from internal dataset for Folsom-pretrained model cross-section generalization tests. poznan_irradiance_19_26_Oct_2024.csv - corresponding irradiance measurements. clear_sky_ineichen_poznan_19_26_Oct_2024.csv - clear-sky irradiance GHI values computed with Ineichen and Perez clear sky model. poznan_periods_15m.pickle - dataset splits used in cross-section generalization tests; period 20-26 Oct 2024 was a test set in this case.