In this repository you can find the pre-print and data associated with the publication titled "The Importance of Being External. Methodological insights for the external validation of machine learning models in medicine. The data is collected in the file "Datasets.zip" and, more in particular: "all-data-processed-v3.xlsx" contains the Brazil-1, Brazil-2 and Brazil-3 datasets "desio_cbc.xls", "bergamo_cbc.xls" and "HSR_novembre.xlsx" contain the Italy-1, Italy-2 and Italy-3 datasets "CBC Italy Octubre (sent 24-12-2020).xlsx" contains the Spain dataset "Etiopia 200 COVID +.xlsx" and "cbc data_anna copy.xlsx" contains the Ethiopia dataset (positive and negative cases, respectively) Code showing how to process the data, as well as all code for producing the visualizations shown in the paper is available at https://github.com/AndreaCampagner/qualiMLpy For any further information, you can contact me at a.campagner@campus.unimib.it