This supplementary material for the article"Evaluating Classifiers in SE Research: The ECSER Pipeline and Two Replication Studies. Dell’Anna, D.; Aydemir, F. B.; and Dalpiaz, F. Empirical Software Engineering. 2022" includes ECSER-ExploratoryStudy.csv: The annotated meta-data of the papers that have been published in ICSE between 2019 and 2021. ECSER_ROCplots+StatTest.ipynb: A python notebook that compares classifiers adn checks the statistical significance of the comparison results. ECSER_SummaryOfReplicationSteps.pdf: This table presents a summary of ECSER steps for the two original studies and our applications on ECSER. ECSER_RE: The directory that holds the datasets and code for the replication of Hay et al. [1] and additional runs on the new data sets. ECSER_FF: The directory that holds the code and data for the replication of Alshammari et al. [2] README.md presents the structure of the supplementary materials. requirements.txt lists the dependencies needed to run the code In the ECSER_RE directory, the code for multiple classifiers that are compared are kept in the "Classifiers" directory. The public data sets are shared in the Datasets directory. "ECSER_RE_Compare_Classifiers.ipynb" python notebook includes the code that runs each classifier. The results are presented in "ECSER_RE_results-Promise-vs-all.csv". In the ECSER_FF directory, the data sets are presented directly under the main directory. The notebook "ECSER-FF-Compare_Classifiers.ipynb" compares the classifiers of the original study and the results are kept under the "ECSER_FF_results" directory. How to cite this repository If you use this repository, please cite the reference paper, and the repository, as below: Dell’Anna, Davide, Fatma Başak Aydemir, and Fabiano Dalpiaz. "Evaluating classifiers in SE research: the ECSER pipeline and two replication studies." Empirical Software Engineering 28.1 (2023): 3. Davide Dell'Anna, Fatma Başak Aydemir, & Fabiano Dalpiaz. (2021). Supplementary Material for "Evaluating Classifiers in SE Research: The ECSER Pipeline and Two Replication Studies" [Data set]. Zenodo. https://doi.org/10.5281/zenodo.6266675 Tobias Hey, Jan Keim, Anne Koziolek, and Walter F. Tichy. 2020. SupplementaryMaterial of "NoRBERT: Transfer Learning for Requirements Classification". https://doi.org/10.5281/zenodo.3874137 Abdulrahman Alshammari, Christopher Morris, Michael Hilton, and JonathanBell. 2021. FlakeFlagger: Predicting Flakiness Without Rerunning Tests. In43rdIEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid,Spain, 22-30 May 2021. IEEE, 1572–1584. https://doi.org/10.1109/ICSE43902.2021.00140