This is the anonymous reviewing version; the source code repository will be added after the review. This dataset provides the results of measuring jetty with the 1,000 artificial regressions with Peass and with JMH. The creation of the artificial regressions and the measurement is defined here: https://anonymous.4open.science/r/jetty-evaluation-6F58/ (Repository named jetty-evaluation, GitHub link will be provided after review) An example regression is contained in https://anonymous.4open.science/r/jetty-experiments-202D We obtained these data from measurement on Intel Xeon CPU E5-2620 v3 @ 2.40GHz. The dataset contains the following data: regression-results-peass-0.tar.xz (Results of the measurement with Peass, part 0) regression-results-peass-1.tar.xz (Results of the measurement with Peass, part 1) regression-results-peass-2.tar.xz (Results of the measurement with Peass, part 2) regression-results-peass-3.tar.xz (Results of the measurement with Peass, part 3) regression-results-jmh.tar.xz (Results of the measurement with JMH) tree-results.tar.xz (Metadata of the trees) To get the data in a usable format, extract the peass data to one folder (the folder will be named $PEASS_RESULT_FOLDER): mkdir peass for file in *; do echo $file; tar -xf $file; done for i in {0..3}; do mv $i/* .; done This will yield to a folder containing 1000 folders named regression-$i, where each consists of deps.tar.xz: The regression test selection results logs.tar.xz: The logs of the test executions results: The traces of the regression test selection and a file named changes_*testcase.json, which contains statistical details of the measured performance change (if present) jetty.project_peass: Detailed measurement data and logs of individual JVM starts To analyse the Peass results, run cd scripts/peass ./analyzeChangeIdentification.sh $PEASS_RESULTS_FOLDER ./analyzeFrequency.sh $PEASS_RESULTS_FOLDER This will take some time, since partial results need to be unpacked for analysis. The first script will create the following results: and the second will yield the following results: Correct Measurement: 587 Not selected changes: 146 Wrong measurement result: 267 Wrong analysis (should be 0): 0 Overall: 1000 Share of changed method on correct measurements: 0.109571 0.11238 32 Method call count on correct measurement: 15638.4 32853.7 32 Average tree depth on correct measurements: 1.1022 2.6875 32 Share of changed method on wrong measurements: 0.17692 0.180365 968 Method call count on wrong measurement: 711415 180438 968 Average tree depth on wrong measurements: 1.23239 2.42252 968 To analyze the JMH data, first extract the metadata (the folder will be named $TREEFOLDER): tar -xf tree-results.tar.xz Afterwards extract the JMH results (the folder will be named $JMH_RESULTS_FOLDER): tar -xvf regression-results-jmh.tar.xz This will yield a folder containing a measurement for each regression with two files: basic.json: The performance measurement result of the basic version regression-$i.json: The performance measurement result of the version containing the regression Afterwards, run the analysis in the jetty-evaluation repository: cd scripts/jmh ./analyzeFrequency.sh $JMH_RESULTS_FOLDER $TREEFOLDER Since the regression are injected in the call tree of the benchmark, there are now unselected changes. The analysis will yield the following results: Share of changed method on correct measurements: 0.184631 0.271968 587 Method call count on correct measurement: 14628.2 4979.6 587 Share of changed method on wrong measurements: 0.180981 0.235614 267 Method call count on wrong measurement: 14902.2 4333.58 267

Data will be de-anonymized after reviews