Does Unit-Tested Code Crash? A Case Study of Eclipse: Replication Package This is a replication package associated with the paper titled “Does Unit-Tested Code Crash? A Case Study of Eclipse”. Below is a description of the package’s contents. Data Data files associated with the paper are provided in the data directory. Text file tested-crashed.txt Data specifying whether methods were tested and whether they crashed (according to the criteria adopted in the study). Extracted from matches.xlsx. The data are used as input for Fisher’s test (RQ1). Spreadsheet matches.xlsx Test coverage data and calculations associated with failed methods, class coverage, and matching method coverage results are provided in an Excel spreadsheet. Below is the description of the spreadsheet’s contents. Worksheet Test Coverage Contains the data regarding the JaCoCo test code coverage analysis. Class: The name of the class in which a method appears in JVM internal form notation Method: The method’s name Parameters: The method’s arguments in JVM parameter descriptor format; required to handle Java’s {} polymporphism Class Has Unit Test: Whether the corresponding class has associated unit test code Class Unit-Test Line Density: The ratio of lines in class’s test code over those in the class’s implementation code Covered Instructions / Branches / Lines: As reported by JaCoCo Total Instructions / Branches / Lines: As reported by JaCoCo Covered Instructions / Branches / Lines ratio: The ratio between the two preceding values; 1 for methods without any branches Top-1 / Top-6 / Top-10 : In how many stack traces the method appears within; the top-10 / top-6 / the very first stack frame(s) Tested: TRUE if the method is considered tested by having a test code coverage above the median (0.966) and an associated test class Crashed: TRUE if the method has crashed as evidenced by its appearance on the topmost stack frame Stack trace file names: in which the method appeared Worksheet Test Existence Contains the data of the analysis regarding the existence of test code. Class: Class containing implementation code TestClassNames: Classes that contain tests for the above Number of relevant tests Lines in class test code Lines of class Class Unit-Test Line Density: The ratio between the two above Worksheet Metrics Contains the derivation of metrics reported in the paper. In the cases of tables these are formatted in LaTeX for direct incorporation into the text. Spreadsheet jacoco.xlsx Complete test coverage data obtained from JaCoCo are provided in an Excel spreadsheet. Below is the description of the spreadsheet’s contents. Worksheet Data Contains the following method code coverage fields as reported by JaCoCo, as well as the calculated percentages. Class Method Parameters Covered Instructions Total Instructions % Covered Instructions Covered Branches Total Branches % Covered Branches Covered Lines Total Lines % Covered Lines Worksheet Metrics Contains the derivation of numbers reported in the preliminary quantitative analysis and Figure 2. Compressed tar archive eclipse-src.tar.gz Contains the Eclipse source code used for running the Eclipse tests with JaCoCo code coverage analysis. It was obtained from the Eclipse source code repositories as follows. Clone the Eclipse aggreagator repository into a directory named z by running: git clone -b master --recursive git://git.eclipse.org/gitroot/platform/eclipse.platform.releng.aggregator.git z In the z directory, checking out the used release by running cd z && git submodule foreach git checkout M20160212-1500 Checking out the release for the main repository by running: git checkout M20160212-1500 Applying the patch eclipse-src.diff Patch file eclipse-src.diff See above. Zip file incidents.zip Contains the 126,026 incidents (crash report stack traces and meta-data) associated with EclipseProduct org.eclipse.epp.package.java.product and BuildID 4.5.2.M20160212-1500. This is a subset from the two million incidents available as the AERI stack traces data set. The subset of incidents was extracted from the full AERI data set with the following command. for f in *; do grep -q org.eclipse.epp.package.java.product $f && grep -q 4.5.2.M20160212-1500 $f && mv $f selected-files/ done Compressed file jacoco.xml.gz Contains the results of the JaCoCo code coverage analysis over the Eclipse testing. Code The following scripts are provided in the src directory extract.py: script for extracting crash (incidents) and coverage (JaCoCo) data unit-tested-classes.py: script for finding the classes with associated unit test code merge.py: script for matching crash (incidents) with coverage (JaCoCo) data fisher.r: R script for running Fisher’s test