This repository contains the data and results from the paper "Detecting and evaluating order-dependent flaky tests in JavaScript" - currently under review. Project Structure The replication package contains the following directories: ./JS-TOD JS-TOD (JavaScript - Test Order-dependency Detector) is a tool to extract and detect order-dependent tests in Jest. The tool contains the following scripts: testReorderRunner.js: reorders tests inside describe blocks of a given project for a defined number (reorder) and saves the results in the ___extracted results___ folder inside the project folder. It will then run all the new test suites for rerun number and save the results as a JSON file in the same folder. The results are named starting with the term testOutput, followed by the name of the test suite and the rerun count. testOutputParser.js: parses the JSON files located in the ___extracted results___ folder of a given project, generated after running testReorderRunner.js. It summarizes the results, including the name of the test suite, along with the number of failing and passing tests, in a CSV file. Each CSV file is named after the corresponding test suite followed by the term output. These CSV files are stored in the output parsing folder within the ___extracted results___ directory. describeReorderRunner.js: reorders describe blocks of a given project for a defined number (reorder) and save the results in the ___extracted results describes___ folder inside the project folder. It will then run all new test suites for rerun number and save the results as a JSON file in the same folder. The results are named starting with the term testOutput, followed by the name of the test suite and the rerun count. describeOutputParser.js: parses the JSON files located in the ___extracted results describes___ folder of a given project, generated after running describeReorderRunner.js. It summarizes the results, including the name of the test suite, along with the number of failing and passing tests, in a CSV file. Each CSV file is named after the corresponding test suite followed by the term output. These CSV files are stored in the output parsing folder within the ___extracted results describes___ directory. testSuitesReorderRunner.js: reorders the test suites of a given project for a defined number (reorder), runs them for rerun number, and saves the results in the ___extracted results test files___ folder inside the project folder. The results are named starting with the term testOutput, followed by the number of reordering and rerun counts. testSuitesOutputParser.js: parses the JSON files located in the ___extracted results test files___ folder of a given project, generated after running testSuitesReorderRunner.js. It summarizes the results, including the number of order, along with the number of failing and passing tests, in a CSV file. Each CSV file is named order and order number followed by the term output. These CSV files are stored in the output parsing folder within the ___extracted results test files___ directory. customSequencer.js is the custom sequencer extended from Jest's testSequencer. It is used by testSuitesReorderRunner.js. ./Data projectsURL contains the name, version, statistics, URL and description of 81 top-starred GitHub JavaScript projects that using Jest. OD-experiment-result contains the result of reordering and rerunning tests in three levels in 81 JavaScript projects. Each contains the following information : - Project Name - Number of tests and test suites before and after reordering the tests, describe blocks and test suites. - Runtime information - Number of flaky tests - A note from each project. testLevelResults contains the result of reordering and rerunning tests in 67 JavaScript projects. describeBlockLevelResults contains the result of reordering and rerunning describe blocks in 32 JavaScript projects. testSuiteLevelResults contains the result of reordering and rerunning test suites in 49 JavaScript projects. ./Reordering Tests contains the projects and new test suites after reordering the tests. ./Reordering Describe Blocks contains the projects and new test suites after reordering the describe blocks. ./Reordering Test Suites contains the projects and the results of running test suites in different orders. Project Dependencies This project relies on the following dependencies: Required modules Node.js Jest npm npx babel Specifically, JS-TOD requires the following utilities: @babel/parser: A JavaScript parser. @babel/traverse: A utility for traversing and manipulating abstract syntax trees (ASTs) generated by Babel. @babel/generator: A utility for generating code from ASTs produced by Babel. is-utf8: A utility to check if a buffer contains UTF-8 encoded text. Installation Before running the scripts, you need to install the necessary dependencies. You can do this by running the following command: npm install jest --global && npm install fs-extra && npm install npm@latest -g && npm install @babel/parser @babel/traverse @babel/generator is-utf8 Reordering and Running tests using JS-TOD Change the directory to where JS-TOD is located: cd /path/to/directory_containing_JS-TOD Choose the level of reordering(tests, describe blocks or test suites). For example for reordering and running tests of a given project: node testReorderRunner.js --project_path="/path/to/project" --rerun= --reorder= project_path sets the path of the project. rerun sets the number of rerunning. The default number of reruns is 10. reorder sets the number of reordering. The default number of reorders is 10.