# Replication Package This is the replication package for our work on "Testing Scratch Programs Automatically". The package contains our raw results and scripts for generating the plots of the paper from the raw data. ## Abstract Block-based programming environments like Scratch foster engagement with computer programming and are used by millions of young learners. Scratch allows learners to quickly create entertaining programs and games, while eliminating syntactical program errors that could interfere with progress. However, functional programming errors may still lead to incorrect programs, and learners and their teachers need to identify and understand these errors. This is currently an entirely manual process. In this paper, we introduce a formal testing framework that describes the problem of Scratch testing in detail. We instantiate this formal framework with the Whisker tool, which provides automated and property-based testing functionality for Scratch programs. Empirical evaluation on real student and teacher programs demonstrates that Whisker can successfully test Scratch programs, and automatically achieves an average of 95.25% code coverage. Although well-known testing problems such as test flakiness also exist in the scenario of Scratch testing, we show that automated and property-based testing can accurately reproduce and replace the manually and laboriously produced grading efforts of a teacher, and opens up new possibilities to support learners of programming in their struggles. ## Contents The replication package is structured into two main directories: * 'data/': raw data and scripts that have been used for collecting the data * 'scripts/': scripts for generating the plots that are presented in the paper ### RAW data * 'data/teacher-data/' data from the scratch workshop: sample solution and scores for student solutions * 'data/code-club-stats/' block counts and input methods of the used Code Club projects * 'data/coverage` code for measuring the coverage of automated input generation * 'data/coverage-results/' coverage measurements on the Code Club projects * 'data/test/' test suites for the projects of the Scratch workshop * 'data/test-results/' test results from the test suites in 'data/test/' * 'data/time/' Scratch programs for time measurement (10x the sample solution from 'data/teacher-data/`) * 'data/time-results/' time measurements on the projects in 'data/time/' ## Reproducing the Plots ### Prerequisites We describe the process based on: * the R statistics package in version 3.5 * an Unix environment (Linux or MacOSX) Following R packages are required: * ggplot2 * dplyr * viridis The package can be installed with the R command "install.packages". ### Generating the Plots Coverage (Figure 10) ./scripts/coverage.R The result is a set of "coverage-*.pdf" files Inconsistency (Figure 9) ./scripts/consistency.R The result is a set of "consistency-*.pdf" files Scatter Plots (Figure 8, Figure 11) ./scripts/scatter.R The result is a set of "scatter-*.pdf" files