Replication Package for: Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud This is our replication package for our study on Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud. All scalability experiments are performed with the scalability benchmarking framework Theodolite at Kiel University's Software Performance Engineering Lab (SPEL) or Google Cloud. With this replication package, we provide: Benchmark execution files in executions, our benchmark (raw) results in results, and analysis script for our results in analysis. Repeating Benchmark Executions All our Theodolite executions are tailored to either the SPEL cluster or the Google Cloud. Kiel University's Software Performance Engineering Lab (SPEL) The SPEL cluster consists of 5 nodes, named kube1-1 to kube1-5 and labeled with env=dev. To run them in your local cluster, make sure to provide the same infrastructure or rename node selectors in the execution files accordingly. To install Theodolite, run: helm install theodolite theodolite/theodolite --version 0.8.6 -f https://raw.githubusercontent.com/cau-se/theodolite/main/helm/preconfigs/extended-metrics.yaml -f se-cluster-dev.yaml or for the vertical scalability experiment: helm install theodolite theodolite/theodolite --version 0.8.6 -f https://raw.githubusercontent.com/cau-se/theodolite/main/helm/preconfigs/extended-metrics.yaml -f se-cluster-dev.yaml -f se-cluster-dev-vertical.yaml See Theodolite's documentation for further usage instructions. Google Cloud In the public cloud baseline experiments, the cluster consists of 5 e2-standard-32 nodes. To install Theodolite, run: helm install theodolite theodolite/theodolite --version 0.8.6 -f https://raw.githubusercontent.com/cau-se/theodolite/main/helm/preconfigs/extended-metrics.yaml -f gcp-cluster-dev.yaml For the experiments testing higher load intensities, the cluster consists of 4 e2-standard-16 nodes labeled with type=infra and 4 or 8 e2-standard-16 nodes with label type=sut. To install Theodolite in this cluster, run: helm install theodolite theodolite/theodolite --version 0.8.6 -f https://raw.githubusercontent.com/cau-se/theodolite/main/helm/preconfigs/extended-metrics.yaml -f gcp-cluster-stress.yaml In both cases, change the maximum load generated per load generator instance: # Generate max. 100000 rec/sec per load generator instance export MAX_RECORDS_PER_INSTANCE=100000 kubectl patch benchmarks uc1-beam-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc1-beam-samza --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc1-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc1-hazelcastjet --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc1-kstreams --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc2-beam-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc2-beam-samza --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc2-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc2-hazelcastjet --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc2-kstreams --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc3-beam-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc3-beam-samza --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc3-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc3-hazelcastjet --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc3-kstreams --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc4-beam-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc4-beam-samza --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc4-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc4-hazelcastjet --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc4-kstreams --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" See Theodolite's documentation for further usage instructions. Repeating Results Analysis To inspect, repeat, or extend our results analysis, see results or run the corresponding notebooks in analysis. For analyzing and visualizing benchmark results, either Docker or a Jupyter installation with Python 3.7 or 3.8 is required (e.g., in a virtual environment). Moreover, we require some Python libraries, which can be installed by: python3.8 -m venv .venv # source .venv/bin/activate pip install -r analysis/requirements.txt