Powered by OpenAIRE graph
Found an issue? Give us feedback
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/ ZENODOarrow_drop_down
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
ZENODO
Other literature type . 2025
License: CC BY
Data sources: ZENODO
ZENODO
Presentation . 2025
License: CC BY
Data sources: Datacite
ZENODO
Presentation . 2025
License: CC BY
Data sources: Datacite
versions View all 2 versions
addClaim

Streaming Genomics at Scale: Efficient CSV Delivery via GraphQL, Brotli and WASM

Authors: Brinkman, Jack;

Streaming Genomics at Scale: Efficient CSV Delivery via GraphQL, Brotli and WASM

Abstract

PLEASE NOTE: Following the conference, it is important to note that the approach described in this abstract was influenced by time and infrastructure constraints. While effective at the scale of ARGA’s datasets in this specific context, embedding large data transfers within a GraphQL response is not generally recommended. In most cases, a more conventional approach - such as generating an export job, writing the result to object storage (e.g., S3 or GCS), and providing download via HTTP - will be more scalable, maintainable, and operationally robust. The Australian Reference Genome Atlas (ARGA), an advanced indexing platform designed to aggregate, filter, and provide access to complex genomics datasets, saw a need to support performant downloads of large datasets without introducing bespoke infrastructure. The primary objective was to enable users to efficiently retrieve large-scale CSV data through the existing GraphQL API. We also sought to avoid standalone download endpoints that would introduce maintenance overhead and architectural complexity. Our solution was two-pronged, leveraging backend and client-side implementations: 1. We leveraged the performance of ARGA s Rust-based backend and the flexibility of GraphQL. The backend retrieves the necessary records directly from ARGA s PostgreSQL database and transforms them into a CSV-formatted string. This CSV content is then compressed using Brotli, a modern compression algorithm known for its high compression ratio and decompression speed. The resulting binary data is encoded into a base64 string to ensure compatibility with the GraphQL transport layer and transmitted to the client through a standard GraphQL response. 2. On the client side, the base64 string is decoded, and WebAssembly (WASM) modules are used to decompress the Brotli-encoded data directly within the browser. This approach leverages WASM s efficiency and sandboxed execution to minimise latency and browser resource consumption. The solution achieved a significant reduction in payload size from 56.1 MB uncompressed down to 12.1 MB compressed for a sample dataset of 265,459 records. In this presentation, we demonstrate the novel utilisation of compression techniques to enhance the performance and efficiency of web-based applications.

Keywords

Tools, Research Software

  • BIP!
    Impact byBIP!
    selected citations
    These citations are derived from selected sources.
    This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    0
    popularity
    This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
    Average
    influence
    This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    Average
    impulse
    This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
    Average
Powered by OpenAIRE graph
Found an issue? Give us feedback
selected citations
These citations are derived from selected sources.
This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Citations provided by BIP!
popularity
This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
BIP!Popularity provided by BIP!
influence
This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Influence provided by BIP!
impulse
This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
BIP!Impulse provided by BIP!
0
Average
Average
Average
Green