Sometimes science happens slowly, sometimes it comes at us like a firehose. Experiments that produce huge volumes of data, e.g. scientific instruments producing streaming data, are becoming more common with time. Capturing, processing, storing, and sharing high-volume, high-speed scientific data is a challenging problem and one that often requires a mix of off-the-shelf tools and custom, bespoke solutions. In this technical note, we will take a short look at the challenges in more detail, highlight some existing tools that are useful to address these challenges and discuss some general strategies and design principles for those times that you have to take data matters into your own hands.

This project is supported by the National Science Foundation Office of Advanced Cyberinfrastructure in the Directorate for Computer Information Science under Grant #2127548. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the funding agency.