ISC 2025 Data Streaming BoF

Real-Time Scientific Data Streaming to HPC Nodes: Challenges and Innovations

ISC 2025 Data Streaming BoF

Tuesday - Jun 10, 2025
TBD CET

The most common way scientific data arrives at HPC facilities today is through a set of border gateway nodes connected to some form of shared file systems. Dataflow orchestration tools like Globus have made this approach popular by being programmable, easy to use, and efficient. This approach works well when the subsequent compute job for data analysis can afford to wait in the scheduler’s queue, with the overall completion time being mostly dominated by that wait time.

However, as HPC centers and experimental and observational facilities become more integrated, workflows will demand immediate, real-time feedback where the latency and performance variability that comes with a shared file system is no longer acceptable. While a few streaming workflows have emerged recently, many HPC, data, or network user facilities are not set out to support these workflows out-of-gate, for policy, scheduler, or hardware reasons. However, these workflows are the cornerstone of modern scientific applications, as their stringent timing requirements benchmark the ultimate integration of HPC, data, and networks into seamless compute-in-the-loop workflows for experimental or observational user facilities.

This BoF discusses this alternative way of using HPC by opening with a science use case that exemplifies this new class of emerging workflows. It addresses a groundbreaking field in supercomputing that has not been previously featured at ISC. Through a set of lightning talks from user facilities, the BoF will survey how HPC centers address this need today and what they have planned for the near future. These presentations aim to seed the ensuing discussion where the audience can ask questions to key staff of HPC facilities, or can bring their specific streaming workflow to the attention of the workflow community, or participate in a workflow challenge where the audience will be divided to represent the various stakeholders of example streaming workflows and have to work together to identify requirements, hurdles and possible solutions.

Agenda