SC25 Data Streaming BoF

Real-Time Scientific Data Streaming to HPC Nodes: Challenges and Innovations

SC25 Data Streaming BoF

Tuesday - Nov 18, 2025
5:15pm—6:45pm CT

Today, scientific data typically arrives at HPC facilities via gateway nodes into shared file systems, a model supported by tools like Globus and suited for workflows that tolerate queue delays. However, as HPC, experimental, and observational facilities become more integrated, time-sensitive workflows increasingly require real-time feedback, where latency and variability from shared storage are unacceptable. Despite early examples of streaming workflows, most facilities remain unprepared to support them due to policy, scheduling, or hardware limitations. Addressing these gaps is critical to enabling seamless, compute-in-the-loop workflows that define the next era of scientific discovery. This BoF addresses this groundbreaking and growing field in supercomputing.

In addition to lightining talks, the BoF will have a very interactive part that we call a "Workflow Challenge Workshop". In this workshop, participants will be divided into small groups representing different stakeholders (facility operators, users, software developers). Each group will receive a real streaming workflow scenario and work together to identify technical, policy, and infrastructure requirements for implementation. Groups will document their solutions using a shared digital whiteboard, allowing cross-group visibility and iteration.