SC26 Data Streaming BoF

As High-Performance Computing enters the Exascale era, the scientific community faces a data paradox: our ability to generate data through high-fidelity simulations and next-generation instruments vastly outpaces our ability to store it. In domains ranging from climate modeling and cosmology to experimental physics, simulation outputs and sensor inputs now routinely reach the petabyte range per run. The traditional "compute-store-analyze" paradigm, which dumps massive datasets to parallel file systems for post-processing, has become an unsustainable bottleneck that introduces unacceptable latency, wastes energy on data movement, and often puts high-frequency data analysis out of practical reach. This BoF is about the architectural shift that follows, from file-based workflows to high-performance streaming, where storage serves as a sink for final scientific results rather than a buffer for post-processing, and analysis, visualization, and reduction happen in-transit or in-situ.

The convergence of HPC and Artificial Intelligence makes this conversation especially timely. Modern scientific workflows increasingly rely on AI to steer simulations, train surrogate models on the fly, and integrate digital twins with real-time data from the edge. Such live interactions require data to flow continuously between compute nodes and inference engines, bypassing the latency of the file system entirely. While a number of streaming workflows have emerged, many HPC, data, and network user facilities are not set up to support them out of the gate, for policy, scheduler, or hardware reasons. Yet these workflows have become a cornerstone of modern scientific applications, and their stringent timing requirements benchmark the ultimate integration of HPC, data, and networks into seamless compute-in-the-loop workflows for experimental and observational user facilities.

This BoF takes another step toward building a community around the software stacks, protocols, and hardware architectures required to make streaming a first-class citizen in HPC, moving the conversation beyond ad-hoc, application-specific implementations toward standardized, interoperable frameworks. It is the sixth edition of its kind, following successful sessions at SC24, ISC25, SC25, SCA26, and ISC26, as well as the Stream2HPC 2026 workshop at ISC26. Those events revealed strong interest in streaming workflows, particularly around integration challenges with existing HPC infrastructure and the need for standardized approaches across facilities. SC26 is an ideal venue, bringing the major HPC user facility providers together with scientific computing users who already run real-time streaming workflows or expect that need to arise soon, and offering an opportunity to survey current solutions and discuss limitations across HPC, data, and network providers. We will also report on the community whitepaper started at ISC26 and invite more people to contribute.