Merlin

Merlin

Enabling Machine Learning HPC Workflows
Last updated: 25 May 2022

v1.8.5
Released on: 07 Mar 2022

The Merlin workflow framework targets large-scale scientific machine learning (ML) workflows in High Performance Computing (HPC) environments. Merlin is a producer-consumer workflow model that enables multi-machine, cross-batch job, dynamically allocated yet persistent workflows capable of utilizing surge-compute resources. Key features are a flexible and intuitive HPC-centric interface, low per-task overhead, multi-tiered fault recovery, and a hierarchical sampling algorithm that allows for highly scalable task execution and queuing to ensembles of millions of tasks.

Project Information