Workflows Systems

Collection of actively-developed open source workflow systems. Currently, 20 workflow systems provide WCI metadata.

Contribute

snakemake

Snakemake

This is the development home of the workflow management system Snakemake. For general information, see

Python
v8.10.7
12 Apr 2024
WCI metadata

nextflow

Nextflow

A DSL for data-driven computational pipelines

Groovy
Version 24.02.0-edge
12 Apr 2024
WCI metadata

covalent

Covalent

Distributed workflows for quantum and HPC

Python
v0.234.0-rc.0
11 Apr 2024
WCI metadata

geoweaver

Geoweaver

a lightweight workflow software to easily orchestrate pipelines from Python and shell scripts and preserve history of every execution

JavaScript
v1.4.6
11 Apr 2024
WCI metadata

common-workflow-language

Common Workflow Language

Interoperable workflow execution of containerized command line tools

YAML
1.2.0
10 Apr 2024
WCI metadata

aiida-core

AiiDA

A workflow manager for computational science with a strong focus on provenance, performance and extensibility.

Python
AiiDA v2.5.0
09 Apr 2024
WCI metadata

compss

PyCOMPSs/COMPSs

Easy task-based parallelization and efficient execution in distributed infrastructures.

['Python', 'Java', 'C/C++']
3.3 Orchid
08 Apr 2024
WCI metadata

SmartSim

SmartSim

Machine learning workflows with HPC applications (Python, C++, C, and Fortran)

Python
0.6.2
07 Apr 2024
WCI metadata

pyiron_base

pyiron

A workflow manager for scientific computing on high performance computing infrastructures

python
pyiron_base 0.8.0
05 Apr 2024
WCI metadata

parsl

Parsl

Productive parallel programming in Python

Python
1.2.0
04 Apr 2024
WCI metadata

cctools

Makeflow

Makeflow is a workflow system for parallel and distributed computing that uses a language very similar to Make.

Make
release/7.9.2
26 Mar 2024
WCI metadata

cctools

TaskVine

An execution system for large scale data intensive dynamic workflows.

Python
release/7.9.2
26 Mar 2024
WCI metadata

scipipe

SciPipe

Robust, flexible and resource-efficient pipelines using Go and the commandline.

Go
v0.12.0: Important bugfix. Minor API change.
26 Mar 2024
WCI metadata

maestrowf

Maestro Workflow Conductor

Orchestrate your HPC workflows with ease

Python
v1.1.10
07 Mar 2024
WCI metadata

merlin

Merlin

Enabling Machine Learning HPC Workflows

Python
Version 1.12.0
27 Feb 2024
WCI metadata

watts

WATTS

Workflow and Template Toolkit for Simulation

Python
0.5.1
18 Jan 2024
WCI metadata

libensemble

libEnsemble

Tool for running dynamic ensembles.

Python
1.2.2
11 Jan 2024
WCI metadata

mpi_list

MPI-list

A package for working with lists distributed over MPI.

Python3
11 Jan 2024
WCI metadata

streamflow

StreamFlow

Towards Cloud-HPC Continuum

Python
0.2.0.dev10
05 Jan 2024
WCI metadata

EE-Demo

Apollo Runtime System

A runtime system for the orchestration of workflows across the cloud-edge-Iot continuum.

Polyglot for Functions, AFCL for workflows
v1.1.0
13 Dec 2021
WCI metadata

galaxy

galaxy

Data intensive science for everyone.

Python
v24.0.0
13 Apr 2024

mlflow

mlflow

Open source platform for the machine learning lifecycle

Python
MLflow 2.11.3
13 Apr 2024

cromwell

cromwell

Scientific workflow engine designed for simplicity & scalability. Trivially transition between one off use cases to massive scale production environments

Scala
86
12 Apr 2024

dask

dask

Parallel computing with task scheduling

Python
12 Apr 2024

jobflow

jobflow

jobflow is a library for writing computational workflows.

Python
v0.1.17
12 Apr 2024

nipype

nipype

Workflows and interfaces for neuroimaging packages

Python
1.8.4
10 Apr 2024

fireworks

fireworks

The Fireworks Workflow Management Repo.

Python
v2.0.3
09 Apr 2024

pegasus

pegasus

Pegasus Workflow Management System - Automate, recover, and debug scientific computations.

Java
31 Mar 2024

swift-t

swift-t

Swift/T: High Performance Parallel Scripting Language

Java
31 Mar 2024

radical.entk

radical.entk

The RADICAL Ensemble Toolkit

Python
v1.42.0
12 Jan 2024

signac-flow

signac-flow

Workflow management for signac-managed data spaces.

Python
v0.29.0
01 Nov 2023

For a comprehensive lists of all known computational workflow systems, see Computational Data Analysis Workflow Systems maintained by the CWL community.