httk-workflow

This site documents the httk-workflow module. For the full documentation of httk₂, see docs.httk.org.

The module implements a recoverable, language-neutral workflow protocol whose source of truth is a single atomically renamed state marker per job. It presents three layers, each with its own import home: the filesystem protocol (httk.workflow.protocol), the execution and authoring surface (httk.workflow — Runner, Attempt — with lower-level helpers in httk.workflow.runtime), and orchestration and management (Workspace, TaskManager, collect, and named submodules for transfers, remotes, and compatibility). Installations register the httk workflow command tree.

Managers are started by the workspace’s manager.launch setting, using the built-in process launcher or a launcher bundle such as slurm. A remote is only the transport for files and commands on another machine.

httk₂ workflows are language-independent: runners, hooks, and postprocess scripts can be written in any language; a workflow is a manifest plus the members it references. Python hooks remain first-class, with an in-process fast path. Successful hook outputs use the same assembly semantics as executable hooks; collector failures differ deliberately: registered .py exceptions abort iteration, while executable-hook errors degrade per job and continue the sweep.

Quick links

New here — start with Quickstart: five commands from an empty directory to a finished relaxation, no runner written and no VASP required.

The filesystem protocol — the language-neutral on-disk contract

• The filesystem protocol API — the httk.workflow.protocol namespace

• Workflow filesystem API — the normative on-disk specification

The execution API — writing and running workflow steps

• Native runner helpers — the Python authoring SDK: Runner, Attempt, steps

• Runner SDKs — the same authoring surface in eight more languages

• Packaged VASP runners — the packaged runners and relaxation report, for campaigns that write none

• Workflow packages — authoring directory packages and their manifest

• Workflow declarations — saying what a workflow is, for a data layer

• Run provenance — turning one JobRecord into one httk.core.Run

• Collecting results — collecting provider-produced outputs and products

• Workflow-language runner realizations — CWL, PWD, jobflow, and httk-v1 runner realizations

• Examples — worked examples as a notebook

Orchestration and management — driving and inspecting a workspace

• Task-manager usage — workspaces, submission, managers, inspection, repair

• Project and workflow command line — the whole httk workflow tree: projects, config, remotes, and launchers

• Sealing jobs, workspaces, and projects — signing finished jobs, workspaces, and projects so tampering is detectable

• Stable ids — keeping entry ids stable across store rebuilds with a sealed id ledger

• Campaigns — partitioning a very large campaign across many workspaces

• Composing workflows — a workflow that calls other workflows as child jobs

• Scale benchmarks — measured local scale snapshot and benchmark methodology

• Collecting results — reading finished jobs back out as records and collected outputs

• Remotes — reaching a machine with a packaged or custom remote adapter

• Launchers — starting managers with a packaged or custom launcher

• Reference — the generated API reference

Migration

• httk v1 migration guide

• httk v1 compatibility

Install

Preferably work in a Python virtual environment:

git clone https://github.com/httk/httk-workflow
cd httk-workflow
python -m pip install -e .

Minimal setup

One workspace, one job of a packaged runner, and one manager that runs it:

httk init --name "Your Name" --email you@example.org
httk project init --name quickstart .
httk workspace init --name default .
httk job new --workflow vasp-relax --input structure=POSCAR --tag silicon
httk workspace settings set --key vasp.command --value "$PWD/examples/mock_vasp.py" default
httk workflow run
httk workflow collect

Quickstart walks through exactly those commands, including how to run them without VASP installed. A complete payload prepared some other way is still submitted directly:

httk job submit --workspace workflow-workspace --placement project/00 prepared-job

Documentation

• Quickstart
  • A structure to start from
  • The eight commands
  • What each command did
  • Looking at a job
  • Many jobs at once
  • Launchers
  • Remotes
  • Moving a project between machines
  • Where to go next
• The filesystem protocol API
  • The contract
• Native runner helpers
• Runner SDKs
  • Python and Bash authoring parity
  • Native Bash runner API
  • Native C runner API
  • Native modern-Fortran runner API
  • Native Rust runner API
  • Native Perl runner API
  • Native Ada runner API
  • Native C++ runner API
  • Native Java runner API
• Packaged VASP runners
  • Submitting a job
  • The command to run
  • Declared inputs and parameters
  • What a run does
  • relaxation-report
  • relaxation-plot
  • Failure codes
  • Formation energies
  • Copying one
• Workflow packages
• Workflow declarations
  • Carried verbatim
  • Declared and observed
  • A job that declares
  • Declaring what a job observed
  • What a collect reports
• Run provenance
  • Declared and observed
• Collecting results
  • What a record guarantees
  • Members
  • Selecting records
  • From the command line
  • Workflow collect hooks
• Workflow-language runner realizations
  • Language packages
  • Bare documents and one-shot jobs
  • Collection
  • CWL supported subset
  • Python API and registry
• Task-manager usage
  • Environment preludes
• Project and workflow command line
• Sealing jobs, workspaces, and projects
  • The three levels
  • Auto-sealing succeeded jobs
  • Key refs
  • What a seal refuses, and what still works
  • Sealing and unsealing in order
  • Verifying
  • Not to be confused with
• Stable ids
  • The three pieces
  • The anchoring rule
  • Using it from collect
  • Bases and numbering
  • Supersession and re-binding
• Campaigns
  • The two rules
  • The partition map
  • Submitting into a campaign
  • Running and collecting across partitions
  • Bounded fan-out: placement recipes
  • Where to go next
• Composing workflows
  • The model
  • What can be called
  • A worked example
  • Passing results between calls
  • Failure semantics
  • Requirements and limits
  • Where to go next
• Scale benchmarks
  • Method
  • Snapshot
  • Linearity check
• Remotes
  • Setting one up
  • From Python
  • Writing a remote adapter
• Launchers
  • Setting one up
  • Several launchers
  • From Python
  • Writing a launcher
• Reference
  • httk.workflow
  • httk.workflow.languages
  • httk.workflow.languages.cwl
  • httk.workflow.languages.pwd
  • httk.workflow.languages.jobflow
  • httk.workflow.languages.httk_v1
• Examples
• Workflow filesystem API
• Migrating an httk v1 workflow to httk₂

Details

• Native runner helpers in detail
• Task-manager usage in detail
• Workflow packages in detail
• Project and workflow command line in detail
• Workflow monitor
• Writing a remote adapter in detail
• Writing a manager launcher in detail
• Workflow filesystem API in detail
• Migrating an httk v1 workflow to httk₂ in detail
• httk v1 task compatibility