Importing workflows written elsewhere¶
For teams whose workflows are already written as PWD or CWL documents.
A workflow that already exists in another language does not have to be rewritten to run on httk₂. Two exchange formats are imported into ordinary jobs:
Format |
Command |
Needs |
|---|---|---|
|
nothing beyond httk-workflow |
|
Common Workflow Language (CWL) |
|
|
Both importers share three properties.
Importing is one way. The imported job carries the document it came from, and nothing translates a job back into the language it was written in. A document that changes afterwards is picked up by importing it again, which submits a new job; the old one keeps running exactly the bytes it was submitted with.
No runner file is written per workflow. Both formats are executed by a
packaged runner — pkg:httk.workflow.compat.pwd/pwd_runner.py and
pkg:httk.workflow.compat.cwl/cwl_runner.py — referenced through the reserved
installed form the manager resolves inside its own module allowlist. The document
is data in the payload; the program that executes it is the same installed bytes
for every imported job, so a partitioned campaign of imported workflows copies no
runner even once.
The workflow runs on httk’s own machinery. No other engine is invoked, no
other engine is bundled, and nothing shells out to cwltool. An imported job is
claimed, leased, retried, checkpointed and journalled like every other job of a
workspace, and shows up in job list, job show, job why and harvest.
Python Workflow Definition¶
A PWD document is a JSON graph: nodes are Python functions named
module.function, literal inputs, and named outputs; edges connect one node’s
output port to another node’s input port. The functions themselves live in a
Python module beside the document.
httk workflow import pwd WORKSPACE workflow.json --module workflow.py --tag arithmetic
httk workflow manager run WORKSPACE --until-idle
--module FILE stages a Python file into the payload, and the runner puts the
payload’s files/ directory first on sys.path, so document and module travel
together as one job. --module-path DIRECTORY adds a further import root that
must exist on the machine that runs the job. --input NAME=VALUE overrides an
input node by name, so one document runs with values it does not itself carry.
From Python:
from httk.workflow import Workspace
from httk.workflow.compat.pwd import import_pwd
workspace = Workspace("workflow-workspace")
job = import_pwd(workspace, "workflow.json", modules=["workflow.py"], workflow_inputs={"x": 3})
print(job.job_key, job.payload)
What a PWD job does¶
The whole graph runs inside one job, sequentially, in topological order. A PWD node is one Python call: it is not worth a claim, a lease and a process of its own, and a multi-node document would otherwise become one job per node with no parallelism to show for it.
The packaged runner registers exactly one step, execute, and keeps the position
in the graph in the job state instead of in a step name — a runner’s step set is
registered when its module is imported and pinned per job, so it cannot depend on
which document a job happens to carry. httk workflow runner describe and
--describe therefore report one honest entry point rather than a step set that
changes per payload.
Restarts are handled by a checkpoint rather than by a step name. Every node’s result is written to the job state as it is produced, in one atomic replace per node, and a repeated attempt skips every node already in the checkpoint. A result that is not JSON cannot be checkpointed, so its node — and everything downstream of it, which comes later in the topological order — is recomputed. Nothing is ever silently reused.
A node that raises fails the job with pwd.node_failed, and that failure is
retryable by default, so the manager repeats the attempt within the job’s attempt
budget (--attempts, three by default) and the repeat resumes from the
checkpoint.
Reading the document, in job show:
Job state key |
What it holds |
|---|---|
|
the checkpoint: one JSON value per completed node, by node id |
|
the node ids completed so far, in completion order |
The named outputs are written to pwd-outputs.json in the job’s workdir, and
published under data/pwd/ when the job was imported with
--data-mode transactional.
Documents too large to embed¶
A job’s inputs object is bounded, so a document larger than half that budget is
staged in the payload as files/pwd.json and the job carries a
pwd_document_path pointer to it instead of the document itself. Both spellings
run identically; import_pwd(..., maximum_embedded_bytes=N) moves the boundary.
Security: a PWD document is code¶
Running a PWD document imports and calls the Python functions it names, with
no sandbox — the format’s entire content is module.function references. There
is no safe way to run an untrusted one, and httk₂ does not pretend otherwise:
importing a document is exactly as dangerous as running the module it names.
The default is therefore to allow what the interpreter can import, because the
person importing the document is the person who would otherwise have run it by
hand. When a document comes from somewhere less trusted, --allow-module PREFIX
(or allowed_modules=[…]) records a prefix allowlist in the job, and the runner
refuses to import anything outside it, failing with pwd.node_failed and naming
the module it would have imported.
Common Workflow Language¶
CWL is supported as a workflow language. A document is parsed and validated
with cwl-utils,
normalized into one self-contained JSON plan with every run: reference inlined,
and executed by the packaged cwl_runner.py.
pip install httk-workflow[cwl]
httk workflow import cwl WORKSPACE flow.cwl job.yml --tag echo
httk workflow manager run WORKSPACE --until-idle
The extra is needed on the machine that imports. The machine that runs the imported job needs neither cwl-utils nor anything else: the plan in the payload is plain JSON, and the runner is one file of an installed httk-workflow.
How a CWL workflow becomes httk jobs¶
CWL |
On httk |
|---|---|
|
one activation of the parent job, executed in |
scattered step |
one labeled child job per shard — |
subworkflow |
one child job labeled |
workflow outputs |
|
The runner’s registered steps — start, enter, advance, collect — are a
fixed dispatch vocabulary, not the workflow’s own step names: which CWL step runs
next is state, so one runner describes itself the same way whatever document it
is given. One CWL step per activation means every step boundary is a journalled
state frame a restart resumes from, and a tool interrupted mid-run is re-run from
a cleaned execution directory rather than from a half-written result.
File inputs are staged into a numbered slot beside the execution directory, never
into it, so a tool’s output globs can never match the files it was given, and two
inputs with one basename — three scatter shards each producing spoken.txt is
the normal case — never become one file.
The supported subset¶
Feature |
Supported |
|---|---|
|
|
|
v1.0, v1.1, v1.2 (older versions are upgraded when cwl-upgrader is installed) |
types |
|
command |
|
input bindings |
|
outputs |
|
redirection |
|
step inputs |
|
scatter |
one input, or several of equal length with |
subworkflows |
any depth |
conditionals |
|
requirements |
|
expressions |
|
Everything else is refused at import time, before anything is submitted, with a message naming the feature and where in the document it was found:
Refused |
Why |
|---|---|
|
httk₂ runs no JavaScript engine |
|
there is nothing to execute but an expression |
|
httk₂ runs an argument vector, never a shell command line |
|
httk₂ stages the files an input names and builds no working directory from a listing |
|
outside the supported type set |
|
only |
|
whole files only, and exactly the files an input names |
|
the files a glob matched are collected unchanged |
|
a tool never writes to its input files |
CWL v1.2 loops |
no loop support |
complex |
one plain parameter reference, nothing that has to be computed |
DockerRequirement is neither refused nor honoured. It is recorded as the
required capability docker on the job — so only a manager started with
--capability docker will claim it — and reported as a warning, because httk₂
runs the command directly and never pulls, builds or enters an image. An
unsupported hint (as opposed to a requirement) is dropped with a warning, which
is what a hint means.
Failures¶
Code |
What happened |
|---|---|
|
the tool exited outside its |
|
a required input was not connected, or its file is gone |
|
a non-optional output’s glob matched nothing |
|
a scattered input is not an array, or a dotproduct’s arrays differ in length |
|
no step of the workflow can run and some are unfinished |
|
a child job published no readable outputs |
None of them are declared retryable: a tool that exited nonzero says the same thing on the next attempt. What repeated attempts are for — a lost node, a killed process — is handled by the manager without any failure being published at all.
From Python¶
from httk.workflow import Workspace
from httk.workflow.compat.cwl import import_cwl
workspace = Workspace("workflow-workspace")
imported = import_cwl(workspace, "flow.cwl", "job.yml", tag="echo", data_mode="transactional")
for warning in imported.warnings:
print(warning)
print(imported.job.job_key, imported.document["class"])
load_cwl_plan() does the parsing and the
subset check alone, without submitting anything, which is the cheap way to find
out whether a document can be imported at all.