Task-manager usage in detail¶

For operators: initializing workspaces, submitting jobs, running managers, and inspecting or repairing what they leave behind.

Every command below is spelled the canonical way, httk workflow …. The httk-taskmanager executable installed beside it is an alias of the same tree (init, submit, run, status, and request); see the project and workflow command line for the mapping.

Initialize a workspace¶

WORKSPACE is optional inside a project: omitting it uses the project’s recorded default, or the per-user default workspace when none is recorded. A project does not contain workspaces; record its routing explicitly with workspace default NAME. Explicit local names are created with workspace init PATH; an existing path is adopted and registered. REMOTE:PATH initializes and names a workspace on that remote.

httk workflow workspace init runs/WORKSPACE --name WORKSPACE

A workspace on a cluster is created there over the adapter; its owning machine registers the basename (or --name) in its own registry. workspace list shows this machine’s names and paths, while workspace list kappa: asks kappa. workspace forget deregisters a name, and workspace delete --force destroys the workspace and deregisters it. A library caller still constructs Workspace(path) directly; the registry is the command-line contract. See Project and workflow command line in detail for the whole workspace group and Campaigns for spreading a very large run across many workspaces.

Protocol publications are synchronized to storage by default. --no-durable turns that off for throwaway workspaces and makes submission and transitions faster at the price of correctness after a node crash: an unsynchronized journal frame can be lost while the marker naming it survives, which leaves a job whose state cannot be read until workspace fsck --repair restores it. --durable is still accepted and does nothing, since it is now the default.

Running on a remote¶

The canonical remote flow keeps scheduler settings with the remote workspace. httk₂ must already be set up on the remote (log in there and install it, e.g. with pipx install httk-workflow); remote check verifies that:

httk workflow remote add kappa --template ssh-slurm
httk workflow remote configure kappa \
    --set host=kappa.example.org --set username=rar \
    --set check_connectivity=yes
httk workflow remote check kappa
httk workflow workspace init kappa:/scratch/rar/httk/runs
httk workflow workspace settings set kappa:runs slurm.partition batch
httk workflow workspace settings set kappa:runs vasp.command "srun -n 32 vasp_std"
httk workflow job new --workflow vasp-relax --input structure=POSCAR --tag silicon
httk workflow transfer default kappa:runs --job JOB-ID
httk workflow run kappa:runs --workers 8
httk workflow workspace status kappa:runs

The remote init command creates and registers runs on kappa; kappa:runs is resolved by kappa at use time. transfer detaches the selected job from the local default workspace, and run submits a manager through the remote adapter. Use transfer kappa:runs default after the remote job stops, then httk workflow collect locally.

Workspace policy¶

Four tunables belong to the workspace rather than to any one process, so that every manager, CLI, and independent implementation attaching it agrees on them. They live in .httk-workflow/format.json and are read and written with:

httk workflow workspace policy show WORKSPACE
httk workflow workspace policy set WORKSPACE visibility_deadline_seconds 60
httk workflow workspace policy set WORKSPACE retention.journal_days 90

Key	Default	Meaning
`visibility_deadline_seconds`	`5.0`	How long a marker rename or a referenced journal frame may take to become visible before it is called damage.
`lease_seconds`	`900.0`	The claim lease of a manager started without `--lease-seconds`.
`journal_segment_bytes`	`67108864`	The size at which a journal writer rotates to its next segment.
`retention`	`{}`	Optional `attempt_control_days`, `journal_days`, and `trash_days` collection limits.

Values are given as JSON and validated on write; an unknown key is refused rather than stored. A change reaches a manager when it attaches, so restart long-running managers after changing policy. Concurrent policy writers are not serialized: the write itself is atomic, but the last writer wins.

Application settings¶

Separate from that engine policy, a workspace also holds application settings: a flat, dotted-name map of small values a runner resolves at run time — the VASP command and a pseudopotential library. The manager submission profile is also a workspace setting, so each workspace can carry its own scheduler requirements.

httk workflow workspace settings set WORKSPACE vasp.command '"srun -n 32 vasp_std"'
httk workflow workspace settings show WORKSPACE

For a Slurm manager, set its profile in the target workspace as well: slurm.account, slurm.partition, slurm.time_limit, slurm.nodes, slurm.cpus_per_task, and slurm.reservation become batch directives, while manager.workers supplies the default worker count. The adapter reads these values from the workspace when it composes the batch script.

A runner reads one through a.setting("vasp.command"), resolved in layers — the job’s inputs, a real HTTK_VASP_COMMAND deployment override, the workspace setting, then the runner’s default. The manager exports scalar workspace settings into each attempt environment (vasp.command becomes HTTK_VASP_COMMAND) and snapshots them into context.json, so a runner sees the values the workspace held when its job was claimed. See Packaged VASP runners and Python and Bash authoring parity.

Readiness and transfer environment advisories¶

Use the read-only precheck before starting managers:

httk workflow precheck WORKSPACE
httk workflow precheck WORKSPACE --json
httk workflow precheck WORKSPACE --runner-search-path PATH

It reports environment entries resolved from the current process environment, workspace settings, or declared defaults, plus runner-reference problems, for pending jobs. It also measures each pending job against the workspace’s live managers: a job no live manager can claim names the closest manager’s unmet requirements (the same wording job why uses, including a runner-module allowlist a manager does not carry), a language job (the collect gate’s workflow_realization = language pair) whose engine modules are absent names the pip extra to install (for example pip install httk-workflow[jobflow]) — a failure only when no live manager serves its executor, since the extras belong on the machine that runs the job; when one does, it is indeterminate and non-failing. A declared required input whose staged destination has gone missing from the payload is flagged. When no manager is live, one workspace-level notice replaces per-job claim findings. An unresolved entry, broken runner, unclaimable job, missing-and-unserved engine, or missing required input gives exit status 1. The repeatable --runner-search-path option checks installed runner references; a plain installed reference without a configured path is indeterminate, not a failure, and does not by itself give exit status 1. The authoritative environment gate is still at attempt start; this report is advisory and can become stale. The HTTK_* layer is this process’s environment, not a promise about the environment of a later compute node.

httk workflow workspace managers WORKSPACE answers “what serves this workspace?” directly — one line per registered manager, live or stale, with its pools, capabilities, executors, and runner modules — rather than by reading it off a job why on an arbitrary job.

Transfers run the environment check against destination settings, job overrides, and declared defaults, without treating the client process environment as the destination. They warn about unresolved default-less entries; add --strict-environment to block before any job state is moved. Remote settings are checked through an isolated read when reachable; an unreachable destination gets one immediate warning and is only a strict-mode failure.

Freeing disk on a quota’d filesystem¶

A manager frees nothing while it runs, by design: it is never required to execute cleanup code, so it can disappear between any two instructions. Over a long campaign the workspace therefore accumulates one control directory per attempt, one journal writer and manager directory per process start, a full copy of every tree a transaction replaced, and an intact bundle for every transfer already acknowledged. On a quota’d HPC filesystem that is what fails first.

Collection is a separate, explicit operation. Configure the retention limits once, then run it from a maintenance job or by hand:

httk workflow workspace policy set WORKSPACE retention.attempt_control_days 14
httk workflow workspace policy set WORKSPACE retention.trash_days 14
httk workflow workspace policy set WORKSPACE retention.journal_days 90
httk workflow workspace gc WORKSPACE --dry-run
httk workflow workspace gc WORKSPACE

It is safe to run against a live workspace: a manager that is still heartbeating keeps its own directory and every journal segment it wrote, no marker or payload is touched beyond the aged attempt-control directories of terminal jobs, and pruning an empty placement mirror that a transition is recreating underneath is an ordinary outcome rather than an error. A limit left unset means keep. See the command guide for the full category table and for what collecting journal history costs.

A long-lived manager can also do this itself, which is convenient where no maintenance job exists:

httk workflow manager run WORKSPACE --gc-interval 3600

The manager then collects at most once per interval, at the end of a tick and never between observing a marker and acting on it, obeying exactly the same policy.retention limits. It is off by default, and a failed collection is logged rather than allowed to disturb scheduling. Keep the interval long: a collection walks the state tree and the journal directory, which is work the scheduling passes do not need done often.

Filesystem visibility¶

A workspace may be attached from several nodes under the same account, which makes metadata visibility part of the filesystem configuration.

Mount options. Renames and directory listings must be seen by other clients promptly, so an aggressively cached mount needs its attribute caching bounded:

NFS: actimeo=5 (or the pair acdirmin=1,acdirmax=5) and lookupcache=positive are a good starting point. The defaults — acdirmax=60 — mean another node may keep serving a stale directory listing for up to a minute, which is legal and must simply be waited out. noac removes the staleness entirely and is correct, but it disables attribute caching and close-to-open optimization altogether and is usually far too slow for a workspace with many jobs. nolock is fine: the protocol never takes a POSIX lock. Use NFSv4.1 or newer where available.
Lustre and GPFS: no special options. Their metadata coherence is strong enough that the local-filesystem defaults apply.
Anything backed by an object store or a FUSE cache without rename atomicity is not a supported workspace filesystem at all: the protocol requires rename(2) to be atomic and to fail rather than silently overwrite.

The visibility deadline. Set it to comfortably exceed the worst-case staleness window of the mount:

Filesystem	Recommended `visibility_deadline_seconds`
Local disk, tmpfs, single node	`5` (the default)
Lustre, GPFS, BeeGFS	`10`
NFS with `actimeo=5`	`30`
NFS with default caching (`acdirmax=60`)	`120`

The deadline costs nothing when nothing is wrong: the schedule starts at 10 ms and stops the moment the rename or frame becomes visible. It is only spent when the filesystem is actually lying to one client.

Clocks. Leases are advisory evidence, not a fence. A manager decides that another manager’s claim has expired by comparing its own wall clock with the heartbeat timestamp that manager wrote, so the nodes sharing a workspace should run NTP; skew larger than lease_seconds will cause premature or delayed recovery of abandoned claims. Safety does not rest on this: the actual fence is the marker rename, which exactly one actor can win, so a mistaken expiry decision costs a lost claim rather than two runners in one job.

Submit a job¶

A prepared payload is a directory containing an immutable job.json and its runner. Submit it at any arbitrary placement:

httk workflow job submit WORKSPACE PAYLOAD --placement project-a/00/17

Submission copies by default. --move performs a same-filesystem rename and consumes the source directory.

Run¶

httk workflow manager run WORKSPACE --workers 8

Safety property: a task manager claims and runs only jobs whose marker, payload directory, and job.json are regular, non-symlink entries owned by the account running that manager. Child jobs belong to the manager’s account; imported jobs belong to the account that imports them.

Without pool configuration, a manager advertises the reserved default pool. Additional routing and capability labels are explicit:

httk workflow manager run WORKSPACE \
  --pool vasp \
  --capability gpu \
  --workers 4

A manager claims work under the workspace’s lease_seconds unless --lease-seconds overrides it for that manager alone.

The default until-idle behavior is useful for batch invocations and tests; pass --idle to keep serving.

One banner, then one summary. Whatever the console log level, run and manager run print one line on startup — the manager id, the workspace, the log file path, and the pools, capabilities, and executors this manager serves — so a normal run is never silent about which manager is doing what and where its log is. When it exits idle it prints one closing summary line that classifies every remaining job: how many succeeded and failed, how many are not claimable here — ready or unregisterable-submitted jobs broken down by the pool, capability, or executor this manager does not serve — how many are waiting on children, how many are paused, and how many committing or cancelling jobs have an unreadable definition. A job this manager cannot progress — including one whose job.json is corrupt — no longer keeps it awake to the idle timeout; it is reported instead. If the manager does hit --idle-timeout, the advice names the actual pool, capability, and executor mismatches, the flags that would clear the pool and capability ones, and points an unreadable definition at workspace fsck, rather than a bare suggestion to raise the timeout.

Taking over another manager’s attempt. An expired lease says that a manager stopped heartbeating, which is not the same as its attempt having stopped, so neither workdir mode relaunches on lease expiry alone:

Workdir mode	What admits a takeover	Relaxed by
`persistent`	The recorded process is provably gone on this host. A second writer would corrupt the shared directory.	`--unsafe-persistent-takeover`
`isolated`	The recorded process is provably gone, or the heartbeat has been silent for `--takeover-grace-factor` leases (default `2.0`). A second attempt corrupts nothing but costs a second allocation.	`--unsafe-isolated-takeover`

Both unsafe options and the evidence of every takeover — which rule admitted it and how old the heartbeat was — are recorded in the new attempt’s state frame, so job log shows exactly why a job was relaunched.

A persistent-workdir attempt whose recorded process ran on another host can never be proven stopped from here — only the launching host can ask its kernel about that process — so a manager on a different host leaves it alone and logs that decision (an info-level line, not a buried debug one). job why says the same truthfully: it reports the job as blocked, names the host the writer ran on, and tells you to run a manager on that host or pass --unsafe-persistent-takeover, rather than claiming the expired lease will be recovered here.

Unresolvable join children. A job waiting on a child that cannot be resolved in this workspace does not wait forever: after --join-grace-seconds (default 3600) it fails with dependency_failure. The grace is measured from the instant a manager first records the child as unresolvable, and that instant is persisted into the waiting job’s state frame, so the deadline survives a manager restart instead of resetting to zero each time a new manager takes over. job why on the waiting job shows the recorded instant and what the grace will do.

Long scans. A manager heartbeats between its scheduling passes and inside long ones, and bounds how many markers of one kind it processes per pass, resuming the rest on the next pass in a stable order. A workspace too large to scan inside one lease is therefore served round-robin instead of making the manager look abandoned to its peers. A pass that still consumes half of the lease is logged as a warning, and nine tenths of it as an error: raise lease_seconds, split the workspace, or reduce what the manager scans.

Every claim, launch, transition, recovery decision, and refused request is logged. The console reports warnings and errors, while the complete info-level record is rotated into .httk-workflow/managers/MANAGER_ID/log. --log-level raises or lowers both, --log-file moves the file, and --json-logs emits one JSON object per line for ingestion.

A manager drains on SIGTERM or SIGINT, which is what a batch system sends at walltime. The first signal stops claiming, terminates the running attempts, and keeps committing their outcomes for --drain-timeout seconds before exiting successfully; a second signal exits immediately. Anything left behind is recovered from its expired lease by the next manager.

Scheduling¶

A manager never reads the whole workspace on a tick. Every scheduling pass discovers its work by streaming the state tree of one active kind — one of submitted, ready, claimed, running, committing, waiting, and cancelling — and never opens the terminal succeeded, failed, or cancelled trees at all. The in-memory marker index and every scheduling scan therefore grow with the active work in flight rather than with the accumulated history of a workspace that has run for years.

Bounded streaming discovery. A pass walks directory entries with os.scandir instead of materializing an rglob of the tree, and it stops early on two independent budgets: it visits at most discovery_budget directory entries — 4096 by default — and it collects at most maximum_pass_markers markers — 256 by default — before it yields the tick. It also takes a heartbeat opportunity every 512 entries inside the walk, so even one enormous flat placement directory keeps a manager’s lease alive from within the scan exactly as crossing many placements does, rather than only between passes. The walk keeps a resume cursor per top-level placement root, held in the manager’s memory alone — nothing is written to disk, so two managers of one workspace never contend on a shared position and a restarted manager simply begins a fresh cycle. The roots are served in a round-robin rotation with per-root resume, so a one large placement subtree can never starve a smaller sibling, and the next tick continues precisely where this one stopped. A concurrent transition that renames or removes a marker underneath the walk is tolerated silently, consistent with how a vanished marker becomes a miss rather than a fault.

The exhaustive workspace operations — fsck, gc, collect, status, and job list — use the same scandir walker in an exhaustive mode with no cursor and no budget, so their semantics are unchanged; only the bounded scheduling passes carry the budgets.

Best-within-window priority. Claiming ready work scans a single bounded window and then claims the best-priority candidates found within that window, in a stable order among equal priorities, up to the number of free worker slots. Priority is therefore best-within-window rather than exact-global: that is the deliberate price of bounded discovery, and the round-robin rotation is what eventually reaches a starved subtree on a later tick. Recovering exact global order would require a derived priority index, which this implementation does not build; it remains a possible future addition only where a deployment measures that it needs one.

Restricting a manager to placement prefixes. A manager may be told to scan only part of the tree, exactly the way pools and capabilities restrict what it claims:

httk workflow manager run WORKSPACE \
  --placement-prefix project-a \
  --placement-prefix project-b/2026

The flag is repeatable, and every scheduling scan — bounded window and exhaustive walk alike — is then confined to those subtrees. With no --placement-prefix a manager scans the whole workspace, which is the default. Overlapping assignments stay safe because the marker rename still arbitrates a claim, so two managers assigned the same subtree never both run one job; disjoint assignments simply divide the scanning, so neither manager pays to walk the other’s trees. The assignment is deployment policy and not a protocol change — placement values remain project-owned semantics that the engine only validates and filters on — and it is recorded in the manager’s manifest, so job why reports a prefix mismatch when a live manager’s placement prefixes exclude the placement of the job being diagnosed. A configured prefix that currently matches no job — whether a typo or simply a manager started before its jobs are submitted — is logged as one honest warning at manager start, naming the prefix and noting that the manager will serve that subtree once work arrives there, so a scan-nothing prefix is a diagnosable condition rather than a silent one.

Laying out placements across a large campaign and assigning their subtrees to managers by a written recipe rather than by hand is out of scope here; the Phase 14 campaign recipes add it.

Inspect and control¶

httk workflow workspace status WORKSPACE
httk workflow workspace status WORKSPACE --json

httk workflow job request WORKSPACE JOB_UUID pause \
  --operator "$USER" --reason "inspection"

httk workflow job request WORKSPACE JOB_UUID continue \
  --operator "$USER" --reason "inputs repaired"

An override_step --step X request is pre-validated on the client: when the job’s state frame already records the runner’s runner_steps (written after its first attempt), a step outside that set is refused before the request is published, listing the recorded steps. --force downgrades that refusal to a stderr note and publishes anyway — a payload runner is mutable, so an operator may have edited it to add the step. Before the first attempt nothing is recorded, so the request is allowed with a note on stderr that it could not be pre-validated; in either allow case the runner, not the manager, refuses the step at the next attempt if it does not implement it (the manager only shape-checks the request).

Requests capture the exact current marker generation and record reference. A delayed request therefore cannot mutate a newer job state. One that can never apply again — because the job has moved on — is moved to .httk-workflow/requests/retired/ with the reason recorded beside it instead of being reread on every pass; a request for a runner executor this manager does not serve is left alone for a manager that does.

When the publishing installation has an operator identity key — created by httk workflow config init — the request also carries a detached Ed25519 signature over its canonical JSON, and the manager records the verified operator_key in the journalled state frame beside operator and reason. The signature is optional in both directions: a request without one is applied exactly as before, so a mixed deployment needs no flag day, while a request whose signature does not verify is quarantined with that reason rather than applied. It is attribution and not authorization; see the project CLI guide.

Cancelling a running job is fenced and verified, not a single signal:

httk workflow job request WORKSPACE JOB_UUID cancel \
  --operator "$USER" --reason "wrong inputs"

The manager first renames the marker running → cancelling, which fences the attempt so it can no longer commit an outcome. Only then does it SIGTERM the process group, SIGKILL it if it has not exited within the grace period, and verify that it is actually gone; only a verified exit moves the job to cancelled, and how it was verified is recorded in the terminal frame. A manager that dies mid-cancellation leaves a cancelling marker, and the next manager finishes exactly the same procedure. A process recorded on another host cannot be proven stopped here: the job stays cancelling, the reason is journaled, and a warning is logged on every retry — which is the safe answer, because cancelled asserts that nothing is still writing the workdir.

Checking and repairing a workspace¶

workspace fsck verifies the one thing a manager cannot route around: that every state marker still resolves to its journal frame.

httk workflow workspace fsck WORKSPACE
httk workflow workspace fsck WORKSPACE --json
httk workflow workspace fsck WORKSPACE --repair
httk workflow workspace fsck WORKSPACE --repair --quarantine-unrepairable

It reads every marker of every state kind and checks that its record reference resolves — within the configured visibility deadline, so a merely slow network filesystem is never mistaken for damage — to a readable frame whose checksum verifies and whose job, kind, and generation agree with the marker name. Each problem is reported with a stable code: missing_segment, short_read, checksum_mismatch, reference_mismatch, identity_mismatch, unparseable_name, and their siblings. Without --repair nothing is written. The command exits 0 when the workspace is clean or everything found was repaired, and 1 when something is left for an operator.

--repair re-points a damaged marker at the last good frame of its job. Since the frame holding the backward link is the unreadable one, the repair scans the journal for readable frames naming that job and adopts the newest one older than the marker’s own generation — never a newer one, which would be either the damaged frame itself or a transition no marker ever committed. It then writes one fsck_repair state frame, chained to the recovered frame and carrying its step, activation, and attempt counters forward, and renames the marker onto it at the next generation. History is added, never rewritten, and the job is schedulable again.

Two things are deliberately never repaired:

a claimed, running, or committing marker whose manager is still heartbeating within its lease is reported and left exactly as it is, because that manager owns the transition that comes next. Stop the manager, or wait for its lease to expire, and run the repair again;
a marker with no readable older frame — typically a job damaged before its second transition — cannot be restored at all. It is reported, and moved into .httk-workflow/quarantine/ with an audit record only if --quarantine-unrepairable is also given.

Run it when a node crashed while writing, when a filesystem was restored from a snapshot, or whenever job show reports that a state frame is not readable.

Inspecting jobs¶

Five commands read one job the way a manager reads it — the authoritative marker, the journal frame that marker names, and the immutable job.json — and none of them writes protocol state:

httk workflow job list WORKSPACE --kind ready --placement project-a
httk workflow job show WORKSPACE JOB
httk workflow job log WORKSPACE JOB --limit 20
httk workflow job why WORKSPACE JOB

JOB is a job UUID, a complete tag--uuid job key, or any unique prefix of either; an ambiguous prefix is refused with the jobs it matched. Every command also accepts --json and prints one object: a report, a frame array, a diagnosis, or a job array.

job show reports the state kind, placement, priority, generation, job digest, runner identity, the budgets of the retry policy against what has been consumed, the current and initial step, any step set the runner declared, the last failure, the join and per-child state of a waiting job, and the payload, workdir, and data paths.

job log walks the journal backward from the marker through previous_record_ref and prints one line per state frame, oldest first, with the timestamp, the transition, the step, the attempt ordinal, the reason, and any failure code. A frame that cannot be read is reported in place; whatever history remains readable is still shown.

job why answers “why is this job not running?” for every state:

submitted: whether any manager has registered it, and which live managers serve its runner executor;
ready: every claim precondition, one line each — runner executor, claim pool, required capabilities, the maintenance lock, the workspace core profile, the attempt budgets, and which live manager would accept the job;
claimed and running: the owning manager, its heartbeat age against the recorded lease, and whether an expired lease means recovery rather than a stuck job;
committing: that a published outcome is being committed and any manager serving the executor resumes it — unless a commit anomaly has repeated for the same attempt, in which case the recorded error is surfaced and the job is reported as a blocked, wedged commit rather than as needing no action;
waiting: the join condition, every child with its label and state, which children block, and which cannot be resolved in this workspace;
failed: the failure, whether an operator continue still fits inside the retry budget, and the error.json breadcrumb of the last attempt;
paused, succeeded, and cancelled: the state and how to proceed.

For ready, running, and failed jobs, job why also folds the journal into one attempt-history line — N attempts across M activations at step 'X'; K after unclean exits — and, when a job under an unlimited retry budget has attempted well past a small threshold, flags it as flapping rather than progressing. A runner-allowlist refusal is reported whenever a live manager’s runner_modules or search paths cannot reach the job’s runner, so a repeating runner_unavailable claim loop is named rather than shown as a manager that “offers everything this job requires”. Any operator request still pending in requests/ready, and the reason recorded for the most recent retired one, are surfaced on the states where they apply.

The job side of every precondition comes from job.json and cannot drift. The other side — pools, capabilities, and served executors — is deployment policy of whichever manager is running and is read from the manifest each manager publishes, so a manager that is not running is reported as absent rather than assumed.

Reading results rather than status is collecting: httk workflow collect WORKSPACE streams CollectedJob summaries, while --raw exposes the JobRecord stream for a data layer; see Collecting results.

The foreground debug runner¶

httk workflow job debug WORKSPACE PAYLOAD --step relax
httk workflow job debug WORKSPACE JOB --follow-children

job debug drives exactly one job to a terminal state in the foreground and streams the attempt’s stdout.log and stderr.log to the console as they grow. Every transition is performed by a private task manager whose scans are restricted to that one job, so the debugged job runs through exactly the code paths a production manager uses and no unrelated work is claimed. Lines are prefixed with the step that produced them, and [debug] marks each transition the polling loop observed; job log always holds the complete record afterwards. --log-level raises the private manager’s own console log, which is quiet by default.

The first argument is either a payload directory, which is submitted fresh at --placement (debug by default), or a selector of a job that already exists. --step overrides the initial step of a fresh payload; overriding the step of a job that already has a history is refused, because rewriting history is what the recorded override_step request is for. --follow-children drives the children a waiting job spawned, depth first, and then resumes the parent.

The exit status is 0 when the job succeeded, 3 when it failed, and 4 when it stopped without finishing — paused, cancelled, or waiting for children without --follow-children. A live maintenance lock is refused up front, since it would stop every launch anyway.

Runner contract¶

The runner executes in the selected persistent or isolated workdir. It reads the context named by HTTK_WORKFLOW_CONTEXT and publishes outcome.tmp.<nonce>/ as outcome.ready/ beneath HTTK_WORKFLOW_CONTROL_DIR. See the Workflow filesystem API in detail for the complete protocol, and Native runner helpers in detail, Native Bash runner API, or the Python and Bash authoring parity table for the two authoring SDKs that implement it.

The local executor starts runners behind a one-byte launch gate. It records the process identity and commits the running marker before releasing that gate. If the manager disappears during this narrow launch interval, the gated process observes end-of-file and exits without executing the runner.

httk workflow manager run executes the normal path runner executor. Converted httk-v1 packages use that same path through their packaged v1 runner; select their taskset claim pool with the manager’s --pool option. See httk v1 task compatibility.