Project and workflow command line in detail¶
For operators and campaign owners: the whole command tree, including projects, configuration, signed manifests, remotes, and work that travels to them.
Installing httk-workflow registers lazy workflow, workspace, and job
commands with httk-core:
httk workflow --help
One command tree¶
httk workflow … remains the spelling for workflow execution and project
commands. Workspace and job management are top-level command trees:
httk workspace … and httk job …. Each group answers --help, each command
answers --help, and a mistyped action is reported by the group it was mistyped
in.
The complete tree¶
httk workspace init | list | default | move | forget | delete | status | managers | workflows | settings show | settings set | settings unset | workflow-prelude show | workflow-prelude set | workflow-prelude unset | policy show | policy set | fsck | gc | unlock | seal | unseal
httk workflow runner publish | describe
httk workflow build [--workspace WORKSPACE] TARGET...
httk job new | submit | request | delete | seal | unseal | list | show | log | why | debug
httk workflow list [--json]
httk workflow describe TARGET [--json]
httk workflow seal verify [PATH] [--json] [--trusted-key KEY] [--shallow]
httk workflow precheck [--workspace WORKSPACE] [--placement P] [--json]
httk workflow collect
httk workflow postprocess
httk workflow run [--workspace WORKSPACE] (the recommended spelling of `manager run`)
httk workflow manager run
httk workflow campaign init | show | submit | collect | start-"managers"
httk workflow v1 collect
httk workflow config show | set | unset | import-v1
httk init | identity (core-owned: per-user configuration and named operator identities)
httk project repair | manifest create | manifest verify | seal | unseal (httk-workflow mounts these beside core init | show | import-v1)
httk workflow remote list | add | configure | check | import-v1 | show | remove
httk workflow transfer [OPTIONS] SRC DST (plus the protocol spellings: receive | offer | retire)
Workspace selection¶
A WORKSPACE is an optional registered local name. When it is omitted, commands
resolve workspaces in this order: the closest enclosing workspace containing
.httk-workspace/format.json, the project’s recorded default, the registry’s
default, then an auto-created per-user default. Explicit local names are created
with workspace init PATH, and remote names use REMOTE:NAME at use time.
The registry is machine-owned: it stores only absolute local paths in
$XDG_CONFIG_HOME/httk/workspaces.json. When a command reaches a remote
workspace, the far side resolves the plain name in its own registry. The Python
API keeps Workspace(path) for library use; the registry is what the command
line speaks.
Remote-capable workspace commands use the adapter; this includes status,
settings, and job request. Most job commands remain local-only. Jobs are
created in the local default workspace, then transfer moves them to a remote
workspace for execution.
job request ACTION --workspace REMOTE:NAME JOB_ID ... asks the owning machine for unsigned envelopes,
signs those envelopes with the control-center identity selected by
--operator, and sends the signed documents back for verbatim publication.
An older far side that cannot parse the additive protocol vectors fails with
its own argparse error; upgrade httk-workflow on the remote. If both are supplied, make
--adapter-timeout longer than --timeout or the adapter may cut the session
off first. An adapter timeout during publication has an indeterminate outcome:
requests may already be published; retrying creates fresh request IDs, and
generation pinning makes such duplicates harmless because stale requests
retire. With prefix or tag selectors, the remote resolves the match; use full
job UUIDs when precise attribution matters.
monitor — inspect and control workspaces interactively¶
Command |
What it does |
Notable options |
|---|---|---|
|
open the curses workspace monitor |
|
workspace — the workspace itself, not its jobs¶
Command |
What it does |
Notable options |
|---|---|---|
|
create or adopt workspaces, registering each name (basename or |
|
|
list local or owning-machine workspaces |
|
|
read or record this project’s default name |
|
|
register copied workspaces on this machine under the names their project’s |
|
|
move a local workspace and update its registry path |
|
|
deregister names, leaving workspaces on disk |
|
|
destroy workspaces and deregister them |
|
|
summarize authoritative markers (remote: over the adapter) |
|
|
list managers serving workspaces, live or stale |
|
|
list the runners a workspace publishes, with each directory package’s workflow identity |
|
|
print application settings, or one selected key |
|
|
store one application setting in each workspace |
|
|
remove one application setting from each workspace |
|
|
print per-workflow preludes, or one selected workflow |
|
|
store one workflow’s prelude in each workspace |
|
|
remove one workflow’s prelude from each workspace |
|
|
print workspace policies |
|
|
store one policy member in each workspace |
|
|
check markers against journal frames; repair modes require names |
|
|
collect what retention policies allow (remote: over the adapter) |
|
|
release maintenance locks |
|
|
record every job’s seal digest under one signed workspace seal |
|
|
remove a workspace’s seal, refused while its project is sealed |
|
Sealing is described in full in Sealing jobs, workspaces, and projects. workspace status gains a
sealed line (and JSON field). workspace seal runs inside the maintenance
guard, so it requires a quiescent workspace; without --force it lists the
still-unsealed jobs and refuses rather than sealing a partial set.
workspace init creates and registers an explicit workspace. A canonical path may
have only one registered name:
httk workspace init --name my-workspace runs/my-workspace
A workspace on a cluster is addressed as REMOTE:NAME; its owning machine
chooses the path supplied to workspace init REMOTE:PATH. --setting KEY=VALUE
seeds an application setting at creation. workspace delete destroys the
workspace (locally, or on its remote over the adapter) and is refused without
--force; workspace forget only removes the name when there are no unretired
outbound transfers. Fetch or retire those first, or pass workspace forget --force to deregister the name anyway.
workspace move NAME DEST_DIR is an atomic same-filesystem rename. It refuses
cross-filesystem moves; stop managers, copy the tree manually, forget the old
name, and re-register it with workspace init --name NAME <newpath> instead.
build — foreground registration of compiled workflow packages¶
Command |
What it does |
Notable options |
|---|---|---|
|
build and register packages, store runners, or jobs’ workspace runners |
|
Directory rows are reported as tree (inferred) because the store format
identifies a tree by its run entry. New nested file publishes named run are
refused; a pre-existing store may still contain an ambiguous legacy layout, so
the marker is an honest inference rather than provenance metadata.
The build command has two forms:
httk workflow build [--workspace WORKSPACE] [--json] TARGET...
httk workflow build [--workspace WORKSPACE] [--json] --list
TARGET may be a workflow package directory, a workspace runner-store path,
or a job reference whose runner is a workspace package. A package directory is
published first, then its source tree is built and its artifacts are registered
for the local platform tag. A store path or job reference builds the already
published source tree. --list does no build and prints the workspace’s
registrations; --json emits machine-readable build or list records. Exit 0
means the registration completed (or the list was read); malformed targets,
probe/build failures, and missing artifacts are nonzero failures.
The build vocabulary and engine come from httk.core.building; this layer keeps
the workspace runner-build store and platform-tagged registrations. The
manager passes registered artifacts through HTTK_WORKFLOW_RUNNER_ARTIFACTS
without modifying the published source tree. A plugin-sourced workflow is first resolved and pinned into
the workspace like any other package, then built with the same command using
its job or store runner target.
job — making jobs, and finding out about them¶
Command |
What it does |
Notable options |
|---|---|---|
|
scaffold and submit jobs from a workflow, runner file, package directory, or command template |
|
|
submit prepared payload directories |
|
|
publish one request per job selector (remote: over the adapter) |
|
|
remove selected job payloads and state markers (remote: over the adapter) |
|
|
seal the payloads of selected quiescent jobs |
|
|
remove the seals of selected jobs, refused while the workspace is sealed |
|
|
list jobs as a cheap table (remote: over the adapter) |
|
|
describe jobs from their state (remote: over the adapter) |
|
|
print transition histories (remote: over the adapter) |
|
|
explain why jobs are not running (remote: over the adapter) |
|
|
drive one job to a terminal state in front of you |
|
When giving more than one JOB_ID, name the workspace explicitly.
job show gains a sealed line — yes with the signer roles, or no — and,
in --json, a sealed boolean plus seal_roles. job seal and job unseal
are the per-job half of Sealing jobs, workspaces, and projects; a job must be quiescent to be sealed.
An operator pause request against claimed, running, or committing is
deferred: the manager records it and pauses the job at the next attempt
boundary; a terminal outcome supersedes it. An older manager that does not
understand this in-flight pause request quarantines it as invalid.
JOB_ID is repeatable, so one command publishes one request per job. --wait
is valid only for pause and exits 0 only when each requested job was observed
paused at some point during the wait; jobs are confirmed individually, not
simultaneously. A concurrent operator may already have resumed an earlier job
when the command exits, and any pause (for example, a runner-declared step
pause), not specifically this command’s published request, may satisfy it.
Terminal, retired, quarantined, or timed-out requests exit 1. --timeout SECONDS requires --wait; timed-out requests remain published.
JOB is a job UUID, a tag--uuid job key, any unique prefix of either, or a
path inside the workspace. A job directory such as jobs/silicon--UUID names
one job; a directory such as jobs names every live job below it, including
nested placements; and a glob such as jobs/silicon* expands to matching
entries. Relative paths are resolved from the command’s current working
directory, and paths must remain inside the selected workspace.
Besides the per-state claim preconditions, job why also folds in, where they
apply: a runner-allowlist refusal when a live manager’s runner_modules or
search paths cannot reach the job’s runner (so a claim would fail with
runner_unavailable); an attempt-history line — N attempts across M activations at step 'X'; K after unclean exits — summarizing the journal; a
flapping flag when an unlimited-budget job has attempted well past a small
threshold without progressing; and any pending operator request still in
requests/ready, or the reason recorded for the most recent retired one.
Language documents use job new --from-runner DOCUMENT; see
Workflow-language runner realizations for PWD, CWL, jobflow, and httk-v1 details.
collect — the finished jobs, as summaries¶
Command |
What it does |
Notable options |
|---|---|---|
|
stream one collected summary per finished job |
|
With --into, a sealed id ledger keeps entry ids stable across rebuilds. It is
on by default at <into>.ids.sqlite; --id-ledger PATH relocates it and
--no-id-ledger disables it (ids then become unstable across rebuilds). See
Stable ids.
--degraded prints only the degraded per-job lines; the trailing summary still
counts the whole sweep, so a filtered listing never hides how many jobs ran. It
cannot be combined with --raw.
Every form except the pure-array --json ends with one
httk-workflow-collect-summary line counting collected, degraded,
unfulfilled_roles, storage_errors, and skipped_unreadable. The command
exits nonzero when any job was degraded, failed to store, or was skipped for an
unreadable job.json; unfulfilled roles alone keep the exit at 0. See
Collecting results for the triage members and --into partial-state semantics.
postprocess — run a curated script¶
Command |
What it does |
Notable options |
|---|---|---|
postprocess [OPTIONS] [JOB…] |
run one declared script for each selected collected job |
–workspace WS, –script NAME (required), –workflow-dir PKG, –state, –placement, –output-dir DIR, –timeout, –json |
httk workflow postprocess --workspace WS --script relaxation-report
httk workflow postprocess --workspace WS --script report --workflow-dir ./my-workflow --json
httk workflow postprocess --script relaxation-plot <job-id>
Output is written outside the job payload, under an output root that is
An optional JOB selector (UUID, key, unique prefix, or a workspace path, the same forms job seal and job delete accept) postprocesses exactly those quiescent jobs and cannot be combined with –state or –placement.
With –json, each result is one JSON object in the
httk-workflow-postprocess wire format, version 2, with workspace_id, job_id,
job_key, script, and either returncode plus output_dir, or an error. Without
–json, each result is tab-separated as
job_key
list — the workflows a name can select¶
Command |
What it does |
Notable options |
|---|---|---|
|
list the workflows |
|
Each text line is WORKFLOW_ID, alias (or -), source (registered or plugin OWNER), and summary (or -); --json reports the same as an array of objects.
A workflow reached only by explicit path — --workflow-dir DIR or --from-runner FILE — is not registered, so it is not listed here; use describe PATH to
report one such workflow directly. To list the runners one workspace has
published, rather than the workflows a name resolves to, use workspace workflows.
describe — inspect a workflow without publishing it¶
Command |
What it does |
Notable options |
|---|---|---|
|
describe a registered id/alias, runner file, or package directory |
|
Resolving a workflow trusts a directory package’s manifest and never executes
anything. describe is a report, so for a directory package it additionally
runs the runner entry’s --describe (with any surrounding attempt context
stripped) and prints a prominent WARNING: step drift line — and a
manifest_step_drift field in --json — when the manifest’s declared steps
disagree with what the runner reports. The drift is reported, not gated:
describe still exits 0.
Directory package authoring, manifest validation, publication, and hook trust tiers are documented in Workflow packages in detail.
Installed-plugin workflow names are included in the registered-workflow
listings — httk workflow list, and the --workflow help and unknown-workflow
hint entries, which carry [plugin PLUGIN_NAME] for their owner; describe
resolves a plugin name and reports its source as installed-package.
precheck — readiness before an attempt¶
httk workflow precheck --workspace WORKSPACE
httk workflow precheck --workspace WORKSPACE --json
httk workflow precheck --workspace WORKSPACE --runner-search-path PATH --runner-search-path OTHER
This read-only report checks submitted, ready, waiting, and paused jobs:
each declared environment entry is shown as resolved, default, or
unresolved, with its source and setting name, and each runner reference is
checked for availability and its pinned digest. --placement restricts the
scan. The authoritative environment gate remains at attempt start;
precheck is advisory and can become stale. Its HTTK_* environment layer is
the current process environment, which may differ on compute nodes; JSON also
carries this caveat once as environment_variable_caveat.
Use repeatable --runner-search-path options to check installed runner
references. A plain installed reference without one is reported as
indeterminate, not as a broken runner, and does not by itself produce exit
status 1.
Beyond the environment and runner reference, precheck measures each pending job against the live managers the workspace actually publishes:
claimability — a job no live manager can claim is a problem naming the closest manager’s unmet requirements exactly as
job whyrenders them (for examplelacks capabilities docker, ordoes not allow runner module …). Runner modules are validated against each manager’s realrunner_modulesallowlist, not a fixed default. When no manager is live at all, one workspace-levelmanager_noticereplaces per-job claim findings, and does not fail the run;language engine — a language job (the collect gate’s pair,
workflow_realization = languagewith aworkflow_language) has each module that language needs checked (without importing it) and names the pip extra to install, for examplepip install httk-workflow[jobflow]. Because the extras belong on the machine that runs the job, an absent module is only a problem when no live manager serves the job’s executor; when one does, the check isindeterminate(the serving manager’s environment may differ, verified only at run time) and does not fail the run;required inputs — a declared required input with a staged
destinationmust still be a member of the payload; a relocated or removed one is a problem.step — a job whose next step is not one of the runner’s recorded
runner_steps(written into the state frame after the runner’s first attempt) is a problem. This is frame-based only; the runner is never executed, so a job that has not recorded its steps yet is never faulted. The frame reflects the last attempt’s runner, so the check is advisory: a mutated payload runner may implement a different set by the next attempt.
The command exits 1 for an unresolved environment, a broken runner reference,
an unclaimable job, a missing-and-unserved language engine, a missing required
input, or a step outside the runner’s recorded set; the indeterminate cases
stay non-failing. The JSON summary carries claim_problems,
language_problems, language_indeterminate, input_problems, and
step_problems alongside the environment and runner counts.
manager — the process that runs the jobs¶
Command |
What it does |
Notable options |
|---|---|---|
|
run managers through the workspace launcher, or keep one serving with |
|
|
run managers through the workspace launcher, or invoke them on a remote workspace |
|
manager run follows the binding: a local workspace uses its manager.launch
setting (the built-in process launcher by default), while a remote workspace
invokes the same command on its owner. --count N means managers at that launch
site, and --workers N means workers each. --worker-resource NAME COUNT is
repeatable and advertises per-manager capacity to the scheduler; local
managers also use SLURM allocation variables when present. With a local
--count N, explicit --worker-resource pairs are passed to every manager
verbatim. Only auto-detected SLURM capacities are split across the N managers,
using quotient-plus-remainder distribution so their aggregate equals the
detected allocation.
Both manager commands run until idle by default; --idle keeps serving. The
top-level run takes --capability and --placement-prefix too, so the quickstart command can
claim a capability-gated job and scope its scan; without them a gated job would
stay unclaimable. Both print one startup banner and, on idle exit, one summary
line that names any jobs left not claimable by the pools, capabilities,
resources, or executors this manager serves, or left committing with an
unreadable definition. --inline forces one in-process manager, and --detach
returns after starting the managers.
The default manager log is the append-only workspace file
.httk-workspace/managers.log; each text line is prefixed with the manager id,
and JSON records carry manager_id. --log-file selects another destination.
The log is rotated when a manager starts or every 1000 records once the file
exceeds 16 MiB; one backup is kept, and a manager that has not yet reopened
the file keeps appending to that backup.
The workspace settings manager.launch, manager.count, manager.workers, and
manager.command control launch-site behavior: manager.launch selects the
built-in process launcher or a named launcher bundle; manager.count supplies
the default number of managers; manager.workers supplies each manager’s
concurrency; and manager.command is used after environment.prelude to find
the manager on the resulting PATH. --count and --workers override the
corresponding settings for the invocation. --inline forces one in-process
manager, while --detach returns after launch. --count always counts managers
at the launch site, not workers or remote adapters.
For a resource-aware run, advertise the manager’s complete allotment:
httk workflow run --workers 4 \
--worker-resource procs 32 --worker-resource mem 128000 \
--worker-resource matlab_license_slots 2
Pair this with the workflow manifest’s per-step resource requirements; see Task-manager usage in detail for the packing and dynamic-requirement example.
v1 — harvesting finished httk v1 trees¶
Command |
What it does |
Notable options |
|---|---|---|
|
harvest a pre-existing v1 result tree |
|
v1 collect ends with one httk-workflow-v1-collect-summary line reporting
finished, unfinished_by_status (tasks the name regex matched that were not
.finished, keyed by status), and skipped_no_rundir (finished tasks with no
dated run directory). Each collected report carries identity_stable: false
for a task whose identity is path-derived because it has no ht.manifest, and a
warning names how many such tasks a harvest saw.
config — the per-user machine configuration¶
Command |
What it does |
Notable options |
|---|---|---|
|
print the configuration, or one member |
|
|
store one member |
|
|
remove one member |
|
|
read a legacy |
Operator identity is no longer a config member. The per-user named identities
and default are set up and managed by the core-owned root commands httk init
and httk identity, documented with httk-core. config import-v1 imports a
legacy name and email as the first named identity when none is configured, and
imports the legacy public key into identity.json through httk-core while
recording only imported_from here.
project — the directory a campaign lives in¶
The project anchor and every project-level verb — init, show, import-v1,
export, repair, manifest create | verify, seal, unseal, and
verify-seal — belong to httk-core, which owns the whole httk project
command. httk-workflow no longer mounts project verbs of its own; instead it
registers the workspace as a project-member kind, so core’s verbs delegate
a workspace’s internals to it: what to leave out of a manifest, how to seal it,
its seal digest, how to verify it, and its health checks. A workspace inside a
project is recorded in httk_project/members.json — registered on
httk workspace init, unregistered on workspace delete / workspace forget,
and its path followed on workspace move. The registered name travels too:
members.json records each workspace’s name, so a project copied to another
machine stays usable. There, httk workspace adopt (per workspace) or
httk project adopt (every member at once) registers the copied workspaces in
the new machine’s per-user registry under their recorded names, joins any that
are missing from members.json, and records a name where one was absent —
idempotently, never touching a sealed project. Even before adopting, resolving a
workspace by a name the local registry does not know falls back to the enclosing
project’s members.json for that one invocation (it never silently writes the
registry). Create a project with httk project init PATH, then give it a
workspace with httk workspace init PATH; seal it with httk workspace seal
and then httk project seal, and verify the whole tree with
httk project verify-seal (or httk seal verify). The project verbs themselves
are documented with httk-core.
seal — verify a sealed tree¶
Writing seals lives beside each level (job seal, workspace seal, project seal); the top-level seal group carries the one verb that belongs to no
single level: verifying a whole sealed tree. Sealing jobs, workspaces, and projects is the full guide.
Command |
What it does |
Notable options |
|---|---|---|
|
verify the seal at |
|
Each line is <level> <subject> <verdict> <reason>, with indented <kind> <path> discrepancy lines beneath a failing entry, then a final status line whose
word and exit code mirror manifest verify: ok / exit 0 when every entry is
valid_trusted, UNTRUSTED / exit 3 when none is invalid but a signer is
untrusted, and FAILED / exit 1 on any invalid entry. By default the project’s
pinned keys and the local identities’ public keys are trusted, so a tree sealed
by its own project or identity verifies as valid_trusted without naming a key.
launcher — the bundles that start managers¶
A launcher is to starting workflow managers what a remote is to reaching a
machine. The built-in process launcher starts detached local processes; named
launchers are versioned bundles, resolved project-first and then globally.
Command |
What it does |
Notable options |
|---|---|---|
|
list manager launchers visible to this project |
|
|
create launchers from a packaged template |
|
|
update launcher settings |
|
|
describe launchers and their settings |
|
|
check a launcher’s required binaries |
|
|
remove launcher bundles |
remote — the adapters that reach other machines¶
Named after git remote: a remote is one machine this project can reach, and
the bundle of adapter operations that reaches it. The name local is reserved
for the built-in remote every workspace registry resolves as “this machine”, so
remote add local is refused — a workspace bound to local must be
unambiguous.
Command |
What it does |
Notable options |
|---|---|---|
|
list the remotes this project can reach |
|
|
create remotes from a packaged adapter template |
|
|
run adapters’ |
|
|
check that |
|
|
map legacy httk v1 computer bundles |
|
|
describe remotes and their settings |
|
|
remove remote bundles |
remote add --template accepts local (same-machine transport) or ssh
(rsync plus command execution over SSH). These templates describe how to reach
a machine; they do not describe how that machine starts managers. Configure
manager launch separately in the target workspace with manager.launch, such
as a packaged slurm launcher.
remote show never prints a credential value: a remote setting stored in
the manifest-excluded credentials.json is reported by name only, so a
description an operator pastes into a bug report cannot carry a password.
remote remove refuses while an unretired transfer still depends on the
remote, because removing it would leave that transfer with no way home;
--force skips the interactive confirmation and nothing else — the refusal
stands either way. Fetch or retire the transfer first.
transfer — moving jobs between two workspaces¶
transfer is one verb that takes two registered workspace names — a source and a
destination — and moves jobs between them, whichever way they point:
httk workflow transfer [--job JOB_ID …] [--state STATE …] [--placement P] \
[--destination-placement P] [--adapter-timeout SECONDS] [--json] SRC DST
Both names resolve through the registry, so which legs run over an adapter and which stay in this filesystem follows entirely from where the two are bound:
Direction |
What happens |
|
|---|---|---|
local → remote |
each named job is detached, its sealed bundle pushed to the remote, and imported there |
UUIDs, prefixes, paths, or globs; required |
remote → local |
the selected jobs are offered, pulled home, imported, and their sources retired |
canonical UUIDs only; optional sweep |
local → local |
each named job is detached from the source and imported into the destination directly, in this filesystem |
UUIDs, prefixes, paths, or globs; required |
remote → remote |
the client relays the selected offers through local staging and pushes them to the destination (v1; a direct source-to-destination path is deferred) |
canonical UUIDs only; optional sweep |
--state (repeatable, default succeeded and failed) chooses which finished
kinds a sweep moves, --placement restricts it to one subtree,
--destination-placement lands the jobs somewhere other than the placement they
had, and --adapter-timeout bounds every adapter operation the move runs.
For a local source, --job accepts a UUID, tag/key prefix, job directory,
placement directory, or glob such as jobs/silicon*; paths and globs are
resolved from the current working directory and must be inside that source
workspace. A remote source accepts only canonical job UUIDs, because its
selectors are resolved on the remote machine.
When --job is supplied, each named job must be eligible before any job is
sealed; by-id moves accept any quiescent state, while an explicit --state
remains an additional filter. With no --job, the sweep remains skip-tolerant
and defaults to succeeded and failed.
--strict-environment blocks before state moves when a checked destination
environment is unresolved or cannot be read. Transfer checks intentionally use
job overrides, destination settings, and declared defaults; they do not use the
client process environment as a destination substitute. A remote settings read
that is unavailable produces one immediate warning in non-strict mode.
For remote → remote, repeated --job values constrain the source offer before
the relay pulls anything; omitting them keeps the skip-tolerant terminal-state
sweep.
Bundles carry sources only for workflows that declare [workflow.build];
compiled artifacts are machine-local and are never transferred. After importing
such a bundle, run httk workflow build --workspace WORKSPACE TARGET on the destination
before starting its managers; the import operation repeats this reminder.
The protocol spellings, and what is gone¶
transfer also carries the frozen argument vectors one machine runs on another
over an adapter. Operator-facing vectors use workspace names; the hidden
--by-path spelling is the path-only protocol form used after the client probes
the owning machine. They are protocol rather than operator interface — a local→remote move invokes
receive on the destination, a fetch invokes offer then retire on the source.
Their spelling is frozen, because the machine that answers may run an httk older
or newer than yours:
httk workflow transfer receive --workspace PATH --bundle BUNDLE
httk workflow transfer offer --destination-workspace-id UUID [--job JOB_ID …] --json PATH
httk workflow transfer retire --destination-workspace-id UUID --json PATH JOB_ID …
httk job request-envelopes ACTION --workspace WORKSPACE --operator=LABEL --reason=TEXT [--priority N] [--step S] [--force] --json JOB_ID …
httk job publish-requests --workspace WORKSPACE --document JSON [--document JSON …] [--wait] [--timeout S] [--durable|--no-durable]
httk workspace status --by-path --json PATH
httk workflow manager run --by-path --workspace PATH
receive is an import half rather than an operator command, so it is not
advertised in --help, but it is its frozen, invocable spelling. The --by-path
switch is likewise hidden: it makes the workspace argument a literal path with no
registry lookup, which is exactly what one machine needs when it addresses another
machine’s workspace.
The pre-release transfer send and transfer fetch and
transfer status verbs are gone — they no longer parse. Move to the single
transfer SRC DST verb, and to manager run --workspace NAME for starting managers (below)
and workspace status NAME for reading a remote workspace’s markers.
Removed |
Now |
|---|---|
|
|
|
|
|
|
An earlier release also renamed two whole groups: httk workflow computer …
became httk workflow remote … (git’s word for the same idea), and
httk workflow tasks … (once httk workflow remote send|fetch|…) became today’s
httk workflow transfer. A job whose runner.path pins the old
pkg:httk.workflow.runners/vasp_* form breaks too: the packaged VASP runners are
now modules of httk.workflow.vasp.runners, and a job pinning the old path fails
with runner_unavailable naming the module it could not resolve — scaffold the
job again, or edit the one runner.path member.
campaign — partitioning a large run across many workspaces¶
Command |
What it does |
Notable options |
|---|---|---|
|
define the project’s partition map and assignment policy |
|
|
show the partition map |
|
|
assign one root job to a partition and submit it there |
|
|
collect every partition, one workspace after another |
|
|
start a manager per selected partition |
|
A campaign is a thin convention over the registered workspaces above: a
partition map, stored in the project, that spreads a very large body of work
across many workspaces without a new scheduler. Each partition names one
registered workspace, roots are assigned to partitions by policy, and spawned
children always inherit their parent’s workspace. See Campaigns.
campaign submit --workflow accepts a workflow name or alias only; use
job new --from-runner or job new --from-command for a file or command.
Creating jobs¶
job new scaffolds and submits jobs from a registered or packaged workflow name,
a runner file, a package directory, or a bare language document — and needs no
prepared payload:
httk job new --workspace WORKSPACE --workflow vasp-relax --input structure=POSCAR --tag silicon
httk job new --workspace WORKSPACE --workflow vasp-relax --input-from structure structures/ --parameter kpoint_density=30.0 --placement project/screening
httk job new --from-runner ./my_runner.py --step characterize --parameter sites=8
httk job new --from-command 'srun --ntasks=10 my_executable {input}' --file input=input_files/a.dat --tag a
--parameter NAME=VALUE supplies an opaque implementation knob;
--environment NAME=VALUE overrides one declared workflow environment entry;
and --format LANG selects the language of a bare document.
With --from-command TEMPLATE, shlex-style words are turned into a published
one-step Bash runner; each {name} placeholder must have a matching
--parameter NAME=VALUE or --file NAME=PATH. Parameter placeholders resolve
through httk_workflow_parameter at run time, while file placeholders resolve
to the staged file below HTTK_WORKFLOW_JOB_DIR. Each --file input is also
copied into the job workdir under its basename before the command runs, so it
can be read there by name as well; existing workdir entries are never replaced.
--files DIR expands every regular file directly inside DIR, sorted by
basename, into the same staging and placeholder behavior as --file for every
job-creation form (--workflow, --workflow-dir, --from-runner, and
--from-command). Symlinks to regular files are followed and staged;
subdirectories, symlinks to directories, broken symlinks, and other non-file
entries are skipped with one stderr warning naming up to five entries (plus
…); a DIR with no regular files errors with no regular files in DIR. For
example:
httk job new --from-command 'srun --ntasks=10 --cpus-per-task=1 vasp_std' --files inputs/si --tag si
The copy in the payload remains the immutable original, and {name} still
resolves to its absolute staged path. The generated runner can be edited and
passed back with --from-runner. The template is an argv-only word list: it has no shell syntax. A placeholder name must match
[A-Za-z_][A-Za-z0-9_.-]*; {{ and }} emit literal braces, and any other
brace text remains literal.
Runner identity is the digest of the rendered text, including the deterministic
sorted workdir staging lines, so staged file names are part of the wrapper
identity even when they are unused. A file name containing / is a valid
staging destination but must be staged under a bare name to use it as a
placeholder.
When a runner is supplied with --from-runner, job new runs it in describe
mode to infer its workflow and initial step; it must call
httk_workflow_runner WORKFLOW STEP... before any work.
--input-from NAME SOURCE... loads a file or the readable files in a directory, realizes the
declared payload destination, and creates one job per file for a batch. A
directory file with no registered reader whose name is a structure convention
(POSCAR*, *.vasp) is read as POSCAR; any remaining unreadable files are
skipped, and one stderr line names them:
httk workflow: skipped N of M files in DIR (no registered reader): …. After a
batch, one final stderr line reports submitted N jobs; if a batch fails partway
it instead reports submitted N of M jobs before failing and exits 2. In a
batch, --tag becomes a prefix combined with each item’s derived tag
(run7-si2o) rather than replacing it; for a single job --tag is the whole
tag. --file NAME=PATH stages anything else, --input NAME=PATH stages one declared input,
and the command prints one
tab-separated job_key<TAB>payload line per job, or --json reports. Any
preparation warning a language raises (for example a CWL DockerRequirement) is
printed once as httk workflow: warning: … on stderr. The runner
file is published into the workspace runner store and pinned by digest unless
--publish installed names a packaged runner where it is installed. See
Quickstart.
Running language documents¶
Run a PWD, CWL, or jobflow document directly with job new --from-runner DOCUMENT;
the document is resolved as a language realization:
httk job new --workspace WS --from-runner flow.cwl --input message=echo
httk job new --workspace WS --from-runner workflow.json --parameter pwd_module_path='["."]'
httk job new --workspace WS --from-runner maker.json
The same --format option accepts cwl, pwd, jobflow, and httk-v1 for
bare document inputs. Manifest packages and registered ids reject the option
because their language is already known.
See Workflow-language runner realizations for package manifests, bare-document rules, the supported CWL subset, PWD security, jobflow Makers, and language collection.
Harvest old v1 results without submitting them:
httk workflow v1 collect --workflow-dir PKG ROOT
httk workflow v1 collect --workflow-dir PKG --into results.sqlite --id-base httk.v1 ROOT
Inspecting and debugging jobs¶
job list, job show, job log, and job why read one workspace without
writing anything, and job debug drives a single job to a terminal state in the
foreground:
httk job list --workspace WORKSPACE --kind ready
httk job show --workspace WORKSPACE JOB
httk job log --workspace WORKSPACE --limit 20 JOB
httk job why --workspace WORKSPACE JOB
httk job debug --workspace WORKSPACE --follow-children PAYLOAD_OR_JOB
JOB is a job UUID, a tag--uuid job key, any unique prefix of either, or a
path inside the workspace (for example jobs or jobs/silicon*), and
each command takes --json. job debug exits 0 on success, 3 on failure, and
4 when the job stopped without finishing. See
Task-manager usage in detail for what each command reports.
job list --json returns format, format_version, jobs, and next_after.
Each row has the existing job_key, job_id, state, step, placement,
priority, generation, and reason members. Use --limit N and pass the
returned cursor to --after to page a large workspace; --tag-contains TEXT
filters on the tag portion of the job key. --counts adds a counts object
with name-based marker totals by selected state kind; malformed
marker-shaped names are included, and fsck reports them. Counts require a
full directory walk and are not free. The human table without --limit
materializes all rows. The cursor format is
<kind>:<placement>/<job_key> and next_after is null when the page is
exhausted. Paging is weakly consistent during concurrent transitions: a job
can appear twice or be missed if it changes kind between pages, so clients
should deduplicate by job_id. Remote job show, job log, and job why
accept canonical lowercase job UUIDs only; keys and prefixes must be resolved
locally first.
httk workflow collect --workspace WORKSPACE streams one CollectedJob summary per finished
job as JSON lines by default. Use --raw to stream JobRecord records for a
data layer; see Collecting results.
Configuration and projects¶
User configuration follows the XDG base-directory convention, and everything per-user this package keeps is configuration:
$XDG_CONFIG_HOME/httk/config.json(machine-level settings such asmachine_names);operator identity in
$XDG_CONFIG_HOME/httk/identity.json, managed by httk-core;identity keys in
$XDG_CONFIG_HOME/httk/keys/;global remote definitions in
$XDG_CONFIG_HOME/httk/remotes/.
HTTK_CONFIG_HOME and HTTK_DATA_HOME can provide explicit deployment or test
overrides. Legacy ~/.httk data is read only through config import-v1; its
64-byte private material is not converted.
httk init --name "A User" --email user@example.org
httk workflow config set machine_names "node-a,node-b"
httk workflow config unset machine_names
httk project init --name example .
The project anchor is owned by httk-core, which provides the umbrella
httk project command and its init, show, and import-v1 leaves. Create the
anchor with httk project init PATH and give it a workspace with
httk workspace init PATH. The workflow-aware verbs — repair, manifest,
seal, and unseal — are mounted by httk-workflow onto the core
httk project command.
config set accepts only the keys the configuration actually has — machine_names
is the sole settable one — and names them when it refuses another, so a typo cannot
become a member that nothing ever reads. Operator name, email, and named identities
are not configuration keys: they live in identity.json and are managed by the
core-owned httk identity commands and httk init. format and format_version describe
the document and are written by httk itself. A configuration whose format or
format_version is missing or something else is refused rather than read as if
its members meant what httk means by them.
A project has httk_project/project.json and a standard 32-byte Ed25519 seed
stored with mode 0600. Its default workspace for workflows is recorded by name and
may live outside the project; commands discover the nearest project in the
working directory’s parent chain.
Describing and checking a project¶
httk project show
httk project show --json
httk project repair --dry-run
httk project repair .
httk project show (core) reports the project’s metadata and keys. project repair applies the safe repairs by default and adopts every member workspace on
this machine — the exact one-shot a freshly copied tree needs; --dry-run
reports without touching anything, and --no-adopt repairs without adopting.
httk workspace status and httk project manifest verify report the live
workspace and manifest state directly.
project repair covers the conditions that quietly break a project later — a
stale maintenance lock, staging leftovers, a workspace on disk missing from the
registry, and a member not yet adopted on this machine — reporting or fixing
each.
--repair fixes the ones that can be fixed automatically, says exactly what it
did, and journals it in the project’s workspace, so the repair is part of that
workspace’s durable history. The command exits 1 only when a check is actually
broken; a warning, such as a project that has no manifest yet, is something to
know about rather than something to fail a script on.
Signed manifests¶
httk project manifest create .
httk project manifest verify
httk project manifest verify --trusted-key keys/collaborator.pub
The httk₂ manifest is deterministic canonical JSON-lines compressed with bzip2.
It records sorted POSIX paths, regular-file sizes and SHA-256 hashes, empty
directories, and symlink targets. Special files are rejected. A
domain-separated body digest is signed with Ed25519. Creation fences manager
launches only when a workspace is co-located with the project, and refuses
active work there. A detached project needs no workspace to create its
manifest. Verification also recognizes the legacy
ht.project/manifest.bz2 format without changing it.
When a directory contains a regular job.json that parses as an
httk-workflow-job whose UUID matches the directory’s valid job key, it is a
job payload and its direct attempts, logs, and .httk-job children are
skipped during both manifest creation and verification; same-named directories
elsewhere are ordinary project content.
What a verified manifest actually proves¶
Be precise about the threat this addresses, because the signing key lives in the
tree it signs. httk_project/keys/project.seed is a file of the project, mode
0600 and excluded from the manifest, but excluded is not absent: anybody who
can write the project directory can re-sign it. A manifest therefore proves
that the tree is exactly the tree somebody with the seed described — it does not
prove that nobody changed the tree, and it is not a tamper seal against an
attacker who had write access.
What it does prove is worth having. Against accidental damage — a truncated
copy, a partial rsync, bit rot on an archive volume, a stray edit in a
directory nobody meant to touch — the digests are exact. Across a copy that
travelled without the seed, and against a replaced tree signed by a different
key, the signature is the check that catches it. That is why verification
compares the signing key with a trust anchor that did not come from the
manifest: the key pinned in project.json at httk project init,
plus any key named with --trusted-key. Reading the key out of the manifest
header and checking the manifest against itself would always say valid.
Verification is therefore three-way, not a boolean:
Verdict |
Exit |
Meaning |
|---|---|---|
|
|
the manifest describes this tree and a pinned key signed it |
|
|
the manifest describes this tree, but nothing here pins the key that signed it |
|
|
the tree does not match the manifest, the signature does not verify, or the manifest names another project |
invalid also covers a manifest whose project_id disagrees with
project.json: a manifest of a different project dropped into this tree is
refused by name however well it verifies internally.
Pinning and adopting keys¶
A project created by httk project init pins its own key at creation, so its
manifests verify as valid_trusted immediately. A project made before pinning
existed has no public_key in project.json, so every manifest of it verifies
as valid_unknown_key until somebody decides which key to trust. That decision
is explicit, because it is the whole trust model in one act:
from httk.workflow.projects import pin_project_key, trust_project_key
pin_project_key("/path/to/project") # adopt keys/project.pub
trust_project_key("/path/to/project", "ed25519:…") # adopt somebody else's key
pin_project_key adopts the key that is in the tree right now — do it only on
a tree you have reason to believe is the one you left. trust_project_key adds
a further anchor to project.json’s trusted_keys; httk project import-v1 fills
that list with the legacy identities of an imported httk v1 project, so its
old ht.project/manifest.bz2 verifies as trusted too. --trusted-key accepts
either an ed25519:BASE64 value or the path of a *.pub file and is the
one-off equivalent that writes nothing.
For attribution between machines — who published this request, who imported this transfer — see the operator identity key below, which is a different key with a different job.
Operator identity¶
Operator identity — the recorded name, email, default, and named identities —
lives in $XDG_CONFIG_HOME/httk/identity.json, managed by httk-core. The
core-owned root command httk init derives a short name from the email local
part, creates the first named identity and its
identity-SHORT.seed/.pub pair below $XDG_CONFIG_HOME/httk/keys/, and makes
it the default. Named identities can also be managed with
httk identity add --name NAME --email EMAIL SHORT; httk identity default SHORT changes the default. Removing an identity leaves its key files on disk;
removing the default with exactly one identity remaining selects that identity
automatically, while removal with multiple remaining identities requires
selecting another default first.
The default signing identity resolves in this order: the default_identity
recorded in identity.json, or the only configured identity when there is
exactly one. With no configured identity, signing is unsigned; multiple
identities without a resolvable default are refused. A selector containing <
is a literal Name <email> attribution label (the name may be empty) and uses
the resolved default identity’s key; other selectors must be configured short
names.
job request records the selected identity’s Name <email> label and signs
the request with that identity’s key. Omitting --operator selects the
configured default; a short name selects that identity, while a literal label
is passed through and signed with the default identity’s key.
For remote requests, the control-center identity builds the signature locally
after the far side returns unsigned envelopes. The remote publishes those
signed documents verbatim, so attribution and signer are the same identity.
If a configured identity’s key file is missing or unreadable, the request fails
loudly; remove and re-add it with httk identity remove SHORT
then httk identity add SHORT ..., or restore the key file.
The two request protocol spellings are additive; when a remote cannot parse
them, upgrade httk-workflow on the remote.
The selected key signs operator requests (httk job request …).
Transfer acknowledgements always use the DEFAULT identity resolution; they do
not carry a per-request identity selection. Signatures are detached, cover the
canonical JSON of the whole document, and are domain-separated from every
other httk signature.
Document signing remains optional for lower-level callers, and a manager or a
transfer source accepts unsigned documents exactly as before. A signature that
is present must verify: a request with a broken signature is quarantined with
the reason, and an acknowledgement with a broken signature will not retire a
sealed bundle. A verified request records its operator_key in the journalled
state frame beside the operator name and reason.
The semantics are attribution, not authorization. The key says which identity published this document; it grants nothing, and no operation is permitted because a document is signed. Anyone who can write the workspace’s request directory can still publish an unsigned request.
The fence is .httk-workspace/maintenance.lock, holding the recording process
identifier, hostname, and creation time. A lock whose same-host process is gone,
whose content is unreadable, or that is older than twenty-four hours is
reclaimed automatically; any other lock is reported with its holder. Operators
can also clear one explicitly:
httk workspace unlock WORKSPACE
httk workspace unlock --force WORKSPACE
Without --force only a stale lock is removed.
Application settings¶
A workspace also carries application settings: a flat, dotted-name map of small
values a runner resolves when it runs — the VASP command, a pseudopotential
library — distinct from the engine policy above, which tunes scheduling. They
are stored in the workspace and edited by name:
httk workspace settings set --key vasp.command --value "srun -n 32 vasp_std" my-workspace
httk workspace settings show my-workspace
httk workspace settings unset --key vasp.command my-workspace
A value that parses as JSON is stored as that scalar; a bare word is stored as a
string. Settings can also be seeded at creation: workspace init --setting KEY=VALUE sets them explicitly, and a workspace bound to a remote is additionally
seeded from that remote definition’s whitelisted remote settings, so a cluster’s
vasp_command becomes the new workspace’s vasp.command without anyone restating
it.
A runner reads a setting through a.setting("vasp.command"), and the value is
resolved in layers, most specific first: the job’s own parameters, a real
HTTK_VASP_COMMAND deployment override, the workspace setting, then the runner’s
default. The manager exports scalar workspace settings into each attempt
environment (vasp.command becomes HTTK_VASP_COMMAND) and snapshots them into
the HTTK_WORKFLOW_CONTEXT JSON value, so a runner sees the values the workspace held when its job was
claimed. See Packaged VASP runners and Python and Bash authoring parity.
Workspace settings are non-secret configuration: they are snapshotted into the
attempt context and exported into the runner environment, so credentials must
not be stored there; remote credentials already live elsewhere.
Workflow preludes¶
Two layers of shell setup run before a job’s runner, both sourced under set -e
so a failing line aborts the job rather than running the calculation in a broken
environment:
environment.prelude— the workspace-wide layer, one shell fragment that applies to every job. It is an ordinary application setting:workspace settings set --key environment.prelude --value "…" NAME.workflow-prelude— the per-workflow layer, keyed by workflow id (the[workflow].idof the manifest,=the job’sworkflow). It applies only to jobs of that workflow and runs after the workspace-wide prelude:httk workspace workflow-prelude set --workflow relax-vasp --value "module load VASP/6.2.1" my-workspace httk workspace workflow-prelude set --workflow relax-vasp --value @prelude.sh my-workspace httk workspace workflow-prelude show my-workspace httk workspace workflow-prelude unset --workflow relax-vasp my-workspace
VALUEis stored verbatim (never JSON-parsed);@FILEreads the shell text from a file, for a multi-line module-load script kept on disk. Without--json,showis line-oriented (WORKFLOW⇥text), so a multi-line prelude’s continuation lines carry no id prefix — machine consumers should use--json.
See Task-manager usage for how each layer is delivered by the workspace’s launcher, and why preludes stay behind when a job is transferred.
Workspace policy and integrity¶
The tunables a workspace shares with every process attaching it — the
visibility deadline, the default lease, the journal segment size, and the
retention limits — are stored in format.json and edited in place:
httk workspace policy show WORKSPACE
httk workspace policy show --json WORKSPACE
httk workspace policy set --key visibility_deadline_seconds --value 60 WORKSPACE
httk workspace policy set --key retention.trash_days --value 14 WORKSPACE
workspace fsck verifies that every state marker still resolves to a readable
journal frame that agrees with it, and can re-point damaged markers at the last
good frame of their job:
httk workspace fsck WORKSPACE
httk workspace fsck --repair --json WORKSPACE
httk workspace fsck --repair --quarantine-unrepairable WORKSPACE
It exits 1 while anything remains for an operator to deal with. See
the task-manager guide for what each problem code means and
for exactly what a repair will and will not touch. When always-safe leftovers
exist, it also prints their total and per-category counts; this informational
line never changes the exit status.
Freeing disk¶
A committing manager removes an attempt’s control directory after its durable
commit and after it has reaped the local process when the actual destination is
ready, waiting, paused, or succeeded; failed and cancelled attempts stay
as evidence. Transaction trash is normally removed with that control tree. An
inherited commit is left for workspace gc. Managers run the always-safe
categories at startup and the full policy-gated collection at clean exit;
artefacts orphaned by a crash still require explicit workspace gc, driven by
the workspace’s policy.retention:
httk workspace gc --dry-run WORKSPACE
httk workspace gc WORKSPACE
httk workspace gc --json WORKSPACE
It prints one row per category with the candidates it found, what it removed,
and an estimate of the bytes reclaimed; --json lists every individual entry
as well. --dry-run touches nothing at all and reports what a real run would
remove. A run that removed anything also appends one httk-workflow-gc frame
to the journal summarizing the same counts, so the collection is itself part of
the workspace’s durable history.
A null or "keep" retention member means keep. On a fresh workspace,
journal_days and trash_days default to one day while
attempt_control_days stays unlimited. The command always prunes what cannot
carry information, plus removable markers that are quiescent and unowned by any
manager (succeeded, failed, cancelled, submitted, and ready) whose
payloads the operator removed;
configure the limits to change the policy-gated categories:
httk workspace policy set --key retention.attempt_control_days --value 14 WORKSPACE
httk workspace policy set --key retention.trash_days --value 14 WORKSPACE
httk workspace policy set --key retention.journal_days --value 90 WORKSPACE
httk workspace policy set --key retention.journal_days --value null WORKSPACE # keep forever
Category |
Retention limit |
What goes |
|---|---|---|
|
|
aged |
|
|
trees a replayed transaction moved aside, once the job left |
|
|
acknowledged transfer bundles below |
|
|
per-transfer receipts below |
|
always safe |
state markers for jobs that are quiescent and unowned by any manager ( |
|
|
segments outside every current non-terminal frame chain (and outside terminal current segments), written by a writer no live manager owns |
|
|
directories of dead managers whose segments are gone |
|
always safe |
empty placement mirrors below |
|
always safe |
staging entries older than 24 hours |
|
always safe |
requests claimed over 30 days ago by a manager now gone, and requests retired over 30 days ago with their |
To remove a finished (succeeded, failed, or cancelled), submitted, or
ready job cleanly, use httk job delete JOB.... It removes both the payload
directory and state marker, confirms on a terminal, and protects join children
referenced by non-terminal parents unless --force is given; --force skips
both confirmation and the join-parent guard. The manual rm -r plus GC or
manager cleanup remains fine for finished jobs. For queued ready or
submitted jobs, prefer job delete, because removing the directory first
can race a manager claiming the job at that instant. A job in any other state
must be cancelled first with job request cancel.
Containment checks are static: a concurrent replacement of a placement
ancestor by the workspace’s own owner is outside the threat model, as it is
for attempts/ and logs/. The hidden --confirmed option is a protocol flag
like --by-path, not part of the user interface.
The collector never touches the quarantine, a sealed transfer bundle, a
persistent workdir, a payload beyond its aged attempt-control directories, a
claimed, running, committing, cancelling, waiting, or paused marker,
a segment in a frame chain protected by a current
non-terminal marker, a manager that is still heartbeating, or the runner store.
Removal is bottom-up and rewrites no state, so a collection killed halfway
leaves the workspace exactly as consistent as it was, and running it again
simply finishes the job.
Collecting journal segments has one honest cost. Every segment in a current
non-terminal marker’s frame chain is protected, while a terminal marker protects
only its current segment. The deep history of a terminal job therefore goes
with aged segments behind it; collect and job log report that timeline with
gaps set, while a non-terminal job’s reachable chain remains intact.
Remote adapters¶
Remote definitions are versioned directories containing remote.json and one
executable adapter. The operation name travels in each versioned JSON request;
the dispatcher prints one JSON result and sends diagnostics to stderr. Commands
and remote commands are always argument arrays. The maintained templates implement that
protocol through httk.workflow.adapter_protocol, which is the public
name of the packaged implementation. Writing a remote adapter in detail is the reference
for writing one of your own: the bundle layout, the exact request and result
document of the six operations (configure, install, invoke, push, pull,
and status), and the rules for a custom adapter.
Maintained local and ssh templates are packaged with
the module. Project definitions shadow global definitions. REMOTE:NAME names
a workspace on a remote. remote import-v1 maps recognized legacy httk v1 computer bundles
by reading assignment-only configuration; legacy shell executables are never
copied or run. Any other kind in a remote.json is refused rather than
executed in the wrong place.
remote import-v1 selects ssh when the legacy configuration contains
REMOTE_HOST, otherwise local. It maps the first legacy submission profile
into legacy_settings; if several config.* profiles exist, the others are
skipped with a warning because submission profiles are now workspace settings.
Legacy SLURM_* values are not remote settings: add a slurm launcher and set
the target workspace’s manager.launch instead. The legacy executables are
never copied or executed. Review the imported legacy_settings and initialize
the workspace path explicitly before transferring jobs.
remote configure --set KEY=VALUE persists only the machine-level keys
check_connectivity, host, httk_command, legacy_settings,
port, username, vasp_command, and vasp_pseudo_library
in the shareable remote.json. Scheduler profile values are workspace
settings: use slurm.account, slurm.partition, slurm.time_limit,
slurm.nodes, slurm.cpus_per_task, slurm.ntasks,
slurm.ntasks_per_node, slurm.mem, slurm.gres, slurm.reservation, and
manager.workers; those scheduler names are refused as remote settings. Other
non-persistable keys are stored in
the remote’s credentials.json with mode 0600 beside it, which project
manifests exclude. Adapters receive both together as the request’s
remote_settings. remote show NAME reports which
file each setting came from, and the name — never the value — of every
credential.
What each kind does¶
local copies files in this filesystem and runs commands as child processes.
Manager launch policy is selected by each workspace’s manager.launch setting.
ssh moves files with rsync over ssh and runs every command on the
configured host. Only ssh and rsync are required locally. Both kinds
implement the same six operations:
Operation |
|
|
|---|---|---|
|
validates pending machine settings |
verifies the host answers with a cheap remote |
|
checks that local |
checks that |
|
runs the argument vector as a child process |
runs it on the configured host and returns status, stdout, and stderr |
|
copies the requested tree or relative file batch locally |
transfers it with |
|
runs the workspace status command locally |
runs |
httk_command overrides how httk is spelled on the far side, for example
httk_command="/proj/venv/bin/httk"; without it the plain httk on the remote
PATH is used, and locally a python3 -m httk.core.cli fallback applies.
Quoting¶
Every subprocess an adapter starts is an argument vector, so no shell ever
parses a value that came from a request or from settings. ssh is the one
exception in the protocol, because it always joins its command words and lets a
login shell on the far side parse the result. All remote command strings are
therefore built by a single helper that quotes element-wise. Manager launcher
bundles own any scheduler-script quoting separately. rsync transfers pass
--protect-args so that even file names travel in the protocol rather than
through the remote shell.
httk on the target: remote check¶
httk is never installed on a remote for you: setting up software on an HPC
account is yours to do, because every cluster does it differently (modules,
venvs, conda, pipx, …). The contract is simply that the connection the
adapter opens — a non-interactive shell — can run httk, with the
httk-workflow package installed beside the core.
remote check verifies exactly that, and running it once after configuring a
new remote is recommended: it confirms the host answers, that httk is found
(also trying python3 -m httk.core.cli), that the workflow command group
exists, and reports the command and version it found. --version alone would
only prove httk-core.
When the check fails, log in on the remote and make sure httk₂ is set up and
available there — for example with pipx install httk-workflow — and note
that it must be reachable from a non-interactive shell: a module load or
conda activation guarded by an interactivity test in .bashrc works when you
log in but not over the adapter’s connection. If httk deliberately lives
elsewhere (a project venv, a wrapper script), point the remote at it with
remote configure --set httk_command="/proj/venv/bin/httk" REMOTE instead.
In the adapter protocol this operation keeps its historical spelling
install; the earlier bootstrap=pip opt-in that attempted a
pip install --user is retired.
Detached transfers¶
A transfer fences an explicit quiescent marker, seals it in the payload,
validates the payload digest at import, publishes the preserved UUID and prior
state only at the destination, and retires the source only after an
idempotent acknowledgement. Transfer UUID and digest checks suppress retries;
sealed and retired bundles are retained for recovery. Repeating the same
transfer SRC DST resumes the matching sealed transfer, including the
copy-before-import and lost-acknowledgement boundaries.
The sealed payload digest pins every path, every file’s content and executable
bit, and the literal target of every symlink, so a runner that arrives without
its executable bit, or a link retargeted in transit, is a detected mismatch
rather than a silent corruption. A symlink is carried as its target string and
must stay inside the payload: an absolute target, or a relative one climbing out
with .., is refused by name, because it would mean something else at the
destination.
Running on a remote and fetching the results¶
Add and configure the machine, make sure httk-workflow is installed there
(log in and set it up, e.g. pipx install httk-workflow), verify with
remote check, create its workspace, then send and run a job:
httk workflow remote add --template ssh kappa
httk workflow remote configure \
--set host=kappa.example.org --set username=rar \
--set check_connectivity=yes kappa
httk workflow remote check kappa
httk workspace init kappa:/scratch/rar/httk/runs
httk workspace settings set --key slurm.partition --value batch kappa:runs
httk workspace settings set --key vasp.command --value "srun -n 32 vasp_std" kappa:runs
httk job new --workflow vasp-relax --input structure=POSCAR --tag silicon
httk workflow transfer --job JOB-ID default kappa:runs
httk workflow run --workspace kappa:runs --workers 8
httk workspace status kappa:runs
The machine that owns a workspace chooses its path; scheduler settings belong to the workspace instead. Remote init sends the path and registers its basename on the owning machine.
transfer default kappa:runs detaches each selected job from the local default
workspace and imports it on the remote, at the placement it had here unless
--destination-placement puts it elsewhere. run --workspace kappa:runs submits the
manager invocation via the remote adapter; the remote invokes
httk workflow manager run --workspace runs --detach on the owning machine.
That command uses the workspace’s manager.launch, just as a command run on the
cluster or through a machine_names alias would. --count means managers at
that launch site, and --workers fixes the worker count of each manager.
manager run is the advanced spelling for the same operation.
Before a transfer moves state, it checks each job’s declared environment against
the destination workspace settings. Unresolved default-less entries produce a
warning; --strict-environment blocks the transfer before detaching. Remote
settings are read through the adapter when reachable. If that read cannot be
completed, one warning says the environment could not be prechecked remotely;
strict mode treats that as a block. The client process environment is not used
as a substitute for destination settings.
To bring stopped jobs home, use the reverse transfer and then collect:
httk workflow transfer --state succeeded --state failed --placement project/screening --json \
kappa:runs default
--state accepts the kinds a stopped job can be in and defaults to succeeded
and failed; --placement restricts the fetch to one subtree; --adapter-timeout
bounds every adapter operation the fetch runs. With --job, any quiescent state
is eligible unless an explicit --state filters it. A fetched job arrives as an
ordinary job of the local default workspace, in its offered state and at the
placement it had on the remote, so httk workflow collect then reports terminal
results exactly like jobs that ran at home.
Under the fetch leg run the two far-side protocol commands, invoked over the adapter but usable on their own on the remote itself. They use literal paths because they bypass the owning machine’s registry:
httk workflow transfer offer --destination-workspace-id UUID [--job JOB_ID …] --json PATH
httk workflow transfer retire --destination-workspace-id UUID PATH JOB_ID ...
offer detaches every selected job into its sealed bundle and prints one entry
per bundle; it requires --destination-workspace-id, because a bundle is sealed
for exactly one destination. --job is repeatable, accepts any quiescent state
when no --state is supplied, and fails all-or-nothing if an id is missing or
filtered. retire moves the sealed source of an already
imported job under .httk-workspace/transfers/retired/ — a rename, never a
delete, so a source is only ever whole or moved whole; its
--destination-workspace-id is optional and, when given, refuses a bundle that
was sealed for somebody else. offer narrows what it seals with the same
--state and --placement fetch passes through; both print their report as
JSON with --json and as tab-separated lines otherwise. A client that sends
--job requires a new far side: an older remote rejects the additive flag with
its argparse error, which the client relays.
Every step is idempotent and the whole pipeline is resumable: offer reports an
already sealed bundle from its ledger instead of sealing it again, a pull onto
a matching staged bundle is a no-op, import returns the acknowledgement it
already wrote, and a retired source is never offered again. An interrupted fetch
is finished by running the same command again, and a fetch that has nothing to
collect does nothing.
Because fetch reads these two commands’ answers back over the adapter’s
invoke, their standard output has to be nothing but the JSON document: a login
banner or a profile’s greeting printed on the far side’s stdout makes the fetch
stop with remote offer did not return a transfer offer document before
anything is pulled or imported. Put such greetings on stderr, or behind a
non-interactive-shell test, on any host a remote adapter reaches.