Workflow filesystem API

For implementers of this protocol, and for anyone who needs to know exactly what a workspace on disk means.

Status and scope

This document specifies the filesystem protocol around which the httk-workflow engine is built. It is the normative on-disk protocol rather than Python API documentation. The current httk.workflow implementation writes and serves the core-v2 profile and supports the transactional-data-v1 and detached-transfer-v1 extensions; relocation and cross-workspace children remain reserved optional extensions and are rejected rather than partially executed.

core-v2 is core-v1 plus four on-disk refinements: every spawn.json entry carries a mandatory unique label; a join summary records typed per-child observations which the next activation also reads as children in its attempt context; runner in job.json may name a shared runner outside the payload through source and sha256; and a runner-declared failure marked retryable is retried within the job’s existing attempt budgets. A core-v1 workspace remains readable, so it can be inspected and exported, but is never mutated in place: the sections below describe core-v2.

The protocol is language independent. A workflow step may be a shell script, a Python program, a compiled executable, or any other program that can read and write files and atomically rename a file or directory.

The design descends from the ht.task.* directories, ht_steps, ht.nextstep, subtasks, and ht.atomic.* replay mechanism in httk v1. It keeps the central property of that design: neither a workflow step nor a task manager is ever required to run cleanup code. Either may disappear between any two instructions. A later task manager must be able to identify the last commit point and continue from it.

The design target is workspaces larger than the measured local snapshot in Scale benchmarks; that target is not a capacity measurement. Metadata inode count, directory fan-out, scheduler scan cost, and manual filesystem inspection are therefore correctness-level design concerns, not later optimizations.

The protocol covers:

• submission and execution of dynamic, multi-step jobs;

• safe competition between any number of task managers;

• automatic and manual continuation;

• dynamic fan-out into child jobs and later joins;

• optional transactional contributions to durable job data;

• explicit, unexpected, and dependency failures;

• durable history and discovery of failed jobs.

It does not promise transactional semantics for effects outside the workflow filesystem. Sending mail, submitting to a second queue, or changing a remote database must be made idempotent in that external system, for example by using the httk job and activation IDs as idempotency keys.

Normative language

The words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are used in their usual specification sense.

Protocol JSON is UTF-8. Unknown object members MUST be ignored when reading a compatible major format version.

Design summary

The core representation is deliberately small:

1. A job has one payload directory containing one required metadata file, job.json. Its parent path is an arbitrary, user-chosen placement path.

2. A job has exactly one small marker file in the global state/ tree.

3. The marker’s location is the sole authority for the job’s current state. It is atomically renamed between submitted, ready, claimed, running, committing, waiting, and terminal state directories.

4. Transition details and history are packed into shared append-only journal segments. There is no state-record file, event directory, failure file, or revision directory per job.

5. An active attempt temporarily adds one small control directory. Application execution may use either one persistent run/ workdir or an isolated run.<attempt-id>/.

6. data/ and replayable transactions are optional. Jobs that opt in receive all-or-none publication at attempt boundaries; jobs such as large VASP runs may instead keep all mutable state directly in persistent run/.

The expected steady-state metadata cost for a job with no retained application data is:

Object	Per job	Lifetime
Job payload directory	1 directory	Job lifetime
job.json	1 file	Job lifetime
Authoritative state marker	1 file	Job lifetime
Shared journal records	0 files	Packed into writer segments
Runner	0 files	Shared and referenced by digest, unless it lives in the payload
.httk-job/ runner job state	0 or 1 directory	Job lifetime, when a runner keeps state across attempts
Attempt control directory	0 or 1 directory	Attempt/retention lifetime
Persistent/isolated workdir	0 or 1 directory	Application policy
Per-state/per-event/per-failure files	0	Not used

Design arithmetic, not a measurement: the permanent floor is therefore three inodes: the payload directory, its job.json, and the one marker. The floor is reachable in practice and not only in theory, because a runner may live outside the payload — runner.source naming the workspace runner store or an installed search path, pinned by runner.sha256 — so a partitioned campaign whose children all execute the same program stores that program once and each child’s payload is exactly its job.json. A payload runner is the alternative, not the requirement, and costs whatever its own files cost. A runner that keeps state across the attempts and steps of one job adds .httk-job/, and a live or retained attempt adds its attempt-control directory and workdir.

Shard and journal directories are shared by many jobs. Application inputs, outputs, code, and logs naturally add their own files; the table only counts workflow metadata. Arbitrary placements can nevertheless make some placement directories unique to one job, and old state kinds can temporarily retain empty copies of those paths. That directory overhead is operationally relevant even though it is not a permanent per-job protocol object.

Concepts

A workflow workspace is one self-contained filesystem tree with an immutable workspace UUID, its jobs, authoritative state markers, and journals.

A watch root is an ordinary directory below which a manager discovers workflow workspaces. A manager may also receive explicit workspace paths and may supervise several workspaces at once.

A job is the durable unit of scheduling, history, and final success or failure.

A job key is the filesystem component [<tag>--]<job-uuid>. The UUID is authoritative; the optional tag is for human navigation.

A placement is an arbitrary relative parent path below a workspace, for example project-17/0/03a. A payload at that placement has the path <placement>/<job-key>.

A step is an application-defined name such as relax or collect. The set of possible step names need not be declared in advance.

An activation is one logical request to execute a step. Advancing from one step to another creates a new activation. Deliberately advancing to the same textual step name also creates a new activation.

An attempt is one physical execution of an activation. Retrying after a timeout or abandoned allocation creates a new attempt of the same activation.

A data generation identifies the committed contents of an optional job data/ tree. It is absent for jobs that do not use transactional data and advances after a successful transaction for jobs that do.

An outcome is the step’s atomically published request to advance, wait, succeed, fail, retry, or pause.

A child job is an ordinary job whose immutable definition names a parent. Children may themselves create children.

Required filesystem semantics

All correctness-critical paths within one workflow workspace MUST reside on one filesystem on which:

1. renaming a file within that filesystem is atomic;

2. renaming a directory within that filesystem is atomic;

3. a successful rename removes the source and installs the destination as one indivisible namespace operation;

4. a failed rename is reported to the caller.

Atomic rename of the exact current marker is the compare-and-swap operation. The protocol never depends on flock, advisory locks, PID uniqueness, or an exit trap.

The baseline guarantee is process-interruption safety. Storage-crash durability additionally requires an implementation to synchronize new file contents and affected parent directories before publishing their names. Implementations claiming storage-crash durability MUST use fsync or an equivalent operation in the order required by the filesystem.

The following require an explicit backend adapter and validation:

• source and destination paths on different mounts;

• object stores that only emulate rename;

• synchronization tools operating on a live root;

• network filesystems without coherent atomic rename;

• filesystems on which a client can indefinitely cache a removed name.

Modification times and wall clocks are evidence for lease expiry, but never provide fencing. Correctness comes from moving the one current state marker.

Conformance profiles

The protocol is deliberately phased. A conforming core implementation of the current profile, core-v2, supports:

• one workflow workspace per job and all references within that workspace;

• submission, validation, claiming, leases, execution, and outcomes;

• persistent and isolated workdirs;

• retries, joins, failures, cancellation, and operator requests;

• packed journals, verified marker renames, startup recovery, and garbage collection.

The following are optional extensions:

Extension	Additional behavior
transactional-data-v1	Replayable contributions to data/.
relocation-v1	Moving payloads between placements in one workspace.
multiworkspace-v1	Cross-workspace children, joins, and coordinated transfer.
detached-transfer-v1	Sealed standalone transfer bundles.

A workspace declares its core profile and enabled extensions in format.json. A manager MUST refuse to mutate a workspace whose core profile or enabled extensions it does not support. It MAY attach such a workspace read-only for inspection. Unknown state kinds are never treated as failed or orphaned jobs.

Withdrawn: priority-bands-v1. An earlier revision defined an extension that sharded state/ready/ into ten p0xx/ through p9xx/ directories. It is withdrawn and MUST NOT be enabled. It bought nothing: the exact priority is in every marker basename already, so nothing needed the directory to know a job’s priority; a scan looking for the best available work still had to walk all ten bands; and because a band is part of the marker path, enabling it was an irreversible decision taken at workspace creation. It also made priority a placement-changing transition, which is the one thing ordinary transitions promise not to do. The p<priority> field of the marker basename stays exactly as specified — nothing about priority encoding changes. A workspace that declares this extension cannot be interpreted, because its ready markers carry one path component that would otherwise read as a placement, so an implementation MUST refuse to attach it — read-only included — and say that the workspace has to be re-initialized rather than migrated.

This split is about implementation scope, not separate data models. Extensions retain the same job definition, marker, journal, and verified-rename rules.

Workspace layout and arbitrary placement

A workspace is an ordinary directory. Its protocol control data are below WORKSPACE/.httk-workflow/; job payload directories may be placed at any valid relative path outside that reserved directory:

WORKSPACE/
├── .httk-workflow/
│   ├── format.json
│   ├── tmp/
│   ├── quarantine/
│   ├── state/
│   │   ├── submitted/<placement>/
│   │   ├── ready/<placement>/
│   │   ├── claimed/<placement>/
│   │   ├── running/<placement>/
│   │   ├── committing/<placement>/
│   │   ├── cancelling/<placement>/
│   │   ├── relocating/<placement>/
│   │   ├── transferring/<placement>/
│   │   ├── waiting/<placement>/
│   │   ├── paused/<placement>/
│   │   ├── succeeded/<placement>/
│   │   ├── failed/<placement>/
│   │   └── cancelled/<placement>/
│   ├── journal/
│   │   └── <writer-id>/<segment-number>.hwj
│   ├── managers/
│   │   └── <manager-id>/
│   │       ├── manager.json
│   │       └── heartbeat.json
│   └── requests/
│       ├── tmp/
│       ├── ready/
│       ├── claimed/
│       └── retired/
└── project-17/
    └── 0/
        └── 03a/
            └── silicon-relax--01234567-89ab-cdef-0123-456789abcdef/
                ├── job.json
                ├── data/
                ├── files/
                ├── run/
                └── .httk-attempt.<attempt-id>/

Here the job placement is project-17/0/03a. Its authoritative marker has a parallel path such as:

.httk-workflow/state/ready/project-17/0/03a/
└── silicon-relax--01234567-89ab-cdef-0123-456789abcdef.p500.g4.<record-ref>

The protocol assigns no meaning to the placement components. They may represent projects, user names, dates, hash shards of any depth, or a mixture. Different jobs in one workspace may use completely different placement schemes.

The layout shows all defined extensions. A core-only workspace need not create relocating/ or transferring/, and no empty state-kind or placement directories are required.

files/, data/, a workdir, and attempt control are created only when required. Empty placeholder directories SHOULD NOT be created. Empty placement directories have no protocol meaning and may be pruned, subject to the rename/prune rules below.

format.json identifies the self-contained workspace:

{
  "format": "httk-workflow-filesystem",
  "format_version": 1,
  "core_profile": "core-v2",
  "extensions": [],
  "record_ref_encoding": "hwref-v1",
  "workspace_id": "b588833b-87ea-4da2-b860-1c9e768cfbc1",
  "created_at": "2026-07-24T12:00:00Z",
  "policy": {
    "visibility_deadline_seconds": 5.0,
    "lease_seconds": 900.0,
    "journal_segment_bytes": 67108864,
    "retention": {}
  }
}

Workspace policy

Everything this specification calls configured is one object in format.json, because a tunable that lives in each process cannot be agreed on by two implementations attaching the same workspace. The policy object is part of format version 1 and holds exactly these members:

Key	Type	Default	Meaning
visibility_deadline_seconds	number	5.0	How long a reader keeps re-probing before it declares a marker rename or a referenced journal frame incoherent.
lease_seconds	number	900.0	The default claim lease of a manager that does not state its own.
journal_segment_bytes	integer	67108864	The size at which a writer rotates to its next journal segment.
retention	object	{}	Optional attempt_control_days, journal_days, and trash_days limits for collection.

An implementation MUST refuse a policy member it does not know rather than ignore it, and MUST read an absent policy object, or an absent member of one, as the default above; a workspace written before this section existed is therefore attached without migration. Values are validated on write: lease_seconds is at least one second, visibility_deadline_seconds is at most one day, and journal_segment_bytes is at least 4096.

Policy is administrative rather than protocol state. A change is an ordinary read-modify-write of format.json through an exclusively created temporary file and a rename, so no reader ever sees a torn object, but concurrent policy writers are not serialized against each other and the last writer wins. A manager reads policy when it attaches; a change reaches already-running managers only when they restart.

There is no configured sharding depth and no priority directory level at all: priority is encoded in marker names, and the withdrawn priority-bands-v1 extension described in “Conformance profiles” is the only thing that ever proposed otherwise. A marker’s path below its state kind is its placement and nothing else.

Managers therefore provide best-effort priority scheduling: a cold start and an incremental scan may temporarily discover lower-priority work first. Strict global priority is not a guarantee of this protocol.

.httk-workflow/tmp/ contains unpublished entries. Task managers MUST ignore it for scheduling. Garbage collection may remove old temporary entries, but correctness MUST NOT depend on cleanup.

Placement components MUST be normalized relative path components. Empty components, ., .., NUL bytes, and .httk-workflow are forbidden. Each component must fit the underlying filesystem’s filename limit. A workspace MAY set policy limits on depth and total relative path length, but these are operational limits rather than a protocol sharding scheme.

Managing and combining workspaces

A core manager may attach several independent workspaces for operational convenience, but jobs and joins remain within their own workspaces. Cross-workspace references and coordinated movement require multiworkspace-v1.

A task manager accepts any combination of:

• explicit workspace paths;

• watch roots below which it discovers directories containing .httk-workflow/format.json.

The manager identifies a workspace by workspace_id, not its current absolute path. The same underlying workspace discovered through two path aliases is attached once. Journal references are workspace-relative and include the workspace ID when used from another workspace.

If two discovered roots declare the same workspace_id, a manager MUST prove that they identify the same underlying directory before treating them as aliases. Suitable evidence is an equal filesystem object identity obtained from open directory handles, device/inode identity where reliable, or an equivalent backend facility. Equal format.json bytes, UUIDs, or path canonicalization alone are insufficient because a copied backup has all three. If equivalence cannot be proved, the manager MUST refuse both roots for mutation and report a loud duplicate-workspace-ID error; it MUST NOT attach an arbitrary winner.

This permits both common arrangements:

# One workspace with projects as placement prefixes
WORKSPACE/project-a/00/17/<job-key>
WORKSPACE/project-b/hash/x9/<job-key>

# A watch root containing self-contained project workspaces
WATCH/project-a/.httk-workflow/format.json
WATCH/project-a/00/17/<job-key>
WATCH/project-b/.httk-workflow/format.json
WATCH/project-b/hash/x9/<job-key>

A manager may schedule from all attached workspaces in one resource pool. Job and child references are (workspace_id, job_id) pairs; the job UUID alone is accepted only when unambiguous among attached workspaces.

Discovery stops at a workspace boundary. Attached workspaces MUST NOT overlap unless an explicit advanced profile defines ownership of every placement prefix; the default manager rejects nested/overlapping workspace roots.

Dynamic attachment

A complete workspace can be built elsewhere on the same filesystem and atomically renamed below a watch root. Its format.json, state tree, journals, and payloads arrive together. The manager discovers the immutable workspace ID and begins scheduling it without restarting.

Filesystem notifications are hints. Managers periodically rescan watch roots so a lost notification cannot hide an attached workspace.

Renaming an attached workspace within watched paths does not create a new workspace. Managers SHOULD hold an open directory handle and update the path associated with the same workspace ID. Copying a live workspace is forbidden: it duplicates authoritative markers and journal identity. Discovery of such a copy is handled by the duplicate-ID refusal above, including when an operator restores a backup beside the live workspace.

Dynamic detachment

Detaching one workspace does not stop a manager from serving its other workspaces. A detach coordinator obtains acknowledgement from every manager with a live heartbeat in that workspace. Each manager:

1. stops new claims from that workspace;

2. completes or releases manager-owned claimed jobs;

3. completes committing replays;

4. waits for, pauses, or explicitly hands off running attempts;

5. closes the workspace journal writer;

6. acknowledges that the workspace is quiescent for it.

Once all live managers acknowledge and no marker is claimed, running, or committing, the workspace directory may be atomically moved out of the watch root. A manager MUST NOT interpret disappearance of a workspace as failure of every job in it. It marks that workspace unavailable until the same workspace ID is reattached.

A raw move of a workspace containing active attempts is supported only when the same supervising managers follow the atomic rename by workspace ID and retain valid directory handles. The portable and recommended operation is controlled detach, move, then attach.

Job UUIDs, names, and path tags

Job IDs are lowercase canonical UUIDs. Human-readable names do not have to be unique and remain in job.json.

An optional tag may be included in the job key:

silicon-relax--01234567-89ab-cdef-0123-456789abcdef

The tag:

• is an immutable convenience label, not identity;

• is 1 through 48 ASCII bytes;

• uses lowercase letters, digits, ., _, and -;

• starts with a letter or digit;

• MUST NOT contain --;

• SHOULD be a short slug derived from the job name;

• need not be unique.

Without a tag, the job key is just the UUID. Parsers identify the final UUID rather than trusting the tag. A lookup by tag may return several jobs; UUID lookup returns at most one job within a workspace.

The current payload path is:

<workspace>/<placement>/<job-key>/

The authoritative state marker uses the same placement and job key. Consequently, an operator can use ordinary shell completion or find to find both payload and state:

find . -path './.httk-workflow' -prune -o -type d -name 'silicon-relax--*' -print
find .httk-workflow/state -type f -name 'silicon-relax--*'

Renaming a tag by hand is forbidden because the payload and state key must agree. A workflow may instead put mutable descriptive labels in application data or an external catalog. Avoiding alias files keeps the per-job inode cost fixed.

The authoritative state tree

One marker, one source of truth

Every submitted job has exactly one regular marker file somewhere below state/. There is no second marker inside the job and no advisory state index. The marker’s directory is the current scheduler state.

The sole exception is an explicitly detached transfer bundle: its same marker inode is sealed inside .httk-transfer/, outside every manager’s state tree, and the bundle is not schedulable until import republishes that marker.

For example:

.httk-workflow/state/ready/project-17/0/03a/
└── silicon-relax--01234567-89ab-cdef-0123-456789abcdef.p500.g4.<record-ref>

The basename has this logical grammar:

<job-key>.p<priority>.g<generation>.<record-ref>

generation is a monotonically increasing unsigned 64-bit state generation, encoded in lowercase base 36 without leading zeroes. It is unrelated to the data generation. record-ref locates the immutable transition record in a packed journal. The initial submitted marker uses g0.init, because its initial information is in job.json. Priority is always a zero-padded three-digit integer from 000 through 999.

The complete relative marker path is the authoritative current state:

• its state directory gives the state kind;

• the directories after the state kind give current placement, and nothing else: no priority, shard, or index level is ever inserted between them;

• its p<priority> component gives current priority;

• its job key gives identity;

• its generation prevents stale operator actions;

• its journal reference gives transition details.

A transition MUST rename the exact old marker path to the exact new marker path. It MUST NOT create a second marker and later delete the first. Therefore:

• two managers racing to claim the same ready marker cannot both win;

• a crash cannot leave old and new authoritative states;

• terminal and non-terminal collections are immediately inspectable;

• no index-repair process is part of ordinary correctness.

Outside the explicitly recoverable relocating and transferring states, a marker without its payload at the mirrored placement is corruption. A payload directory without a marker is unsubmitted temporary/orphan data, not a queued job, unless it is a sealed detached bundle containing .httk-transfer/manifest.json and its marker.

State-marker rename

Before a transition, the actor:

1. prepares and synchronizes the new journal record;

2. derives the new marker basename from its journal reference;

3. creates the destination placement parents if absent;

4. attempts to rename the exact old marker to the unique destination;

5. resolves the observed filesystem state before deciding whether it won.

The return status of rename is not itself a transition result. In particular, ENOENT can mean that the source vanished or that a destination parent was pruned, and a network filesystem can execute a rename even when the caller receives a failure after retransmission. Every implementation MUST use this verified-transition algorithm after a rename error or ambiguous response, and MAY use it unconditionally:

1. Reopen or refresh the source and destination parent directories rather than relying on one cached negative lookup.

2. If the actor’s unique expected destination exists and its basename resolves to the prepared journal record, the actor won. It proceeds even if rename reported failure.

3. Otherwise, if the exact source still exists, the actor recreates any missing destination parents and retries the same rename.

4. Otherwise, the actor locates and validates the job’s current marker. A different valid marker proves that another transition won.

5. If source, expected destination, and another valid current marker all remain absent after bounded reopen/backoff retries, the workspace is unavailable or corrupt. The actor MUST stop mutation and report that condition; it MUST NOT silently classify the result as a lost race.

Record references are globally unique within a workspace, so the expected destination MUST initially be absent. An implementation SHOULD use no-replace rename where available. An already present expected destination is success only when it names the actor’s prepared record; any conflicting entry is corruption.

This algorithm applies to every correctness-critical rename in this specification: marker transitions, submission, outcome publication, transaction operations, child registration, relocation, and transfer.

When a loser prepared a journal record but another transition won, its unreferenced record stays in the journal. That record is off-chain and is harmless by construction, because every reconstruction of a job’s history starts at the authoritative marker and walks previous_record_ref backwards. An orphan is on nobody’s chain — no marker ever referenced it and no frame ever named it as its predecessor — so it cannot be observed as a transition, and no implementation may reconstruct history by scanning the journal forwards. Readers therefore need no special handling of orphan records, and none is specified: no tombstone, no supersession record, no compaction requirement.

The one operation that legitimately reads the journal without a chain to walk is marker repair, because there the frame holding the backward link is exactly the unreadable one. Repair is bounded instead: it considers only frames of the same job at a generation strictly lower than the damaged marker’s own, never walks forward onto a frame no marker committed, and records the reference of the frame it adopted in the repair frame it writes. An orphan always shares its generation with the transition that actually won, so a repair that reaches back across a lost race MAY adopt the loser’s members; the recorded fsck_repair.recovered_record_ref is what makes that visible and auditable afterwards.

Empty state-placement parents MAY be removed only with operations that fail when a directory is nonempty. A pruner racing a transition either observes the new marker and fails to remove the directory, or removes the empty directory first and causes the transition to recreate it and retry. Broad recursive deletion is forbidden.

A pruner MUST make at most one removal attempt per candidate directory in one scan and MUST NOT immediately retry a failed removal. It backs off until a later scan. Transition parent-creation/rename retries are also bounded; if repeated confirmed prune collisions would exhaust that budget, the manager suppresses pruning for the affected subtree or workspace and retries the transition before reporting storage failure. Pruning is optional and MUST yield to state mutation. Implementations MAY simply disable placement-directory pruning while managers are attached.

The marker file is created once at submission and then only renamed. A normal job lifetime therefore consumes one state-marker inode regardless of its number of steps, retries, failures, or manual continuations.

Packed transition journal

Creating one JSON file for every state, event, failure, and output revision would be prohibitive at this scale. Transition metadata are instead frames in shared append-only journal segments:

journal/<writer-id>/<segment-number>.hwj

writer-id is a fresh random UUID generated for every task-manager process incarnation. The manager-id used in claims and heartbeats is likewise a fresh process-incarnation UUID; any stable administrative name is a separate manager_label. The writer and manager UUIDs MAY be equal but neither may be reused by a restarted process. On startup a process MUST create a new writer directory and its first segment with exclusive creation. It MUST NOT reopen any existing segment for append, even if it believes that segment belonged to a previous incarnation of the same manager. A zombie and its replacement therefore write different paths and heartbeats.

Each process is the sole writer of its own segments and never appends to another writer’s segment. Segments rotate at the size policy.journal_segment_bytes configures, not per job. Submission needs no journal file per invocation because the initial marker uses init.

A core journal segment begins with the 12 bytes 48 54 54 4b 2d 48 57 4a 2d 56 31 0a (HTTK-HWJ-V1 plus newline). Segment filenames are lowercase base-36 numbers without leading zeroes. Each frame then contains:

1. an eight-byte unsigned big-endian payload length;

2. exactly that many bytes of one UTF-8 JSON record;

3. the 32-byte SHA-256 checksum over the eight length bytes and payload;

4. the same eight-byte big-endian length as a trailer.

The marker’s compact record-ref identifies writer, segment, byte offset, length, and checksum. Format version 1 mandates the canonical filename-safe hwref-v1 encoding:

w<writer-uuid-hex>-s<segment-base36>-o<offset-base36>-l<length-base36>-h<checksum128-hex>

The writer UUID is 32 lowercase hexadecimal digits without hyphens. Segment numbers are unsigned 32-bit integers; byte offsets and frame lengths are unsigned 64-bit integers. Numeric fields use lowercase base 36 without leading zeroes except for zero itself. checksum128 is the first 128 bits of the SHA-256 checksum over the encoded frame length and payload, rendered as 32 lowercase hexadecimal digits. The complete checksum remains in the frame. No alternate encoding is permitted in a core workspace, so independent implementations resolve marker names identically.

The worst-case noninitial marker basename is 213 ASCII bytes:

86 job-key + 5 ".p999" + 15 (".g" + 13 generation digits)
+ 1 "." + 106 maximum hwref-v1 = 213

The 86-byte job key is a 48-byte tag, --, and a 36-byte UUID. The reference budget is 33 for w and the writer UUID, 9 for -s and a seven-digit base-36 segment number, 15 each for -o/offset and -l/length, and 34 for -h plus the checksum: 33 + 9 + 15 + 15 + 34 = 106. A conforming workspace MUST support at least 213 bytes per filename component and MUST validate this at initialization. These field limits and the tag limit MUST NOT be enlarged within format version 1.

Before a state-marker rename, the writer MUST flush the entire frame and, in the storage-durable profile, synchronize the segment. A marker can therefore never legally reference a torn tail. An unreferenced partial final frame is ignored and may be truncated during journal repair.

This implementation runs the storage-durable profile by default. Durability is not only a manager-side property of journals and markers: the runner-side artifacts that publish work are synchronized to the same standard, because a marker or journal frame that claims a committed outcome is worthless if the outcome its storage should hold was never flushed. In the durable profile, each of the following is synchronized — its file contents, then the directory entries that name it — before the rename that makes it authoritative:

Artifact	Synchronized before
Journal frame	the state-marker rename that references it
State marker / request / submitted payload	it becomes visible in state/ (its parent directory is flushed)
Attempt context.json, process.json	the runner is launched against it
Outcome bundle (outcome.json, runner_steps, failure detail, its sealed transaction manifest and staged payload, its child job.json bundles)	the outcome.tmp.<nonce> → outcome.ready rename; the whole draft tree is flushed in one batch, then the attempt-control directory after the rename
Committed transaction data (data/)	the manager appends the destination state frame and renames the marker out of committing; every replayed destination and each parent directory it touched, including the trash a removal moved into, is flushed first
Registered child payloads and their submitted markers	the parent’s marker leaves committing
Job state (.httk-job/state.json), observed declarations, runner-steps.json	each atomic replace returns
Sealed replayable workdir batch (.httk-runner/workdir-ready/) and its replay into the workdir	the batch is published, then retired as applied

Two runner-side artifacts keep only process-interruption safety even in the durable profile, because per-line synchronization would dominate their cost and neither is authoritative workflow state: the append-only run log (.httk-runner/runlog.jsonl) and the runner’s captured stdout.log and stderr.log. They are evidence for an operator, not markers or committed data; losing their tail to a power cut costs a diagnostic line, never a lost outcome or a half-applied transaction.

In the non-durable profile every one of these keeps process-interruption safety only: a torn write or an interrupted rename is still never observed, but a node that loses power may lose any of the above — a journal frame, a marker, a published outcome, half of a “committed” transaction, or a registered child. The opt-out (--no-durable) exists for throwaway and test workspaces where that trade buys speed. Without it, a node that loses power can leave a marker naming a frame or an outcome its storage never received, which is exactly the damage workspace fsck below has to repair.

On a network filesystem, visibility of a marker and visibility of the newly extended journal segment may reach different clients at different times. A reader that sees a referenced frame as absent, short, or checksum-incomplete MUST close and reopen or otherwise refresh the segment and retry with bounded backoff. It declares corruption only after the workspace’s configured visibility deadline — policy.visibility_deadline_seconds in format.json — and it MUST NOT mutate the marker while the referenced frame is temporarily unreadable. Damage that no amount of waiting can repair, such as a frame whose checksum is present and wrong, is reported at once rather than waited out. The same deadline bounds the retries of a verified state-marker rename and of any other metadata-visibility probe.

Every state frame includes:

{
  "format": "httk-workflow-state",
  "format_version": 1,
  "workspace_id": "b588833b-87ea-4da2-b860-1c9e768cfbc1",
  "job_id": "01234567-89ab-cdef-0123-456789abcdef",
  "job_key": "silicon-relax--01234567-89ab-cdef-0123-456789abcdef",
  "placement": "project-17/0/03a",
  "state_generation": 4,
  "kind": "ready",
  "previous_record_ref": "...",
  "created_at": "2026-07-24T12:00:00Z",
  "step": "collect",
  "activation_id": "e7f86a0e-34d6-45a7-b92d-3f4b2dc98c54",
  "activation_ordinal": 3,
  "attempt_ordinal": 1,
  "total_attempts": 4,
  "data_generation": 2,
  "priority": 500,
  "reason": "advance"
}

data_generation is omitted or null when job.json declares data.mode: "none".

State frames form a backwards-linked history across writer segments. Failure, join, operator, and outcome details are embedded in the applicable state frame or in another journal frame referenced by it. They do not create per-job metadata files.

Journal segments are append-only and retained according to history policy. They may be compressed only into a random-access archive format that preserves record references. A derived SQL database or in-memory map MAY accelerate queries, but it is a cache: the state tree and referenced journal frames remain authoritative.

Three rules make such a cache safe, and they are what this implementation’s job-id-to-marker index obeys:

1. A hit is confirmed before it is used. The cached location is a path; the marker either is there or is not, and a marker another actor moved is detected by the check rather than reported as current state.

2. A miss is never an answer. Absence is reported only after the ladder in Waiting and joining has been walked to its end, including one complete scan. Any other rule would let a job that another manager has just published be reported as nonexistent.

3. It is process-local. This implementation keeps the index in memory, per attached workspace, built lazily from one scan and updated by every rename the same process performs. There is deliberately no on-disk index file: a workspace has many concurrent writers and no protocol-level way to order their updates to a shared derived file, so a durable cache would need a synchronization and repair story that the authoritative state tree already provides for nothing. An implementation that wants a shared derived database MAY build one, under exactly the same three rules.

Job definition and submission

A minimal job.json is:

{
  "format": "httk-workflow-job",
  "format_version": 1,
  "id": "01234567-89ab-cdef-0123-456789abcdef",
  "tag": "silicon-relax",
  "name": "Silicon relaxation",
  "workflow": "example.vasp-relax",
  "runner": {
    "backend": "path",
    "path": "files/runner",
    "arguments": []
  },
  "workdir": {
    "mode": "persistent",
    "path": "run"
  },
  "data": {
    "mode": "none"
  },
  "initial_step": "prepare",
  "priority": 500,
  "claim": {
    "pool": "default",
    "required_capabilities": []
  },
  "retry_policy": {
    "maximum_attempts_per_activation": 10,
    "maximum_total_attempts": 100,
    "maximum_activations": 50,
    "retry_on": ["lease_lost", "timeout", "process_failure"]
  },
  "resources": {},
  "parent": null
}

job.json is the only required metadata file in the payload directory and is immutable after submission. Small workflow-specific parameters SHOULD be stored directly in it instead of one file per parameter.

The optional top-level inputs member is the place for them: a JSON object with string keys and application-defined values, opaque to the protocol and covered by the immutable job digest like every other member. An implementation MUST reject an inputs object whose serialization exceeds 262144 bytes; bulk content belongs in the payload or in transactional data/. A parent that synthesizes a child job varies normally only the child’s initial_step and its inputs.

The optional top-level declarations member carries workflow declarations: a JSON object mapping a declaration name to one declaration document. A declaration document states what a workflow is — its inputs, its method, its outputs — without describing a graph, and is intended to feed provenance.

The document is carried verbatim and is opaque to this protocol. An implementation MUST validate that each member is a JSON object and MUST NOT interpret, wrap, normalize, or version it any further: versioning and self-description live inside the document itself, as the $id-style members of the property-definition conventions it follows. A declaration name MUST match [A-Za-z0-9_][A-Za-z0-9._-]{0,63}, because it is also one file basename in the payload area below. An implementation MUST reject a declarations object whose serialization exceeds 262144 bytes, which is the inputs allowance again and separate from it. Being members of job.json, declarations are immutable after submission and covered by the immutable job digest like every other member. Nothing is inherited: a synthesized child job carries the declarations its parent gave it and no others.

runner.backend selects an installed execution adapter and defaults to path when omitted. The core task manager implements path; managers MUST leave jobs using an unavailable or disallowed backend unclaimed. This permits specialized managers to share a workspace without either one accidentally running the other’s job profile. Backend-specific immutable fields belong in job.json, and their submission validation is owned by that backend.

runner.source selects the root runner.path is resolved against and defaults to payload:

runner.source	Root of runner.path
payload	the job’s own payload directory
workspace	<workspace>/.httk-workflow/runners/
installed	one ordered runner search path configured in the manager

runner.path MUST remain beneath its root under every source. arguments is an argument vector, never a shell command string. A backend may treat the path as a backend-specific program while retaining these path-containment rules.

A payload runner is already pinned by the immutable job digest, so runner.sha256 MUST be absent for it. Every other source names one file, or one tree, shared across jobs by design, so runner.sha256 is REQUIRED and pins it: the digest of a file is over its bytes, and the digest of a tree is the canonical tree digest. Sharing one runner file across a large partitioned campaign is the purpose of the non-payload sources.

A manager MUST NOT execute a shared runner in place. It resolves the runner, copies it below the attempt control directory as runner, verifies runner.sha256 against that staged copy, and executes only the copy. A digest disagreement fails the job with runner_mismatch; a runner that cannot be resolved fails it with runner_unavailable. Both are ordinary continuable failures, never silent substitutions. A staged tree is entered at its top-level run file.

An installed path may also use the reserved form pkg:<module>/<resource>, which resolves inside an installed Python package. A manager MUST restrict that form to an explicit module allowlist, httk.workflow by default.

The immutable job digest a manager records is the SHA-256 over the stored job.json file bytes exactly as submitted. Nothing renormalizes those bytes, so any implementation with a hash utility reproduces the digest.

files/ is optional and contains submitted code, templates, or immutable input objects that require separate files. Small inputs SHOULD be embedded in job.json.

workdir.mode is persistent or isolated:

• persistent reuses the declared workdir directory across step activations and attempts. The workflow program owns recovery and cleanup of partial application files.

• isolated creates a new run.<attempt-id>/ for every attempt. It may be initialized from submitted files or transactional data/.

The mode MUST be explicit in job.json; the protocol has no implicit default.

data.mode is none or transactional and is also explicit. With none, the job may keep all mutable and final application data in its persistent workdir. It never needs to create data/, publish a transaction, or increment a data generation. With transactional, data/ and the transaction protocol below are available when the workspace enables transactional-data-v1.

claim and its pool are required; required_capabilities may be empty. claim.pool is the scheduling pool or queue from which the job may be claimed. Pool and capability labels use the same conservative component syntax as tags. A manager advertises its pools and capabilities and MUST claim a job only when both match. This is claim eligibility, not a workflow name and not a substitute for quantitative resources.

The literal pool name default is reserved for jobs requiring no explicit routing. A manager started without pool configuration MUST advertise default. Thus a trivial deployment uses the value shown above without any out-of-band pool agreement; sites opt into other pool names deliberately.

Retry limits are independent optional safeguards:

• maximum_attempts_per_activation limits retries of one logical activation;

• maximum_total_attempts limits physical executions over the whole job;

• maximum_activations limits initial plus advanced activations, including advances back to the same textual step.

The counters are durable state-frame values. Before creating an activation or attempt, the manager checks the applicable limits. A request that would exceed one transitions to failed with budget_exhausted; it is not launched. An omitted limit is unbounded.

Priority is 0 through 999, where 0 is highest. It affects scheduling, not correctness.

Submission protocol

To submit:

1. Create a complete job below .httk-workflow/tmp/, including job.json and any initial files/ or data/.

2. Choose any placement and atomically rename it to <workspace>/<placement>/<job-key>/.

3. Create a temporary zero-length marker named <job-key>.p<priority>.g0.init.

4. Atomically rename that marker to .httk-workflow/state/submitted/<placement>/<job-key>.p<priority>.g0.init.

Step 4 is the submission commit point. Before it, the placed payload is an unsubmitted orphan and may be completed, retried, or eventually collected. After it, a fully populated submitted job exists.

A task manager registers the job by validating job.json, appending its first ready state frame, and renaming the same marker from submitted to the mirrored path below state/ready/. A crash before this rename leaves the job visibly submitted; another manager repeats validation.

g0.init describes the marker submission creates, not every marker that may be found in submitted. A job can be acted on before any manager registers it — set_priority and pause both apply to submitted — and such a request is an ordinary verified transition, so it appends a frame and renames the marker to <job-key>.p<new-priority>.g1.<record-ref> in the same state directory. Registration MUST therefore accept both forms of a submitted marker: generation 0 referencing init, whose priority is the one in job.json, and a later generation referencing a real frame, whose priority is the operator’s and may differ from job.json. The marker is authoritative for priority in both cases; only at generation 0 does the job.json priority have to agree with it.

If the submitted marker is well formed but job.json, its immutable files, or their relationship to the marker fails validation, the manager appends a failed frame with class protocol_error and moves that same marker to state/failed/<placement>/. The frame uses the UUID and job key from the validated submitted marker when the document’s identity cannot be trusted and records bounded validation details. The payload is retained for diagnosis. Implementations MUST NOT leave invalid jobs indefinitely in submitted.

An entry whose marker name or placement is itself not parseable cannot enter the normal job state machine. A workspace repair tool moves it to a shared .httk-workflow/quarantine/ area outside state/; managers report it loudly and never schedule it. Names in quarantine MUST preserve or record the original relative path without permitting collisions or traversal. Quarantine is exceptional workspace corruption, not another job state.

Retrying submission with an existing job UUID succeeds only when the existing immutable job digest is identical and its marker already exists. A different definition under the same UUID is an error.

State machine

The state kinds are:

Kind	Meaning
submitted	Complete job awaiting manager validation and registration.
ready	Eligible to be claimed when resources permit.
claimed	A manager won the claim but has not launched the attempt.
running	The current attempt may have a live process.
committing	The attempt is fenced; its outcome is being replayed.
cancelling	The attempt is fenced by a cancellation; its process is being stopped and its exit verified.
relocating	relocation-v1: no attempt may run; placement is changing.
transferring	Transfer extensions: a quiescent payload is moving or detaching.
waiting	Waiting for a declared child join.
paused	Requires an operator request to continue.
succeeded	Successful terminal state.
failed	Failed until explicit operator continuation.
cancelled	Cancelled terminal state.

“Terminal” is used in two senses in this document, and they are not the same thing. Terminal for scheduling means that no manager will move the marker on its own: the three kinds succeeded, failed, and cancelled. That is the sense a join condition uses, the sense state/failed/ is a complete collection in, and the sense in which a failure’s own frame is the job’s last automatic one. Permanent means that nothing moves the marker again at all, and only succeeded and cancelled are permanent: a failed job is terminal for scheduling but explicitly revivable by an operator continue or override_step, which is the whole point of tracking broken jobs in the state tree. Where this document says “terminal” without qualification it means terminal for scheduling. cancelling is neither: it is a live state in which a fenced attempt is being stopped.

The scheduling cycle is:

the cycle
  submitted ─validate─> ready ─claim─> claimed ─launch─> running ─outcome─> committing
                          ▲                                                       │
                          └──────────── advance or retry ────────────────────────┘

  committing ─────> ready | waiting | succeeded | failed | paused

back to ready
  claimed ─release────────────────────────────────────────────────> ready
  running ─retry within budget────────────────────────────────────> ready
  waiting ─join satisfied, or on_impossible advance───────────────> ready
  failed | paused ─operator continue or override_step─────────────> ready

into failed
  submitted ─protocol_error───────────────────────────────────────> failed
  claimed   ─prepare failure──────────────────────────────────────> failed
  running   ─lease loss, process failure, or an exhausted budget──> failed
  waiting   ─dependency_failure, or a join that cannot be read────> failed

cancellation
  running ─operator cancel─> cancelling ─verified exit─> cancelled
                                 ▲   │
                                 └───┘ the exit is not proven yet: stay fenced
  any other nonterminal state ─operator cancel───────────────────> cancelled

operator moves that do not change the kind, or only queue the job
  submitted | ready | waiting ─pause──────────────────────────────> paused
  submitted | ready | waiting | paused | failed ─set_priority────> the same kind

Every transition of the core profile, exhaustively:

From	To	Trigger
submitted	ready	Validation and registration.
submitted	failed	protocol_error: the payload, job.json, or its relationship to the marker is invalid.
ready	claimed	A manager won the claim.
claimed	running	The attempt was launched.
claimed	ready	Release: the manager may no longer launch it — a maintenance lock appeared, its budget changed — and it undoes the attempt counters it took.
claimed	failed	The attempt could not be prepared: protocol_error, process_failure, runner_unavailable, or runner_mismatch.
running	committing	A valid outcome was published; the rename fences the attempt.
running	ready	Retry within budget after lease_lost or process_failure.
running	failed	Retry budget exhausted, or a failure the policy does not retry.
running	cancelling	Operator cancel of a job that may have a live process.
committing	ready	advance to a new activation, or retry of this one.
committing	waiting	A wait outcome naming a child join.
committing	succeeded	A succeed outcome.
committing	failed	A fail outcome, an unusable one, or an exhausted budget.
committing	paused	A pause outcome.
waiting	ready	The join was satisfied, or became impossible and on_impossible names a step.
waiting	failed	dependency_failure: the join became impossible with no on_impossible, or a named child stayed unresolvable past the join grace.
waiting	failed	protocol_error: the recorded join itself cannot be read.
failed, paused	ready	Operator continue (retry the activation) or override_step (new activation).
cancelling	cancelled	The exit of the fenced attempt was verified.
any nonterminal	cancelled	Operator cancel where no live attempt has to be stopped first.
submitted, ready, waiting	paused	Operator pause.
submitted, ready, waiting, paused, failed	the same kind	Operator set_priority, which renames the marker to a new priority at the next generation.

Each row is one verified rename of the same marker; no other rename of a marker is defined, and the two extension states relocating and transferring are described in their own sections.

advance may name any next step, including the same textual name. It creates a new activation. Retry retains the activation ID and increments the attempt ordinal.

Any non-terminal state can be cancelled by an authorized operator. Cancellation of running first fences the attempt by moving its marker to cancelling; a late outcome from that attempt can no longer commit. cancelling is not a terminal state and not a quiescent one: it is the interval in which the fenced process is stopped and its exit is verified. See Cancellation.

When relocation-v1 is enabled, quiescent states may also pass through relocating and return to the same logical state at a different placement.

Cancellation

Cancelling a job that may have a live process is three ordered steps, and the order is the guarantee:

1. Fence first. The manager renames the exact running marker to cancelling. From that instant no manager accepts an outcome from that attempt, because the state tree no longer names it as current. Nothing has been signalled yet, so this step cannot be skipped by a race.

2. Then stop the process. The owning manager — or, if it has died, any manager that can see the process — sends SIGTERM to the recorded process group, and SIGKILL after a configured grace period. The cancelling frame names the attempt, its attempt-control directory, and its previous owner, so a recovering manager has everything it needs.

3. Only then finish, against evidence. The marker moves to cancelled only once the exit has actually been verified, and the verification is recorded in the cancellation member of the terminal frame.

A manager MUST NOT publish cancelled for an attempt it has merely signalled. Acceptable evidence is:

cancellation.verified	Meaning
process_exited	The manager launched the process and reaped it; the exit status is recorded.
process_group_absent	The process was recorded on this host and its process group no longer exists.
no_launched_process	No process identity was ever durably recorded, so the launch gate guarantees the runner never executed.
no_live_attempt	The job was cancelled from a state that has no live attempt at all.

A cancellation whose process cannot be proven stopped — typically one recorded on a different host — MUST leave the marker in cancelling, journal why, and retry. Staying in cancelling is the safe outcome: the attempt remains fenced, so it can produce no state, and an operator sees a stalled cancellation rather than a terminal state that falsely asserts that nothing is still writing the workdir. A site whose batch system can confirm that an allocation has ended MAY treat that confirmation as evidence and record it in the same member.

Because the fence is a marker rename, cancellation composes with everything else by construction: an attempt that publishes an outcome after being fenced finds its marker in cancelling rather than running, and the outcome is ignored exactly like any other outcome from a fenced attempt.

Claiming, leases, and fencing

To claim a ready job, manager M:

1. Selects the exact ready marker.

2. Reads the state frame and stable job.json.

3. Generates attempt and claim IDs.

4. Appends and synchronizes a claimed state frame.

5. Renames that exact ready marker to the claimed state path whose basename references the new frame.

6. Applies the verified-transition algorithm and proceeds only if its unique claimed destination is confirmed.

All competing managers rename the same source path to different unique destinations. At most one transition can remove that source; each caller uses verified-transition resolution rather than the rename return status to learn whether it won.

The claimed frame names:

• manager, writer-incarnation, and attempt IDs;

• step, activation, and attempt ordinal;

• input data generation when transactional data are enabled;

• lease duration and start time;

• matched pool and capabilities;

• resource allocation;

• preceding record reference.

The manager creates .httk-attempt.<attempt-id>/, prepares the selected persistent or isolated workdir, appends a running frame, and renames the claimed marker to running immediately before launching the application. A local executor MAY first create a process blocked on a launch gate, durably record that process identity, commit the running marker, and only then release the gate to execute the application. If its manager disappears before release, the gate MUST cause that process to exit without executing the application. This closes the otherwise ambiguous crash window between a running transition and recording the new process identity.

Managers update managers/<manager-id>/heartbeat.json by atomic replacement. A recoverer uses the state frame, heartbeat, batch scheduler when available, and a configured grace period. It MUST NOT steal a job merely because one delayed metadata read appears stale.

A heartbeat is a statement about the manager, not about its scheduling pass, so a manager MUST NOT let one pass over the state tree hold its heartbeat: it takes heartbeat opportunities between the state kinds it scans and within long scans of one kind. A manager whose workspace is too large to serve inside one lease SHOULD also bound how many markers of a kind it processes per pass and resume the rest on the next pass, in a stable order so that no marker starves. An implementation SHOULD report a pass that consumes a large fraction of its own lease, because that is the condition under which a healthy manager begins to look abandoned.

Once policy determines that an attempt is abandoned, a recoverer appends a new claimed frame and renames the exact old claimed or running marker. That rename fences the old attempt. Even if its process later wakes, no task manager accepts its outcome because the state tree names a different attempt.

A manager taking immediate ownership transitions directly to a new claimed state. A manager merely releasing work transitions back to ready. Both retain the activation ID for a retry.

Filesystem fencing cannot prevent a partitioned old process from producing external side effects or modifying a persistent workdir, and it does not make a duplicated attempt free: two allocations burning for one activation is a real cost whichever workdir mode the job uses. Lease expiry alone is therefore never sufficient evidence to relaunch an attempt, in either mode.

For a persistent workdir, the default safe policy MUST establish that the previous writer can no longer modify the workdir—for example, by scheduler-confirmed allocation expiry or cancellation and process-group termination—before launching a replacement. A site MAY enable lease-only persistent takeover as an explicit unsafe policy.

For an isolated workdir, a replacement corrupts nothing, so the requirement is weaker but not absent: a manager MUST have one of

• the recorded process is provably gone on this host,

• a scheduler confirmation that the allocation has ended, or

• a heartbeat that has been silent for a configured multiple of the lease — the takeover grace, by default twice the lease.

A manager SHOULD terminate the old process group or cancel its batch allocation before relaunching when it can. The grace exists because an expired lease says only that a manager is slow: a manager whose scan of a very large workspace overruns one lease is alive and its attempts are running, and taking those over at the first expired lease is how one slow node becomes twice the bill.

Every takeover MUST record its evidence in the new attempt frame — which rule admitted it, and the observed heartbeat age — and every use of an explicitly unsafe policy MUST be recorded there too.

Attempt control, workdirs, and runner contract

Attempt control is separate from application workdir:

<workspace>/<placement>/<job-key>/
├── .httk-attempt.<attempt-id>/
│   ├── context.json
│   ├── runner                   # the verified staged copy of a shared runner
│   ├── outcome.tmp.<nonce>/
│   └── outcome.ready/
├── .httk-job/                   # optional runner-private job state
│   └── declarations/            # optional observed workflow declarations
│       └── <declaration-name>.json
└── run/                         # persistent mode

In isolated mode the application directory is run.<attempt-id>/ instead of run/. The runner’s current working directory is the selected application workdir, not the attempt control directory.

.httk-job/ is runner-private state that belongs to the job rather than to one attempt: it survives retries, step advances, and isolated workdirs, and it travels with the payload when the job is transferred. A runner MAY store what it needs there, atomically. Both .httk-job/ and every .httk-attempt.<attempt-id>/ are excluded from every payload digest — submission, child registration, and detached transfer alike — so publishing an outcome and writing job state can never disturb an immutability check of the payload.

.httk-job/declarations/<declaration-name>.json is the one reserved name inside that area: the observed workflow declaration of that name, the runtime-refined counterpart of what job.json declared. A campaign that discovers its outputs only while running writes what it observed there, and the last write of a name wins. The document is carried verbatim and is opaque to the protocol exactly as in job.json, the basename before .json is the declaration name and MUST therefore satisfy the same syntax, and being below .httk-job/ the file is excluded from every payload digest. An implementation MUST report the declared and the observed document side by side and MUST NOT merge them; a document that cannot be read is reported as absent, with the reading tool’s damage flag set.

Separating them matters in persistent mode: a late process from an old attempt can only publish beneath its own attempt-control name. Its outcome cannot replace or impersonate the new attempt’s outcome.

Persistent workdir

The same declared workdir is used across normal step advances and retries. The manager does not clean, copy, snapshot, or transactionally inspect its application files.

This mode supports, for example, a VASP workflow that retains a very large WAVECAR in run/, modifies it over several steps, and decides for itself whether an interrupted calculation can continue. data/ need not exist.

On retry, the manager first fences and attempts to terminate the old process, then invokes the same step activation in the same directory with a new attempt context. The workflow code examines the reason and the existing files and may:

• continue directly;

• remove known partial outputs and restart;

• repair inputs and request another retry;

• declare the job failed.

A planned advance to another step also reuses the directory, but is not a restart: the new activation has attempt ordinal 1 and is_restart: false.

Persistent mode deliberately gives up workdir isolation. If an old process cannot be established incapable of further writes, it may still modify run/ after being fenced from publishing an outcome. The safe default is therefore to wait, force scheduler cancellation, or pause for manual action. Only an explicitly configured unsafe takeover policy may accept concurrent-writer risk, as specified under fencing above.

Isolated workdir

Every attempt receives a new run.<attempt-id>/. The manager may populate it from submitted files or transactional data/ by copying, reflinking, or a filesystem snapshot. Writes through an isolated workdir MUST NOT mutate committed data/ before a transaction.

Old isolated workdirs may be retained for diagnosis or collected later. A manual continuation can import selected files into a new isolated workdir, with the choice recorded in history.

Context and restart detection

The manager MUST set:

HTTK_WORKFLOW_CONTEXT=<absolute path to attempt context.json>
HTTK_WORKFLOW_CONTROL_DIR=<absolute attempt-control path>
HTTK_WORKFLOW_WORKSPACE_DIR=<absolute workspace path>
HTTK_WORKFLOW_JOB_DIR=<absolute current payload path>
HTTK_WORKFLOW_WORKDIR=<absolute selected workdir path>
HTTK_WORKFLOW_IS_RESTART=0|1
HTTK_WORKFLOW_UNCLEAN_RESTART=0|1
HTTK_WORKFLOW_DURABLE=0|1
HTTK_WORKFLOW_ATTEMPT_REASON=<reason>
HTTK_WORKFLOW_STEP=<current step>
HTTK_WORKFLOW_PYTHON=<manager Python interpreter>
HTTK_WORKFLOW_BASH_API=<absolute native workflow Bash library>

HTTK_WORKFLOW_DATA_DIR is additionally set only for transactional-data jobs. The JSON file is the source of truth; scalar environment variables are language-neutral conveniences.

HTTK_WORKFLOW_DURABLE, and the durable member of the attempt context, carry the workspace’s durability mode to the runner. A runner that publishes an outcome, a transaction, or a child bundle synchronizes it before the rename that makes it authoritative exactly when this is 1. A runner that does not compose outcomes on storage itself may ignore it; the packaged Python and Bash SDKs honour it automatically, and a context written before the member existed reads as 0.

A manager MAY export further variables that belong to the runner libraries it ships rather than to this protocol. They are SDK or application conveniences: a conforming manager that exports none of them is still conforming, and a runner that needs one MUST treat its absence as a missing dependency of that library rather than as a protocol violation. The ones this implementation exports are:

HTTK_WORKFLOW_VASP_BASH_API=<absolute native VASP Bash library>

An unclean persistent retry context is:

{
  "format": "httk-workflow-attempt-context",
  "format_version": 1,
  "workspace_id": "b588833b-87ea-4da2-b860-1c9e768cfbc1",
  "job_id": "01234567-89ab-cdef-0123-456789abcdef",
  "placement": "project-17/0/03a",
  "step": "relax",
  "activation_id": "e7f86a0e-34d6-45a7-b92d-3f4b2dc98c54",
  "attempt_id": "a6c2c973-29e1-44e2-9649-ae419e340ac4",
  "attempt_ordinal": 2,
  "is_restart": true,
  "is_unclean_restart": true,
  "attempt_reason": "lease_lost",
  "previous_attempt_id": "31fa431e-e01c-49aa-8fa8-e8af23b73c52",
  "activation_reason": "advance",
  "workdir_mode": "persistent",
  "workdir_reused": true,
  "data_generation": null,
  "durable": true,
  "resources": {},
  "join": null,
  "children": []
}

attempt_ordinal > 1 and is_restart mean that the same activation is being retried. is_unclean_restart specifically means that the preceding attempt did not publish and complete a valid outcome. Reasons include lease_lost, timeout, and process_failure. requested_retry and an orderly manual_continue may have is_unclean_restart: false.

Thus a shell or Python step does not infer restart from leftover filenames. It reads HTTK_WORKFLOW_CONTEXT (or the scalar variables), then uses the existing files according to application policy.

Attempt-control directories are transient metadata. Persistent workdirs are application data and MUST NOT be garbage-collected merely because an attempt ended. Isolated workdirs may be collected under their retention policy. No state transition depends on cleanup.

Publishing an outcome

Exit codes are not the workflow protocol. A step communicates in the attempt-control directory named by HTTK_WORKFLOW_CONTROL_DIR:

1. Create outcome.tmp.<nonce>/.

2. Write outcome.json and any transaction or child bundles inside it.

3. Close all files, and in the storage-durable profile synchronize the whole draft tree — the outcome, its transaction manifest and staged payload, and its child bundles — in one batch.

4. Atomically rename the directory to outcome.ready/, and in the durable profile synchronize the attempt-control directory so the new name survives a crash.

5. Exit.

The directory rename is the outcome publication point. Temporary outcomes are ignored. The fixed destination is nonempty, so a second publication MUST fail rather than replace the first. The draft is unreferenced until step 4, so a crash before it discards a partial outcome whole; the single batched synchronization at step 3 is why a published outcome can never name staged data the storage never received.

A minimal outcome is:

{
  "format": "httk-workflow-outcome",
  "format_version": 1,
  "job_id": "01234567-89ab-cdef-0123-456789abcdef",
  "activation_id": "e7f86a0e-34d6-45a7-b92d-3f4b2dc98c54",
  "attempt_id": "a6c2c973-29e1-44e2-9649-ae419e340ac4",
  "action": "advance",
  "next_step": "collect",
  "message": "relaxation converged"
}

expected_data_generation is required only when the outcome contains a transaction. It MUST equal the generation supplied in the attempt context. An outcome MAY contain priority, an integer from 0 through 999. The manager uses it for the marker produced by the committed action; omission preserves the current priority. This allows a workflow decision and its scheduling preference to share one atomic marker transition.

An outcome MAY also contain runner_steps, the array of step names the publishing runner implements. A manager copies a valid array verbatim into the state frame it writes and carries it forward, and ignores a malformed one with a log entry: the member is evidence for an operator or a tool drawing the reachable steps of a job, never an input of a manager decision. A runner that declares it SHOULD do so in the first outcome the job publishes.

Actions are:

Action	Required information	Effect after commit
advance	next_step	Create a ready activation for that step.
wait	next_step, join	Register children and wait for the join.
succeed	none	Enter successful terminal state.
fail	failure	Record a declared failure and enter failed.
retry	retry.reason	Retry this activation under policy.
pause	pause.reason	Require an operator request.

advance and retry first apply the total and per-activation budgets from job.json. If the requested next activation or attempt would exceed a limit, the committed effect is failed/budget_exhausted rather than another ready activation.

After observing a valid outcome, the manager appends a committing frame and renames running to committing before modifying durable job data or registering children. This fences the step and makes interrupted outcome replay explicit in the state tree.

If a process exits without an outcome:

• exit status zero is a protocol_error, because success is ambiguous;

• nonzero exit is process_failure;

• manager timeout is timeout;

• loss of manager/allocation is lease_lost.

Retry policy decides whether these create another attempt or retry_exhausted. A declared fail is permanent by default. A step requesting a managed retry uses retry.

A valid current outcome is authoritative over the process exit status. An outcome from a fenced attempt is retained only for diagnosis.

Optional transactional contributions to data/

Visibility guarantee

This section applies only when job.json declares "data": {"mode": "transactional"}. Such a step MUST NOT modify committed data/ directly. It publishes a replayable transaction in its outcome.

A job with data.mode equal to none omits data/ and the transaction bundle. In particular, a persistent-workdir job may freely update application files such as WAVECAR in run/; those files are not workflow-protocol metadata.

The required guarantee is:

A later attempt starts only after either none of a transaction or all of it has been applied to data/.

POSIX cannot atomically rename several unrelated paths. Raw observers looking inside data/ during committing may therefore see replay in progress. The atomic boundary is between workflow attempts: no runner is launched while the marker is committing.

Applications that require a simultaneously atomic tree for external readers MAY contribute one complete version directory and atomically replace a single current name. That is an application-level use of the same protocol, not a mandatory per-step revision directory.

Transaction bundle

.httk-attempt.<attempt-id>/outcome.tmp.<nonce>/
├── outcome.json
└── transaction/
    ├── manifest.json
    ├── payload/
    │   ├── results/energy.json
    │   └── restart/CHGCAR
    └── trash/

Example:

{
  "format": "httk-workflow-transaction",
  "format_version": 1,
  "id": "6fc3852b-f1df-4edf-a92c-7f81c3e02465",
  "expected_data_generation": 3,
  "operations": [
    {
      "id": "energy",
      "op": "put-file",
      "source": "payload/results/energy.json",
      "path": "results/energy.json",
      "sha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
    },
    {
      "id": "old-scratch",
      "op": "remove",
      "path": "scratch/obsolete.dat",
      "missing_ok": true
    }
  ]
}

Paths are normalized relative POSIX paths. Absolute paths, empty components, ., .., NUL bytes, and paths into protocol control data are forbidden. Operations MUST NOT overlap. Operation IDs are unique normalized protocol components and determine all replay scratch paths.

Required operation types are:

• make-dir: create an application directory and any explicitly declared missing parents;

• put-file: atomically install or replace one regular file;

• remove: atomically rename an existing path into transaction trash;

• put-tree: install a complete directory tree, normally at a previously absent path;

• replace-tree: move the old tree to transaction trash, then install the new tree.

Put operations declare a content digest. Replacement and removal operations MAY declare an expected old digest or require absence. Preconditions prevent a replayed or stale transaction from overwriting an unexpected workdir. Symlinks, devices, sockets, and FIFOs are forbidden by default.

Every removal uses the deterministic destination transaction/trash/<operation-id>/removed. Every replace-tree moves the old tree to transaction/trash/<operation-id>/old before installing its deterministic payload source. These destinations MUST NOT contain preexisting unrelated data. Replayers create the operation directory idempotently; no random trash name or replayer identity may affect the paths.

Idempotent replay

While the marker is committing, a manager applies operations in manifest order using atomic renames and the verified-transition algorithm:

• source present, destination not yet new: validate and rename source to destination;

• declared directory absent: create it; an already matching directory means make-dir already happened;

• source absent, destination has the declared new digest: the put already happened;

• removal target absent, matching trash entry present: removal already happened;

• both old tree and new source present during replace-tree: continue its defined two-rename sequence;

• any other combination: stop with transaction corruption rather than guess.

More than one manager may inspect an abandoned committing marker. The deterministic source, destination, and trash paths make concurrent replay converge: one rename wins and every other replayer verifies the same resulting path state. A replayer MUST perform the bounded visibility retries from the verified-transition algorithm before declaring an impossible combination.

The manifest and deterministic operation IDs supply all replay information. No per-operation progress marker files are required.

After all operations validate as applied, the manager:

1. appends the destination state frame with incremented data generation when the transaction changed data;

2. renames the exact committing marker to ready, waiting, succeeded, failed, or paused;

3. only then permits transaction trash and, in isolated mode, an old isolated workdir to be collected.

If interrupted after the data changes but before the final marker rename, a new manager sees committing, reads the outcome named by the committing frame, and idempotently completes replay. It does not rerun the step.

This retains the useful httk v1 ht.atomic.* principle while avoiding one permanent revision and manifest hierarchy per step.

Relocating and transferring jobs

This section requires relocation-v1, multiworkspace-v1, or detached-transfer-v1 as indicated. A core implementation does not create relocating or transferring states.

Arbitrary placement is dynamic. A job may move after submission, but a raw mv of an authoritative payload is not a state transition: the marker would still name the old placement. Relocation is therefore a short replayable protocol.

Relocation within one workspace

Only a quiescent job may relocate. ready, waiting, paused, failed, succeeded, and cancelled are quiescent. A claimed or running job must first be released or fenced; a committing job must finish replay.

To move from placement A to placement B, a manager:

1. validates that B/<job-key> does not exist;

2. appends a relocating frame containing source A, destination B, the prior logical state, priority, and exact expected record;

3. renames the exact marker from state/<old-kind>/A/ to state/relocating/A/, fencing scheduling;

4. atomically renames payload directory A/<job-key> to B/<job-key>;

5. appends a destination frame with placement B and the prior logical state;

6. renames the same marker from state/relocating/A/ to state/<old-kind>/B/.

The marker remains the sole authority throughout. Recovery from relocating uses these cases:

Source payload	Destination payload	Recovery
Present	Absent	Perform the payload rename and finish.
Absent	Present with matching job ID/digest	Payload rename happened; finish the marker transition.
Present	Present	Stop: destination collision or non-atomic copy.
Absent	Absent	Stop: payload loss.

Creation and later removal of empty placement parents are not state changes.

A batch relocation MAY move a common project/shard prefix with one directory rename. It first moves every affected job marker to relocating and journals one batch ID and the complete member set. Only after all members are fenced does it rename the common payload prefix. It then returns each marker to its mirrored destination placement. Recovery uses the batch record; no per-job coordinator files are required.

Moving new jobs into a running workspace

Files may be copied or generated under .httk-workflow/tmp/ while managers continue working. The complete payload is renamed to any chosen placement and becomes schedulable only when its submitted marker is published. A partial copy has no marker and is invisible.

A whole project tree containing many complete, unsubmitted payloads may be renamed into a placement prefix at once. Their markers are then published from a validated batch manifest. Marker publication is intentionally one job at a time; a batch requiring an all-at-once scheduling barrier should initially publish its members paused and release them through an explicit batch request.

If the source is on another filesystem, the copy must finish and be validated inside the destination workspace before marker publication. Atomic rename is relied on only for the final same-filesystem publication.

Moving jobs between workspaces

An individual job can transfer atomically between two workspaces only when both control trees and the payload source/destination are on the same filesystem. A coordinator attached to both workspaces:

1. moves the quiescent source marker to transferring, recording source and destination workspace IDs, placements, transfer ID, and prior logical state;

2. renames the payload into the destination workspace;

3. appends the import frame to a destination-workspace journal;

4. renames the same marker inode from the source transferring tree to the mirrored destination state tree.

Until step 4, the source transferring marker remains authoritative and points to both possible payload locations for recovery. The destination cannot run the job early.

Moving across filesystems is necessarily copy-and-acknowledge rather than one atomic filesystem transaction. The source job must first be sealed in transferring; the destination publishes a marker only after a complete copy and transfer-token validation; the source is retired only after explicit acknowledgement. A backend implementing this profile must document its duplicate-suppression and failure policy.

If the job may be named by an unresolved cross-workspace join, the source workspace MUST retain a packed forwarding record keyed by source workspace ID, job ID, job key, and old placement, naming the destination workspace and placement. It is retained for at least the maximum join-history period. A transfer implementation without such lookup support MUST reject transfer of a job participating in an unresolved join.

To detach one job without an immediately attached destination workspace, the manager moves it to transferring, writes one compact .httk-transfer/manifest.json inside the payload, and renames the authoritative marker into .httk-transfer/ inside that payload. The directory is then a sealed, nonschedulable bundle that can be moved out. Import places and validates the complete bundle first, appends an import frame, then renames the embedded marker into the target workspace’s state tree. The extra transfer metadata exists only while the job is detached or retained for transfer provenance.

For moving whole projects, a self-contained workspace is preferable: controlled detach and attach carry its state tree, journals, and all arbitrary placements together.

Dynamic branching and joins

Child publication

A step places complete child bundles in its outcome:

.httk-attempt.<attempt-id>/outcome.tmp.<nonce>/
├── outcome.json
└── children/
    ├── spawn.json
    └── jobs/
        ├── branch-a--<child-uuid>/
        │   ├── job.json
        │   └── ...
        └── branch-b--<child-uuid>/
            ├── job.json
            └── ...

Each child job.json names parent workspace, parent job, parent activation, and spawn ID, and — a clean pre-release requirement — the parent’s placement. spawn.json chooses the target workspace and arbitrary placement for each child. The parent placement lets the decided-join revival guard resolve the parent by probing its exact state set rather than scanning the workspace: a manager deciding whether a continue would revive a child a join already consumed reads the parent at that placement alone and never falls back to a whole-workspace scan inside a scheduling tick. The guard is advisory, so a child written before spawns carried the parent placement makes it probe nothing rather than rescan; a fresh child MUST carry it. Child UUIDs and tags are chosen before outcome publication. In the core profile, the target workspace MUST be the parent’s workspace; cross-workspace children require multiworkspace-v1.

Every spawn.json entry MUST also carry a label: a nonempty tag-syntax name that is unique within that spawn set. A gathering step selects its inputs by label, so a missing or ambiguous label makes the parent’s own join unusable and the manager rejects the published outcome with protocol_error instead of registering any child of the set.

While the parent is committing, the manager:

1. moves each complete child bundle to its chosen <workspace>/<placement>/<job-key> path;

2. creates its one g0.init marker at the mirrored target-workspace path below state/submitted;

3. treats an identical existing child plus marker as already registered;

4. fails on the same UUID with different immutable content.

Children are registered before the parent leaves committing. A crash may expose only some children, but replay registers the deterministic missing set. Registration is not rolled back. Published children may start before the parent has completed its transition; that is safe.

When a target workspace is on another filesystem, the child is first copied into that workspace’s temporary area and validated there. Its placement rename and submitted-marker publication then occur within the target filesystem while the parent remains committing.

Each child is an ordinary independently schedulable job with one job file, one state marker, its own attempts, and the ability to create more children.

Waiting and joining

A wait outcome explicitly names its child set:

{
  "action": "wait",
  "next_step": "aggregate",
  "join": {
    "children": [
      {
        "workspace_id": "b588833b-87ea-4da2-b860-1c9e768cfbc1",
        "job_id": "411d9e6e-c050-451d-a851-e20f2570d7c5",
        "job_key": "branch-a--411d9e6e-c050-451d-a851-e20f2570d7c5",
        "placement_hint": "project-17/0/041"
      },
      {
        "workspace_id": "b588833b-87ea-4da2-b860-1c9e768cfbc1",
        "job_id": "7ead0705-e1bb-4290-9ebd-fc1b24df9005",
        "job_key": "branch-b--7ead0705-e1bb-4290-9ebd-fc1b24df9005",
        "placement_hint": "project-17/0/042"
      }
    ],
    "condition": "all_succeeded",
    "on_impossible": {
      "action": "advance",
      "next_step": "handle_child_failure"
    }
  }
}

Supported conditions are:

• all_succeeded;

• all_terminal;

• any_succeeded;

• at_least, with a successful-child count.

The waiting journal frame contains each exact child identity, its spawn placement as placement_hint, and the condition. The placement is explicitly a lookup hint rather than identity. No join file or child marker is added to the parent directory. New unrelated descendants cannot affect the join.

A manager resolves each named child through this ladder, in order:

1. the finite set of state kinds at the child’s placement_hint, when the reference carries one — ordinary state transitions never change placement, so this is a bounded number of directory lookups and resolves the normal case;

2. its in-memory job-id-to-marker index, confirmed against the filesystem before it is used;

3. under relocation-v1 or multiworkspace-v1, any packed relocation or transfer forwarding record;

4. a complete marker scan of the named workspace.

A cache is never the only recovery path, and a cache miss is never the answer. In the core profile a clean pre-release break makes step 1 mandatory: every join child reference MUST carry both job_key and placement, join evaluation resolves each child by probing the finite state set at that exact placement alone, and it never falls back to a whole-workspace scan from inside a scheduling tick. A reference that lacks a placement is a protocol error of whatever published it, and the manager rejects the outcome rather than rescanning the workspace per child. The whole-workspace scan of step 4 survives for the extension forwarding of step 3 and for interactive resolution — job show, job why, job log, and harvest resolve — where locating an arbitrary and possibly finished job by one exhaustive scan is acceptable; it is never entered on the scheduling hot path. The consequence that matters at scale is that a waiting parent’s per-tick cost is independent of how many children its join names — each child is one hint lookup or one confirmed index hit — rather than one complete scan per child. Failure to find the child after bounded workspace visibility retries is an unavailable/corrupt dependency, not evidence that a join condition is impossible. Such a join is nevertheless never allowed to wait forever: because children are registered before their parent leaves committing, a child that stays unresolvable past a bounded grace is corrupt evidence, and the manager MUST fail the parent with dependency_failure and a message naming the missing child. This implementation records the first unresolved observation in memory and fails the parent after join_grace_seconds, one hour by default; a restarted manager restarts that grace, which is safe because the grace exists only to absorb transient nonvisibility.

In the core profile, a child named by an unresolved join MUST NOT relocate, so its placement hint remains valid. Extension implementations MAY relocate it only when they provide the forwarding and full-scan fallback above.

When satisfied, the manager appends a ready frame for a new activation and renames waiting to ready. Its attempt context summarizes the exact child state generation, terminal data generation, and failure information. Each observation is one object carrying the child’s workspace_id, label, job_id, job_key, placement, kind, state_generation, record_ref, its published failure object when it ended failed or cancelled, its data_generation, and the workspace-relative payload_path and workdir_path through which its results are read. The observations appear in the attempt context both as the join summary and as the children array, which is empty for an activation that follows no join. Child committed data may be exposed through read-only paths named in the context. A transactional parent imports selected child data through its own transaction. A non-transactional workflow may instead let application code inspect or copy child data into its persistent workdir according to its own conventions.

Join evaluation records an observation vector containing every child’s exact marker generation, state-frame reference, and state kind. The vector need not be a simultaneous filesystem snapshot; it is the complete evidence on which the parent transition was based, and each entry must have been the child’s current authoritative marker when observed. Once the parent marker transition commits, later manual continuation of a child does not change or retract that decision.

Success is currently impossible when the recorded observation vector contains enough terminal nonsuccess states to make the condition false:

• all_succeeded: any child is failed or cancelled;

• any_succeeded: every child is terminal and none succeeded;

• at_least N: succeeded children plus nonterminal children is less than N;

• all_terminal: never impossible merely because a child failed.

A manually continuable failed child still counts as terminal nonsuccess in the current vector. If success is currently impossible, on_impossible selects an error-handling step or the parent fails with dependency_failure. A later revival of that child does not retract the committed on_impossible transition. Cancellation is terminal but not successful.

on_impossible is optional and has exactly one defined form:

{"action": "advance", "next_step": "handle_child_failure"}

advance is the only legal action, and next_step MUST be a valid step name — any application-defined name, including the step the parent just ran. The member behaves exactly like an advance outcome: it starts a new activation of the parent at that step, carrying the observation vector as the join summary, so the error-handling step sees precisely what the satisfied step would have seen. An absent on_impossible, or one whose action is anything else, means the parent fails with dependency_failure; a manager MUST NOT invent another action, and MUST NOT treat an unrecognized one as a reason to keep waiting.

Reviving a child a decided join consumed

A join decision is final, but a child of a decided join is still an ordinary job, and continue or override_step on it would start writing its workdir and payload again — possibly while the parent activation that consumed it is reading exactly those files. Nothing in the protocol can order those two writers.

A manager MUST therefore refuse a continue or override_step request for a job that a decided join has already observed, unless the request explicitly carries "force": true. The check is a cheap, exact one and needs no index: a child’s job.json names its parent, and the parent’s current state frame carries the join_summary of the activation it is in, so the refusal applies precisely while the consuming activation is the parent’s current one. A refused request is retired with the reason, which names the parent, so an operator can continue the parent instead. With "force": true the manager applies the request and MUST record the hazard it accepted in the resulting state frame, as a revival_hazard object naming the parent, its state, and the observation that consumed this child.

An advance or succeed outcome may also publish detached children without a join.

Failures and tracking broken jobs

A step declares itself broken with a fail outcome:

{
  "action": "fail",
  "failure": {
    "code": "vasp.nonconvergent",
    "message": "electronic minimization did not converge",
    "details": {"iterations": 61, "log": "vasp/convergence.log"},
    "retryable": false
  }
}

There is exactly one failure object shape, used identically by application runners, by every language binding, and by the manager itself:

Member	Type	Required	Meaning
code	string	yes	Stable machine identity, one token without whitespace, at most 128 bytes. Matched against retry_policy.retry_on.
message	string	yes	One nonempty human sentence.
details	object	no	Structured evidence, such as exit_status or application counters.
retryable	boolean	no, default false	Advisory operator evidence. It never overrides retry_on.

No other member is permitted. A manager MUST validate a runner-published failure before recording it and MUST NOT store an unvalidated object. A malformed failure is itself a protocol violation: the job enters failed with the manager’s own protocol_error code and a message naming the defect, so a broken runner can never make a failure invisible to a consumer.

The manager embeds the structured failure in the failed state frame and moves the one marker to state/failed/<placement>/. No failure.json, ht.reason, per-job event directory, or second failure index marker is created.

Codes emitted by this manager itself are reserved. Those currently in use are:

• protocol_error — invalid submission, an outcome the protocol forbids, a malformed published failure, an unusable join, or a runner that exited successfully without publishing an outcome;

• process_failure — a runner that could not be launched, or that exited nonzero without publishing an outcome;

• lease_lost — the owning manager’s heartbeat expired;

• retry_exhausted — maximum_attempts_per_activation reached during retry;

• budget_exhausted — an attempt or activation budget exceeded;

• dependency_failure — a join became impossible, or a named join child stayed unresolvable past the manager’s bounded grace;

• transaction_corruption — a published transaction could not be replayed;

• runner_unavailable — a runner outside the payload could not be resolved, staged, or entered at all;

• runner_mismatch — the staged copy of such a runner did not match the runner.sha256 the job pinned.

A runner library that dispatches steps on a runner’s behalf publishes ordinary runner failures, so its codes are reserved too. Those of the runner libraries shipped with this implementation are:

• no_outcome — the step handler returned without publishing an outcome;

• unknown_step — the job asked for a step this runner does not implement;

• declared_failure — a legacy ht_steps task declared itself broken, published by the httk v1 compatibility runner only.

resource_unsatisfiable, manager_error, and cancelled remain reserved for manager use but are not currently emitted by this implementation. timeout is likewise reserved for a manager-enforced attempt timeout, which this manager does not yet apply; the httk v1 compatibility runner does publish it for a legacy task that exceeded its own timeout. Application codes SHOULD be namespaced, as in vasp.nonconvergent, to keep them distinct from the reserved set.

The failure frame contains job, step, activation, and attempt IDs; the failure object with its code, message, and details such as exit status or signal; retry history; manager ID; relevant retained log paths; and data generation. For a job with data.mode equal to none, data generation is null.

Current broken jobs are exactly the markers below state/failed/. This directory tree is authoritative and requires no reconciliation.

If a failed job is manually continued, its marker moves elsewhere, but its failed frame remains in the backwards-linked journal history. “Ever failed” queries are answered from a compact journal scan, an optional derived database, or a periodically generated report. They intentionally do not consume another per-job marker inode.

Logs are evidence, not state. Missing or truncated logs cannot prevent recovery.

Manual continuation and control requests

Operators MUST NOT edit state markers, journal segments, or job.json. They write one complete request file in requests/tmp/ and atomically rename it into requests/ready/ under the same name, so a manager never reads a partially written request and the publication is the same verified rename as every other one in this protocol. A request names:

• job UUID, job key, and the job’s placement;

• exact expected marker generation and record reference;

• requested action;

• operator identity and reason;

• optionally force, the operator’s explicit acceptance of a hazard the manager would otherwise refuse;

• any selected retained files for manual import;

• for relocation or transfer, the exact destination workspace and placement plus a unique operation ID.

Actions include:

• continue: retry the current activation;

• override_step: create a new activation at a named step;

• cancel;

• set_priority;

• pause;

• relocate, when relocation-v1 is enabled;

• transfer, when multiworkspace-v1 or detached-transfer-v1 is enabled.

Action validity is deliberately narrow:

• continue and override_step apply to failed or paused;

• set_priority applies only to submitted, ready, waiting, paused, or failed;

• an operator pause applies only to submitted, ready, or waiting;

• relocate and transfer apply only to the quiescent states permitted by their extension;

• cancel may target any nonterminal state and uses the explicit fencing and process-termination procedure of Cancellation for a live attempt. A second cancel of a job already in cancelling is not an error and changes nothing: the first one is still being carried out.

continue and override_step remain subject to job budgets unless the request contains an authorized, auditable budget change under site policy. Because job.json is immutable, such an override lives in the operator journal frame, not by editing the job definition. They are additionally refused, without force, for a job that a decided join already consumed; see Reviving a child a decided join consumed.

In particular, set_priority and operator pause MUST NOT rename a claimed, running, or committing marker behind its owning manager. An operator who must stop live work uses cancel; a pause can be requested after that work is released or fenced.

A manager claims the request by rename, verifies the exact expected current marker, appends the new journal frame, and renames that marker. It resolves the target by probing the finite state set at the request’s exact placement, the same clean pre-release break the join ladder makes, and never by scanning the workspace from a scheduling tick; a request that carries no placement is a protocol error, claimed and quarantined as malformed input rather than allowed to trigger a global lookup. A delayed request cannot apply to a newer state because its expected generation no longer matches. All request-induced marker moves use verified transitions. A manager whose live transition loses to cancellation rereads the current marker and stops the fenced attempt; it does not infer ownership merely from an errno.

The result is written to the shared journal. The transient request file may be removed after retention policy permits; there is no per-job request directory.

A claimed request lives in requests/claimed/<manager-id>/ until the manager that claimed it has decided about it: a request that was applied is removed, and one that can never become actionable is retired.

A request that has been claimed and can never become actionable — its expected generation or record reference no longer matches, the job moved while the request was being applied, or it asks for something the protocol refuses such as reviving a job a decided join already consumed — MUST be retired rather than left where it will be read again. Retiring means writing a retirement record and then moving the claimed file to requests/retired/; a retired request is never rescanned. The record sits beside the request under the request’s own name plus .retirement and is one object:

{
  "format": "httk-workflow-retired-request",
  "format_version": 1,
  "request": "3f2b0a1c-6c2e-4a2a-9a94-1b7c3e5f0d21.json",
  "manager_id": "b0d1e2f3-4455-6677-8899-aabbccddeeff",
  "reason": "the job is at generation 7, not the expected one",
  "retired_at": "2026-07-26T09:12:44.512314Z"
}

Both the retired request and its record are transient evidence for an operator, not protocol state: nothing reads them back, and “Retention gates and always-safe collection” below collects them after a month. This is distinct from two neighbouring cases that MUST NOT be retired:

• a request whose job is served by a runner backend this manager does not serve is left in requests/ready/ for a manager that does serve it. A manager that has decided this about a request SHOULD remember the decision rather than reread the file on every pass;

• a request that cannot be applied right now — a held maintenance lock, an unavailable workspace — stays actionable and is simply retried.

A request that violates the protocol, including one naming a job that does not exist, is quarantined as malformed input rather than retired.

Manual continuation preserves failure history. In persistent mode it reuses the declared workdir in place; the new attempt context records is_restart: true, its cleanliness, and attempt_reason: "manual_continue". No file import or transaction is required. In isolated mode, selected retained files may instead be imported into the new workdir; if they also become committed data/, that contribution uses a transaction.

Required outcome-processing order

For a valid current outcome, a manager:

1. validates job, activation, and attempt, plus expected data generation when a transaction is present;

2. appends a committing frame;

3. renames the exact running marker to committing, fencing the attempt;

4. applies or verifies the transaction, if present, and in the durable profile synchronizes every replayed destination and the directories it touched;

5. registers or verifies every child, synchronizing each published payload and marker’s directory in the durable profile;

6. computes failure or join information;

7. appends and synchronizes the destination state frame;

8. renames the exact committing marker to the destination;

9. performs optional cleanup later.

Steps 4 and 5 synchronize before steps 7 and 8, so a durable workspace has the committed data and the registered children on storage before the marker rename that claims them: a power cut can never leave a marker out of committing that names a transaction its storage only half received.

Interruption recovery follows directly:

Interruption point	Recovery
Before marker submission	No job was submitted; placed orphan may be retried or collected.
In submitted	Revalidate; move the same marker to ready or to failed with protocol_error.
Before outcome.ready	Ignore temporary outcome; retry under policy.
After outcome publication, before committing	Apply that outcome; do not rerun the step.
While a transaction is replayed	State remains committing; infer completed operations from source, destination, trash, and digests.
While children are registered	Verify existing children and register the missing set.
After destination frame, before marker rename	Frame is prepared but not current; replay and rename.
After marker rename	New state is already authoritative; cleanup is optional.
Old fenced process publishes late	Never apply it.

Before recovering a stale claimed or running job, a manager MUST inspect its named attempt-control directory for a valid published outcome. A committed decision must not be mistaken for an abandoned attempt.

Task-manager startup and recovery

A manager:

1. discovers explicit and watched workspaces and validates each format.json;

2. rejects duplicate workspace IDs and unsupported enabled extensions before mutation;

3. creates a manager record, fresh writer-incarnation journal, and heartbeat in each attached workspace;

4. resumes markers in committing and, when enabled, relocating and transferring;

5. resumes markers in cancelling: every one of them names a fenced attempt that still has to be stopped and verified, whether this manager fenced it or inherited it from one that died mid-cancellation;

6. examines possibly abandoned claimed and running markers;

7. evaluates waiting joins, including cross-workspace references when enabled;

8. handles submitted jobs and operator requests;

9. claims eligible ready work in pool, capability, priority, and resource order;

10. continues watching for workspaces being attached, renamed, or detached.

It does not need to reconcile a separate state index. Listing state/ is listing the authoritative scheduler state.

A high-scale implementation SHOULD:

• scan only non-terminal state placement prefixes relevant to its configured projects or job set;

• keep an in-memory job-key-to-marker map while running;

• process arbitrary placement subtrees incrementally rather than repeatedly scanning every attached workspace;

• use filesystem notifications only as hints, since notifications may be lost;

• workspace task-manager query caches outside per-job directories.

Discovering the globally highest-priority ready job requires enumerating the ready tree. Incremental scans MAY therefore cause long-lived operational priority inversion, especially after cold start; priority is a preference rather than a strict global ordering guarantee. This is not a gap an extension should close by adding directory levels — see the withdrawn priority-bands-v1 above — and no scan strategy changes claim correctness.

A manager MAY restrict every scan to a set of placement prefixes, the same kind of deployment policy by which it advertises pools and capabilities, so that disjoint managers divide a large workspace without walking each other’s trees. This is a restriction a manager places on itself and not a protocol change: placement values remain project-owned semantics that the engine only validates and filters on, overlapping assignments stay safe because the marker rename still arbitrates a claim, and a manager records its assigned prefixes in its manifest so a diagnosis can report when a live manager’s prefixes exclude a job’s placement. A manager with no assignment scans the whole workspace.

Workspace check and marker repair

A marker that no longer resolves to its journal frame is the one form of damage a manager cannot route around: the marker is authoritative, and everything beyond its kind, priority, and generation lives in the frame. A workspace check walks every marker of every state kind and verifies that its record reference resolves, within the configured visibility deadline, to a readable frame whose checksum verifies and whose workspace_id, job_id, job_key, kind, and state_generation agree with the marker name. The init reference of a submitted marker resolves to nothing by definition and is verified as such. Each unresolved marker is reported with a stable problem code: missing_segment, short_read, checksum_mismatch, length_mismatch, trailer_mismatch, reference_mismatch, invalid_header, undecodable_frame, invalid_record_ref, identity_mismatch, or unparseable_name for a marker-shaped entry whose basename cannot be interpreted at all.

Repair is optional, separate, and conservative. Because the frame that holds previous_record_ref is precisely the unreadable one, a repair cannot follow the chain of the job; it walks the journal segments instead, collects the readable frames naming that job, and adopts the newest one whose state_generation is strictly older than the marker’s. Adopting a frame at or beyond the marker’s own generation is forbidden: such a frame is either the damaged one or a transition that was appended and never committed by a rename, and publishing it would invent state no marker ever carried. The repair then appends one ordinary state frame with reason: "fsck_repair", whose previous_record_ref is the recovered frame and whose members are carried forward from it, keeping the marker’s own kind, placement, and priority, and renames the marker onto that frame at the next generation. A repair therefore adds history rather than rewriting it, and the job becomes loadable and schedulable again.

A repair MUST NOT touch a claimed, running, committing, or cancelling marker whose owning manager — identified from the readable frames of that job — is still heartbeating within its lease. Such a marker is reported only; its manager owns the transition that follows. cancelling belongs in that set for a reason of its own: the marker is the fence a live manager put in place and is still acting on, and re-pointing it at an older frame would take away the attempt identity that manager needs to finish stopping and verifying the process. A marker with no readable older frame is unrepairable and is likewise reported, and may be moved into .httk-workflow/quarantine/ only when an operator asks for that explicitly.

Garbage collection and compaction

The following may be collected under the explicit retention policy the workspace publishes in policy.retention:

• unpublished workspace temporary entries;

• placed payload directories that never reached submitted state, except sealed transfer bundles;

• journal frames not referenced by any marker or history chain;

• incomplete outcome directories;

• obsolete attempt-control directories;

• abandoned and completed isolated workdirs;

• transaction trash after the destination marker transition;

• retained diagnostic application files.

That list is permissive: it bounds what a conforming collector may touch, not what one must. The httk workflow workspace gc implementation collects the subset tabulated in “Retention gates and always-safe collection” below, and adds the retired transfer bundles and per-transfer receipts the detached-transfer-v1 extension accumulates. It deliberately leaves isolated workdirs, incomplete outcome directories, and payloads that never reached submitted alone, because each of those is the only remaining evidence of a job that went wrong.

Journal compaction operates on shared segments, not by creating files per job. It must preserve every record reference reachable from a current marker and the configured amount of history.

Terminal job.json, committed application data, the one state marker, and required journal history are retained according to site policy. Age alone is not permission to delete a non-terminal job.

A persistent workdir is application data, not attempt scratch. It is retained until an explicit job or site retention rule permits its removal, including after failure or manual continuation.

A payload containing .httk-transfer/manifest.json and its sealed marker is not an orphan even though it has no marker in a workspace state tree. Generic temporary/orphan GC MUST NOT collect, alter, or unseal it. Only an explicit transfer import, abort, or transfer-specific retention action may do so.

Entries in .httk-workflow/quarantine/ are likewise outside generic orphan GC. They are removed only by an explicit repair decision or a separately configured quarantine-retention policy that preserves an audit record.

Empty placement-directory pruning follows the nonrecursive race rules in “State-marker rename.” Implementations should account for these directories: deep or job-unique placements can temporarily leave one empty hierarchy under several state kinds even though the marker inode count remains one per job.

No recovery operation begins by broadly deleting tmp, run, or unknown files. Cleanup is separate from correctness.

Retention gates and always-safe collection

An absent member of policy.retention means keep. A collector MUST NOT prune a category whose limit is unconfigured; a workspace whose operator has said nothing therefore loses nothing.

Three categories are exempt because the entries in them cannot carry information, and are collected whatever policy.retention says:

• an empty placement mirror below a state kind, pruned by rmdir alone;

• an entry still sitting in .httk-workflow/tmp/ or .httk-workflow/requests/tmp/ more than 24 hours after it was written, since every publication renames its staging entry away within one operation;

• a request in .httk-workflow/requests/claimed/<manager-id>/ more than 30 days old whose manager is no longer heartbeating;

• a request in .httk-workflow/requests/retired/, and its .retirement record, more than 30 days old. A retired request was already decided about and is never rescanned, so it is evidence for an operator and nothing else.

The remaining categories are gated as follows.

Category	Gate	Additional condition
Attempt-control directory	attempt_control_days	The job is in succeeded, failed, or cancelled, and this is not its newest attempt-control directory.
Transaction trash	trash_days	The job’s marker has reached a quiescent kind, so the destination transition has happened and no replay consults the trash again.
Retired transfer bundle	trash_days	Below transfers/retired/; the ledger describing it is kept.
Import acknowledgement and import record	trash_days	Below transfers/acks/ and transfers/imported/.
Journal segment	journal_days	No current marker — nor the sealed marker of a bundle awaiting handover — references it, and its writer belongs to no manager heartbeating within its lease.
Manager directory	journal_days	The manager’s heartbeat is expired and none of its writer’s segments were retained.

The newest attempt-control directory of a terminal job is retained regardless of age: it holds the outcome, the failure breadcrumb, and the runner logs of the attempt that decided the job.

A collector MUST NOT prune the runner store. A runner is referenced by digest from job.json and from transfer manifests, an attached workspace can gain a job referring to one at any time, and the store is small; a future version may add an explicit runner-retention rule, but generic collection never applies one.

The cost of collecting journal history

Only the segment a marker’s record_ref points into is protected, not the whole chain behind it. Collecting older segments of the same writer is exactly what journal_days buys, and the consequence must be stated plainly: the deep history of an old job goes with those segments. harvest and job log then report that job’s timeline from whatever frames remain and set gaps — the job’s current state, marker, payload, and terminal outcome are unaffected, because they never depended on the older frames.

Crash safety and journaling of a collection

A collection MUST be interruptible at any instruction. It therefore removes entries bottom-up, renames nothing, and rewrites no protocol state: the remains of an interrupted removal are scratch that the next collection removes again. Empty-placement pruning tolerates ENOTEMPTY and ENOENT as ordinary outcomes, never as faults, because a concurrent transition recreating exactly that path is expected.

A collection that removed anything appends one summarizing frame to the journal:

{
  "format": "httk-workflow-gc",
  "format_version": 1,
  "workspace_id": "…",
  "collected_at": "2026-07-26T09:12:44.512314Z",
  "retention": {"journal_days": 30},
  "removed": 41,
  "bytes_reclaimed": 918273,
  "categories": {"attempt_control": {"candidates": 12, "removed": 12, "bytes_reclaimed": 40960}}
}

This is not a state frame, so every reader that walks the journal for job history ignores it. A collection that removed nothing writes no frame at all, since opening a journal writer creates a writer directory and an empty collection must not create the garbage it came to collect.

Manual filesystem inspection

The layout is intentionally legible without a database:

• state/ready/<arbitrary-placement>/ is the runnable queue, with p000 through p999 encoded in marker names and no priority directory level;

• state/running/ is everything presently believed to execute;

• state/committing/ is exactly the replay backlog;

• state/cancelling/ is every attempt already fenced by a cancellation whose exit has not been verified yet;

• state/waiting/ is the join backlog;

• state/failed/ is the current broken-job collection;

• state/succeeded/ is the finalized-success collection;

• .httk-workflow/quarantine/ contains malformed entries requiring workspace repair rather than workflow scheduling;

• every marker begins with the optional human tag plus UUID job key;

• the matching payload is at the same relative placement outside .httk-workflow;

• a first placement component such as project-17 groups a project without imposing a protocol-specific hierarchy.

Operators may read all of these paths. They must use a workflow control command rather than mv for state changes, because a correct transition must append the matching journal record and validate the expected old generation.

An inspection tool should accept any of:

• UUID;

• workspace UUID;

• exact job key;

• marker path;

• current payload path;

• placement-prefix query;

• tag query, returning all matches.

It follows the marker’s journal reference and backwards links to render state, step, retries, manager ownership, failure, children, and manual intervention history.

Security and hostile input

Task code is arbitrary code and should run under an appropriate OS, container, or batch-scheduler boundary. Managers additionally MUST:

• open paths relative to already opened job/run directories where possible;

• reject path traversal and forbidden file types;

• avoid following symlinks during protocol validation;

• limit JSON size, file count, nesting, journal-frame size, and payload size;

• treat all runner IDs and paths as untrusted;

• never interpolate a runner string through a shell;

• verify that outcome job, activation, and attempt IDs match the current frame;

• prevent a runner from writing immutable job metadata, the state tree, or the journal directly;

• for transactional jobs, additionally prevent direct writes to committed data/. Writes to the declared application workdir are allowed.

Persistent VASP restart example

A VASP workflow that wants traditional in-place execution can declare:

{
  "workdir": {"mode": "persistent", "path": "run"},
  "data": {"mode": "none"}
}

The behavior is then:

1. The first activation runs with run/ as its working directory, is_restart: false, and HTTK_WORKFLOW_UNCLEAN_RESTART=0.

2. VASP writes WAVECAR, CHGCAR, OUTCAR, and any other application files directly in run/. The manager does not copy, rename, or interpret them.

3. If execution disappears without publishing an outcome, the manager fences that attempt. After establishing that the old writer cannot still modify the workdir, it starts a new attempt in the same run/.

4. The replacement context says is_restart: true, is_unclean_restart: true, and, for example, attempt_reason: "lease_lost". The step may validate or remove partial files, adjust INCAR, and resume from WAVECAR using domain-specific logic.

5. If the step publishes advance, the next workflow step may also use the same run/, but it receives is_restart: false: this is its first attempt, not a restart merely because the workdir already contains files.

6. A later operator continue again reuses run/ and is explicitly identified as a restart. No transactional data/ output is involved.

For a POSIX shell step, the essential test can be as small as:

if [ "$HTTK_WORKFLOW_UNCLEAN_RESTART" = 1 ]; then
    repair_or_validate_vasp_restart_files
fi
exec vasp_std

The context JSON remains the full, versioned interface; the environment variables are convenient projections for simple runners.

Worked example

This example deliberately uses isolated workdirs in a workspace with transactional-data-v1 enabled. Job silicon-relax--J starts at prepare, creates two calculations, joins them, and finalizes:

1. Submission publishes its one marker as state/submitted/project-17/0/03a/<job-key>.p500.g0.init.

2. A manager validates job.json, journals ready record R1, and renames the same marker to state/ready/project-17/0/03a/...p500.g1.R1.

3. Manager M1 journals claim R2 and wins the rename to claimed.

4. M1 creates run.A1, journals running R3, renames the marker, and launches prepare.

5. prepare publishes one outcome containing transaction T1, children branch-a--C1 and branch-b--C2, and an all_succeeded join.

6. M1 journals R4, renames running to committing, replays T1, and publishes both child job directories and their one markers.

7. M1 journals waiting record R5 and renames committing to waiting.

8. Children run independently and may restart without changing the parent marker.

9. Once both succeed, a manager journals R6 and moves the parent marker to ready for aggregate.

10. aggregate publishes an advance outcome and transaction T2. The manager moves through committing, applies all files, increments data generation, and queues finalize.

11. finalize publishes succeed. The same marker that existed at submission is renamed to state/succeeded/....

12. Completed isolated workdirs are later collected. The payload retains one job file and application data; its one external state marker mirrors its arbitrary placement, and history is packed with many other jobs in journal segments.

At every interruption point, the marker location selects exactly one recovery rule.

Relationship to httk v1

The httk-v1 runner backend and httk-v1-taskmanager compatibility executor implement the following mapping for instantiated httk v1 task templates:

httk v1	This protocol
ht.task.<set>.<id>.<step>...<status>	One global state marker plus a packed state frame
Task-manager -s <set> eligibility	claim.pool and manager pool membership
Rename to *.running	Exact marker rename to claimed/running
Stale directory ctime	Lease evidence followed by marker fencing
ht.run.current	Persistent run/ or isolated run.<attempt-id>/
ht.nextstep plus exit code 2	advance outcome
Exit code 3 / waitsubtasks	wait outcome with explicit child set
Exit code 4 / broken	fail outcome and failed journal frame
ht.reason	Structured failure in packed history plus retained log
ht.tmp.task.* to ht.task.*	Child bundle to placed payload plus one marker
ht.tmp.atomic.* / ht.atomic.*	Idempotent adapter preflight replay before an httk₂ outcome
Restart count in pathname	Activation ID and attempt ordinal in state frame
ht.run.resume	Audited manual continuation, reusing a persistent workdir or explicitly importing into an isolated one

The adapter preserves the httk v1 ordering rule that a published ht_finished/broken decision or pending atomic transaction is completed without rerunning ht_steps.

httk v1’s restart counter was carried across steps but incremented only when a stale running task was adopted; it did not bound an indefinitely advancing clean workflow. httk₂ keeps the per-activation retry limit and adds optional total attempt and activation budgets for that separate concern.

Join migration is intentionally not a pathname-for-pathname emulation. httk v1 waitsubtasks searched the whole nested task subtree and could notice descendants that appeared later. An httk₂ join waits only for its immutable, explicit child set. A child that publishes detached grandchildren may complete without those grandchildren, so a migrated workflow that requires subtree completion must make the child join its own descendants before succeeding or name the additional jobs explicitly in the ancestor’s join.

The shipped adapter makes each discovered direct subtask an explicit child. Each such child applies the same rule recursively, so ordinary nested httk v1 task trees retain subtree completion. It uses all_terminal, rather than all_succeeded, because httk v1 resumed a waitsubtasks parent once no descendant remained in an active state; broken descendants did not keep it waiting. Legacy ht.task.*.<status> symlinks are derived operator views only. The httk₂ marker and journal remain authoritative.

The principal improvements are:

• one authoritative, atomically moving state entry rather than a state plus a potentially stale index;

• a constant one-marker state inode cost per job;

• transition, failure, and event metadata packed into shared journal segments;

• optional human-readable path tags without weakening UUID identity;

• arbitrary project/shard placement and crash-safe relocation;

• dynamic multi-workspace attachment for combining projects;

• explicit attempt fencing and restart detection;

• explicit child sets and join policies;

• transactional replay with no permanent revision-metadata tree per step;

• structured, durable manual and failure history.

The filesystem remains the interoperability layer. Its steady-state inode cost is a design property described by the arithmetic above, while capacity beyond the measured snapshot is an operational question for the target filesystem and should be evaluated with Scale benchmarks before a campaign is sized.