Migrating an httk v1 workflow to httk₂ in detail¶
For maintainers of an httk v1 workflow, moving it to httk₂ at whatever pace suits.
This guide takes an existing ht_steps or ht_run workflow from httk v1 to
httk-workflow. You can keep the workflow unchanged in a converted package,
migrate one job type at a time, or replace the legacy API completely with the
native httk₂ Bash or Python API.
The central rule is:
Migrate task definitions and newly instantiated task directories, not a live httk v1 task-manager queue.
The httk₂ manager never claims or rewrites an existing httk v1 queue tree. Once a task
has been prepared and submitted to an httk₂ workspace, its job.json, marker, and
journal are authoritative.
1. Choose a migration route¶
You do not need to migrate every workflow at once.
Route |
Workflow changes |
Manager |
Best use |
|---|---|---|---|
Converted package |
None, normally |
normal |
Establish an httk₂ operational baseline quickly |
Mixed |
Per job type |
the normal manager on one workspace |
Incremental migration with a direct fallback |
Native Bash |
Replace |
|
Preserve a shell-oriented workflow |
Native Python |
Replace the runner with Python calls |
|
New development and more structured logic |
Start with a converted package unless you already have tests that describe the workflow’s inputs, outputs, restart behavior, and child-task behavior. A package run gives you a useful reference result before semantics change.
2. Inventory the httk v1 workflow¶
Make a copy of an instantiated task directory and record:
whether the entry point is
ht_stepsorht_run;every sourced helper, especially
$HTTK_DIR/Execution/tasks/ht_tasks_api.shand$HTTK_DIR/Execution/tasks/vasp/vasptools.sh;use of
HT_TASK_ATOMIC_*,HT_TASK_CREATE,HT_TASK_SUBTASKS,HT_TASK_STORE_VAR, controlled processes, checkers, templates, or compression;files that must survive a retry, such as
WAVECAR,CONTCAR, or application checkpoints;files that are final results rather than attempt scratch;
task-set, priority, timeout, retry-limit, and resource assumptions;
any
ht.instantiate.pyimports from the old httk Python package (convert them as described in section 12);automatic VASP remedies on which the workflow depends;
child tasks that may still be running independently.
Do not infer completion from a legacy task-directory suffix alone. Capture the actual input files, logs, expected results, and exit behavior.
3. Run the workflow unchanged in an httk₂ workspace¶
Initialize an httk₂ workspace:
httk workflow workspace init workflow-workspace
The standalone workspace alias is equivalent:
httk-taskmanager init workflow-workspace
Wrap the task in a converted package and submit it through the normal path:
httk workflow job new workflow-workspace --workflow-dir ./legacy-package \
--placement migration/reference/silicon-relax
httk workflow run workflow-workspace --pool vasp --workers 4
The exact old source paths remain available below the compatibility
HTTK_DIR. An unchanged step may therefore continue to use:
source "$HTTK_DIR/Execution/tasks/ht_tasks_api.sh"
source "$HTTK_DIR/Execution/tasks/vasp/vasptools.sh"
Inspect the result through the httk₂ source of truth:
httk workflow workspace status workflow-workspace --json
The packaged v1 runner preserves the persistent ht.run.current/ workdir,
translates httk v1 decisions and dynamic subtasks, and completes published
httk v1 atomic sections after interruption. See
httk v1 task compatibility for the precise
compatibility boundary.
4. Migrate project, configuration, and remotes separately¶
These imports do not migrate workflow code or task queues.
Import safe user configuration explicitly:
httk workflow config import-v1
Create httk₂ project metadata from a local ht.project without modifying it:
httk workflow project import-v1 . --source ./ht.project
This imports safe metadata and public identities. It does not import private
keys or the httk v1 queue. Imported project metadata records
legacy_queue_imported: false. The imported project can record a workspace
default, but the core-v2 workspace itself remains outside the project; detached
transfer and transactional data are available to native jobs.
The workspace registry is machine-owned in httk₂. An old workspaces.json
is refused with a teaching error; remove it and re-register local workspaces
with workspace init PATH (remote names are registered on their owning
machine). workspace default NAME replaces project workspace bindings: it
records only the name in project.json, while the workspace remains outside
the project.
Recognized httk v1 computer definitions can be mapped explicitly into httk₂ remotes:
httk workflow remote import-v1 ~/.httk/computers/cluster-a \
--name cluster-a
httk workflow workspace init cluster-a:/remote/path/to/workflow-workspace \
--name default
workspace_root is retired. workspace init REMOTE:PATH performs the remote
initialization and registration; workspace settings set REMOTE:NAME … then
sets scheduler and application settings on that workspace. remote import-v1
does not create a workspace: it preserves the legacy Runs hint in the remote’s
legacy_settings so an operator can choose the path explicitly.
Review every generated adapter before installation. Legacy shell executables and credentials are not copied or executed by the importer.
5. Run converted and native jobs side by side¶
Converted v1 packages and native workflows are both ordinary jobs, so one
normal manager can serve them from the same httk₂ workspace. Select the
converted package’s taskset with the manager pool:
httk workflow run workflow-workspace --pool vasp --workers 2
httk workflow run workflow-workspace --pool vasp-native --workers 2
The shared core-v2 workspace already provides transactional data and detached transfer for native jobs.
Give the first native version a new job UUID and preferably a distinct tag and
placement. Do not edit the immutable job.json of an already submitted job to
change its runner executor.
Migrate one representative task first. Compare it with the compatibility reference before moving a larger batch.
6. Replace the httk v1 control flow with native Bash¶
A typical httk v1 runner looks like:
#!/usr/bin/env bash
source "$HTTK_DIR/Execution/tasks/ht_tasks_api.sh"
source "$HTTK_DIR/Execution/tasks/vasp/vasptools.sh"
HT_TASK_INIT "$@"
case "$STEP" in
prepare)
VASP_PREPARE_CALC
HT_TASK_NEXT run
;;
run)
VASP_PRECLEAN
VASP_RUN_CONTROLLED 86400 vasp_std
HT_TASK_NEXT collect
;;
collect)
HT_TASK_FINISHED
;;
esac
A relaxation may need no runner at all
The workflow below ships with the module, in Bash and in Python, as
vasp_relax.sh and vasp_relax.py. A campaign that wants the ordinary relaxation
submits jobs naming the installed file and writes nothing: see
Packaged VASP runners. Write your own when your practice differs from the packaged
one — starting from a copy of it.
The native Bash equivalent sources paths supplied by the manager and publishes structured outcomes:
#!/usr/bin/env bash
set -euo pipefail
source "$HTTK_WORKFLOW_BASH_API"
source "$HTTK_WORKFLOW_VASP_BASH_API"
httk_workflow_runner vasp.relax prepare run collect
step_prepare() {
local input
for input in POSCAR INCAR; do
if [ ! -e "$input" ]; then
cp -- "$HTTK_WORKFLOW_JOB_DIR/files/$input" "$input"
fi
done
httk_vasp_prepare \
--options "$HTTK_WORKFLOW_JOB_DIR/files/vasp-options.json"
httk_workflow_advance run
}
step_run() {
local status=0
httk_vasp_preclean --keep WAVECAR
if httk_vasp_run \
--timeout 86400 \
--report vasp-run-report.json \
-- vasp_std; then
httk_workflow_advance collect
return
fi
status=$?
if httk_vasp_remedy_plan \
vasp-run-report.json \
--output remedy.json; then
httk_vasp_remedy_apply remedy.json
httk_workflow_retry "reviewed VASP remedy applied"
return
fi
httk_workflow_fail vasp.failed \
"VASP stopped with status $status"
}
step_collect() {
httk_workflow_put OUTCAR results/OUTCAR >/dev/null
httk_workflow_put OSZICAR results/OSZICAR >/dev/null
httk_workflow_succeed
}
httk_workflow_main
The outcome functions publish exactly one decision and then return:
httk_workflow_main owns the process exit status. Do not additionally return a
legacy decision code or write ht.nextstep.
The collect example uses transactional data. Its workspace is core-v2:
httk workflow workspace init native-workspace
If the job uses data.mode: "none", omit the transaction and keep restartable
working files in its persistent workdir instead.
Prepare the native payload¶
Put the runner and static inputs below one payload directory:
mkdir -p native-job/files
cp run.sh vasp-options.json POSCAR INCAR native-job/files/
chmod +x native-job/files/run.sh
Create the immutable job.json through the Python builder:
from httk.workflow import JobSpec, prepare_job_payload
prepare_job_payload(
"native-job",
JobSpec(
name="silicon relaxation",
workflow="example.vasp-relax",
runner_path="files/run.sh",
initial_step="prepare",
tag="silicon-relax",
workdir_mode="persistent",
data_mode="transactional",
priority=700,
claim_pool="vasp-native",
maximum_attempts_per_activation=5,
maximum_total_attempts=20,
),
)
Submit and run it:
httk workflow job submit native-workspace native-job \
--placement migration/native/silicon-relax
httk workflow manager run native-workspace \
--pool vasp-native \
Static payload files are available below HTTK_WORKFLOW_JOB_DIR; the selected
workdir is HTTK_WORKFLOW_WORKDIR. Copy or link static inputs into the
workdir in an explicit preparation step when necessary.
7. Translate the commonly used task helpers¶
The native API is intentionally not a spelling change of the httk v1 API.
httk v1 operation |
Native Bash |
Native Python |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
run-log helpers |
|
|
|
|
|
|
|
|
compress/uncompress |
explicit |
|
|
declared attempt resources |
|
State values are JSON, not sourced shell assignments:
httk_workflow_state_set relaxation_index 3
httk_workflow_state_set phase '"ionic"'
relaxation_index=$(httk_workflow_state_get relaxation_index)
Templates use string.Template placeholders and an explicit JSON value file:
# INCAR.template
ENCUT = $ENCUT
SYSTEM = $SYSTEM
{
"ENCUT": 520,
"SYSTEM": "silicon"
}
httk_template_render INCAR.template INCAR template-values.json
There is no shell eval in the native template or arithmetic implementation.
8. Replace controlled-run checkers¶
For simple programs, use argv-only supervision directly:
if httk_workflow_run \
--timeout 3600 \
--report process-report.json \
--stdout program.out \
--stderr program.err \
-- simulation --input input.dat; then
httk_workflow_advance collect
else
status=$?
httk_workflow_retry "simulation stopped with status $status"
fi
For application-specific monitoring, write a checker spec:
{
"format": "httk-workflow-checker-spec",
"format_version": 1,
"argv": ["./checker.py"],
"required": true,
"sources": [
{
"path": "progress.log",
"name": "progress",
"inactivity_timeout": 600
}
]
}
This example assumes the executable checker and its specification were copied from the immutable job payload into the workdir during preparation.
The executable reads httk-workflow-checker-event JSON lines from stdin and
emits versioned results on stdout:
#!/usr/bin/env python3
import json
import sys
for line in sys.stdin:
event = json.loads(line)
if event["event"] == "line" and "FATAL" in event.get("line", ""):
print(
json.dumps(
{
"format": "httk-workflow-checker-result",
"format_version": 1,
"code": "application_fatal",
"severity": "fatal",
"summary": "application reported a fatal error",
"source": event["source"],
"evidence": event["line"],
"stop": True,
}
),
flush=True,
)
Invoke it without constructing a shell command:
httk_workflow_run \
--checker checker.json \
--timeout 3600 \
-- simulation --input input.dat
Checker diagnostics belong on stderr. Do not reproduce the httk v1 signal, temporary-message-file, or process-discovery convention.
9. Replace VASP helpers¶
VASP input choices can be recorded in JSON:
{
"kpoint_density": 40.0,
"centering": "Gamma",
"accuracy_per_atom": 0.001,
"pseudopotential_library": "/data/vasp/potpaw_PBE",
"parallel_tag": "NPAR",
"parallel_value": 4,
"normalize_handedness": true
}
The native Bash operations include:
httk_vasp_prepare --options vasp-options.json
httk_vasp_get_tag EDIFF INCAR
httk_vasp_set_tag ISYM 0 INCAR
httk_vasp_prepare_kpoints 40 --centering Gamma
httk_vasp_prepare_potcar /data/vasp/potpaw_PBE
httk_vasp_nbands --divisor 4
httk_vasp_preclean --keep WAVECAR
httk_vasp_run --timeout 86400 -- vasp_std
httk_vasp_energy OSZICAR
httk_vasp_volume vasprun.xml
httk_vasp_promote_contcar
httk_vasp_clean_outcar
Important behavioral differences are:
diagnostics work with VASP 5 and VASP 6 output;
input diagnosis never changes files;
remedies are bounded proposals under the explicit
reviewed-v1policy;httk_vasp_remedy_applyis a separate, auditable mutation;remedy history records before and after input digests;
rerun cleanup is explicit and can preserve named files;
commands are argv arrays and never interpolated shell strings.
The old Python modules httk.task.ht_tasks_api and
httk.task.vasptools are not available in httk₂. Use the public functions in
httk.workflow instead. Their independent design and the prior httk v1
contributor work are described in the packaged compatibility NOTICE.
10. Migrate dynamic subtasks¶
Do not recreate the httk v1 ht.task.<set>...waitstart filename protocol in a
native workflow. Prepare explicit child payloads and publish their identities
with the parent outcome.
A Python parent can create a fixed child set as follows:
import shutil
import tempfile
import uuid
from pathlib import Path
from httk.workflow import JobSpec, Runner, prepare_job_payload
run = Runner("example.volume-scan")
@run.step
def branch(a):
with tempfile.TemporaryDirectory(dir=a.workdir) as draft_root:
for index, parameter in enumerate(("0.95", "1.00", "1.05")):
child = Path(draft_root) / f"child-{index}"
shutil.copytree(a.payload / "files" / "child-template", child)
(child / "parameter.txt").write_text(
parameter + "\n",
encoding="utf-8",
)
child_id = uuid.uuid5(
uuid.UUID(a.context.job_id),
f"volume-{index}",
)
prepare_job_payload(
child,
JobSpec(
name=f"volume point {index}",
workflow="example.volume-point",
runner_path="files/run.py",
tag=f"volume-{index}",
job_id=str(child_id),
initial_step="run",
claim_pool="vasp-native",
),
)
a.spawn(
child,
label=f"volume-{index}",
placement=f"volume-scan/{index:03d}",
)
a.gather("collect", when="all_terminal")
A child whose steps live in the same runner needs no payload at all: publish the
runner once in the workspace and spawn a ChildSpec, which synthesizes the whole
child job from its step and parameters and inherits the parent’s runner reference.
a.spawn(ChildSpec(step="run", parameters={"scale": parameter}), label=f"volume-{index}")
Use all_succeeded when any failed child should make the join impossible.
all_terminal most closely matches the compatibility behavior in which a
broken descendant no longer counts as active. Other native conditions are
any_succeeded and at_least.
A Bash step spawns the same children by step and parameters, and gathers exactly the ones it spawned:
for index in 000 001; do
httk_workflow_spawn "volume-$index" \
--step run \
--parameter scale="0.$index" \
--placement "volume-scan/$index" >/dev/null
done
httk_workflow_gather collect --when all_terminal
The complete child set is sealed with the outcome. A native parent cannot add untracked children after publication. Child UUIDs must remain stable if the parent recreates the same unpublished outcome after an interrupted attempt.
11. Migrate to a Python runner¶
A native Python VASP runner can express the same steps without a shell facade:
#!/usr/bin/env python3
import shutil
from httk.workflow import (
Runner,
VaspPreparationOptions,
apply_vasp_remedy,
plan_vasp_remedy,
prepare_vasp_inputs,
run_vasp,
)
run = Runner("example.vasp-relax")
@run.step
def prepare(a):
for name in ("POSCAR", "INCAR"):
destination = a.workdir / name
if not destination.exists():
shutil.copy2(a.payload / "files" / name, destination)
prepare_vasp_inputs(
VaspPreparationOptions(
kpoint_density=40,
centering="Gamma",
pseudopotential_library="/data/vasp/potpaw_PBE",
parallel_tag="NPAR",
parallel_value=4,
),
directory=a.workdir,
)
a.advance("run")
@run.step(name="run")
def run_step(a):
report = run_vasp(["vasp_std"], directory=a.workdir, timeout=86400)
if report.classification == "completed":
a.advance("collect")
return
history = ".httk-vasp/remedies.json"
decision = plan_vasp_remedy(report.diagnostics, history_path=history)
if not decision.give_up:
apply_vasp_remedy(decision, directory=a.workdir, history_path=history)
a.retry("reviewed VASP remedy applied")
return
a.fail(
"vasp_failure",
f"VASP stopped with classification {report.classification}",
details=report.as_mapping(),
)
@run.step
def collect(a):
a.put(a.workdir / "OUTCAR", "results/OUTCAR")
a.put(a.workdir / "OSZICAR", "results/OSZICAR")
a.succeed()
if __name__ == "__main__":
raise SystemExit(run.main())
There is no step-dispatch chain and no unknown_step branch to write: Runner.main
dispatches the step the manager asked for, and reports an unimplemented step, a step
that published nothing, and a step that raised as the corresponding outcome.
As in the Bash example, the transaction requires a transactional-data job in a core-v2 workspace.
12. Converting your ht.instantiate.py¶
In v1, create_batch_task copied the template, changed into the new task
directory, and exec’d ht.instantiate.py with the args dictionary as its
globals. The script wrote the task files and could set finalname in args to
name the task.
The script only wrote the structure¶
This is the overwhelmingly common case: every template shipped with v1 did exactly this. You need no code. Use a packaged template, or declare the input on your own Python runner:
run = Runner("example.structure", inputs={"structure": "POSCAR"})
Callers pass the structure object. The scaffold writes it to files/POSCAR
using the registered writer (the httk-atomistic distribution
provides the POSCAR writer). A path input is copied instead; for example,
the Python API uses new_job(ws, workflow, inputs={"structure": obj}),
and a batch uses new_jobs(...) items such as
{"inputs": {"structure": obj}}. The command-line spellings are
documented in Project and workflow command line in detail.
The script produced derived creation-time files¶
If everything the script produced was derivable from its arguments, declare a
input for every declared object that defines workflow equivalence. Use a
destination for values that can be staged directly and None for values
consumed by a creation hook. Keep implementation knobs outside the declaration
as job parameters. Move the remaining logic to @run.instantiate:
from httk.workflow import Runner
run = Runner(
"example.supercell",
inputs={"structure": "POSCAR"},
)
@run.instantiate
def instantiate(ctx):
from httk.atomistic import build_supercell
from httk.core import save
result = build_supercell(ctx.inputs["structure"], ctx.parameters["supercell"])
save(result.structure, ctx.payload / "files" / "POSCAR")
ctx.suggest_tag("supercell")
For a v1 script whose args contained structure and supercell, the
mapping is:
v1 |
v2 |
|---|---|
|
declared |
script current directory |
|
writing a file |
a declared destination, or a write below |
|
|
a value the runner needs later |
|
Declarative staging happens before the hook. ctx.inputs is read-only;
ctx.parameters is mutable and becomes the job’s opaque parameter mapping. The hook runs
in-process on the creating machine. See Native runner helpers in detail for the hook
reference and Project and workflow command line in detail for --parameter NAME=VALUE and
--input-from NAME SOURCE....
The work belongs at run time¶
If the script’s work is really run-time preparation — for example, deriving
INCAR values or k-points, as v1 ht_steps commonly did — put it in the
runner’s prepare step. Use the instantiate hook only for creation-time work
that needs the supplied domain objects or must happen before submission.
The hook is template code imported and executed at creation on the creating
machine, with the same trust and locality implications as v1’s exec. The
template’s Python file must therefore be available there. Bash runners do not
support @run.instantiate.
13. Validate before switching production work¶
For each migrated job type:
Run one fixed input through the converted package’s normal path runner.
Run the same fixed input through the native runner under a new UUID.
Compare prepared
INCAR,KPOINTS, POTCAR metadata, final energies, structures, and retained result files.Compare failure classification and retry limits.
Stop the runner during preparation, execution, remedy application, and result publication; verify that restart does not duplicate work or lose the authoritative outcome.
Exercise a failed child as well as an all-successful child set.
Verify
httk workflow workspace status workflow-workspace --jsonand the journal rather than relying on directory names.Run several jobs with the intended pool, capabilities, resources, and worker count.
Keep the original template and the known-good converted payload until the native result has passed these checks.
14. Cut over and retire compatibility deliberately¶
Stop instantiating new converted jobs first. Let submitted httk v1 jobs reach a terminal state or cancel them through recorded operator requests:
httk workflow job request workflow-workspace JOB_UUID cancel \
--operator "$USER" \
--reason "replaced by validated native workflow"
Then stop the normal manager serving the converted package’s pool while leaving the native pool’s manager running. Retain the legacy source, its attribution, and reference results for reproducibility.
Do not delete or reinterpret the old queue as part of cutover. Archive it read-only according to the project’s provenance and retention policy.
15. Wrap an existing template as a package¶
An existing template directory can become a package without first rendering it:
silicon-relax/
├── httk_workflow.toml
├── ht_steps.template
├── ht.instantiate.py
├── INCAR.template
└── collect.py
Use a manifest that makes the v1 contract explicit:
[workflow]
id = "legacy.silicon-relax"
[workflow.runner]
language = "httk-v1"
taskset = "vasp"
attempts = 10
[workflow.inputs.structure]
entry_type = "structures"
[workflow.parameters.encut]
type = "number"
default = 520
[workflow.collect]
file = "collect.py"
Pre-rename alpha jobs carrying the workflow_postprocess parameter lose
package-hook collection; re-scaffold them.
Submit a one-shot job or a structure campaign:
httk workflow job new WS --workflow ./silicon-relax \
--format httk-v1 --input-from structure structures/*.cif --parameter encut=520
httk workflow run WS --pool vasp
httk workflow collect WS
At preparation, the package is snapshotted and each job gets its own rendered
payload. ht_steps or ht_run must be executable after rendering. The v1
template engine is intentionally trusted and supports $name, $(expr),
${code}, escaped \$, and .template filenames. Its implementation is
available as apply_templates in httk.workflow.compat.v1.templates.
ht.instantiate.py receives declared inputs and parameters as globals. A
path-valued input with entry_type is loaded through httk.core.load before
that execution, which is why a CIF campaign can feed structure objects. The
script is still v1 Python, not a compatibility promise for every old import:
an ht.instantiate.py written against v1’s own Python API (for example v1
Structure) works only when it receives objects compatible with that API. Port
the script or supply compatible objects when it crosses this boundary.
16. Harvest old result trees¶
Use finished_tasks to inspect a tree and collect_finished_tree to run its
package collector against every finished task:
from httk.workflow.compat.v1 import collect_finished_tree, finished_tasks
for task in finished_tasks("/archive/ht-results"):
print(task.task_id, task.rundir, task.code_name, task.code_version)
collected = collect_finished_tree(
"/archive/ht-results", workflow_dir="./silicon-relax"
)
The CLI equivalent is:
httk workflow v1 collect /archive/ht-results \
--workflow-dir ./silicon-relax --into results.sqlite
The harvester selects the latest dated ht.run.*, reads code metadata from
lines 2–3 of ht_steps or ht_run, and calls the authored hook using
run_directory, code_of, and task_file from
httk.workflow.compat.v1. A per-task hook failure degrades that task and the
sweep continues. Manifest-backed UUIDv5 identity survives tree relocation;
path-derived identity does not.
17. What stays behind¶
The compatibility layer does not recreate every v1 subsystem:
v1 surface left behind |
v2 replacement |
|---|---|
ssh/rsync computer templates and send/receive transport |
v2 remotes and transfer protocol |
openmaterialsdb submission and signing arc |
v2 project manifests, keys, and remote transfer |
|
declared workflow inputs, opaque parameters, and manager policy |
|
explicit v2 managers and workers |
runtime priority rewrites |
immutable job priority and recorded operator requests |
These are migration boundaries, not hidden package options. Keep a v1 installation only where the packaged runner’s trusted runtime or a template still needs it.
Migration checklist¶
An instantiated httk v1 task runs successfully through its converted package.
Project/configuration/remote imports were reviewed separately.
No live httk v1 queue is being treated as an httk₂ workspace.
Persistent scratch and committed result files are distinguished.
The core-v2 workspace matches the native job’s data model.
Every
HT_TASK_*andVASP_*dependency has an explicit replacement.Automatic remedies became explicit plan-and-apply decisions.
Child jobs use stable identities and an explicit join condition.
Native Bash commands use quoted argv elements and no
eval.Compatibility and native reference results agree.
Restart and interruption boundaries were exercised.
Every
ht.instantiate.pyis converted to declared parameters or@run.instantiate.New production submissions use native payloads and new UUIDs.
For API details, continue with Native Bash runner API, Native runner helpers in detail, and Workflow filesystem API in detail.