httk.workflow.scaffold

Scaffolding submitted jobs from a template, some files, and some inputs.

A job is a payload directory plus a job.json that names the runner to execute, and building one by hand means knowing the runner’s workflow name, its initial step, its digest, and where its inputs live in the payload. This module is the short way: new_job() takes a template — a packaged runner a domain registered by name, or the path of a runner file of your own — stages the files the runner reads, writes the job.json, and submits the result, all in one call.

Packaged templates are not known to this module. A domain or compat engine registers each one it ships with register_template(), supplying only the generic description of a starting point — its name, the runner it starts from, the workflow and steps that runner declares, the modes a job of it defaults to, and what it does — so the scaffold resolves and pins a template without ever importing the science that owns it.

from httk.workflow import Workspace
from httk.workflow.scaffold import new_job

workspace = Workspace.initialize("workflow-workspace", extensions=["transactional-data-v1"])
job = new_job(workspace, "some-template", files={"input": "input"}, tag="example")
print(job.job_key, job.payload)

By default the runner file is published into the workspace runner store, and the job references it there by digest. That is what makes a scaffolded job durable: the bytes that will run are pinned in the workspace, so upgrading the installed httk-workflow underneath a queued campaign cannot change what its jobs execute. Publication is content addressed — the store name carries the digest of the bytes — so scaffolding the same template twice publishes nothing the second time, and a later, different version of a packaged runner lands beside the old one instead of replacing it. publish="installed" instead references a packaged template through the reserved pkg: form, which copies nothing at all.

new_jobs() is the same operation for a campaign: one template resolution and one publication amortized over every job, and a lazy iterator over the results. By design, generating a partitioned campaign costs one payload and one marker per job and never materializes a list of them.

Attributes

JOB_SCAFFOLD_FORMAT
DEFAULT_PLACEMENT
FILES_DIRECTORY
STRUCTURE_PATTERNS

Classes

TemplateProvider	One packaged template a domain or compat engine offers by name.
JobItem	What one job of a new_jobs() campaign varies from the shared values.
JobTemplate	One resolved starting point for a job: a runner file and how to run it.
ScaffoldedJob	One job this module submitted, and everything needed to look at it again.

Functions

register_template(→ None)	Register one packaged template, replacing any registered under its name.
registered_templates(→ tuple[str, Ellipsis])	Return the name of every registered template, in registration order.
template_provider(→ TemplateProvider | None)	Return the provider name names, by template name or runner file.
describe_runner(→ dict[str, object])	Return the self-description one runner file prints, by running it.
packaged_template(→ JobTemplate | None)	Return the registered template name names, or None.
resolve_template(→ JobTemplate)	Return the JobTemplate template names.
structure_files(→ list[pathlib.Path])	Return every structure file of one directory, in a stable order.
structure_tag(→ str | None)	Return the job tag one structure file name suggests, or None.
new_job(→ ScaffoldedJob)	Scaffold, submit, and describe one job of template.
new_jobs(→ collections.abc.Iterator[ScaffoldedJob])	Scaffold and submit one job per member of items, lazily.
payload_relative(→ pathlib.PurePosixPath)	Return where one staged file lands inside a payload.

Module Contents

httk.workflow.scaffold.JOB_SCAFFOLD_FORMAT = 'httk-workflow-job-scaffold'

httk.workflow.scaffold.DEFAULT_PLACEMENT = 'jobs'

httk.workflow.scaffold.FILES_DIRECTORY = 'files'

httk.workflow.scaffold.STRUCTURE_PATTERNS = ('POSCAR*', '*.vasp')

class httk.workflow.scaffold.TemplateProvider

One packaged template a domain or compat engine offers by name.

A provider is the generic description of a starting point. It names the packaged runner it starts from by the package the runner file is a module of and the file beside that module — so the reserved pkg: form and the digest are resolved from the provider alone — and it declares the runner’s workflow, its steps, the modes a job of it defaults to, and what it does. Declaring the steps here rather than running the runner to ask keeps scaffolding cheap; the owning domain’s tests hold the declaration to what the runner really describes.

name: str

runner_package: str

runner_file: str

workflow: str

initial_step: str

steps: tuple[str, Ellipsis] = ()

data_mode: DataMode = 'none'

workdir_mode: WorkdirMode = 'persistent'

summary: str = ''

httk.workflow.scaffold.register_template(provider: TemplateProvider) → None

Register one packaged template, replacing any registered under its name.

A domain calls this once per template it ships when its package is imported, which is how httk workflow job new --template NAME resolves a packaged runner the generic scaffold never names.

httk.workflow.scaffold.registered_templates() → tuple[str, Ellipsis]

Return the name of every registered template, in registration order.

httk.workflow.scaffold.template_provider(name: str) → TemplateProvider | None

Return the provider name names, by template name or runner file.

class httk.workflow.scaffold.JobItem

Bases: TypedDict

What one job of a new_jobs() campaign varies from the shared values.

Every member is optional, and a member that is absent takes the value new_jobs() was called with. inputs and files are merged over the shared mappings key by key; everything else replaces the shared value.

inputs: collections.abc.Mapping[str, object]

files: collections.abc.Mapping[str, str | os.PathLike[str]]

tag: str | None

name: str

placement: str | pathlib.PurePosixPath

priority: int | None

class httk.workflow.scaffold.JobTemplate

One resolved starting point for a job: a runner file and how to run it.

A template is either one of the runners a domain registered by name — see registered_templates() — or a runner file of your own, which is described by running it — every native runner answers --describe with its workflow and its steps — so a scaffolded job never guesses either.

name: str

source: pathlib.Path

workflow: str

initial_step: str

steps: tuple[str, Ellipsis] = ()

data_mode: DataMode = 'none'

workdir_mode: WorkdirMode = 'persistent'

packaged: str | None = None

summary: str = ''

property store_name: str

The content-addressed name this template takes in a runner store.

The digest of the bytes is part of the name, so publishing is idempotent for identical bytes and never overwrites a name a submitted job pinned: an upgraded packaged runner is published beside the version its queued jobs still reference.

class httk.workflow.scaffold.ScaffoldedJob

One job this module submitted, and everything needed to look at it again.

job_id: str

job_key: str

tag: str | None

placement: pathlib.PurePosixPath

payload: pathlib.Path

marker: pathlib.Path

workflow: str

initial_step: str

template: str

runner: collections.abc.Mapping[str, object]

as_mapping() → dict[str, object]

Return the machine-readable report of this job.

httk.workflow.scaffold.describe_runner(runner: str | os.PathLike[str]) → dict[str, object]

Return the self-description one runner file prints, by running it.

Every native runner — Python or Bash — answers HTTK_WORKFLOW_DESCRIBE=1 with its workflow name and its registered steps and exits without touching anything, which is how a runner nobody wrote a template for is still scaffolded without being told what it implements.

httk.workflow.scaffold.packaged_template(name: str) → JobTemplate | None

Return the registered template name names, or None.

httk.workflow.scaffold.resolve_template(template: str | os.PathLike[str], *, workflow: str | None = None, step: str | None = None, data_mode: DataMode | None = None) → JobTemplate

Return the JobTemplate template names.

template is the name of a packaged template, the file name of a packaged runner, or the path of a runner file. A runner file is described by running it, so its workflow name and its steps come from the runner itself; workflow and step override what it said, and step is required when a runner registers several steps and none of them is start.

httk.workflow.scaffold.structure_files(directory: str | os.PathLike[str]) → list[pathlib.Path]

Return every structure file of one directory, in a stable order.

A structure is a file matching one of STRUCTURE_PATTERNS — the VASP conventions POSCAR, POSCAR.something, and something.vasp — which is what makes a directory of structures one campaign.

httk.workflow.scaffold.structure_tag(path: str | os.PathLike[str]) → str | None

Return the job tag one structure file name suggests, or None.

The tag is the part of the name that identifies the structure — Si2O of POSCAR.Si2O, fcc-al of fcc-al.vasp — reduced to the tag syntax the protocol allows. A name that says nothing beyond POSCAR suggests no tag.

httk.workflow.scaffold.new_job(workspace: httk.workflow.workspace.Workspace, template: str | os.PathLike[str], *, inputs: collections.abc.Mapping[str, object] | None = None, files: collections.abc.Mapping[str, str | os.PathLike[str]] | None = None, tag: str | None = None, placement: str | pathlib.PurePosixPath = DEFAULT_PLACEMENT, priority: int | None = None, workdir_mode: WorkdirMode = 'persistent', data_mode: DataMode | None = None, publish: PublishMode = 'workspace', step: str | None = None, workflow: str | None = None, name: str | None = None) → ScaffoldedJob

Scaffold, submit, and describe one job of template.

template is a registered template name — see registered_templates() — or the path of a runner file. files maps payload names to the files to stage there: a bare name lands in the payload’s FILES_DIRECTORY, which is where a packaged runner reads its inputs, and a name with a directory in it is used verbatim. inputs becomes the job’s inputs object, whose members are documented per runner.

data_mode defaults to what the template needs — transactional for a template whose runner publishes collected results, and none for a runner that said nothing. publish workspace publishes the runner file into the workspace runner store and pins its digest; installed references a packaged runner through the reserved pkg: form instead and copies nothing.

httk.workflow.scaffold.new_jobs(workspace: httk.workflow.workspace.Workspace, template: str | os.PathLike[str], items: collections.abc.Iterable[JobItem], *, inputs: collections.abc.Mapping[str, object] | None = None, files: collections.abc.Mapping[str, str | os.PathLike[str]] | None = None, tag: str | None = None, placement: str | pathlib.PurePosixPath = DEFAULT_PLACEMENT, priority: int | None = None, workdir_mode: WorkdirMode = 'persistent', data_mode: DataMode | None = None, publish: PublishMode = 'workspace', step: str | None = None, workflow: str | None = None, name: str | None = None) → collections.abc.Iterator[ScaffoldedJob]

Scaffold and submit one job per member of items, lazily.

Every keyword is the shared value of the whole campaign, and every member of one JobItem is what that job varies: inputs and files are merged over the shared mappings, and tag, name, placement, and priority replace the shared value.

This is the pattern for a campaign of any size. The template is resolved once and its runner published once, however many jobs follow, so every job costs exactly one payload directory and one state marker; items is consumed as an iterator and the results are yielded as they are submitted, so a structure generator can be turned into jobs without either side of the loop ever being materialized.

def structures():
    for path in sorted(Path("structures").glob("POSCAR.*")):
        yield {"files": {"POSCAR": path}, "tag": structure_tag(path)}

for job in new_jobs(workspace, "some-template", structures(), inputs={"kpoint_density": 30.0}):
    print(job.job_key)

httk.workflow.scaffold.payload_relative(name: str) → pathlib.PurePosixPath

Return where one staged file lands inside a payload.

A bare name lands in FILES_DIRECTORY, so POSCAR becomes files/POSCAR — where the packaged runners read it — and a name that carries a directory of its own is used exactly as it is written.