httk.workflow.scaffold ====================== .. py:module:: httk.workflow.scaffold .. autoapi-nested-parse:: Scaffolding submitted jobs from a template, some files, and some inputs. A job is a payload directory plus a ``job.json`` that names the runner to execute, and building one by hand means knowing the runner's workflow name, its initial step, its digest, and where its inputs live in the payload. This module is the short way: :func:`new_job` takes a *template* — a packaged runner a domain registered by name, or the path of a runner file of your own — stages the files the runner reads, writes the ``job.json``, and submits the result, all in one call. Packaged templates are not known to this module. A domain or compat engine registers each one it ships with :func:`~httk.workflow.scaffold.register_template`, supplying only the generic description of a starting point — its name, the runner it starts from, the workflow and steps that runner declares, the modes a job of it defaults to, and what it does — so the scaffold resolves and pins a template without ever importing the science that owns it. .. code-block:: python from httk.workflow import Workspace from httk.workflow.scaffold import new_job workspace = Workspace.initialize("workflow-workspace", extensions=["transactional-data-v1"]) job = new_job(workspace, "some-template", files={"input": "input"}, tag="example") print(job.job_key, job.payload) By default the runner file is *published into the workspace runner store*, and the job references it there by digest. That is what makes a scaffolded job durable: the bytes that will run are pinned in the workspace, so upgrading the installed *httk-workflow* underneath a queued campaign cannot change what its jobs execute. Publication is content addressed — the store name carries the digest of the bytes — so scaffolding the same template twice publishes nothing the second time, and a later, different version of a packaged runner lands beside the old one instead of replacing it. ``publish="installed"`` instead references a packaged template through the reserved ``pkg:`` form, which copies nothing at all. :func:`new_jobs` is the same operation for a campaign: one template resolution and one publication amortized over every job, and a lazy iterator over the results. By design, generating a partitioned campaign costs one payload and one marker per job and never materializes a list of them. Attributes ---------- .. autoapisummary:: httk.workflow.scaffold.JOB_SCAFFOLD_FORMAT httk.workflow.scaffold.DEFAULT_PLACEMENT httk.workflow.scaffold.FILES_DIRECTORY httk.workflow.scaffold.STRUCTURE_PATTERNS Classes ------- .. autoapisummary:: httk.workflow.scaffold.TemplateProvider httk.workflow.scaffold.JobItem httk.workflow.scaffold.JobTemplate httk.workflow.scaffold.ScaffoldedJob Functions --------- .. autoapisummary:: httk.workflow.scaffold.register_template httk.workflow.scaffold.registered_templates httk.workflow.scaffold.template_provider httk.workflow.scaffold.describe_runner httk.workflow.scaffold.packaged_template httk.workflow.scaffold.resolve_template httk.workflow.scaffold.structure_files httk.workflow.scaffold.structure_tag httk.workflow.scaffold.new_job httk.workflow.scaffold.new_jobs httk.workflow.scaffold.payload_relative Module Contents --------------- .. py:data:: JOB_SCAFFOLD_FORMAT :value: 'httk-workflow-job-scaffold' .. py:data:: DEFAULT_PLACEMENT :value: 'jobs' .. py:data:: FILES_DIRECTORY :value: 'files' .. py:data:: STRUCTURE_PATTERNS :value: ('POSCAR*', '*.vasp') .. py:class:: TemplateProvider One packaged template a domain or compat engine offers by name. A provider is the generic description of a starting point. It names the packaged runner it starts from by the package the runner file is a module of and the file beside that module — so the reserved ``pkg:`` form and the digest are resolved from the provider alone — and it declares the runner's workflow, its steps, the modes a job of it defaults to, and what it does. Declaring the steps here rather than running the runner to ask keeps scaffolding cheap; the owning domain's tests hold the declaration to what the runner really describes. .. py:attribute:: name :type: str .. py:attribute:: runner_package :type: str .. py:attribute:: runner_file :type: str .. py:attribute:: workflow :type: str .. py:attribute:: initial_step :type: str .. py:attribute:: steps :type: tuple[str, Ellipsis] :value: () .. py:attribute:: data_mode :type: DataMode :value: 'none' .. py:attribute:: workdir_mode :type: WorkdirMode :value: 'persistent' .. py:attribute:: summary :type: str :value: '' .. py:function:: register_template(provider: TemplateProvider) -> None Register one packaged template, replacing any registered under its name. A domain calls this once per template it ships when its package is imported, which is how ``httk workflow job new --template NAME`` resolves a packaged runner the generic scaffold never names. .. py:function:: registered_templates() -> tuple[str, Ellipsis] Return the name of every registered template, in registration order. .. py:function:: template_provider(name: str) -> TemplateProvider | None Return the provider *name* names, by template name or runner file. .. py:class:: JobItem Bases: :py:obj:`TypedDict` What one job of a :func:`new_jobs` campaign varies from the shared values. Every member is optional, and a member that is absent takes the value :func:`new_jobs` was called with. ``inputs`` and ``files`` are merged over the shared mappings key by key; everything else replaces the shared value. .. py:attribute:: inputs :type: collections.abc.Mapping[str, object] .. py:attribute:: files :type: collections.abc.Mapping[str, str | os.PathLike[str]] .. py:attribute:: tag :type: str | None .. py:attribute:: name :type: str .. py:attribute:: placement :type: str | pathlib.PurePosixPath .. py:attribute:: priority :type: int | None .. py:class:: JobTemplate One resolved starting point for a job: a runner file and how to run it. A template is either one of the runners a domain registered by name — see :func:`~httk.workflow.scaffold.registered_templates` — or a runner file of your own, which is described by running it — every native runner answers ``--describe`` with its workflow and its steps — so a scaffolded job never guesses either. .. py:attribute:: name :type: str .. py:attribute:: source :type: pathlib.Path .. py:attribute:: workflow :type: str .. py:attribute:: initial_step :type: str .. py:attribute:: steps :type: tuple[str, Ellipsis] :value: () .. py:attribute:: data_mode :type: DataMode :value: 'none' .. py:attribute:: workdir_mode :type: WorkdirMode :value: 'persistent' .. py:attribute:: packaged :type: str | None :value: None .. py:attribute:: summary :type: str :value: '' .. py:property:: store_name :type: str The content-addressed name this template takes in a runner store. The digest of the bytes is part of the name, so publishing is idempotent for identical bytes and never overwrites a name a submitted job pinned: an upgraded packaged runner is published beside the version its queued jobs still reference. .. py:class:: ScaffoldedJob One job this module submitted, and everything needed to look at it again. .. py:attribute:: job_id :type: str .. py:attribute:: job_key :type: str .. py:attribute:: tag :type: str | None .. py:attribute:: placement :type: pathlib.PurePosixPath .. py:attribute:: payload :type: pathlib.Path .. py:attribute:: marker :type: pathlib.Path .. py:attribute:: workflow :type: str .. py:attribute:: initial_step :type: str .. py:attribute:: template :type: str .. py:attribute:: runner :type: collections.abc.Mapping[str, object] .. py:method:: as_mapping() -> dict[str, object] Return the machine-readable report of this job. .. py:function:: describe_runner(runner: str | os.PathLike[str]) -> dict[str, object] Return the self-description one runner file prints, by running it. Every native runner — Python or Bash — answers ``HTTK_WORKFLOW_DESCRIBE=1`` with its workflow name and its registered steps and exits without touching anything, which is how a runner nobody wrote a template for is still scaffolded without being told what it implements. .. py:function:: packaged_template(name: str) -> JobTemplate | None Return the registered template *name* names, or ``None``. .. py:function:: resolve_template(template: str | os.PathLike[str], *, workflow: str | None = None, step: str | None = None, data_mode: DataMode | None = None) -> JobTemplate Return the :class:`JobTemplate` *template* names. *template* is the name of a packaged template, the file name of a packaged runner, or the path of a runner file. A runner file is described by running it, so its workflow name and its steps come from the runner itself; *workflow* and *step* override what it said, and *step* is required when a runner registers several steps and none of them is ``start``. .. py:function:: structure_files(directory: str | os.PathLike[str]) -> list[pathlib.Path] Return every structure file of one directory, in a stable order. A structure is a file matching one of :data:`STRUCTURE_PATTERNS` — the VASP conventions ``POSCAR``, ``POSCAR.something``, and ``something.vasp`` — which is what makes a directory of structures one campaign. .. py:function:: structure_tag(path: str | os.PathLike[str]) -> str | None Return the job tag one structure file name suggests, or ``None``. The tag is the part of the name that identifies the structure — ``Si2O`` of ``POSCAR.Si2O``, ``fcc-al`` of ``fcc-al.vasp`` — reduced to the tag syntax the protocol allows. A name that says nothing beyond ``POSCAR`` suggests no tag. .. py:function:: new_job(workspace: httk.workflow.workspace.Workspace, template: str | os.PathLike[str], *, inputs: collections.abc.Mapping[str, object] | None = None, files: collections.abc.Mapping[str, str | os.PathLike[str]] | None = None, tag: str | None = None, placement: str | pathlib.PurePosixPath = DEFAULT_PLACEMENT, priority: int | None = None, workdir_mode: WorkdirMode = 'persistent', data_mode: DataMode | None = None, publish: PublishMode = 'workspace', step: str | None = None, workflow: str | None = None, name: str | None = None) -> ScaffoldedJob Scaffold, submit, and describe one job of *template*. *template* is a registered template name — see :func:`~httk.workflow.scaffold.registered_templates` — or the path of a runner file. *files* maps payload names to the files to stage there: a bare name lands in the payload's :data:`~httk.workflow.scaffold.FILES_DIRECTORY`, which is where a packaged runner reads its inputs, and a name with a directory in it is used verbatim. *inputs* becomes the job's ``inputs`` object, whose members are documented per runner. *data_mode* defaults to what the template needs — ``transactional`` for a template whose runner publishes collected results, and ``none`` for a runner that said nothing. *publish* ``workspace`` publishes the runner file into the workspace runner store and pins its digest; ``installed`` references a packaged runner through the reserved ``pkg:`` form instead and copies nothing. .. py:function:: new_jobs(workspace: httk.workflow.workspace.Workspace, template: str | os.PathLike[str], items: collections.abc.Iterable[JobItem], *, inputs: collections.abc.Mapping[str, object] | None = None, files: collections.abc.Mapping[str, str | os.PathLike[str]] | None = None, tag: str | None = None, placement: str | pathlib.PurePosixPath = DEFAULT_PLACEMENT, priority: int | None = None, workdir_mode: WorkdirMode = 'persistent', data_mode: DataMode | None = None, publish: PublishMode = 'workspace', step: str | None = None, workflow: str | None = None, name: str | None = None) -> collections.abc.Iterator[ScaffoldedJob] Scaffold and submit one job per member of *items*, lazily. Every keyword is the shared value of the whole campaign, and every member of one :class:`~httk.workflow.scaffold.JobItem` is what that job varies: ``inputs`` and ``files`` are merged over the shared mappings, and ``tag``, ``name``, ``placement``, and ``priority`` replace the shared value. This is the pattern for a campaign of any size. The template is resolved once and its runner published once, however many jobs follow, so every job costs exactly one payload directory and one state marker; *items* is consumed as an iterator and the results are yielded as they are submitted, so a structure generator can be turned into jobs without either side of the loop ever being materialized. .. code-block:: python def structures(): for path in sorted(Path("structures").glob("POSCAR.*")): yield {"files": {"POSCAR": path}, "tag": structure_tag(path)} for job in new_jobs(workspace, "some-template", structures(), inputs={"kpoint_density": 30.0}): print(job.job_key) .. py:function:: payload_relative(name: str) -> pathlib.PurePosixPath Return where one staged file lands inside a payload. A bare name lands in :data:`~httk.workflow.scaffold.FILES_DIRECTORY`, so ``POSCAR`` becomes ``files/POSCAR`` — where the packaged runners read it — and a name that carries a directory of its own is used exactly as it is written.