httk.workflow.protocol ====================== .. py:module:: httk.workflow.protocol .. autoapi-nested-parse:: The language-neutral filesystem protocol surface of *httk₂* workflows. This module is the one deliberate public home of the on-disk protocol: the shapes, validators, and primitives an implementation in any language reads and writes to interoperate through a workspace. The normative specification is the filesystem protocol reference in the httk-workflow documentation; everything named here is what that document describes, and an independent inspection or verification tool should be able to work from this namespace and that document alone. Nothing here is manager bookkeeping, a subprocess wrapper, a CLI handler, or a scheduling pass — those live in their own modules and are not part of the protocol. The implementations are owned by the modules re-exported below (:mod:`~httk.workflow.models`, :mod:`~httk.workflow.journal`, :mod:`~httk.workflow.transactions`, and the runtime builders), which are internal detail from the protocol's point of view; import the names from here. Attributes ---------- .. autoapisummary:: httk.workflow.protocol.ATTEMPTS_DIRECTORY httk.workflow.protocol.CARRIED_STATE_MEMBERS httk.workflow.protocol.CORE_PROFILE httk.workflow.protocol.CORE_STATE_KINDS httk.workflow.protocol.JOB_STATE_DIRECTORY httk.workflow.protocol.LOGS_DIRECTORY httk.workflow.protocol.QUIESCENT_KINDS httk.workflow.protocol.RUNNER_SOURCES httk.workflow.protocol.STATE_KINDS httk.workflow.protocol.SUPPORTED_EXTENSIONS httk.workflow.protocol.TERMINAL_KINDS httk.workflow.protocol.JoinCondition httk.workflow.protocol.OutcomeAction Exceptions ---------- .. autoapisummary:: httk.workflow.protocol.FormatError httk.workflow.protocol.RunnerResolutionError httk.workflow.protocol.TransactionError httk.workflow.protocol.TransitionLostError httk.workflow.protocol.UnsupportedExtensionError httk.workflow.protocol.WorkflowError httk.workflow.protocol.WorkspaceCorruptionError httk.workflow.protocol.WorkspaceUnavailableError Classes ------- .. autoapisummary:: httk.workflow.protocol.JournalFrame httk.workflow.protocol.RecordVerification httk.workflow.protocol.Failure httk.workflow.protocol.JobDefinition httk.workflow.protocol.Marker httk.workflow.protocol.RetentionPolicy httk.workflow.protocol.RetryPolicy httk.workflow.protocol.StateFrame httk.workflow.protocol.WorkspacePolicy httk.workflow.protocol.AttemptContext httk.workflow.protocol.ChildReference httk.workflow.protocol.JobSpec httk.workflow.protocol.OutcomeDraft httk.workflow.protocol.ReplayableWorkdirBatch httk.workflow.protocol.RunLog httk.workflow.protocol.TransactionBuilder httk.workflow.protocol.MarkerFault Functions --------- .. autoapisummary:: httk.workflow.protocol.encode_record_ref httk.workflow.protocol.iter_journal_frames httk.workflow.protocol.iter_segment_frames httk.workflow.protocol.parse_record_ref httk.workflow.protocol.read_record httk.workflow.protocol.segment_path httk.workflow.protocol.verify_record httk.workflow.protocol.canonical_uuid httk.workflow.protocol.is_payload_private httk.workflow.protocol.job_digest httk.workflow.protocol.make_job_key httk.workflow.protocol.marker_basename httk.workflow.protocol.normalize_placement httk.workflow.protocol.parse_job_key httk.workflow.protocol.parse_package_runner httk.workflow.protocol.to_base36 httk.workflow.protocol.validate_attempt_control httk.workflow.protocol.validate_declaration_name httk.workflow.protocol.validate_declarations httk.workflow.protocol.validate_failure httk.workflow.protocol.validate_label httk.workflow.protocol.validate_parameters httk.workflow.protocol.validate_resources httk.workflow.protocol.validate_runner_path httk.workflow.protocol.validate_sha256 httk.workflow.protocol.validate_step httk.workflow.protocol.join_mapping httk.workflow.protocol.prepare_job_payload httk.workflow.protocol.replay_transaction Module Contents --------------- .. py:exception:: FormatError Bases: :py:obj:`WorkflowError`, :py:obj:`ValueError` A workspace, job, journal frame, outcome, or request is malformed. .. py:exception:: RunnerResolutionError(code, message) Bases: :py:obj:`WorkflowError` A shared runner cannot be resolved or verified. The failure carries the exact protocol failure ``code`` the manager records, so an unresolvable runner (``runner_unavailable``), a runner whose bytes disagree with the digest the job pinned (``runner_mismatch``), a missing registration (``runner_not_built``), and a failed foreground build (``runner_build_failed``) stay distinguishable to an operator. :param code: Protocol failure code recorded by the manager. :param message: Human-readable failure description. .. py:attribute:: code .. py:exception:: TransactionError Bases: :py:obj:`WorkflowError` A transactional-data manifest cannot be safely replayed. .. py:exception:: TransitionLostError Bases: :py:obj:`WorkflowError` Another actor committed a transition from the expected marker. .. py:exception:: UnsupportedExtensionError Bases: :py:obj:`WorkflowError` A workspace requires an extension this implementation does not support. .. py:exception:: WorkflowError Bases: :py:obj:`RuntimeError` Base class for workflow protocol failures. .. py:exception:: WorkspaceCorruptionError Bases: :py:obj:`WorkflowError` The authoritative filesystem state is internally inconsistent. .. py:exception:: WorkspaceUnavailableError Bases: :py:obj:`WorkflowError` The workspace cannot currently provide a coherent protocol view. .. py:class:: JournalFrame Describe one intact frame found by walking a segment from its header. :param record_ref: Identify the canonical record reference. :param writer_id: Identify the journal writer. :param segment: Identify the journal segment. :param offset: Locate the frame within the segment. :param frame: Hold the decoded journal record. .. py:attribute:: record_ref :type: str .. py:attribute:: writer_id :type: str .. py:attribute:: segment :type: int .. py:attribute:: offset :type: int .. py:attribute:: frame :type: dict[str, Any] .. py:class:: RecordVerification Report the outcome of reading one referenced frame without raising. :param record_ref: Identify the record that was checked. :param frame: Hold the verified frame when reading succeeded. :param problem: Name the verification problem when reading failed. :param detail: Explain the verification result. .. py:attribute:: record_ref :type: str .. py:attribute:: frame :type: dict[str, Any] | None .. py:attribute:: problem :type: str | None .. py:attribute:: detail :type: str .. py:property:: ok :type: bool Report whether the referenced frame was read and verified. :return: ``True`` when the frame is present and valid. .. py:function:: encode_record_ref(writer_id, segment, offset, length, checksum) Encode one canonical ``hwref-v2`` reference. :param writer_id: Identify the journal writer. :param segment: Identify the journal segment. :param offset: Locate the frame within the segment. :param length: Record the frame payload length. :param checksum: Supply the frame checksum. :return: The canonical record reference. .. py:function:: iter_journal_frames(control_dir) Yield every intact frame of every segment of every writer. :param control_dir: Locate the workspace control directory. :yield: Each intact journal frame. .. py:function:: iter_segment_frames(path, writer_id, segment) Yield every intact frame of one segment. The walk is deliberately forgiving. A damaged frame whose framing is still intact is skipped, because the frames behind it remain locatable and are exactly what a repair is looking for; a torn or partially visible tail is the normal state of a segment a live writer is appending to and simply ends the walk. :param path: Locate the journal segment. :param writer_id: Identify the journal writer. :param segment: Identify the journal segment number. :yield: Each intact frame found in the segment. .. py:function:: parse_record_ref(record_ref) Parse one canonical ``hwref-v2`` reference. :param record_ref: Supply the record reference to parse. :return: The writer, segment, offset, length, and checksum components. :raises httk.workflow.errors.FormatError: If the reference is not canonical. .. py:function:: read_record(control_dir, record_ref, *, deadline_seconds = None) Read and verify a journal record, retrying visibility-short reads. A frame that is absent, short, or undecodable may be an extension of a segment that has not reached this client yet, so it is retried with bounded backoff until *deadline_seconds* — the workspace's configured visibility deadline — expires. Damage that no amount of waiting can repair is reported at once. :param control_dir: Locate the workspace control directory. :param record_ref: Identify the journal record to read. :param deadline_seconds: Bound retries for metadata visibility. :return: The verified journal record. :raises httk.workflow.errors.FormatError: If the record reference is invalid. :raises httk.workflow.errors.WorkspaceCorruptionError: If the record is permanently damaged. :raises httk.workflow.errors.WorkspaceUnavailableError: If the record remains incoherently visible. .. py:function:: segment_path(control_dir, writer_id, segment) Return the segment file one record reference names. :param control_dir: Locate the workspace control directory. :param writer_id: Identify the journal writer. :param segment: Identify the journal segment. :return: The segment file path. .. py:function:: verify_record(control_dir, record_ref, *, deadline_seconds = None) Read one referenced frame, reporting damage rather than raising. This is the reading half of a workspace check: it distinguishes a segment that is gone from one that is truncated, corrupt, or simply not holding the frame the reference names, which is what a repair decision needs. :param control_dir: Locate the workspace control directory. :param record_ref: Identify the journal record to verify. :param deadline_seconds: Bound retries for metadata visibility. :return: The verification result. .. py:data:: ATTEMPTS_DIRECTORY :value: 'attempts' .. py:data:: CARRIED_STATE_MEMBERS :value: ('step', 'activation_id', 'activation_ordinal', 'attempt_id', 'attempt_ordinal',... .. py:data:: CORE_PROFILE :value: 'core-v2' .. py:data:: CORE_STATE_KINDS .. py:data:: JOB_STATE_DIRECTORY :value: '.httk-job' .. py:data:: LOGS_DIRECTORY :value: 'logs' .. py:data:: QUIESCENT_KINDS .. py:data:: RUNNER_SOURCES .. py:data:: STATE_KINDS :value: ('submitted', 'ready', 'claimed', 'running', 'committing', 'cancelling', 'relocating',... .. py:data:: SUPPORTED_EXTENSIONS :type: frozenset[str] .. py:data:: TERMINAL_KINDS .. py:class:: Failure One canonical structured failure record. Every failure published by a runner, a bridge, or the manager itself uses exactly this shape: a stable machine ``code``, one human ``message``, optional structured ``details``, and the advisory ``retryable`` flag. Retry policy uses ``retry_on`` for manager-detected failures, while a runner-declared ``retryable`` failure is retry-eligible regardless of ``retry_on`` when budget remains. :param code: The stable machine failure code. :param message: The human-readable failure message. :param details: Optional structured failure details. :param retryable: Whether repeating the attempt could help. .. py:attribute:: code :type: str .. py:attribute:: message :type: str .. py:attribute:: details :type: collections.abc.Mapping[str, object] | None :value: None .. py:attribute:: retryable :type: bool :value: False .. py:method:: as_mapping() Return the canonical JSON representation of this failure. :return: The JSON failure mapping. .. py:class:: JobDefinition The immutable declaration and execution settings of one job. The mapping carried in :attr:`parameters` contains opaque implementation knobs; declared staged objects belong to the SDK's input declarations. .. py:attribute:: id :type: str .. py:attribute:: tag :type: str | None .. py:attribute:: name :type: str .. py:attribute:: workflow :type: str .. py:attribute:: runner_executor :type: str .. py:attribute:: runner_source :type: str .. py:attribute:: runner_path :type: pathlib.PurePosixPath .. py:attribute:: runner_sha256 :type: str | None .. py:attribute:: runner_arguments :type: tuple[str, ...] .. py:attribute:: workdir_mode :type: str .. py:attribute:: workdir_path :type: pathlib.PurePosixPath .. py:attribute:: data_mode :type: str .. py:attribute:: initial_step :type: str .. py:attribute:: priority :type: int .. py:attribute:: claim_pool :type: str .. py:attribute:: required_capabilities :type: frozenset[str] .. py:attribute:: retry_policy :type: RetryPolicy .. py:attribute:: resources :type: collections.abc.Mapping[str, int] .. py:attribute:: step_resources :type: collections.abc.Mapping[str, collections.abc.Mapping[str, int]] .. py:attribute:: parameters :type: collections.abc.Mapping[str, object] .. py:attribute:: environment :type: collections.abc.Mapping[str, object] .. py:attribute:: declarations :type: collections.abc.Mapping[str, collections.abc.Mapping[str, object]] .. py:attribute:: declared :type: collections.abc.Mapping[str, collections.abc.Mapping[str, collections.abc.Mapping[str, object]]] .. py:attribute:: parent :type: collections.abc.Mapping[str, object] | None .. py:attribute:: raw :type: collections.abc.Mapping[str, object] .. py:attribute:: stored_digest :type: str | None :value: None .. py:property:: job_key :type: str Return the stable key of this job. .. py:property:: digest :type: str Return the immutable job digest. Normatively the digest is :func:`~httk.workflow.models.job_digest` over the stored ``job.json`` file bytes exactly as submitted, which is what every definition read through :meth:`from_bytes` carries. A definition composed in memory has no stored bytes yet, so its canonical serialization is hashed instead; the two agree as soon as that serialization is what gets written. .. py:method:: from_bytes(data, *, name = 'job.json') :classmethod: Parse stored ``job.json`` bytes, pinning the normative job digest. :param data: The stored job document bytes. :param name: The document name used in validation errors. :return: The parsed job definition. :raises httk.workflow.errors.FormatError: If the bytes do not contain a valid job document. .. py:method:: from_path(path) :classmethod: Read one stored ``job.json``, pinning the normative job digest. :param path: The path of the stored job document. :return: The parsed job definition. :raises httk.workflow.errors.FormatError: If the file cannot be read or is invalid. .. py:method:: from_mapping(value) :classmethod: Parse one job definition mapping. :param value: The job document mapping. :return: The parsed job definition. :raises httk.workflow.errors.FormatError: If the mapping does not satisfy the job protocol. .. py:class:: Marker The state marker locating one job transition in a workspace. :param kind: The state kind encoded by the marker. :param placement: The workspace placement of the job. :param job_key: The stable job key. :param priority: The marker priority. :param generation: The state generation. :param record_ref: The transition record reference. :param path: The marker path. .. py:attribute:: kind :type: str .. py:attribute:: placement :type: pathlib.PurePosixPath .. py:attribute:: job_key :type: str .. py:attribute:: priority :type: int .. py:attribute:: generation :type: int .. py:attribute:: record_ref :type: str .. py:attribute:: path :type: pathlib.Path .. py:property:: job_id :type: str Return the job identifier encoded in this marker. .. py:method:: from_path(state_root, path) :classmethod: Parse one marker path below a workspace state root. :param state_root: The workspace state directory. :param path: The marker path to parse. :return: The parsed state marker. :raises httk.workflow.errors.FormatError: If the path does not use marker syntax. .. py:class:: RetentionPolicy How long a workspace keeps the history it is allowed to collect. ``journal_days`` and ``trash_days`` default to one day. An explicit ``null`` or ``"keep"`` value means that category has no limit; omitted ``attempt_control_days`` likewise remains unlimited. The collector that acts on these numbers is a separate concern; the workspace only carries them so that every implementation attaching to it agrees on what may be removed and when. :param attempt_control_days: The retention period for attempt controls. :param journal_days: The retention period for journal history. :param trash_days: The retention period for discarded workspace entries. .. py:attribute:: attempt_control_days :type: float | None :value: None .. py:attribute:: journal_days :type: float | None :value: 1.0 .. py:attribute:: trash_days :type: float | None :value: 1.0 .. py:method:: from_mapping(value, name = 'policy.retention') :classmethod: Validate one retention policy mapping. :param value: The policy mapping to validate. :param name: The field name used in validation errors. :return: The validated retention policy. :raises httk.workflow.errors.FormatError: If the mapping contains unsupported or invalid members. .. py:method:: as_mapping() Return the JSON representation, omitting only unset attempt retention. :return: The JSON policy mapping. .. py:class:: RetryPolicy The attempt budgets of one job and the failures it retries within them. Two independent rules make a failure retry-eligible, and both are bounded by exactly the same budgets: * ``retry_on`` lists failure codes. A manager-detected failure — a lost lease, a process failure, an unusable outcome — is retried when its code appears in this set. * A runner-declared failure published with ``retryable: true`` is retried whether or not its code appears in ``retry_on``, because the runner that produced the failure is the authority on whether repeating the attempt can help. Neither rule can exceed ``maximum_attempts_per_activation`` or ``maximum_total_attempts``: an exhausted budget always ends the job. :param maximum_attempts_per_activation: The per-activation attempt budget. :param maximum_total_attempts: The total attempt budget. :param maximum_activations: The activation budget. :param retry_on: Failure codes eligible for manager-detected retry. .. py:attribute:: maximum_attempts_per_activation :type: int | None .. py:attribute:: maximum_total_attempts :type: int | None .. py:attribute:: maximum_activations :type: int | None .. py:attribute:: retry_on :type: frozenset[str] .. py:method:: from_mapping(value) :classmethod: Validate one job retry policy mapping. :param value: The retry policy mapping to validate. :return: The validated retry policy. :raises httk.workflow.errors.FormatError: If the mapping contains invalid retry settings. .. py:class:: StateFrame The members of one state frame, typed for the manager that uses them. The frame is held exactly as it is on disk, so every member round-trips verbatim — including one written by a newer implementation, by an enabled extension, or by a workspace older than this code. What this implementation reads and writes goes through the typed accessors and through :meth:`replace`, so a mistyped member name is a type error at the call site rather than a silently defaulted value at runtime. The envelope members ``format``, ``workspace_id``, ``job_id``, ``kind``, ``state_generation``, and their siblings belong to the transition that publishes a frame and are supplied by the workspace, never here. :param members: The state members carried by this frame. .. py:attribute:: members :type: collections.abc.Mapping[str, Any] .. py:method:: from_mapping(value, name = 'state frame') :classmethod: Read one stored state frame, keeping every member verbatim. :param value: The state-frame mapping to read. :param name: The field name used in validation errors. :return: The state frame. :raises httk.workflow.errors.FormatError: If the value is not a mapping. .. py:method:: as_mapping() Return the JSON representation, member for member. :return: The state-frame member mapping. .. py:method:: has(name) Report whether the frame carries *name* at all, null included. :param name: The member name to check. :return: Whether the member is present. .. py:method:: replace(base = None, *, step = _UNSET, activation_id = _UNSET, activation_ordinal = _UNSET, attempt_id = _UNSET, attempt_ordinal = _UNSET, total_attempts = _UNSET, data_generation = _UNSET, resources = _UNSET, join_summary = _UNSET, runner_steps = _UNSET, manager_id = _UNSET, writer_id = _UNSET, claim_id = _UNSET, attempt_control = _UNSET, lease_seconds = _UNSET, matched_pool = _UNSET, matched_capabilities = _UNSET, process = _UNSET, started_at = _UNSET, workdir = _UNSET, outcome_action = _UNSET, child_digests = _UNSET, child_labels = _UNSET, next_step = _UNSET, join = _UNSET, pause = _UNSET, pause_requested = _UNSET, failure = _UNSET, job_digest = _UNSET, join_unresolved = _UNSET, unclean_restart = _UNSET, unsafe_persistent_takeover = _UNSET, takeover_evidence = _UNSET, cancellation = _UNSET, previous_attempt_id = _UNSET, operator = _UNSET, operator_key = _UNSET, operator_reason = _UNSET, request_id = _UNSET, revival_hazard = _UNSET, reason = _UNSET) :classmethod: Return *base* with the named members set, absent ones untouched. Every member a manager writes is one declared keyword, so the complete vocabulary of a state frame is visible in one signature and no call site can invent a member by misspelling one. Passing ``None`` writes the JSON null the protocol distinguishes from an absent member. :param base: The frame to update, or an empty frame when omitted. :param step: The activation step. :param activation_id: The activation identifier. :param activation_ordinal: The activation ordinal. :param attempt_id: The attempt identifier. :param attempt_ordinal: The attempt ordinal. :param total_attempts: The total attempt count. :param data_generation: The transactional data generation. :param resources: The dynamic requirement of this activation. :param join_summary: The children observed by the activation. :param runner_steps: The runner's registered step names. :param manager_id: The owning manager identifier. :param writer_id: The writer identifier. :param claim_id: The claim identifier. :param attempt_control: The attempt-control directory name. :param lease_seconds: The claim lease duration. :param matched_pool: The pool selected for the claim. :param matched_capabilities: The capabilities matched by the claim. :param process: The launched process identity. :param started_at: The attempt start timestamp. :param workdir: The attempt workdir. :param outcome_action: The published outcome action. :param child_digests: The child payload digests. :param child_labels: The child labels. :param next_step: The next activation step. :param join: The child join condition. :param pause: The pause record. :param pause_requested: The deferred operator pause request. :param failure: The failure record. :param job_digest: The immutable job digest. :param join_unresolved: The persisted first-unresolved child and timestamp of a waiting join. :param unclean_restart: Whether the previous attempt ended uncleanly. :param unsafe_persistent_takeover: Whether persistent takeover was unsafe. :param takeover_evidence: Evidence for the persistent takeover. :param cancellation: The cancellation record. :param previous_attempt_id: The previous attempt identifier. :param operator: The operator identity. :param operator_key: The operator key. :param operator_reason: The operator reason. :param request_id: The request identifier. :param revival_hazard: Evidence of a revival hazard. :param reason: The transition reason. :return: The updated state frame. .. py:method:: carried() Return only the members every transition of this activation repeats. :return: The carried state frame. .. py:method:: select(names) Return only the named members this frame actually carries. :param names: The member names to retain. :return: A frame containing the selected members. .. py:property:: step :type: str | None Return the activation step, when present. .. py:property:: activation_id :type: str | None Return the activation identifier, when present. .. py:property:: activation_ordinal :type: int | None Return the activation ordinal, when present. .. py:property:: attempt_id :type: str | None Return the attempt identifier, when present. .. py:property:: attempt_ordinal :type: int | None Return the attempt ordinal, when present. .. py:property:: total_attempts :type: int | None Return the total attempt count, when present. .. py:property:: data_generation :type: int | None Return the transactional data generation, when present. .. py:property:: resources :type: dict[str, int] | None Return the validated dynamic resource requirement, when present. .. py:property:: join_summary :type: object Return the observed child summary, when present. .. py:property:: join_unresolved :type: collections.abc.Mapping[str, object] | None Return the persisted first-unresolved join child and timestamp. A waiting frame records this once, the first time a manager finds a join child unresolvable, so the grace before the join fails is measured from that instant and survives a manager restart rather than resetting. .. py:property:: manager_id :type: str | None Return the owning manager, refusing anything that is not one. The value is joined below ``managers/`` to reach a heartbeat, so a frame that does not name a canonical manager UUID is a protocol violation rather than a path to try. .. py:property:: attempt_control :type: str | None Return the validated attempt-control component of this frame. .. py:property:: lease_seconds :type: float | None Return the claim lease duration, when present. .. py:property:: started_at :type: str | None Return the attempt start timestamp, when present. .. py:property:: workdir :type: str | None Return the attempt workdir, when present. .. py:property:: next_step :type: str | None Return the next activation step, when present. .. py:property:: join :type: collections.abc.Mapping[str, object] | None Return the child join condition, when present. .. py:property:: failure :type: collections.abc.Mapping[str, object] | None Return the failure record, when present. .. py:property:: pause :type: object Return the pause record, when present. .. py:property:: pause_requested :type: collections.abc.Mapping[str, object] | None Return the deferred operator pause request, when present. .. py:property:: cancellation :type: collections.abc.Mapping[str, object] | None Return the cancellation record, when present. .. py:property:: child_digests :type: collections.abc.Mapping[str, object] | None Return the child digests, when present. .. py:property:: previous_attempt_id :type: str | None Return the previous attempt identifier, when present. .. py:property:: unclean_restart :type: bool Report whether the previous attempt ended uncleanly. .. py:property:: unsafe_persistent_takeover :type: bool Report whether persistent takeover was unsafe. .. py:property:: reason :type: str | None Return the transition reason, when present. .. py:class:: WorkspacePolicy The tunables every implementation attaching to one workspace shares. These are workspace properties rather than per-process options: two managers on different hosts must agree on how long a marker may take to become visible and on how long an unheartbeaten lease means anything. They live in ``format.json`` beside the format and profile declarations, and a workspace written before this section existed simply reads as the defaults. :param visibility_deadline_seconds: The marker visibility deadline. :param lease_seconds: The manager claim lease duration. :param journal_segment_bytes: The journal segment size. :param retention: The workspace retention policy. .. py:attribute:: visibility_deadline_seconds :type: float :value: 5.0 .. py:attribute:: lease_seconds :type: float :value: 900.0 .. py:attribute:: journal_segment_bytes :type: int :value: 67108864 .. py:attribute:: retention :type: RetentionPolicy .. py:method:: from_mapping(value, name = 'policy') :classmethod: Validate one complete policy object, filling in absent members. :param value: The policy mapping to validate. :param name: The field name used in validation errors. :return: The validated retention policy. :raises httk.workflow.errors.FormatError: If the mapping contains unsupported or invalid members. .. py:method:: as_mapping() Return the complete JSON representation stored in ``format.json``. :return: The JSON workspace policy mapping. .. py:method:: updated(changes, name = 'policy') Return this policy with *changes* applied and revalidated. :param changes: Policy members to replace. :param name: The field name used in validation errors. :return: The updated workspace policy. :raises httk.workflow.errors.FormatError: If the changes contain unsupported or invalid members. .. py:function:: canonical_uuid(value, name = 'id') Validate and return a canonical lowercase UUID. :param value: The value to validate. :param name: The field name used in validation errors. :return: The canonical UUID text. :raises httk.workflow.errors.FormatError: If the value is not canonical UUID text. .. py:function:: is_payload_private(name) Report whether one payload entry name is runner-private scratch. A runner-private entry is excluded from every payload digest, so publishing an outcome or writing job state can never change the digest of a payload that a manager, a transfer, or a registration check must still recognize. :param name: The payload entry name to classify. :return: Whether the name is runner-private. .. py:function:: job_digest(data) Return the normative immutable job digest of stored ``job.json`` bytes. The digest of a job is the SHA-256 over the ``job.json`` file bytes exactly as submitted. Nothing rewrites or renormalizes those bytes, so the digest is reproducible by any implementation with only a hash utility. :param data: The stored ``job.json`` bytes. :return: The lowercase SHA-256 digest. .. py:function:: make_job_key(job_id, tag) Compose the stable job key from an identifier and optional tag. :param job_id: The job UUID text. :param tag: The optional job tag. :return: The job key used by workspace markers. .. py:function:: marker_basename(job_key, priority, generation, record_ref) Build one bounded state-marker basename. :param job_key: The stable job key. :param priority: The marker priority. :param generation: The state generation. :param record_ref: The transition record reference. :return: The marker basename. :raises httk.workflow.errors.FormatError: If a component is invalid or the basename exceeds the profile limit. .. py:function:: normalize_placement(value) Validate and normalize one relative POSIX placement. :param value: The placement to validate. :return: The normalized relative placement. :raises httk.workflow.errors.FormatError: If the placement is absolute, empty, unsafe, or too long. .. py:function:: parse_job_key(value) Split a job key into its optional tag and job identifier. :param value: The job key to parse. :return: The tag and job identifier. :raises httk.workflow.errors.FormatError: If the key does not use the protocol syntax. .. py:function:: parse_package_runner(value) Split the reserved ``pkg:/`` installed runner form. Return ``None`` when *value* is an ordinary relative runner path, so callers can treat the reserved form as one alternative spelling of ``runner.path`` rather than as a separate protocol member. :param value: The runner path to inspect. :return: The package module and resource, or ``None`` for an ordinary path. :raises httk.workflow.errors.FormatError: If the reserved package form is malformed. .. py:function:: to_base36(value) Encode a nonnegative integer in lowercase base 36. :param value: The integer to encode. :return: The base-36 representation. :raises ValueError: If the value is negative. .. py:function:: validate_attempt_control(value, name = 'attempt_control') Validate one attempt-control directory name read from a state frame. The name is joined below a job payload to reach the control directory of an attempt, so it is exactly the canonical ``attempts/`` shape. Validating it before it is joined is what keeps a hostile or damaged frame from naming a path outside the job. :param value: The attempt-control name to validate. :param name: The field name used in validation errors. :return: The validated attempt-control name. :raises httk.workflow.errors.FormatError: If the value is not a canonical attempt-control name. .. py:function:: validate_declaration_name(value, name = 'declaration name') Validate one declaration name of a job. The name keys the ``declarations`` object of ``job.json`` and is also the basename of the runtime-refined document below ``.httk-job/declarations/``, so it must be a safe single path component and nothing else. :param value: The declaration name to validate. :param name: The field name used in validation errors. :return: The validated declaration name. :raises httk.workflow.errors.FormatError: If the value is not a safe declaration name. .. py:function:: validate_declarations(value, name = 'declarations') Validate the optional ``declarations`` object of a job. Each member is one workflow-declaration document carried verbatim: the protocol checks that a declaration is a JSON object and never looks inside it, because what the members mean is owned by the vocabulary the document names itself — the OPTIMADE workflow-declaration work is standardizing exactly that, and an engine that reinterpreted it would only be able to disagree with it. The bytes live in ``job.json`` and are therefore covered by the immutable job digest like every other member. :param value: The declarations mapping to validate. :param name: The field name used in validation errors. :return: The validated declaration documents keyed by name. :raises httk.workflow.errors.FormatError: If a declaration name, document, or size limit is invalid. .. py:function:: validate_failure(value, name = 'failure') Validate one published failure object. :param value: The failure mapping to validate. :param name: The field name used in validation errors. :return: The validated failure record. :raises httk.workflow.errors.FormatError: If the failure is missing required members or has invalid members. .. py:function:: validate_label(value, name) Validate and return one protocol label. :param value: The label to validate. :param name: The field name used in validation errors. :return: The validated label. :raises httk.workflow.errors.FormatError: If the value is not valid label text. .. py:function:: validate_parameters(value, name = 'parameters') Validate the optional application-defined ``parameters`` object of a job. The member is opaque to the protocol: only its shape, its key syntax, and its serialized size are checked. Its bytes are part of ``job.json`` and are therefore covered by the immutable job digest like every other member. :param value: The parameters mapping to validate. :param name: The field name used in validation errors. :return: The validated parameters mapping. :raises httk.workflow.errors.FormatError: If the mapping or its serialized contents are invalid. .. py:function:: validate_resources(value, name = 'resources') Validate one quantitative resource-requirement mapping. Resource names are protocol labels and values are opaque non-negative integers. The protocol does not assign units or meanings to names. :param value: The resource mapping to validate. :param name: The field name used in validation errors. :return: The validated resource mapping. :raises httk.workflow.errors.FormatError: If the mapping, names, or values are invalid. .. py:function:: validate_runner_path(value, source) Validate ``runner.path`` against the root implied by ``runner.source``. Every source resolves the same relative path below a different root: the job payload, the workspace runner store, or one configured installed-runner search path. The path must therefore stay below its root under every source, and only an installed runner may use the reserved ``pkg:`` form. :param value: The runner path to validate. :param source: The runner source that determines the permitted path form. :return: The validated runner path. :raises httk.workflow.errors.FormatError: If the path is absolute, unsafe, or incompatible with its source. .. py:function:: validate_sha256(value, name) Validate one lowercase hexadecimal SHA-256 digest string. :param value: The digest to validate. :param name: The field name used in validation errors. :return: The validated digest. :raises httk.workflow.errors.FormatError: If the value is not a lowercase hexadecimal digest. .. py:function:: validate_step(value, name = 'step') Validate and return one workflow step name. :param value: The step name to validate. :param name: The field name used in validation errors. :return: The validated step name. :raises httk.workflow.errors.FormatError: If the value is not a valid step name. .. py:class:: AttemptContext Describe the immutable identity and restart evidence for one running attempt. :param workspace_id: Identify the workspace. :param job_id: Identify the job. :param job_key: Identify the job payload. :param placement: Locate the job placement. :param payload: Record the absolute job payload path. :param step: Identify the runner step. :param activation_id: Identify the activation. :param attempt_id: Identify the attempt. :param activation_ordinal: Record the activation sequence position. :param attempt_ordinal: Record the attempt sequence position. :param total_attempts: Record the planned attempt count. :param is_restart: Mark whether this attempt is a restart. :param is_unclean_restart: Mark whether the restart followed an unclean exit. :param attempt_reason: Explain why this attempt was selected. :param previous_attempt_id: Identify the preceding attempt when present. :param activation_reason: Explain why the activation was selected. :param workdir_mode: Describe how the work directory was selected. :param workdir_reused: Mark whether the work directory was reused. :param unsafe_persistent_takeover: Record whether persistent takeover was enabled. :param data_generation: Record the data generation at claim time. :param durable: Record whether storage-crash durability was enabled. :param settings: Record workspace application settings at claim time. :param resources: Record the resources assigned to the attempt. :param join: Record the job's join description. :param raw: Preserve the complete decoded context. .. py:attribute:: workspace_id :type: str .. py:attribute:: job_id :type: str .. py:attribute:: job_key :type: str .. py:attribute:: placement :type: str .. py:attribute:: payload :type: str .. py:attribute:: step :type: str .. py:attribute:: activation_id :type: str .. py:attribute:: attempt_id :type: str .. py:attribute:: activation_ordinal :type: int | None .. py:attribute:: attempt_ordinal :type: int | None .. py:attribute:: total_attempts :type: int | None .. py:attribute:: is_restart :type: bool .. py:attribute:: is_unclean_restart :type: bool .. py:attribute:: attempt_reason :type: str | None .. py:attribute:: previous_attempt_id :type: str | None .. py:attribute:: activation_reason :type: str | None .. py:attribute:: workdir_mode :type: str | None .. py:attribute:: workdir_reused :type: bool .. py:attribute:: unsafe_persistent_takeover :type: bool .. py:attribute:: data_generation :type: int | None .. py:attribute:: durable :type: bool .. py:attribute:: settings :type: collections.abc.Mapping[str, object] .. py:attribute:: resources :type: collections.abc.Mapping[str, int] .. py:attribute:: join :type: object .. py:attribute:: raw :type: collections.abc.Mapping[str, Any] .. py:method:: from_mapping(value) :classmethod: Validate a manager-written attempt context mapping. :param value: Decoded attempt context. :return: The validated attempt context. :raises ValueError: If the context format or required values are invalid. .. py:class:: ChildReference Identify one child in a native join. :param workspace_id: Identify the child's workspace. :param job_id: Identify the child job. :param job_key: Identify the child payload and markers. :param placement_hint: Locate the child within its workspace. .. py:attribute:: workspace_id :type: str .. py:attribute:: job_id :type: str .. py:attribute:: job_key :type: str .. py:attribute:: placement_hint :type: str .. py:method:: as_mapping() Return the protocol mapping for this child. :return: The serialized child reference. .. py:class:: JobSpec Values needed to create an immutable native job definition. A ``payload`` runner is a file inside the job payload. A ``workspace`` or ``installed`` runner lives outside the payload and must therefore pin its own ``runner_sha256``, which is how one published runner serves a whole campaign of jobs without being copied per job. :param name: Set the job display name. :param workflow: Name the workflow. :param runner_path: Locate the runner. :param initial_step: Name the starting step. :param tag: Set the optional job tag. :param job_id: Preserve a job id when resuming or spawning. :param runner_executor: Select the runner executor. :param runner_source: Select where the runner lives. :param runner_sha256: Pin a runner outside the payload by digest. :param runner_arguments: Supply runner arguments. :param workdir_mode: Select the workdir mode. :param workdir_path: Name the workdir below the job payload. :param data_mode: Select the job data mode. :param priority: Set the scheduling priority. :param claim_pool: Select the claim pool. :param required_capabilities: Require these manager capabilities. :param maximum_attempts_per_activation: Bound attempts in one activation. :param maximum_total_attempts: Bound attempts across the job. :param maximum_activations: Bound job activations. :param retry_on: Name manager-detected failure codes eligible for retry. :param resources: Supply resource requirements. :param step_resources: Supply per-step resource requirements. :param parameters: Supply opaque job parameters. :param environment: Supply declared environment metadata and overrides. :param declarations: Supply workflow declarations. :param declared: Supply the declared parameter and input metadata sections. :param compatibility: Supply an optional compatibility profile. .. py:attribute:: name :type: str .. py:attribute:: workflow :type: str .. py:attribute:: runner_path :type: str .. py:attribute:: initial_step :type: str :value: 'start' .. py:attribute:: tag :type: str | None :value: None .. py:attribute:: job_id :type: str | None :value: None .. py:attribute:: runner_executor :type: str :value: 'path' .. py:attribute:: runner_source :type: Literal['payload', 'workspace', 'installed'] :value: 'payload' .. py:attribute:: runner_sha256 :type: str | None :value: None .. py:attribute:: runner_arguments :type: tuple[str, ...] :value: () .. py:attribute:: workdir_mode :type: Literal['persistent', 'isolated'] :value: 'persistent' .. py:attribute:: workdir_path :type: str :value: 'run' .. py:attribute:: data_mode :type: Literal['none', 'transactional'] :value: 'none' .. py:attribute:: priority :type: int :value: 500 .. py:attribute:: claim_pool :type: str :value: 'default' .. py:attribute:: required_capabilities :type: tuple[str, ...] :value: () .. py:attribute:: maximum_attempts_per_activation :type: int | None :value: None .. py:attribute:: maximum_total_attempts :type: int | None :value: None .. py:attribute:: maximum_activations :type: int | None :value: None .. py:attribute:: retry_on :type: tuple[str, ...] :value: () .. py:attribute:: resources :type: collections.abc.Mapping[str, int] .. py:attribute:: step_resources :type: collections.abc.Mapping[str, collections.abc.Mapping[str, int]] .. py:attribute:: parameters :type: collections.abc.Mapping[str, object] .. py:attribute:: environment :type: collections.abc.Mapping[str, object] .. py:attribute:: declarations :type: collections.abc.Mapping[str, collections.abc.Mapping[str, object]] .. py:attribute:: declared :type: collections.abc.Mapping[str, object] .. py:attribute:: compatibility :type: collections.abc.Mapping[str, object] | None :value: None .. py:method:: as_mapping(*, parent = None) Return the validated job-definition mapping. :param parent: Identify the parent job when this is a spawned child. :return: The mapping written to ``job.json``. :raises ValueError: If runner placement or digest settings are invalid. .. py:type:: JoinCondition :canonical: Literal['all_succeeded', 'all_terminal', 'any_succeeded', 'any_terminal', 'at_least'] .. py:type:: OutcomeAction :canonical: Literal['advance', 'retry', 'wait', 'succeed', 'fail', 'pause'] .. py:class:: OutcomeDraft(context, control, root = None, *, durable = False) One unpublished outcome bundle below an attempt control directory. The draft is the single place that writes the protocol shapes of an outcome: its transaction, its spawn set, and the atomic rename that publishes it. It is bound to nothing but the attempt identity and the control directory, so the authoring SDK and the Bash bridge publish through exactly one implementation. :param context: Provide the attempt identity and generation. :param control: Locate the attempt control directory. :param root: Locate an existing unpublished draft, when resuming one. :param durable: Synchronize the draft before publishing it. .. py:attribute:: context .. py:attribute:: control .. py:attribute:: durable :value: False .. py:attribute:: root .. py:method:: transaction() Create the transaction builder for this outcome. :return: The outcome's transaction builder. :raises ValueError: If the attempt has no transactional data. :raises RuntimeError: If the outcome already has a transaction. .. py:method:: add_child(payload, placement, *, label = None) Register one prepared payload directory as a child of this outcome. :param payload: Locate the prepared child payload. :param placement: Place the child within the workspace. :param label: Set the child's unique spawn label. :return: The registered child reference. .. py:method:: add_child_job(job, placement, *, label) Register one synthesized child that needs no prepared payload. A child whose runner lives outside the payload — a workspace or installed runner — is completely described by its ``job.json``, so a partitioned campaign can spawn children without copying a payload tree per child. :param job: Supply the synthesized child job definition. :param placement: Place the child within the workspace. :param label: Set the child's unique spawn label. :return: The registered child reference. .. py:property:: children :type: tuple[ChildReference, ...] Return the children registered in this outcome. :return: The child references in registration order. .. py:method:: publish(action, *, next_step = None, priority = None, failure = None, retry = None, join = None, pause = None, message = None, expected_data_generation = None, runner_steps = None, resources = None) Publish this outcome atomically. :param action: Select the next job action. :param next_step: Name the next step for ``advance`` or ``wait``. :param priority: Override the next marker priority. :param failure: Supply canonical failure details for ``fail``. :param retry: Supply retry details for ``retry``. :param join: Supply join details for ``wait``. :param pause: Supply pause details for ``pause``. :param message: Attach an optional human-readable message. :param expected_data_generation: Confirm the transaction generation. :param runner_steps: Record the runner steps available to the manager. :param resources: Set the requirement of the next activation for ``advance`` or ``wait``. :return: The authoritative published outcome path. :raises FileExistsError: If an outcome is already published. :raises ValueError: If the action's required details are invalid. .. py:class:: ReplayableWorkdirBatch(workdir, root, *, durable = False) Build a sealed, idempotently replayable set of workdir changes. :param workdir: Locate the workdir receiving the changes. :param root: Locate the batch staging directory. :param durable: Synchronize staged and applied batch directories. .. py:attribute:: workdir .. py:attribute:: root .. py:attribute:: durable :value: False .. py:attribute:: transaction .. py:method:: initialize(workdir, *, durable = False) :classmethod: Create a new workdir batch. :param workdir: Locate the workdir receiving the changes. :param durable: Synchronize the batch publications. :return: The new replayable batch. .. py:method:: seal() Seal the batch for recovery. :return: The sealed ready-directory path. .. py:method:: commit() Replay and retire the sealed batch. :return: The applied batch path. .. py:method:: recover(workdir, *, durable = False) :staticmethod: Replay every ready batch found in a workdir. :param workdir: Locate the workdir containing ready batches. :param durable: Synchronize replayed publications. :return: The applied batch paths. .. py:class:: RunLog(payload) Append structured application evidence in a job payload. :param payload: Locate the payload receiving the run log. .. py:attribute:: path .. py:method:: append(kind, message, *, files = ()) Append one structured run-log event. :param kind: Name the event kind. :param message: Record the event message. :param files: Attach existing regular files as evidence. :raises ValueError: If *kind* is empty or contains NUL. .. py:method:: append_record(path, record) :staticmethod: Append one already-formed run-log record with one operating-system write. :param path: Locate the append-only run log. :param record: Provide the complete JSON record to append. .. py:class:: TransactionBuilder(root, *, expected_generation, durable = False) Build a validated replayable transaction manifest. :param root: Locate the transaction staging directory. :param expected_generation: Pin the data generation the transaction applies to. :param durable: Synchronize the sealed manifest before returning it. .. py:attribute:: root .. py:attribute:: expected_generation .. py:attribute:: durable :value: False .. py:method:: resume(root, *, expected_generation, durable = False) :classmethod: Reattach to a transaction an earlier process of this attempt sealed. A Bash runner publishes one outcome through many short-lived processes, so the staged manifest on disk — not any in-memory counter — is what carries the operations of a draft from one call to the next. Resuming reads it back, so appending an operation continues the same sequence and the same overlap checks as the process that staged the first one. :param root: Locate the sealed transaction directory. :param expected_generation: Require this data generation in the manifest. :param durable: Preserve the durability setting for later sealing. :return: The resumed transaction builder. :raises ValueError: If the manifest format or generation is invalid. .. py:method:: make_dir(operation_id, path) Stage creation of one directory. :param operation_id: Identify the transaction operation. :param path: Name the relative destination directory. .. py:method:: put_file(operation_id, source, path) Stage one regular file for installation. :param operation_id: Identify the transaction operation. :param source: Locate the regular source file. :param path: Name the relative destination path. :raises ValueError: If the source is not a regular file. .. py:method:: put_tree(operation_id, source, path, *, replace = False) Stage one directory tree for installation. :param operation_id: Identify the transaction operation. :param source: Locate the regular source directory. :param path: Name the relative destination path. :param replace: Replace the destination tree instead of merging it. :raises ValueError: If the source tree contains an unsupported entry. .. py:method:: remove(operation_id, path, *, missing_ok = False) Stage removal of one path. :param operation_id: Identify the transaction operation. :param path: Name the relative path to remove. :param missing_ok: Allow the destination to be absent during replay. .. py:method:: seal() Seal the staged operations into a manifest. :return: The sealed manifest path. .. py:function:: join_mapping(children, condition = 'all_succeeded', count = None, on_impossible_step = None, additional_children = ()) Return the validated ``join`` member of one waiting outcome. :param children: Identify the child jobs to join. :param condition: Select the condition that decides the join. :param count: Set the success threshold for ``at_least``. :param on_impossible_step: Name the step to advance to when the join is impossible. :param additional_children: Add already registered child references to the join. :return: The validated join mapping. :raises ValueError: If the child set or condition arguments are invalid. .. py:function:: prepare_job_payload(destination, spec, *, parent = None, durable = False) Create and validate ``job.json`` in an existing prepared payload. *durable* synchronizes the written ``job.json`` for a caller preparing a payload directly on durable storage; it defaults to ``False`` because a payload prepared here is not yet a workspace artifact, and its submission is what makes it authoritative and durable. :param destination: Locate the prepared payload directory. :param spec: Supply the immutable job definition values. :param parent: Identify the parent job when preparing a child. :param durable: Synchronize ``job.json`` before returning. :return: The validated job definition. :raises FileExistsError: If ``job.json`` already exists. :raises ValueError: If the job definition or payload runner is invalid. .. py:function:: replay_transaction(transaction_dir, data_dir, *, expected_generation, durable = False) Idempotently apply one published transaction. Returns whether the manifest contains operations and therefore advances the data generation. When *durable* is set, every destination this replay installs and every directory whose entries it changes — including the parents that gained or lost a name, and the trash a removal moved into — is synchronized before this call returns. The manager relies on that ordering: it appends the destination state frame and renames the marker out of ``committing`` only after replay returns, so a committed transaction is on storage before the marker that claims it is. :param transaction_dir: Locate the published transaction directory. :param data_dir: Locate the workspace data directory to update. :param expected_generation: Require this current data generation. :param durable: Synchronize installed data before returning. :return: Whether the manifest contains operations. :raises httk.workflow.errors.FormatError: If the transaction manifest or an operation is invalid. :raises httk.workflow.errors.TransactionError: If the generation, source, or destination is invalid. :raises httk.workflow.errors.WorkspaceUnavailableError: If an operation cannot be resolved after retries. .. py:class:: MarkerFault Describe one state entry shaped like a marker that cannot be interpreted. :param path: Identify the unusable state entry. :param reason: Explain why the entry could not be interpreted. .. py:attribute:: path :type: pathlib.Path .. py:attribute:: reason :type: str