httk.store.backend.sql.store

The SQL store: save and fetch storable frozen dataclasses through a Backend.

SqlStore is the object-level storage API on top of the schema IR (httk.store.backend.schema), the value codecs (httk.store.backend.codecs), the content identity (httk.core.storage), and the SQLAlchemy table mapping (httk.store.backend.sql.mapping):

  • SqlStore.save() writes an instance (recursing into referenced and child-element storables) and returns its integer sid, deduplicating per the class’s dedup policy;

  • SqlStore.fetch() reconstructs the instance stored under a sid — exactly, via the *_exact companion columns for rationals — as a lazy row by default (fields decode on first access) or, with eager=True, fully materialized; repeated live default fetches of one sid return the same object, with a materialized instance taking precedence over a proxy;

  • SqlStore.transaction() scopes several operations into one database transaction (commit on exit, roll back on exception); outside of it every operation autocommits;

  • SqlStore.referring() finds join-objects (tags, references) pointing at a stored instance, replacing v1’s implicit codependent-data machinery;

  • SqlStore.searcher() starts a query through the search DSL (httk.store.backend.sql.searcher), implementing the httk.store.query protocols.

Deduplication semantics (ported from v1): under "content_id" an equal instance maps to the existing row (children are not re-inserted); under "by_value" a row matching all parent-table columns is reused — child table contents are not part of the match, mirroring v1 which matched key columns only; under "none" every save inserts a new row.

Identity caches are best-effort; content-addressed SqlStore.sid_of() lookups fall back to the database.

Exceptions

EntryDispatchIntegrityError

A persisted entry dispatch row does not name exactly its expected backing.

EntryIdConflictError

An entry id is already owned by a different lineage or alternative group.

EntryMetadataConflictError

Stored identity-excluded metadata differs from a repeated save.

EntryReplacementError

A replacement deduplicated onto a row from a different lineage.

StoreClockRegressionError

A writable store clock is behind its process-local timestamp mark.

Classes

SqlStore

Object storage for storable frozen dataclasses in a relational Backend.

Module Contents

exception httk.store.backend.sql.store.EntryDispatchIntegrityError

Bases: RuntimeError

A persisted entry dispatch row does not name exactly its expected backing.

exception httk.store.backend.sql.store.EntryIdConflictError(table_name, entry_id, existing_logical_id, requested_logical_id)

Bases: ValueError

An entry id is already owned by a different lineage or alternative group.

Parameters:

  • table_name (str) – The table containing the conflicting identifier.

  • entry_id (str) – The conflicting entry identifier.

  • existing_logical_id (int | None) – The lineage or group already owning the identifier.

  • requested_logical_id (int | None) – The lineage or group requesting it, when known.

table_name
entry_id
existing_logical_id
requested_logical_id
exception httk.store.backend.sql.store.EntryMetadataConflictError

Bases: ValueError

Stored identity-excluded metadata differs from a repeated save.

exception httk.store.backend.sql.store.EntryReplacementError(table_name, predecessor_logical_id, conflicting_logical_id)

Bases: ValueError

A replacement deduplicated onto a row from a different lineage.

Parameters:

  • table_name (str) – The table (or collection) whose replacement failed.

  • predecessor_logical_id (int) – The logical_id of the intended predecessor.

  • conflicting_logical_id (int) – The logical_id of the row actually hit.

table_name
predecessor_logical_id
conflicting_logical_id
exception httk.store.backend.sql.store.StoreClockRegressionError(mark_ns, clock_ns)

Bases: RuntimeError

A writable store clock is behind its process-local timestamp mark.

class httk.store.backend.sql.store.SqlStore(database, *, entry_records=None, entry_families=None, entry_ids=None, store_timestamps=True, store_timestamp_resolution=1000, allow_clock_regression=False, clock_regression_grace=True, upgrade=False)

Object storage for storable frozen dataclasses in a relational Backend.

A store starts with an explicit, versioned entry declaration. Ordinary unconfigured frozen-dataclass tables remain on-demand, but only after the layout marker has been initialized on a physically empty database. Schemas edited out-of-band fail at use time with the database’s own errors.

The first open of a database requires entry_records or entry_families. The store stamps the canonical JSON declaration and protocol version, then trusts that declaration on reopen: a supplied declaration must be byte-identical, and mismatches raise StorageLayoutUpgradeRequiredError. Reopening does not diff or migrate record schemas. Read paths never issue DDL; missing ordinary tables behave as empty results or missing rows, while table creation happens only through writes or ensure_tables().

Parameters:

  • database (httk.store.backend.sql.engine.Backend) – The database used for storage.

  • entry_records (collections.abc.Mapping[type, type | tuple[type, Ellipsis]] | None) – The required entry-family declaration when first opening a database.

  • entry_families (collections.abc.Sequence[httk.store.storage_layout.EntryFamilyDeclaration] | None) – Application-owned declarations which bypass global registration.

  • entry_ids (httk.store.store_common.EntryIdScheme | None) – Optional scheme used to mint ids for defined entry families.

  • store_timestamps (bool) – Whether parent rows carry store-managed timestamps.

  • store_timestamp_resolution (int) – Nanoseconds represented by one stored unit.

  • allow_clock_regression (bool) – Whether to disable the process-local clock guard.

  • clock_regression_grace (bool) – Whether to wait briefly for sub-millisecond regressions.

  • upgrade (bool) – Whether to apply a purely additive schema-fingerprint change (new nullable columns, new lazily created tables) on reopen instead of raising; non-additive or non-schema differences still raise.

Raises:
bulk_ingest_finalize_default: Literal['auto', 'parity', 'deferred'] = 'auto'
property layout: httk.store.backend.sql.layout.StorageLayout

Return the immutable persisted entry declaration and resolved classes.

Returns:

The persisted storage layout.

Return type:

httk.store.backend.sql.layout.StorageLayout

property backend_facts: httk.store.backend.sql.layout.BackendFacts

Return the dialect capabilities resolved when this store was opened.

property write_profile: Literal['transactional', 'degraded', 'bulk-fenced']

Return the persisted permanentization write profile.

property store_timestamps: bool

Whether parent rows carry store-managed timestamps.

property store_timestamp_resolution: int | None

Return nanoseconds per stored timestamp unit, or None when disabled.

property entry_layout: tuple[httk.store.backend.sql.layout.EntryFamilyLayout, Ellipsis]

Return configured entry-family layouts in deterministic stable-name order.

Returns:

The configured entry-family layouts.

Return type:

tuple[httk.store.backend.sql.layout.EntryFamilyLayout, Ellipsis]

property entry_records: collections.abc.Mapping[type, tuple[type, Ellipsis]]

Return configured entry-family classes mapped to ordered concrete records.

Returns:

The entry-family to concrete-record mapping.

Return type:

collections.abc.Mapping[type, tuple[type, Ellipsis]]

bulk_ingest(*, chunk_size=100000, verify_metadata=True, index_strategy='auto', on_progress=None, workers=1, finalize='auto', track_sids=True, id_series=None)

Return a context manager that appends a stream of objects into this store.

The returned BulkIngest exposes save(obj, *, as_record=None, promote=None) -> int mirroring save(), but buffers encoded rows with pre-assigned sids and appends them in executemany batches. On a physically empty store the record tables are created index-less and their separable indexes are built once the stream completes; on a populated store each flushed chunk is staged and resolved set-wise against the existing rows (content-id anti-join with metadata verification, by_value whole-column anti-join, and a sid remap of the surviving references) before it is appended. While the context is open the store’s ordinary write path is exclusively owned: save(), ensure_tables() and transaction() raise RuntimeError.

A sid returned by save inside the context is provisional: a record that deduplicates against a pre-existing row is remapped at flush, so its durable sid is obtained from resolved_sid() once the context has exited cleanly.

save(..., promote=RecordClass) additionally makes every nested occurrence of that record class a top-level entry without a second projection or worker transfer. An iterable promotes several classes.

Parameters:

  • chunk_size (int) – The number of top-level saves buffered before a flush.

  • verify_metadata (bool) – Whether content-id hits compare identity-excluded metadata.

  • index_strategy (Literal['auto', 'keep', 'rebuild']) – How an existing table’s separable indexes are handled during the append — "keep" appends through them, "rebuild" drops and recreates them, and "auto" picks per table by the staged-to-existing row ratio. On DuckDB, which reserves a dropped index’s name until commit, "rebuild" instead keeps the indexes and verifies content-id uniqueness with a duplicate scan; the final indexes are the same either way.

  • on_progress (collections.abc.Callable[[int, int], None] | None) – An optional (records_buffered_total, rows_flushed_total) callback invoked after each flush.

  • workers (int) – The number of worker processes. 1 (the default) is the serial path with byte-for-byte unchanged semantics; >1 encodes the stream in forked worker processes and merges their per-table shards set-wise. Parallel mode requires a physically empty target store (the offline-build use case) and, on DuckDB, the httk-store[parallel] extra (pyarrow); incremental appends stay on the serial path.

  • finalize (Literal['auto', 'parity', 'deferred']) – "parity" selects the historical in-database ingest; "deferred" stages a physically empty ingest outside the store and finalizes it at context exit; "auto" selects deferred only for a physically empty, supported serial ingest and otherwise selects parity (including workers>1). A subclass may override bulk_ingest_finalize_default for "auto" calls.

  • track_sids (bool) – Whether to retain provisional-to-durable sid mappings.

  • id_series (str | None) – Override the configured entry-id series for minted bulk rows.

Returns:

A bulk-ingest context manager bound to this store.

Return type:

httk.store.backend.sql.bulk.BulkIngest

ensure_tables(*classes)

Create the requested tables as an explicit write operation.

Parameters:

*classes (type) – The storable classes whose tables should exist.

Returns:

None.

Raises:

RuntimeError – If a bulk_ingest() context is currently open.

Return type:

None

transaction()

Return a context manager for one database transaction.

Returns:

A transaction context manager that commits on normal exit and rolls back on failure.

Return type:

contextlib.AbstractContextManager[None]

steal_lease()

Conditionally replace the current degraded-store writer lease.

The compare-and-swap includes the complete observed value so a stale caller can never overwrite a newer owner.

save(obj, *, as_record=None, id_series=None, alternative_of=None, alternative_kind=None, links=None)

Store obj (deduplicating per its class’s policy) and return its integer sid.

An opted-in domain object is projected through its exact __httk_storage_record__; as_record selects an alternate record representation explicitly. Referenced records and record-valued child elements are saved recursively without constructing intermediate record instances.

A content-id deduplication hit compares metadata marked with IdentitySkip in schema order. Nested plans are cached per record type, and a mismatch raises EntryMetadataConflictError without replacing the row.

Passing alternative_of (a stored main entry’s id) with alternative_kind saves obj as a named ALTERNATIVE representation of that main: it copies the main’s public id, joins the main’s alternative group, and hashes with the group identity folded in so its content never dedups onto the main. The main must live in obj’s own backing table and must itself be a main (not another alternative).

Parameters:

  • obj (Any) – The object to store.

  • as_record (type | None) – The alternate record representation to use, if any.

  • id_series (str | None) – Override the configured entry-id series for minted ids.

  • alternative_of (str | None) – The stored main entry’s id this record is an alternative of, if any.

  • alternative_kind (str | None) – The alternative kind name (grammar [a-z][a-z0-9_]*); required with alternative_of.

  • links (collections.abc.Mapping[str, object] | None) – Weak links to add after saving, mapping each declared link name to a target or iterable of targets; the save and every link are applied in one atomic transaction.

Returns:

The stored row’s sid.

Raises:
Return type:

int

fetch[T](cls, sid, *, eager=False)

Reconstruct the cls instance stored under sid.

By default a lazy row is returned: the parent row is loaded now, but every child, reference and derived field decodes only when first accessed (recursively, so a lazy record’s children are lazy too). Pass eager=True to fully materialize the base dataclass up front — the behaviour required for records that must outlive the fetching transaction, connection or engine.

Repeated default fetches of a live (class, sid) return the same object; a live materialized instance takes precedence over creating a new proxy. Mixing eager and lazy access may hand out two distinct but equal objects when a caller still holds the older one, and internal cache maintenance (a failed write, dedup compensation) may re-materialize a later fetch — strict is identity across arbitrary call sequences is not promised.

Raises KeyError (carrying the class and sid) when no such parent row exists. A missing table therefore has the same result as a missing row. Under the lazy default, abnormal external deletion of a referenced row surfaces at attribute access as StaleResultError; abnormally deleted child rows are indistinguishable from an empty sequence.

Parameters:

  • cls (type[T]) – The storable class to reconstruct.

  • sid (int) – The stored row identifier.

  • eager (bool) – Whether to fully materialize the record instead of returning a lazy row.

Returns:

The reconstructed instance.

Raises:

KeyError – If no row exists for cls and sid.

Return type:

T

fetch_many[T](cls, sids, *, eager=False)

Reconstruct every cls instance stored under sids in one batch.

The batched counterpart of fetch(): child-element and reference reads are shared across the requested rows instead of re-queried per sid. By default lazy rows are returned; they share one RowHydrator, so a deferred child or reference read stays chunk-batched (one SELECT per child table per 500-row chunk on first touch) exactly as the eager path batches it, merely deferred. Pass eager=True to fully materialize every record up front.

Mirroring fetch(), a live cached object (proxy or materialized) is returned for any (class, sid) still alive without touching the database (so a fully cached call issues no SQL); the remaining rows share one connection. Memory is O(len(sids)) — every chunk stays pinned for the batch — so callers pass bounded pages.

Parameters:

  • cls (type[T]) – The storable class to reconstruct.

  • sids (collections.abc.Sequence[int]) – The stored row identifiers to reconstruct.

  • eager (bool) – Whether to fully materialize each record instead of returning lazy rows.

Returns:

The reconstructed instances in sids order.

Raises:

KeyError – If any requested row does not exist.

Return type:

list[T]

fetch_by_content_id[T](cls, key, *, eager=False)

Return the cls instance whose content identity is key, or None if not stored.

Only classes with the "content_id" dedup policy carry a content identity column; SchemaError is raised for any other class. A lazy row is returned by default; pass eager=True to fully materialize it.

Parameters:

  • cls (type[T]) – The storable class to search.

  • key (str) – The content identity to find.

  • eager (bool) – Whether to fully materialize the record instead of returning a lazy row.

Returns:

The stored instance, or None when no row matches.

Raises:

httk.store.backend.schema.SchemaError – If the class does not use content-id deduplication.

Return type:

T | None

fetch_entry(family_cls, content_id, *, eager=False)

Return the concrete configured record for an entry-family content identity.

The result is the actual frozen record class, not the family protocol. A single-record family can query that record directly; only multi-record families use their reserved one-of-many dispatch table, whose constraint permits exactly one backing sid per content identity. A lazy row is returned by default; pass eager=True to fully materialize it.

Parameters:

  • family_cls (type) – The configured entry-family class.

  • content_id (str) – The entry content identity to find.

  • eager (bool) – Whether to fully materialize the record instead of returning a lazy row.

Returns:

The concrete stored record, or None when no row matches.

Raises:
Return type:

object | None

sid_of(obj, *, as_record=None)

Return this store’s sid for obj’s record identity, if present.

Parameters:

  • obj (Any) – The object whose stored identity should be looked up.

  • as_record (type | None) – The alternate record representation to use, if any.

Returns:

The stored sid, or None when no matching row is known.

Return type:

int | None

searcher(*, as_of=None, only_latest=False, only_main_alt=True)

Return a new SqlSearcher querying this store.

The searcher runs on this store’s read path — inside an open transaction() block it sees uncommitted writes — and reconstructs matched objects as lazy rows, decoding each field on first access exactly as the lazy default of fetch() does.

Parameters:

  • as_of (object) – Optional historic cutoff in canonical timestamp form.

  • only_latest (bool) – Whether root variables are restricted to the latest row of each logical_id lineage by sid (bounded by as_of when given). Reference/child variables stay unfiltered. Does not require store_timestamps=True.

  • only_main_alt (bool) – Whether root variables are restricted to mains (alt_kind IS NULL), hiding named alternatives. Defaults to True; pass False to reveal alternatives.

Returns:

A new SQL searcher bound to this store.

Return type:

httk.store.backend.sql.searcher.SqlSearcher

fsck(*, repair=True, collect_garbage=True, repair_conflicts=False, clamp_future_timestamps=False, known_types=(), exclusive=False)

Repair dispatches and reclaim permanentization residue.

Only tables attributable to the persisted layout or known_types are swept; unrelated application tables make collection refuse.

stored_property_plan(family)

Return the wire-form SQL stored-property plan for one configured entry family.

This method is a serving edge: like the entry providers, it serves the family’s definition in OPTIMADE wire form (served_form()), so a prefixed family whose __httk_stored_properties__ keys are served names plans correctly. served_form() is idempotent for standard (unprefixed) families, so their plans are byte-identical.

Parameters:

family (type) – The logical entry-family class to plan.

Returns:

The validated SQL stored-property plan.

Return type:

Any

referring(cls, *, field, to, eager=False)

Return all stored cls instances whose reference field field points at to.

field must be a reference field of cls targeting to’s class (SchemaError otherwise), and to must be known to this store — saved or fetched through it — else ValueError is raised. Results are ordered by sid. Lazy rows are returned by default (batched over the matched sids); pass eager=True to fully materialize them.

Parameters:

  • cls (type) – The storable class whose references should be searched.

  • field (str) – The reference field to match.

  • to (Any) – The stored target instance.

  • eager (bool) – Whether to fully materialize the records instead of returning lazy rows.

Returns:

The referring stored instances ordered by sid.

Raises:
Return type:

list[Any]

replace(predecessor, obj, *, id_series=None, links=None)

Store obj as a logical replacement of predecessor and return its sid.

The saved row copies predecessor’s logical_id (its lineage identity) instead of starting a fresh one, so both rows share the lineage history() walks. Nothing is updated or deleted: plain fetch() and searcher() queries keep returning both rows, and the lineage’s latest row is simply the one with the highest sid. predecessor need not itself be the latest row of its lineage — replacing an already-replaced row is allowed and extends the same lineage. obj is saved through the ordinary save() path, so its dedup policy, timestamp capture, identity caching and entry-family dispatch all behave exactly as they do there.

If obj’s content deduplicates onto an existing row (under the "content_id" or "by_value" policies), that row’s lineage is compared with predecessor’s: an equal lineage (including obj equalling predecessor itself) is an idempotent no-op returning the existing sid, while a different lineage raises EntryReplacementError.

links optionally adds weak links to the replacement’s lineage, as in save(); the replace and every link are applied in one atomic transaction.

Parameters:

  • predecessor (Any) – The stored instance or lazy proxy being replaced; it must have been stored or fetched through this store.

  • obj (Any) – The replacement object to store.

  • id_series (str | None) – Override the configured entry-id series when an id must be minted.

  • links (collections.abc.Mapping[str, object] | None) – Weak links to add, mapping each declared link name to a target or iterable of targets.

Returns:

The stored replacement row’s sid.

Raises:

  • ValueError – If predecessor is not known to this store, or obj’s record table differs from predecessor’s.

  • EntryReplacementError – If obj deduplicates onto a row from a different lineage.

Return type:

int

Assert a weak link named name from source’s lineage to target’s lineage.

Weak links live in a dedicated append-only link table and bind lineages: both endpoints always resolve to their latest revision. The operation is idempotent per the pair (source lineage, target lineage) — a live pair is a no-op, a retracted pair is revived, an absent pair founds a fresh link lineage — and duplicate pair lineages (from concurrent writers) are tolerated: the pair is live if any lineage is live.

Parameters:

  • source (Any) – The stored source instance or lazy proxy declaring the link.

  • name (str) – The declared weak-link name.

  • target (Any) – The stored target instance whose type matches the link declaration.

Returns:

None.

Raises:
Return type:

None

Retract the weak link named name from source’s lineage to target’s lineage.

Retracts every live lineage of the pair (duplicate-tolerant); an absent or already-retracted pair is a no-op. Retraction appends a revision — no row is deleted, and history/as_of still see the earlier live rows.

Parameters:

  • source (Any) – The stored source instance or lazy proxy declaring the link.

  • name (str) – The declared weak-link name.

  • target (Any) – The stored target instance whose type matches the link declaration.

Returns:

None.

Raises:
Return type:

None

linked(source, name, *, eager=False)

Return the latest revisions of the targets currently linked from source under name.

Targets are deduplicated by lineage and ordered by first-link order (the link lineage’s root logical_id, ascending; stable across retract+relink). Each returned object is the latest revision of its target lineage, hydrated through the ordinary fetch machinery (a lazy row by default, fully materialized when eager is true).

Parameters:

  • source (Any) – The stored source instance or lazy proxy declaring the link.

  • name (str) – The declared weak-link name.

  • eager (bool) – Whether to fully materialize each target instead of returning a lazy row.

Returns:

The linked targets’ latest revisions, deduplicated and ordered by first-link order.

Raises:
Return type:

tuple[Any, Ellipsis]

history(obj)

Return every record in obj’s replacement lineage, oldest first.

The lineage is the set of rows sharing obj’s logical_id — the fresh record that started it and every replace() of it — ordered by sid ascending (the fresh record first, the latest replacement last). Records are reconstructed through the same machinery as fetch(), lazily by default.

Parameters:

obj (Any) – A stored instance or lazy proxy whose lineage to walk; it must have been stored or fetched through this store.

Returns:

The lineage’s records ordered by ascending sid.

Raises:

ValueError – If obj is not known to this store.

Return type:

tuple[Any, Ellipsis]