Storing data in a database

Collected results (page Fetching results back) become durable records in a httk-store database. The easy path needs no code: collect straight into a store.

httk workflow collect --workspace WORKSPACE --into results.sqlite

This stores each collected job’s entries plus a provenance Run record — succeeded jobs by default; add --state failed (repeatable --state) to include fetched failures. It is deduplicated on re-collection, so re-running is always safe — a job whose records already exist is skipped rather than duplicated.

The direct store API

To store your own records, open a Backend, declare a store once, and save inside a transaction:

from httk.store import Backend, SqlStore

db = Backend.sqlite("results.sqlite")
store = SqlStore(db, entry_records={})
with store.transaction():
    sid = store.save(record)

Records are frozen dataclasses. Identity is content-addressed: content_id is computed from the record’s canonical JSON, so two records with identical content deduplicate to one row. The returned sid is only the local relational id of that row and can differ between stores. The first open of a store declares the durable representations it may hold; reopen later with just SqlStore(db).

Beyond that storage identity, a defined entry family carries store-minted public ids: an entry id shared by every revision of a lineage, a per-revision immutable_id written <id>~<n>, and named alternative representations addressed as <id>~<kind>. Searches and revision streams return main entries only by default (only_main_alt=True); only_latest=True narrows to each lineage’s latest revision.

In httk v1

The v1 store was backend = httk.db.backend.Sqlite('example.sqlite'); store = httk.db.store.SqlStore(backend); store.save(struct), and a storable result class was declared with @httk.httk_typed_init({...}) on an HttkObject subclass. httk₂ replaces the typed-init classes with ordinary frozen dataclass records carrying storage markers, and the store declaration happens once, at the first open.

In httk v1

Deduplication keyed on an object’s hexhash (struct.hexhash). httk₂ uses content ids (content_id) computed from canonical JSON, and you look a record up with store.fetch_by_content_id(cls, key).

Backends and vocabulary

SQLite, DuckDB, and PostgreSQL sit behind one Backend API — Backend.sqlite(...), Backend.duckdb(...), Backend.postgresql(url) — with the same store surface. MongoDB is available through httk.store.backend.mongo when MongoDB is already the operational data service. Property and entry-type definitions come from the OPTIMADE definition vocabulary, so what you store is what you can later serve.

In httk v1

Querying was a store.searcher() DSL: bind a class to a variable, add conditions, declare outputs, iterate. That shape survives in spirit rather than being replaced — it moved with the store.

See Analyzing the data for querying the stored data with the httk₂ searcher.

If you have an existing v1 database, there is a dedicated migration guide that walks every v1 construct beside its httk₂ replacement.