Storing data in a database

Collected results (page Fetching results back) become durable records in a httk-store database. The easy path needs no code: collect straight into a store.

httk workflow collect --workspace WORKSPACE --into results.sqlite

This stores each collected job’s entries plus a provenance Run record — succeeded jobs by default; add --state failed (repeatable --state) to include fetched failures. It is deduplicated on re-collection, so re-running is always safe — a job whose records already exist is skipped rather than duplicated.

The direct store API

To store your own records, open a store for the chosen database and save:

from httk.store import SqliteStore

store = SqliteStore("results.sqlite", entry_records={})
sid = store.save(record)
store.close()

Records are frozen dataclasses. Identity is content-addressed: content_id is computed from the record’s canonical JSON, so two records with identical content deduplicate to one row. The returned sid is only the local relational id of that row and can differ between stores. The first open of a store declares the durable representations it may hold; reopen later with just SqliteStore("results.sqlite"). Use with store.transaction(): when several writes should commit together. The convenience store owns its connection; close it explicitly or use a with SqliteStore(...) as store: block.

For records intended for OPTIMADE serving, see the three-file example. It uses @entry_record and DataEntryRecord to supply the common metadata and property mappings, and records=[Result] on both creation and reopening. That explicit application class declaration is required each time; the database does not import it.

Beyond that storage identity, a defined entry family carries store-minted public ids: an entry id shared by every revision of a lineage, a per-revision immutable_id written <id>~<n>, and named alternative representations addressed as <id>~<kind>. Searches and revision streams return main entries only by default (only_main_alt=True); only_latest=True narrows to each lineage’s latest revision.

In httk v1

The v1 store was backend = httk.db.backend.Sqlite('example.sqlite'); store = httk.db.store.SqlStore(backend); store.save(struct), and a storable result class was declared with @httk.httk_typed_init({...}) on an HttkObject subclass. httk₂ replaces the typed-init classes with ordinary frozen dataclass records carrying storage markers, and the store declaration happens once, at the first open.

In httk v1

Deduplication keyed on an object’s hexhash (struct.hexhash). httk₂ uses content ids (content_id) computed from canonical JSON, and you look a record up with store.fetch_by_content_id(cls, key).

Backends and vocabulary

SQLite, DuckDB, PostgreSQL, and ClickHouse each have a dedicated store class — SqliteStore(...), DuckdbStore(...), PostgresqlStore(url), and ClickhouseStore(url). ClickHouse supports bulk ingestion and read serving, subject to its write restrictions. The two-object form SqlStore(Backend.sqlite(...)) remains for custom SQLAlchemy engines or a Backend shared across several stores. MongoDB is available through httk.store.backend.mongo when MongoDB is already the operational data service. Property and entry-type definitions come from the OPTIMADE definition vocabulary, so what you store is what you can later serve.

In httk v1

Querying was a store.searcher() DSL: bind a class to a variable, add conditions, declare outputs, iterate. That shape survives in spirit rather than being replaced — it moved with the store.

See Analyzing the data for querying the stored data with the httk₂ searcher.

If you have an existing v1 database, there is a dedicated migration guide that walks every v1 construct beside its httk₂ replacement.