Migrating from httk v1

The database layer of httk v1 (httk.db, with httk.db.backend.Sqlite and httk.db.store.SqlStore) and the storage layer of httk₂ (httk.data.db, see Database storage) solve the same problem, and the shape of a query survives the port almost unchanged: open a searcher, bind classes to variables, add conditions, declare outputs, iterate. What changed is how a class is declared storable, and that several v1 constructs whose meaning depended on context — most of all add versus add_all — were replaced by ones that mean the same thing everywhere.

v1 snippets on this page are historical

Every block marked httk v1 is shown only so you can recognise it in your own scripts. It requires httk v1 and does not run under httk₂. The blocks marked httk₂ are the ones to copy.

What stays the same

store.searcher() still opens a query; search.variable(Cls) still binds a class to a variable; search.add(...) still adds a condition; search.output( variable, name) still declares an output; iterating still runs the query. Attribute access on a variable still builds the joins for you, two variables of the same class still self-join, comparing a reference field against another variable (result.structure == structure) is still how you write a join condition, and set_limit / add_offset / count() are unchanged.

First: add versus add_all

This is the one v1 idiom that silently changes meaning if you port it literally. In v1 the same expression meant different things depending on which method it was passed to:

# httk v1 — historical, does not run under httk₂
search.add(search_struct.formula_symbols.is_in('Na'))              # contains ANY of
search.add_all(search_struct.formula_symbols.is_in('O','Ca','Ti')) # contains ONLY these

Both lines are real v1 code (Examples/4_database/1_store_retrieve.py and Tutorial/Step6/step6_part5.py respectively), and the first even carries the comment “cf. the search for a material system in tutorial step6, where all symbols must be in a set, ‘add’ vs. ‘add_all’”.

In httk₂ the reading is in the expression, not in the method. There is one add(), and each expression carries its own placement:

# httk₂
search.add(s.symbols.has_any("Na"))               # existential: some symbol is Na
search.add(s.symbols.has_only("O", "Ca", "Ti"))   # for-all: every symbol is in the set

s.symbols.is_in("O", "Ca", "Ti") on a child (list-valued) field is the same for-all reading as has_only; on a root field is_in is plain membership. Against records NaCl, CaTiO3 and NaTiO2, has_any matches NaCl and NaTiO2, while has_only and child-field is_in both match CaTiO3 alone.

Old → new

httk v1

httk₂

httk.db.backend.Sqlite(path) then httk.db.store.SqlStore(backend)

Database.sqlite(path) then SqlStore(database)

httk.db.backend.Duckdb(path)

Database.duckdb(path)

store.delay_commit() … store.commit()

with store.transaction():

subclass httk.HttkObject, @httk.httk_typed_init({...}, index=[...], skip=[...])

plain frozen dataclass with Annotated markers (Indexed, Unique, Skip, Shape, Related) and an optional __httk_storage__: ClassVar[StorageInfo]

@httk_typed_property(t)

@stored_property (value type read from the return annotation)

store.save(obj), obj.db.store(store)

store.save(obj)

search.add_all(expr) (a separate post-filter position)

gone — one add(); the expression decides where it applies

col.has_inv_any(...), col.has_inv_only(...)

~col.has_any(...), ~col.has_only(...)

col.like('%x%') with raw SQL wildcards

col.contains("x"), col.startswith(...), col.endswith(...), matching literal text (% and _ match themselves)

add_sort(expr, direction='ASC')

add_sort(field, descending=False)

for match, header in search: then match[0]

iteration yields a SearchResult; use result.values / result.names — result[0][0] still works

obj.db.sid

store.sid_of(obj)

hexhash deduplication

content-id deduplication; look a record up with store.fetch_by_content_id(cls, key)

struct.get_tags(), struct.get_refs() (lazy codependent fetch)

explicit store.referring(TagCls, field="structure", to=struct)

One v1 trap is worth naming because it looked like a query bug: because the v1 table name came from cls.__name__, the storable class had to be re-declared verbatim in the querying script, and search.variable(cls) would silently create the table when it did not exist — so a renamed or slightly mistyped class returned no matches instead of an error. httk₂ still creates a class’s tables on first use, but you import the class rather than re-declare it, and StorageInfo(storage_name="…") pins the stored name so renaming the class cannot orphan the data.

Converting a store-and-search script

The v1 original (Examples/4_database/1_store_retrieve.py) stored two structures with a tag and searched for anything containing sodium:

# httk v1 — historical, does not run under httk₂
import httk, httk.db
from httk.atomistic import Structure

backend = httk.db.backend.Sqlite('example.sqlite')
store = httk.db.store.SqlStore(backend)
tablesalt = httk.load('NaCl.cif')
tablesalt.add_tag('common name', 'salt')
arsenic = httk.load('As.cif')
store.save(tablesalt)
store.save(arsenic)

search = store.searcher()
search_struct = search.variable(Structure)
search.add(search_struct.formula_symbols.is_in('Na'))
search.output(search_struct, 'structure')

for match, header in list(search):
    struct = match[0]
    print("Found structure", struct.formula, [str(struct.get_tags()[x]) for x in struct.get_tags()])

In httk₂ the record class is declared where you use it — a plain frozen dataclass — and the tag becomes an ordinary join class stored like any other, looked up with store.referring:

# httk₂
from dataclasses import dataclass
from typing import Annotated, ClassVar

from httk.core import Indexed, StorageInfo
from httk.data.db import Database, SqlStore


@dataclass(frozen=True)
class Structure:
    formula: Annotated[str, Indexed()]
    symbols: tuple[str, ...]


@dataclass(frozen=True)
class StructureTag:
    __httk_storage__: ClassVar[StorageInfo] = StorageInfo(dedup="by_value")

    structure: Structure
    tag: Annotated[str, Indexed()]
    value: str


store = SqlStore(Database.sqlite("example.sqlite"))

tablesalt = Structure("NaCl", ("Na", "Cl"))
arsenic = Structure("As", ("As",))
with store.transaction():
    store.save(tablesalt)
    store.save(arsenic)
    store.save(StructureTag(tablesalt, "common name", "salt"))

search = store.searcher()
s = search.variable(Structure)
search.add(s.symbols.has_any("Na"))
search.output(s, "structure")
for result in search:
    structure = result.values[0]
    tags = store.referring(StructureTag, field="structure", to=structure)
    print("Found structure", structure.formula, [f"{tag.tag}={tag.value}" for tag in tags])

Point by point: add(... .is_in('Na')) became add(... .has_any("Na")); delay_commit/commit became transaction(); add_tag/get_tags became a stored join class and store.referring; and for match, header in list(search) became a plain loop over SearchResult objects, of which result.values[0] is the first declared output. Note that symbols is a tuple, not a list: an instance with a list field is unhashable, and the store’s object→sid cache (which store.referring and store.sid_of consult) is equality-keyed.

Looking records up by identity

v1 identified a stored object through the obj.db plugin and deduplicated on hexhash. httk₂ has no plugin on the object: the store answers both questions.

# httk₂
from httk.data.db import content_id

nacl = structures[0]
sid = store.sid_of(nacl)                                  # was nacl.db.sid
found = store.fetch_by_content_id(Structure, content_id(nacl))   # was the hexhash lookup

sid_of consults the store’s in-memory cache only — it answers for instances that passed through save() or fetch() and are still alive — while fetch_by_content_id queries the database.

No longer available

These v1 features have no httk₂ replacement:

  • the obj.db plugin and the Storable base class — storability is non-intrusive, and all storage operations go through the store;

  • DictStore and TrivialStore;

  • searcher.store_table(name);

  • searcher.sql() and searcher.sql_query() SQL introspection;

  • search.function(name) custom SQL functions;

  • output(..., concat=True) (GROUP_CONCAT);

  • arithmetic operators on columns, and the ^ (xor) and // (string concatenation) operators;

  • searcher.reset() and searcher.duplicate(other) — build a new searcher;

  • explicit join keys, i.e. variable(Cls, parent=..., parentkey='Compound_id', subkey='compound_Compound_sid'); variable() takes only the class and derives the joins from the schema;

  • the DATABASE_DEBUG and DATABASE_DEBUG_SLOW environment variables.