Data management¶

httk-data is the data-management capability layer of httk₂. httk-core defines the neutral httk.core.EntryProvider contract, the OPTIMADE property/entry-type definition model, and the stdlib-only record dataclasses (Reference, File, Calculation) — but ships no concrete providers and no third-party dependencies. httk-data provides capabilities built on those models:

entry providers that serve the record models through the provider contract,
property-definition validation on jsonschema, and
the database storage layer httk.data.db (see Database storage), which stores plain frozen dataclasses relationally and serves them through the same provider contract.

Entry providers¶

ReferenceEntryProvider, FileEntryProvider, and CalculationEntryProvider map a {id: record} mapping — where each record is a httk-core dataclass (or a plain mapping coerced into one) — onto the neutral contract. A provider answers three questions: which entry types it serves (each described by a first-class httk.core.EntryTypeDefinition), which record key holds each served property (property_keys()), and what the records are (records(), plain JSON-able mappings).

from httk.core import Reference
from httk.data import ReferenceEntryProvider

provider = ReferenceEntryProvider(
    {
        "ref-1": Reference(title="A study of gallium titanium compounds", doi="10.1234/demo.2021.1"),
        "ref-2": {"title": "Silicon dioxide polymorphs revisited", "year": "2019"},
    }
)

# The served entry type is the vendored OPTIMADE 'references' definition:
entry_types = provider.entry_types()
assert list(entry_types) == ["references"]
assert entry_types["references"].properties["title"].optimade_type == "string"

# property_keys() maps served property name -> record key; records() are JSON-able:
property_keys = provider.property_keys("references")
assert property_keys["id"] == "__id" and property_keys["type"] == "type"

records = list(provider.records("references"))
assert {r["__id"] for r in records} == {"ref-1", "ref-2"}
assert records[0]["type"] == "references"

Discovery through the registry¶

The three providers self-register when httk.core discovers the module, under the names data-references, data-files, and data-calculations (the database-backed provider of Database storage registers alongside them as data-db-store). A serving module (such as httk-optimade) can therefore find them through the registry without importing httk-data directly:

import httk.core
from httk.core import known_entry_providers
from httk.core._plugins import resolve_callable
from httk.core.register import entry_providers

registered = set(known_entry_providers())
assert {"data-references", "data-files", "data-calculations"} <= registered

# The registered value is a lazy factory reference; resolve and instantiate it:
factory = resolve_callable(entry_providers.require("data-references").handler)
provider = factory({"ref-1": {"title": "A study"}})
assert list(provider.entry_types()) == ["references"]

To actually serve these providers over HTTP as an OPTIMADE API, hand them to httk-optimade’s adapter_from_providers (see that module’s documentation); neither module imports the other — the httk-core contract is the only coupling.

Validation¶

validate_property checks a single value against one property definition, and validate_record checks a whole record against an entry-type definition. Both build a jsonschema Draft 2020-12 validator directly from the definition’s OPTIMADE document. The definitions are self-contained (no $ref), and the document’s $schema meta-schema reference is removed before validation, so no network access or schema resolution ever happens — validation is fully offline.

from httk.core import standard_entry_type
from httk.data import validate_property, validate_record

references = standard_entry_type("references")

# A single value against one property definition (references.year is a string):
validate_property(references.properties["year"], "2021")

# Nullable properties accept None:
assert references.properties["year"].nullable
validate_property(references.properties["year"], None)

# A whole record against the entry-type definition. Properties absent from the
# record are simply not checked (serving a subset is normal); 'id' and 'type'
# must be present:
validate_record(
    references,
    {"id": "ref-1", "type": "references", "title": "A study", "year": "2021"},
)

Failing validation¶

A value that does not match its definition raises PropertyValidationError (a ValueError). The underlying jsonschema error is kept as the chained cause:

from httk.core import standard_entry_type
from httk.data import PropertyValidationError, validate_property, validate_record

references = standard_entry_type("references")

# 'year' is a string property, so an integer is rejected:
try:
    validate_property(references.properties["year"], 2021)
except PropertyValidationError as exc:
    assert exc.name == "year"
    assert exc.__cause__ is not None  # the jsonschema ValidationError

# Unknown property names are rejected, naming them and the entry type:
try:
    validate_record(references, {"id": "r", "type": "references", "sprocket": 3})
except PropertyValidationError as exc:
    assert "sprocket" in str(exc)

# 'id' and 'type' are required:
try:
    validate_record(references, {"type": "references"})
except PropertyValidationError as exc:
    assert exc.name == "id"

Database-backed serving¶

The entry providers above are in-memory. To store records in a database and serve them the same way, see Database storage: httk.data.db.SqlStore stores plain frozen dataclasses in SQLite or DuckDB, and StoreEntryProvider (registered as data-db-store) serves them through the identical provider contract.