httk.data

httk-data: the data-management capability layer for httk v2.

Built on the stdlib-only contracts and models in httk-core, httk-data supplies capabilities:

• in-memory EntryProvider implementations for the standard OPTIMADE entry types (ReferenceEntryProvider, FileEntryProvider, CalculationEntryProvider), serving httk-core’s record models through the neutral provider contract; and

• property-definition validation (validate_property(), validate_record(), PropertyValidationError) built on jsonschema (Draft 2020-12), checking record values against their OPTIMADE property definitions fully offline; and

• the store/searcher query protocols (httk.data.query) — the backend-agnostic query contract implemented by httk data stores and consumed by serving modules; and

• the generic OPTIMADE filter translation (httk.data.optimade_query) — turning filter syntax trees parsed by httk.core.parse_optimade_filter() into search expressions over the query protocols (translate_filter_ast(), filter_searcher()), with neutral FilterTranslationError categories; and

• the database storage layer (httk.data.db, requiring the httk-data[db] extra) — relational storage and querying of plain frozen dataclasses (SqlStore over SQLite or DuckDB), served through the provider contract by StoreEntryProvider.

The providers self-register (under httk.handlers.data, as data-references/data-files/data-calculations/data-db-store) when httk.core discovers the module, so a serving module (such as httk-optimade) can find them through the registry.

Submodules

• httk.data.db
• httk.data.entry_providers
• httk.data.optimade_query
• httk.data.query
• httk.data.validation

Attributes

FilterTranslationCategory	Why a filter could not be translated (see FilterTranslationError).
HandlerTable	Per-property translation callables, keyed by property name.
RelatedPropertyResolver	Resolve a depth-1 relationship filter to the matching related-entry ids.
invert_op

Exceptions

FilterTranslationError	A filter cannot be translated into a search expression.
PropertyValidationError	A value did not conform to its OPTIMADE property definition.

Classes

CalculationEntryProvider	Serves OPTIMADE calculations from a mapping of id to Calculation.
FileEntryProvider	Serves OPTIMADE files from a mapping of id to File.
ReferenceEntryProvider	Serves OPTIMADE references from a mapping of id to Reference.
Searcher	A single query under construction, and its results once iterated.
SearchExpression	Base class for protocol classes.
SearchField	A queryable field of a search variable.
SearchResult	One match: the declared output values, and the names they were declared under.
SearchVariable	A query variable bound to a target type; attribute access yields fields.
Store	Base class for protocol classes.

Functions

filter_searcher(, relationship_targets, Ellipsis] =, ...)	Build a Searcher over store applying an OPTIMADE filter.
format_value(→ Any)	Convert a filter constant node to a Python value of the property's fulltype.
relationship_id_handler(→ collections.abc.Mapping[str, ...)	A '<type>.id' handler-table entry matching related ids over a list-valued field.
simple_property_handlers(→ dict[str, ...)	Build a filter handler table for an entry type from a property-key map.
translate_filter_ast(, related_property_resolver)	Translate one filter syntax-tree node into a search expression.
validate_property(→ None)	Validate a single value against definition's JSON-Schema payload.
validate_record(→ None)	Validate every property present in record against entry_type.

Package Contents

class httk.data.CalculationEntryProvider(entries: collections.abc.Mapping[str, httk.core.Calculation | collections.abc.Mapping[str, Any]], *, relationships: collections.abc.Mapping[str, collections.abc.Iterable[httk.core.RelatedEntry]] | None = None)

Bases: httk.core.EntryProvider

Serves OPTIMADE calculations from a mapping of id to Calculation.

relationships optionally maps a calculation id to its related entries (RelatedEntry values, served flat per id) — e.g. its input/output files, expressed via the role metadata.

entry_types() → collections.abc.Mapping[str, httk.core.EntryTypeDefinition]

Return the served entry types keyed by name.

Each value is an EntryTypeDefinition describing the entry type and its properties. The subset a provider actually serves is named by property_keys(); a definition may describe more properties than are served.

property_keys(entry_type: str) → collections.abc.Mapping[str, str]

Return the served-property-name to record-key map for entry_type.

The mapping MUST include entries for at least id and type. Every key names a property described by entry_types(); every value names the key under which that property’s value is found in a record from records().

records(entry_type: str) → collections.abc.Iterable[collections.abc.Mapping[str, Any]]

Yield the records for entry_type as plain JSON-able mappings.

Each record is a mapping keyed by the record keys named in property_keys(); values are JSON-able (strings, numbers, booleans, None, or nested lists/dicts of the same).

relationships(entry_type: str) → collections.abc.Mapping[str, tuple[httk.core.RelatedEntry, Ellipsis]]

Return the related entries for each record of entry_type.

The result maps an entry id to a flat tuple of RelatedEntry values, e.g. {"struct-1": (RelatedEntry("references", "ref-1"), RelatedEntry("references", "ref-2", description="Cites the method"))}. Grouping the related entries by related entry type is the serving layer’s concern (JSON:API groups them at render time). This is the neutral source of an OPTIMADE relationships block: a consumer turns each related entry into a resource identifier under its entry type (carrying the description/role metadata when present), and an include=<type> request then embeds those related resources. The default implementation returns an empty mapping (no relationships); a provider overrides it to declare them. Ids referring to records this provider (or a sibling provider serving the related type) does not supply are simply not resolvable by the consumer.

class httk.data.FileEntryProvider(entries: collections.abc.Mapping[str, httk.core.File | collections.abc.Mapping[str, Any]], *, relationships: collections.abc.Mapping[str, collections.abc.Iterable[httk.core.RelatedEntry]] | None = None)

Bases: httk.core.EntryProvider

Serves OPTIMADE files from a mapping of id to File.

relationships optionally maps a file id to its related entries (RelatedEntry values, served flat per id) — e.g. the calculations a file is input/output of.

entry_types() → collections.abc.Mapping[str, httk.core.EntryTypeDefinition]

Return the served entry types keyed by name.

Each value is an EntryTypeDefinition describing the entry type and its properties. The subset a provider actually serves is named by property_keys(); a definition may describe more properties than are served.

property_keys(entry_type: str) → collections.abc.Mapping[str, str]

Return the served-property-name to record-key map for entry_type.

The mapping MUST include entries for at least id and type. Every key names a property described by entry_types(); every value names the key under which that property’s value is found in a record from records().

records(entry_type: str) → collections.abc.Iterable[collections.abc.Mapping[str, Any]]

Yield the records for entry_type as plain JSON-able mappings.

Each record is a mapping keyed by the record keys named in property_keys(); values are JSON-able (strings, numbers, booleans, None, or nested lists/dicts of the same).

relationships(entry_type: str) → collections.abc.Mapping[str, tuple[httk.core.RelatedEntry, Ellipsis]]

Return the related entries for each record of entry_type.

The result maps an entry id to a flat tuple of RelatedEntry values, e.g. {"struct-1": (RelatedEntry("references", "ref-1"), RelatedEntry("references", "ref-2", description="Cites the method"))}. Grouping the related entries by related entry type is the serving layer’s concern (JSON:API groups them at render time). This is the neutral source of an OPTIMADE relationships block: a consumer turns each related entry into a resource identifier under its entry type (carrying the description/role metadata when present), and an include=<type> request then embeds those related resources. The default implementation returns an empty mapping (no relationships); a provider overrides it to declare them. Ids referring to records this provider (or a sibling provider serving the related type) does not supply are simply not resolvable by the consumer.

class httk.data.ReferenceEntryProvider(entries: collections.abc.Mapping[str, httk.core.Reference | collections.abc.Mapping[str, Any]], *, relationships: collections.abc.Mapping[str, collections.abc.Iterable[httk.core.RelatedEntry]] | None = None)

Bases: httk.core.EntryProvider

Serves OPTIMADE references from a mapping of id to Reference.

relationships optionally maps a reference id to its related entries (RelatedEntry values, served flat per id).

entry_types() → collections.abc.Mapping[str, httk.core.EntryTypeDefinition]

Return the served entry types keyed by name.

Each value is an EntryTypeDefinition describing the entry type and its properties. The subset a provider actually serves is named by property_keys(); a definition may describe more properties than are served.

property_keys(entry_type: str) → collections.abc.Mapping[str, str]

Return the served-property-name to record-key map for entry_type.

The mapping MUST include entries for at least id and type. Every key names a property described by entry_types(); every value names the key under which that property’s value is found in a record from records().

records(entry_type: str) → collections.abc.Iterable[collections.abc.Mapping[str, Any]]

Yield the records for entry_type as plain JSON-able mappings.

Each record is a mapping keyed by the record keys named in property_keys(); values are JSON-able (strings, numbers, booleans, None, or nested lists/dicts of the same).

relationships(entry_type: str) → collections.abc.Mapping[str, tuple[httk.core.RelatedEntry, Ellipsis]]

Return the related entries for each record of entry_type.

The result maps an entry id to a flat tuple of RelatedEntry values, e.g. {"struct-1": (RelatedEntry("references", "ref-1"), RelatedEntry("references", "ref-2", description="Cites the method"))}. Grouping the related entries by related entry type is the serving layer’s concern (JSON:API groups them at render time). This is the neutral source of an OPTIMADE relationships block: a consumer turns each related entry into a resource identifier under its entry type (carrying the description/role metadata when present), and an include=<type> request then embeds those related resources. The default implementation returns an empty mapping (no relationships); a provider overrides it to declare them. Ids referring to records this provider (or a sibling provider serving the related type) does not supply are simply not resolvable by the consumer.

type httk.data.FilterTranslationCategory = Literal['unrecognized-property', 'not-implemented', 'type-mismatch', 'internal']

Why a filter could not be translated (see FilterTranslationError).

exception httk.data.FilterTranslationError(message: str, category: FilterTranslationCategory, detail: str | None = None)

Bases: Exception

A filter cannot be translated into a search expression.

The exception message describes the failure; category classifies it neutrally (this module knows nothing about transports, so consumers map each category onto their own error codes):

• "unrecognized-property" — the filter names an unknown property carrying a recognized prefix (a caller error);

• "type-mismatch" — a filter constant does not match the property’s declared type (a caller error);

• "not-implemented" — the filter uses a construct this translation (or the supplied handler table) does not support;

• "internal" — an inconsistency in the translation itself.

detail optionally carries extra machine-readable context.

category: FilterTranslationCategory

detail = None

type httk.data.HandlerTable = Mapping[str, Mapping[str, Callable[..., Any]]]

Per-property translation callables, keyed by property name.

The inner mapping’s keys name the filter-operation families: 'comparison' (=, !=, <, <=, >, >=), 'stringmatching' (CONTAINS/STARTS/ENDS), 'HAS' (the set operations), 'length' (LENGTH), and 'unknown' (IS KNOWN/IS UNKNOWN). A 'HAS' handler is called as handler(property, ops, values, search_variable, has_type) and returns a plain SearchExpression. The caller applies NOT as ~; handlers do not receive negation state or report post-filter placement. Dotted '<type>.id' entries provide relationship-id filtering (see relationship_id_handler()).

type httk.data.RelatedPropertyResolver = Callable[[str, FilterAst], tuple[str, ...]]

Resolve a depth-1 relationship filter to the matching related-entry ids.

Called as resolver(related_type, sub_ast) where sub_ast is the filter node with the <related_type>. prefix stripped from its identifier (and without any surrounding NOT); returns the ids of the related entries matching the sub-filter, evaluated against the related type’s own properties.

httk.data.filter_searcher(store: httk.data.query.Store, target: Any, filter_string: str | httk.core.FilterAst, *, entry_type: str, property_fulltypes: collections.abc.Mapping[str, str], property_keys: collections.abc.Mapping[str, str] | None = None, handlers: HandlerTable | None = None, recognized_prefixes: tuple[str, Ellipsis] = (), relationship_targets: tuple[str, Ellipsis] = (), related_property_resolver: RelatedPropertyResolver | None = None) → httk.data.query.Searcher

Build a Searcher over store applying an OPTIMADE filter.

filter_string is an OPTIMADE filter string (parsed with httk.core.parse_optimade_filter()) or an already-parsed FilterAst. The searcher binds one search variable to target (the store-specific query target, declared as the searcher output named entry_type) and applies the translated filter. When handlers is not supplied, a default table is built with simple_property_handlers() from property_keys (or, when property_keys is also None, from an identity map over property_fulltypes). The remaining keyword arguments are passed through to translate_filter_ast().

Raises:

• FilterTranslationError – When the filter cannot be translated.

• httk.core.ParserSyntaxError – When a filter string does not parse.

httk.data.format_value(fulltype: str, val: tuple[Any, Ellipsis], allow_null: bool = False) → Any

Convert a filter constant node to a Python value of the property’s fulltype.

Raises:

FilterTranslationError – With category "type-mismatch" when the constant does not fit the declared type, or "not-implemented" for dictionary-typed properties.

httk.data.invert_op

httk.data.relationship_id_handler(rel_key: str) → collections.abc.Mapping[str, collections.abc.Callable[Ellipsis, Any]]

A '<type>.id' handler-table entry matching related ids over a list-valued field.

rel_key names a list-valued backend field holding the related entry ids; the returned {'HAS': ...} mapping serves <type>.id HAS ... filters (and the semi-join rewrites of other dotted filters) with the standard set-operation semantics of set_handler().

httk.data.simple_property_handlers(entry_type: str, property_keys: collections.abc.Mapping[str, str], property_fulltypes: collections.abc.Mapping[str, str]) → dict[str, collections.abc.Mapping[str, collections.abc.Callable[Ellipsis, Any]]]

Build a filter handler table for an entry type from a property-key map.

Always provides the standard id (matched against the __id field) and type (a constant equal to entry_type) handlers. For every property named in property_keys (which maps property names to backend field names), handlers are generated from the property’s fulltype in property_fulltypes (default "string"): string properties get comparison and stringmatching handlers; integer and float properties get a numeric comparison handler; list of ... properties get a HAS (set membership) handler. Every generated property also gets a known unknown handler.

httk.data.translate_filter_ast(node: httk.core.FilterAst, search_variable: httk.data.query.SearchVariable, entry_type: str, property_fulltypes: collections.abc.Mapping[str, str], handlers: HandlerTable, recognized_prefixes: tuple[str, Ellipsis], *, recursion: int = 0, relationship_targets: tuple[str, Ellipsis] = (), related_property_resolver: RelatedPropertyResolver | None = None) → httk.data.query.SearchExpression

Translate one filter syntax-tree node into a search expression.

node is a FilterAst node (as produced by httk.core.parse_optimade_filter()); search_variable is the backend search variable the expression is built against; property_fulltypes, handlers, and recognized_prefixes drive the translation as described in the module docstring; recursion counts the nesting depth.

Returns the translated expression, ready to hand to add(). NOT translates to the backend’s ~; a backend whose set predicates need a second (post-filter) evaluation position derives that from the expression itself, so the caller never has to.

Relationship filtering: an identifier dotted with a name in relationship_targets (e.g. ('Identifier', 'references', 'doi')) filters on the properties of related entries. <type>.id HAS ... is served directly by the '<type>.id' handler-table entry. Every other depth-1 dotted filter — comparisons (including <type>.id = ...), string matching, IS KNOWN/IS UNKNOWN, and the HAS family over related list properties — is resolved by a two-phase semi-join: the <type>. prefix is stripped from the node, the resulting sub-filter is passed to related_property_resolver (without any surrounding NOT — inversion applies to the resulting id-set membership), and the returned ids are substituted back as <type>.id HAS ANY <ids> (an empty id set translates to a constant-false expression). Each dotted node is resolved independently: references.doi CONTAINS "x" AND references.year >= 2000 matches entries where some related reference matches the doi condition and some — possibly different — related reference matches the year condition. Without a resolver, dotted filters other than <type>.id HAS ... are not implemented; nested (deeper than depth-1) paths and dotted LENGTH filters are never supported.

Raises:

FilterTranslationError – See FilterTranslationError for the failure categories.

class httk.data.Searcher

Bases: Protocol

A single query under construction, and its results once iterated.

Iteration yields one SearchResult per match, so item[0][0] is the first declared output of the match (typically the matched row object). The expressions received by add are always ones produced by this same backend’s search variables, so implementations may type them as their own expression class; a backend that needs a second (post-filter) evaluation position decides that from the expression itself, not from the caller.

offset: int

variable(target: Any) → Any

output(variable: Any, name: str) → None

add(expression: Any) → None

count() → int

set_limit(limit: int) → None

add_offset(offset: int) → None

add_sort(field: Any, descending: bool) → None

class httk.data.SearchExpression

Bases: Protocol

Base class for protocol classes.

Protocol classes are defined as:

class Proto(Protocol):
    def meth(self) -> int:
        ...

Such classes are primarily used with static type checkers that recognize structural subtyping (static duck-typing).

For example:

class C:
    def meth(self) -> int:
        return 0

def func(x: Proto) -> int:
    return x.meth()

func(C())  # Passes static type check

See PEP 544 for details. Protocol classes decorated with @typing.runtime_checkable act as simple-minded runtime protocols that check only the presence of given attributes, ignoring their type signatures. Protocol classes can be generic, they are defined as:

class GenProto[T](Protocol):
    def meth(self) -> T:
        ...

class httk.data.SearchField

Bases: Protocol

A queryable field of a search variable.

In addition to the methods below, fields support the rich comparison operators (==, !=, <, <=, >, >=), returning SearchExpression. The handlers invoke those via getattr(field, '__eq__')(value) since the comparison dunders cannot be typed as expression-returning.

The three string-matching methods take literal text: no wildcard or pattern syntax whatsoever crosses this contract, so % and _ (and any other metacharacter) match themselves. A backend is therefore free to implement them with SQL LIKE over an escaped pattern, with a regular expression, or with a full-text index — the choice is invisible here.

has_any(*values: Any) → SearchExpression

has_only(*values: Any) → SearchExpression

contains(text: str) → SearchExpression

Match values containing text as a literal substring.

startswith(prefix: str) → SearchExpression

Match values beginning with the literal prefix.

endswith(suffix: str) → SearchExpression

Match values ending with the literal suffix.

class httk.data.SearchResult

Bases: NamedTuple

One match: the declared output values, and the names they were declared under.

values holds one entry per Searcher.output() call in declaration order; it is a tuple, so values, names = result and result[0][0] both work.

values: tuple[Any, Ellipsis]

names: tuple[str, Ellipsis]

class httk.data.SearchVariable

Bases: Protocol

A query variable bound to a target type; attribute access yields fields.

always_true/always_false are reserved names: they are real methods of the variable, never stored fields resolved through __getattr__. They exist so a translation layer can express a constant truth value without inventing a probe field. A field == field probe is NULL-unsound, since it yields NULL (not true) for a NULL field.

always_true() → SearchExpression

An expression that matches every row.

always_false() → SearchExpression

An expression that matches no row.

class httk.data.Store

Bases: Protocol

Base class for protocol classes.

Protocol classes are defined as:

class Proto(Protocol):
    def meth(self) -> int:
        ...

Such classes are primarily used with static type checkers that recognize structural subtyping (static duck-typing).

For example:

class C:
    def meth(self) -> int:
        return 0

def func(x: Proto) -> int:
    return x.meth()

func(C())  # Passes static type check

See PEP 544 for details. Protocol classes decorated with @typing.runtime_checkable act as simple-minded runtime protocols that check only the presence of given attributes, ignoring their type signatures. Protocol classes can be generic, they are defined as:

class GenProto[T](Protocol):
    def meth(self) -> T:
        ...

searcher() → Searcher

exception httk.data.PropertyValidationError(name: str, message: str)

Bases: ValueError

A value did not conform to its OPTIMADE property definition.

Carries the offending property name and a human-readable message. For single-value failures the message wraps the underlying jsonschema error message, and that jsonschema.exceptions.ValidationError is preserved as the chained __cause__.

name

message

httk.data.validate_property(definition: httk.core.PropertyDefinition, value: Any) → None

Validate a single value against definition’s JSON-Schema payload.

Builds a jsonschema.Draft202012Validator directly from the definition’s document (with the $schema meta-schema reference removed) and validates value against it. Returns None on success; raises PropertyValidationError on failure, chaining the underlying jsonschema.exceptions.ValidationError as the cause. No network access or registry lookup ever happens.

httk.data.validate_record(entry_type: httk.core.EntryTypeDefinition, record: collections.abc.Mapping[str, Any]) → None

Validate every property present in record against entry_type.

Each key in record must be described by entry_type; unknown property names are rejected with a PropertyValidationError naming them and the entry type. id and type must both be present. Properties described by the definition but absent from record are simply not checked (serving a subset of the described properties is normal). The value of every property that is present is validated via validate_property(). Returns None on success.