httk.data ========= .. py:module:: httk.data .. autoapi-nested-parse:: httk-data: the data-management capability layer for httk v2. Built on the stdlib-only *contracts and models* in *httk-core*, httk-data supplies *capabilities*: - in-memory :class:`~httk.core.EntryProvider` implementations for the standard OPTIMADE entry types (:class:`ReferenceEntryProvider`, :class:`FileEntryProvider`, :class:`CalculationEntryProvider`), serving httk-core's record models through the neutral provider contract; and - **property-definition validation** (:func:`validate_property`, :func:`validate_record`, :class:`PropertyValidationError`) built on ``jsonschema`` (Draft 2020-12), checking record values against their OPTIMADE property definitions fully offline; and - the **store/searcher query protocols** (:mod:`httk.data.query`) — the backend-agnostic query contract implemented by httk data stores and consumed by serving modules; and - the **generic OPTIMADE filter translation** (:mod:`httk.data.optimade_query`) — turning filter syntax trees parsed by :func:`httk.core.parse_optimade_filter` into search expressions over the query protocols (:func:`~httk.data.optimade_query.translate_filter_ast`, :func:`filter_searcher`), with neutral :class:`FilterTranslationError` categories; and - the **database storage layer** (:mod:`httk.data.db`, requiring the ``httk-data[db]`` extra) — relational storage and querying of plain frozen dataclasses (:class:`~httk.data.db.store.SqlStore` over SQLite or DuckDB), served through the provider contract by :class:`~httk.data.db.entry_provider.StoreEntryProvider`. The providers self-register (under ``httk.handlers.data``, as ``data-references``/``data-files``/``data-calculations``/``data-db-store``) when ``httk.core`` discovers the module, so a serving module (such as *httk-optimade*) can find them through the registry. Submodules ---------- .. toctree:: :maxdepth: 1 /reference/autoapi/httk/data/db/index /reference/autoapi/httk/data/entry_providers/index /reference/autoapi/httk/data/optimade_query/index /reference/autoapi/httk/data/query/index /reference/autoapi/httk/data/validation/index Attributes ---------- .. autoapisummary:: httk.data.FilterTranslationCategory httk.data.HandlerTable httk.data.RelatedPropertyResolver httk.data.invert_op Exceptions ---------- .. autoapisummary:: httk.data.FilterTranslationError httk.data.PropertyValidationError Classes ------- .. autoapisummary:: httk.data.CalculationEntryProvider httk.data.FileEntryProvider httk.data.ReferenceEntryProvider httk.data.Searcher httk.data.SearchExpression httk.data.SearchField httk.data.SearchResult httk.data.SearchVariable httk.data.Store Functions --------- .. autoapisummary:: httk.data.filter_searcher httk.data.format_value httk.data.relationship_id_handler httk.data.simple_property_handlers httk.data.translate_filter_ast httk.data.validate_property httk.data.validate_record Package Contents ---------------- .. py:class:: CalculationEntryProvider(entries: collections.abc.Mapping[str, httk.core.Calculation | collections.abc.Mapping[str, Any]], *, relationships: collections.abc.Mapping[str, collections.abc.Iterable[httk.core.RelatedEntry]] | None = None) Bases: :py:obj:`httk.core.EntryProvider` Serves OPTIMADE ``calculations`` from a mapping of id to :class:`~httk.core.Calculation`. ``relationships`` optionally maps a calculation id to its related entries (:class:`~httk.core.RelatedEntry` values, served flat per id) — e.g. its ``input``/``output`` files, expressed via the ``role`` metadata. .. py:method:: entry_types() -> collections.abc.Mapping[str, httk.core.EntryTypeDefinition] Return the served entry types keyed by name. Each value is an :class:`~httk.core.property_definitions.EntryTypeDefinition` describing the entry type and its properties. The subset a provider actually serves is named by :meth:`property_keys`; a definition may describe more properties than are served. .. py:method:: property_keys(entry_type: str) -> collections.abc.Mapping[str, str] Return the served-property-name to record-key map for ``entry_type``. The mapping MUST include entries for at least ``id`` and ``type``. Every key names a property described by :meth:`entry_types`; every value names the key under which that property's value is found in a record from :meth:`records`. .. py:method:: records(entry_type: str) -> collections.abc.Iterable[collections.abc.Mapping[str, Any]] Yield the records for ``entry_type`` as plain JSON-able mappings. Each record is a mapping keyed by the record keys named in :meth:`property_keys`; values are JSON-able (strings, numbers, booleans, ``None``, or nested lists/dicts of the same). .. py:method:: relationships(entry_type: str) -> collections.abc.Mapping[str, tuple[httk.core.RelatedEntry, Ellipsis]] Return the related entries for each record of ``entry_type``. The result maps an entry id to a flat tuple of :class:`~httk.core.RelatedEntry` values, e.g. ``{"struct-1": (RelatedEntry("references", "ref-1"), RelatedEntry("references", "ref-2", description="Cites the method"))}``. Grouping the related entries by related entry type is the serving layer's concern (JSON:API groups them at render time). This is the neutral source of an OPTIMADE **relationships** block: a consumer turns each related entry into a resource identifier under its entry type (carrying the ``description``/``role`` metadata when present), and an ``include=`` request then embeds those related resources. The default implementation returns an empty mapping (no relationships); a provider overrides it to declare them. Ids referring to records this provider (or a sibling provider serving the related type) does not supply are simply not resolvable by the consumer. .. py:class:: FileEntryProvider(entries: collections.abc.Mapping[str, httk.core.File | collections.abc.Mapping[str, Any]], *, relationships: collections.abc.Mapping[str, collections.abc.Iterable[httk.core.RelatedEntry]] | None = None) Bases: :py:obj:`httk.core.EntryProvider` Serves OPTIMADE ``files`` from a mapping of id to :class:`~httk.core.File`. ``relationships`` optionally maps a file id to its related entries (:class:`~httk.core.RelatedEntry` values, served flat per id) — e.g. the calculations a file is ``input``/``output`` of. .. py:method:: entry_types() -> collections.abc.Mapping[str, httk.core.EntryTypeDefinition] Return the served entry types keyed by name. Each value is an :class:`~httk.core.property_definitions.EntryTypeDefinition` describing the entry type and its properties. The subset a provider actually serves is named by :meth:`property_keys`; a definition may describe more properties than are served. .. py:method:: property_keys(entry_type: str) -> collections.abc.Mapping[str, str] Return the served-property-name to record-key map for ``entry_type``. The mapping MUST include entries for at least ``id`` and ``type``. Every key names a property described by :meth:`entry_types`; every value names the key under which that property's value is found in a record from :meth:`records`. .. py:method:: records(entry_type: str) -> collections.abc.Iterable[collections.abc.Mapping[str, Any]] Yield the records for ``entry_type`` as plain JSON-able mappings. Each record is a mapping keyed by the record keys named in :meth:`property_keys`; values are JSON-able (strings, numbers, booleans, ``None``, or nested lists/dicts of the same). .. py:method:: relationships(entry_type: str) -> collections.abc.Mapping[str, tuple[httk.core.RelatedEntry, Ellipsis]] Return the related entries for each record of ``entry_type``. The result maps an entry id to a flat tuple of :class:`~httk.core.RelatedEntry` values, e.g. ``{"struct-1": (RelatedEntry("references", "ref-1"), RelatedEntry("references", "ref-2", description="Cites the method"))}``. Grouping the related entries by related entry type is the serving layer's concern (JSON:API groups them at render time). This is the neutral source of an OPTIMADE **relationships** block: a consumer turns each related entry into a resource identifier under its entry type (carrying the ``description``/``role`` metadata when present), and an ``include=`` request then embeds those related resources. The default implementation returns an empty mapping (no relationships); a provider overrides it to declare them. Ids referring to records this provider (or a sibling provider serving the related type) does not supply are simply not resolvable by the consumer. .. py:class:: ReferenceEntryProvider(entries: collections.abc.Mapping[str, httk.core.Reference | collections.abc.Mapping[str, Any]], *, relationships: collections.abc.Mapping[str, collections.abc.Iterable[httk.core.RelatedEntry]] | None = None) Bases: :py:obj:`httk.core.EntryProvider` Serves OPTIMADE ``references`` from a mapping of id to :class:`~httk.core.Reference`. ``relationships`` optionally maps a reference id to its related entries (:class:`~httk.core.RelatedEntry` values, served flat per id). .. py:method:: entry_types() -> collections.abc.Mapping[str, httk.core.EntryTypeDefinition] Return the served entry types keyed by name. Each value is an :class:`~httk.core.property_definitions.EntryTypeDefinition` describing the entry type and its properties. The subset a provider actually serves is named by :meth:`property_keys`; a definition may describe more properties than are served. .. py:method:: property_keys(entry_type: str) -> collections.abc.Mapping[str, str] Return the served-property-name to record-key map for ``entry_type``. The mapping MUST include entries for at least ``id`` and ``type``. Every key names a property described by :meth:`entry_types`; every value names the key under which that property's value is found in a record from :meth:`records`. .. py:method:: records(entry_type: str) -> collections.abc.Iterable[collections.abc.Mapping[str, Any]] Yield the records for ``entry_type`` as plain JSON-able mappings. Each record is a mapping keyed by the record keys named in :meth:`property_keys`; values are JSON-able (strings, numbers, booleans, ``None``, or nested lists/dicts of the same). .. py:method:: relationships(entry_type: str) -> collections.abc.Mapping[str, tuple[httk.core.RelatedEntry, Ellipsis]] Return the related entries for each record of ``entry_type``. The result maps an entry id to a flat tuple of :class:`~httk.core.RelatedEntry` values, e.g. ``{"struct-1": (RelatedEntry("references", "ref-1"), RelatedEntry("references", "ref-2", description="Cites the method"))}``. Grouping the related entries by related entry type is the serving layer's concern (JSON:API groups them at render time). This is the neutral source of an OPTIMADE **relationships** block: a consumer turns each related entry into a resource identifier under its entry type (carrying the ``description``/``role`` metadata when present), and an ``include=`` request then embeds those related resources. The default implementation returns an empty mapping (no relationships); a provider overrides it to declare them. Ids referring to records this provider (or a sibling provider serving the related type) does not supply are simply not resolvable by the consumer. .. py:type:: FilterTranslationCategory :canonical: Literal['unrecognized-property', 'not-implemented', 'type-mismatch', 'internal'] Why a filter could not be translated (see :class:`FilterTranslationError`). .. py:exception:: FilterTranslationError(message: str, category: FilterTranslationCategory, detail: str | None = None) Bases: :py:obj:`Exception` A filter cannot be translated into a search expression. The exception message describes the failure; :attr:`category` classifies it neutrally (this module knows nothing about transports, so consumers map each category onto their own error codes): - ``"unrecognized-property"`` — the filter names an unknown property carrying a recognized prefix (a caller error); - ``"type-mismatch"`` — a filter constant does not match the property's declared type (a caller error); - ``"not-implemented"`` — the filter uses a construct this translation (or the supplied handler table) does not support; - ``"internal"`` — an inconsistency in the translation itself. ``detail`` optionally carries extra machine-readable context. .. py:attribute:: category :type: FilterTranslationCategory .. py:attribute:: detail :value: None .. py:type:: HandlerTable :canonical: Mapping[str, Mapping[str, Callable[..., Any]]] Per-property translation callables, keyed by property name. The inner mapping's keys name the filter-operation families: ``'comparison'`` (``=``, ``!=``, ``<``, ``<=``, ``>``, ``>=``), ``'stringmatching'`` (``CONTAINS``/``STARTS``/``ENDS``), ``'HAS'`` (the set operations), ``'length'`` (``LENGTH``), and ``'unknown'`` (``IS KNOWN``/``IS UNKNOWN``). A ``'HAS'`` handler is called as ``handler(property, ops, values, search_variable, has_type)`` and returns a plain :class:`~httk.data.query.SearchExpression`. The caller applies ``NOT`` as ``~``; handlers do not receive negation state or report post-filter placement. Dotted ``'.id'`` entries provide relationship-id filtering (see :func:`~httk.data.optimade_query.relationship_id_handler`). .. py:type:: RelatedPropertyResolver :canonical: Callable[[str, FilterAst], tuple[str, ...]] Resolve a depth-1 relationship filter to the matching related-entry ids. Called as ``resolver(related_type, sub_ast)`` where ``sub_ast`` is the filter node with the ``.`` prefix stripped from its identifier (and without any surrounding ``NOT``); returns the ids of the related entries matching the sub-filter, evaluated against the related type's own properties. .. py:function:: filter_searcher(store: httk.data.query.Store, target: Any, filter_string: str | httk.core.FilterAst, *, entry_type: str, property_fulltypes: collections.abc.Mapping[str, str], property_keys: collections.abc.Mapping[str, str] | None = None, handlers: HandlerTable | None = None, recognized_prefixes: tuple[str, Ellipsis] = (), relationship_targets: tuple[str, Ellipsis] = (), related_property_resolver: RelatedPropertyResolver | None = None) -> httk.data.query.Searcher Build a :class:`~httk.data.query.Searcher` over ``store`` applying an OPTIMADE filter. ``filter_string`` is an OPTIMADE filter string (parsed with :func:`httk.core.parse_optimade_filter`) or an already-parsed :py:type:`~httk.core.FilterAst`. The searcher binds one search variable to ``target`` (the store-specific query target, declared as the searcher output named ``entry_type``) and applies the translated filter. When ``handlers`` is not supplied, a default table is built with :func:`simple_property_handlers` from ``property_keys`` (or, when ``property_keys`` is also None, from an identity map over ``property_fulltypes``). The remaining keyword arguments are passed through to :func:`~httk.data.optimade_query.translate_filter_ast`. :raises FilterTranslationError: When the filter cannot be translated. :raises httk.core.ParserSyntaxError: When a filter string does not parse. .. py:function:: format_value(fulltype: str, val: tuple[Any, Ellipsis], allow_null: bool = False) -> Any Convert a filter constant node to a Python value of the property's ``fulltype``. :raises FilterTranslationError: With category ``"type-mismatch"`` when the constant does not fit the declared type, or ``"not-implemented"`` for dictionary-typed properties. .. py:data:: invert_op .. py:function:: relationship_id_handler(rel_key: str) -> collections.abc.Mapping[str, collections.abc.Callable[Ellipsis, Any]] A ``'.id'`` handler-table entry matching related ids over a list-valued field. ``rel_key`` names a list-valued backend field holding the related entry ids; the returned ``{'HAS': ...}`` mapping serves ``.id HAS ...`` filters (and the semi-join rewrites of other dotted filters) with the standard set-operation semantics of :func:`~httk.data.optimade_query.set_handler`. .. py:function:: simple_property_handlers(entry_type: str, property_keys: collections.abc.Mapping[str, str], property_fulltypes: collections.abc.Mapping[str, str]) -> dict[str, collections.abc.Mapping[str, collections.abc.Callable[Ellipsis, Any]]] Build a filter handler table for an entry type from a property-key map. Always provides the standard ``id`` (matched against the ``__id`` field) and ``type`` (a constant equal to ``entry_type``) handlers. For every property named in ``property_keys`` (which maps property names to backend field names), handlers are generated from the property's fulltype in ``property_fulltypes`` (default ``"string"``): string properties get comparison and stringmatching handlers; integer and float properties get a numeric comparison handler; ``list of ...`` properties get a HAS (set membership) handler. Every generated property also gets a ``known`` unknown handler. .. py:function:: translate_filter_ast(node: httk.core.FilterAst, search_variable: httk.data.query.SearchVariable, entry_type: str, property_fulltypes: collections.abc.Mapping[str, str], handlers: HandlerTable, recognized_prefixes: tuple[str, Ellipsis], *, recursion: int = 0, relationship_targets: tuple[str, Ellipsis] = (), related_property_resolver: RelatedPropertyResolver | None = None) -> httk.data.query.SearchExpression Translate one filter syntax-tree node into a search expression. ``node`` is a :py:type:`~httk.core.FilterAst` node (as produced by :func:`httk.core.parse_optimade_filter`); ``search_variable`` is the backend search variable the expression is built against; ``property_fulltypes``, ``handlers``, and ``recognized_prefixes`` drive the translation as described in the module docstring; ``recursion`` counts the nesting depth. Returns the translated expression, ready to hand to :meth:`~httk.data.query.Searcher.add`. ``NOT`` translates to the backend's ``~``; a backend whose set predicates need a second (post-filter) evaluation position derives that from the expression itself, so the caller never has to. **Relationship filtering:** an identifier dotted with a name in ``relationship_targets`` (e.g. ``('Identifier', 'references', 'doi')``) filters on the properties of related entries. ``.id HAS ...`` is served directly by the ``'.id'`` handler-table entry. Every other depth-1 dotted filter — comparisons (including ``.id = ...``), string matching, ``IS KNOWN``/``IS UNKNOWN``, and the HAS family over related list properties — is resolved by a two-phase semi-join: the ``.`` prefix is stripped from the node, the resulting sub-filter is passed to ``related_property_resolver`` (without any surrounding ``NOT`` — inversion applies to the resulting id-set membership), and the returned ids are substituted back as ``.id HAS ANY `` (an empty id set translates to a constant-false expression). Each dotted node is resolved **independently**: ``references.doi CONTAINS "x" AND references.year >= 2000`` matches entries where *some* related reference matches the doi condition and *some* — possibly different — related reference matches the year condition. Without a resolver, dotted filters other than ``.id HAS ...`` are not implemented; nested (deeper than depth-1) paths and dotted ``LENGTH`` filters are never supported. :raises FilterTranslationError: See :class:`FilterTranslationError` for the failure categories. .. py:class:: Searcher Bases: :py:obj:`Protocol` A single query under construction, and its results once iterated. Iteration yields one :class:`SearchResult` per match, so ``item[0][0]`` is the first declared output of the match (typically the matched row object). The expressions received by ``add`` are always ones produced by this same backend's search variables, so implementations may type them as their own expression class; a backend that needs a second (post-filter) evaluation position decides that from the expression itself, not from the caller. .. py:attribute:: offset :type: int .. py:method:: variable(target: Any) -> Any .. py:method:: output(variable: Any, name: str) -> None .. py:method:: add(expression: Any) -> None .. py:method:: count() -> int .. py:method:: set_limit(limit: int) -> None .. py:method:: add_offset(offset: int) -> None .. py:method:: add_sort(field: Any, descending: bool) -> None .. py:class:: SearchExpression Bases: :py:obj:`Protocol` Base class for protocol classes. Protocol classes are defined as:: class Proto(Protocol): def meth(self) -> int: ... Such classes are primarily used with static type checkers that recognize structural subtyping (static duck-typing). For example:: class C: def meth(self) -> int: return 0 def func(x: Proto) -> int: return x.meth() func(C()) # Passes static type check See PEP 544 for details. Protocol classes decorated with @typing.runtime_checkable act as simple-minded runtime protocols that check only the presence of given attributes, ignoring their type signatures. Protocol classes can be generic, they are defined as:: class GenProto[T](Protocol): def meth(self) -> T: ... .. py:class:: SearchField Bases: :py:obj:`Protocol` A queryable field of a search variable. In addition to the methods below, fields support the rich comparison operators (``==``, ``!=``, ``<``, ``<=``, ``>``, ``>=``), returning :class:`SearchExpression`. The handlers invoke those via ``getattr(field, '__eq__')(value)`` since the comparison dunders cannot be typed as expression-returning. The three string-matching methods take **literal** text: no wildcard or pattern syntax whatsoever crosses this contract, so ``%`` and ``_`` (and any other metacharacter) match themselves. A backend is therefore free to implement them with SQL ``LIKE`` over an escaped pattern, with a regular expression, or with a full-text index — the choice is invisible here. .. py:method:: has_any(*values: Any) -> SearchExpression .. py:method:: has_only(*values: Any) -> SearchExpression .. py:method:: contains(text: str) -> SearchExpression Match values containing ``text`` as a literal substring. .. py:method:: startswith(prefix: str) -> SearchExpression Match values beginning with the literal ``prefix``. .. py:method:: endswith(suffix: str) -> SearchExpression Match values ending with the literal ``suffix``. .. py:class:: SearchResult Bases: :py:obj:`NamedTuple` One match: the declared output values, and the names they were declared under. ``values`` holds one entry per :meth:`Searcher.output` call in declaration order; it is a tuple, so ``values, names = result`` and ``result[0][0]`` both work. .. py:attribute:: values :type: tuple[Any, Ellipsis] .. py:attribute:: names :type: tuple[str, Ellipsis] .. py:class:: SearchVariable Bases: :py:obj:`Protocol` A query variable bound to a target type; attribute access yields fields. ``always_true``/``always_false`` are reserved names: they are real methods of the variable, never stored fields resolved through ``__getattr__``. They exist so a translation layer can express a constant truth value without inventing a probe field. A ``field == field`` probe is NULL-unsound, since it yields NULL (not true) for a NULL field. .. py:method:: always_true() -> SearchExpression An expression that matches every row. .. py:method:: always_false() -> SearchExpression An expression that matches no row. .. py:class:: Store Bases: :py:obj:`Protocol` Base class for protocol classes. Protocol classes are defined as:: class Proto(Protocol): def meth(self) -> int: ... Such classes are primarily used with static type checkers that recognize structural subtyping (static duck-typing). For example:: class C: def meth(self) -> int: return 0 def func(x: Proto) -> int: return x.meth() func(C()) # Passes static type check See PEP 544 for details. Protocol classes decorated with @typing.runtime_checkable act as simple-minded runtime protocols that check only the presence of given attributes, ignoring their type signatures. Protocol classes can be generic, they are defined as:: class GenProto[T](Protocol): def meth(self) -> T: ... .. py:method:: searcher() -> Searcher .. py:exception:: PropertyValidationError(name: str, message: str) Bases: :py:obj:`ValueError` A value did not conform to its OPTIMADE property definition. Carries the offending property ``name`` and a human-readable ``message``. For single-value failures the message wraps the underlying ``jsonschema`` error message, and that ``jsonschema.exceptions.ValidationError`` is preserved as the chained ``__cause__``. .. py:attribute:: name .. py:attribute:: message .. py:function:: validate_property(definition: httk.core.PropertyDefinition, value: Any) -> None Validate a single ``value`` against ``definition``'s JSON-Schema payload. Builds a ``jsonschema.Draft202012Validator`` directly from the definition's document (with the ``$schema`` meta-schema reference removed) and validates ``value`` against it. Returns ``None`` on success; raises :class:`PropertyValidationError` on failure, chaining the underlying ``jsonschema.exceptions.ValidationError`` as the cause. No network access or registry lookup ever happens. .. py:function:: validate_record(entry_type: httk.core.EntryTypeDefinition, record: collections.abc.Mapping[str, Any]) -> None Validate every property present in ``record`` against ``entry_type``. Each key in ``record`` must be described by ``entry_type``; unknown property names are rejected with a :class:`PropertyValidationError` naming them and the entry type. ``id`` and ``type`` must both be present. Properties described by the definition but absent from ``record`` are simply not checked (serving a subset of the described properties is normal). The value of every property that *is* present is validated via :func:`validate_property`. Returns ``None`` on success.