httk.serve.optimade.backend =========================== .. py:module:: httk.serve.optimade.backend .. autoapi-nested-parse:: Public backend adapters, stores, and filter translation helpers. Submodules ---------- .. toctree:: :maxdepth: 1 /reference/autoapi/httk/serve/optimade/backend/adapter/index /reference/autoapi/httk/serve/optimade/backend/execution/index /reference/autoapi/httk/serve/optimade/backend/handlers/index /reference/autoapi/httk/serve/optimade/backend/memory_store/index /reference/autoapi/httk/serve/optimade/backend/partial/index /reference/autoapi/httk/serve/optimade/backend/protocols/index /reference/autoapi/httk/serve/optimade/backend/providers/index /reference/autoapi/httk/serve/optimade/backend/stores/index /reference/autoapi/httk/serve/optimade/backend/translation/index Classes ------- .. autoapisummary:: httk.serve.optimade.backend.Searcher httk.serve.optimade.backend.SearchExpression httk.serve.optimade.backend.SearchField httk.serve.optimade.backend.SearchResult httk.serve.optimade.backend.SearchVariable httk.serve.optimade.backend.Store httk.serve.optimade.backend.BackendAdapter httk.serve.optimade.backend.EntrySource httk.serve.optimade.backend.StoreResults httk.serve.optimade.backend.InMemoryStore httk.serve.optimade.backend.PartialDimension httk.serve.optimade.backend.PartialValue httk.serve.optimade.backend.QueryFunction httk.serve.optimade.backend.QueryResults httk.serve.optimade.backend.StoredBackendAdapter Functions --------- .. autoapisummary:: httk.serve.optimade.backend.execute_query httk.serve.optimade.backend.simple_property_handlers httk.serve.optimade.backend.adapter_from_providers httk.serve.optimade.backend.providers_from_registry httk.serve.optimade.backend.adapter_from_store httk.serve.optimade.backend.adapter_from_stores httk.serve.optimade.backend.translate_filter httk.serve.optimade.backend.translate_filter_node Package Contents ---------------- .. py:class:: Searcher Bases: :py:obj:`Protocol` Build one query and iterate its results. Iteration yields one :class:`SearchResult` per match, so ``item[0][0]`` is the first declared output of the match (typically the matched row object). The expressions received by ``add`` are always ones produced by this same backend's search variables, so implementations may type them as their own expression class; a backend that needs a second (post-filter) evaluation position decides that from the expression itself, not from the caller. .. py:attribute:: offset :type: int .. py:method:: variable(target) Bind a query variable to ``target``. .. py:method:: output(variable, name) Declare ``variable`` as a named result output. .. py:method:: add(expression) Add a filter expression to the query. .. py:method:: count() Return the exact count of the current query. .. py:method:: set_limit(limit) Set the query limit. .. py:method:: add_offset(offset) Add an offset to the query. .. py:method:: add_sort(field, descending) Add a field sort to the query. .. py:method:: results(**outputs) Return a result set for the requested named outputs. .. py:class:: SearchExpression Bases: :py:obj:`Protocol` Require composable backend search expressions. .. py:class:: SearchField Bases: :py:obj:`Protocol` Expose a queryable field of a search variable. In addition to the methods below, fields support the rich comparison operators (``==``, ``!=``, ``<``, ``<=``, ``>``, ``>=``), returning :class:`SearchExpression`. The handlers invoke those via ``getattr(field, '__eq__')(value)`` since the comparison dunders cannot be typed as expression-returning. The three string-matching methods take **literal** text: no wildcard or pattern syntax whatsoever crosses this contract, so ``%`` and ``_`` (and any other metacharacter) match themselves. A backend is therefore free to implement them with SQL ``LIKE`` over an escaped pattern, with a regular expression, or with a full-text index — the choice is invisible here. .. py:method:: has(value) Match a list field containing ``value``. .. py:method:: has_any(*values) Match a list field containing any of ``values``. .. py:method:: has_only(*values) Match a list field containing no values outside ``values``. .. py:method:: is_in(*values) Match a root scalar field whose value is one of ``values``. ``None`` is an explicit member: it matches a null field value, and its negation excludes nulls rather than inheriting SQL's three-valued ``NOT IN (..., NULL)`` behavior. Backends define the corresponding semantics for child or set fields; for example, a backend may use the existing ``has_only``-style all-values reading for a child field. .. py:method:: contains(text) Match values containing ``text`` as a literal substring. .. py:method:: startswith(prefix) Match values beginning with the literal ``prefix``. .. py:method:: endswith(suffix) Match values ending with the literal ``suffix``. .. py:class:: SearchResult Bases: :py:obj:`NamedTuple` Represent one match with declared output values and names. ``values`` holds one entry per :meth:`Searcher.output` call in declaration order; it is a tuple, so ``values, names = result`` and ``result[0][0]`` both work. .. py:attribute:: values :type: tuple[Any, Ellipsis] .. py:attribute:: names :type: tuple[str, Ellipsis] .. py:class:: SearchVariable Bases: :py:obj:`Protocol` Bind a query variable to a target type whose attributes yield fields. ``always_true``/``always_false`` are reserved names: they are real methods of the variable, never stored fields resolved through ``__getattr__``. They exist so a translation layer can express a constant truth value without inventing a probe field. A ``field == field`` probe is NULL-unsound, since it yields NULL (not true) for a NULL field. .. py:method:: always_true() An expression that matches every row. .. py:method:: always_false() An expression that matches no row. .. py:class:: Store Bases: :py:obj:`Protocol` Require a store that can create a query searcher. Implementations predating the ``as_of`` keyword may omit it and remain usable for current-state queries, but cannot honor historic queries. .. py:method:: searcher(*, as_of = None) Create an empty searcher, optionally at a historic cutoff. :param as_of: Optional canonical historic timestamp cutoff. :return: An empty query searcher. .. py:class:: BackendAdapter Bind a store to the OPTIMADE entry endpoints it serves. ``sources`` maps entry endpoint names (e.g. ``'structures'``) to the sources queried for that endpoint; an endpoint with several sources (e.g. several calculation result types) is queried across all of them. ``schema`` is required: it declares the served entry types and properties. ``field_handlers`` maps each entry type to its filter-handler table. When omitted (left empty) it is derived from ``schema`` via :func:`~httk.store.query.optimade_filters.simple_property_handlers`, using an identity property-key map (each property is filtered against a backend field of the same name); a backend whose field names differ, or that wants finer control, supplies its own tables instead. :param store: Store implementing the neutral query protocol. :param sources: Queryable sources keyed by entry endpoint. :param schema: Required schema describing served entries and properties. :param field_handlers: Optional filter handlers keyed by entry endpoint. .. py:attribute:: store :type: httk.store.query.Store .. py:attribute:: sources :type: collections.abc.Mapping[str, collections.abc.Sequence[EntrySource]] .. py:attribute:: schema :type: httk.serve.optimade.schema.served.ServedSchema .. py:attribute:: field_handlers :type: collections.abc.Mapping[str, httk.store.query.optimade_filters.HandlerTable] .. py:method:: query_function() Return the callback that executes queries through this adapter. :return: Query callback consumed by the OPTIMADE request engine. .. py:class:: EntrySource Describe one queryable source behind an OPTIMADE entry endpoint. ``target`` is what gets passed to ``searcher.variable()``; ``fields`` maps OPTIMADE response-field names to extractors applied to matched row objects. ``relationships``, when set, is an extractor mapping a matched row to a dictionary keyed by related entry type, each value a list of ``{'id': str, 'description': str?, 'role': str?}`` dictionaries. ``sort_keys`` maps response-field names to the backend field names to sort on. ``property_metadata`` maps response-field names to extractors returning the per-property metadata dictionary for a matched row (or ``None`` when there is no metadata for that row). :param target: Store-specific target passed to ``searcher.variable``. :param fields: Response-field extractors applied to matched rows. :param sort_keys: Response-field to backend-sort-field mappings. :param relationships: Optional extractor for related-resource data. :param property_metadata: Optional per-property metadata extractors. .. py:attribute:: target :type: Any .. py:attribute:: fields :type: collections.abc.Mapping[str, FieldExtractor] .. py:attribute:: sort_keys :type: collections.abc.Mapping[str, str] .. py:attribute:: relationships :type: FieldExtractor | None :value: None .. py:attribute:: property_metadata :type: collections.abc.Mapping[str, FieldExtractor] .. py:class:: StoreResults(pairs, response_fields, unknown_response_fields, limit, offset, total_count, recognized_prefixes) Results of a query over one or more searchers. Implements the :class:`~httk.serve.optimade.model.results.QueryResults` protocol. Iteration yields one :class:`~httk.serve.optimade.model.results.ResultRow` per entry, whose values map response-field names to values extracted from the matched row objects. :param pairs: Sources and already-configured searchers to iterate. :param response_fields: Recognized fields to extract. :param unknown_response_fields: Unknown fields to return as null. :param limit: Maximum number of results to yield. :param offset: Number of results to skip. :param total_count: Total matches before pagination. :param recognized_prefixes: Prefixes for dynamic row attributes. .. py:attribute:: pairs .. py:attribute:: recognized_prefixes .. py:attribute:: cur :type: collections.abc.Iterator[tuple[httk.serve.optimade.backend.adapter.EntrySource, Any]] | None .. py:attribute:: limit .. py:attribute:: response_fields .. py:attribute:: unknown_response_fields .. py:attribute:: offset .. py:attribute:: more_data_available :value: True .. py:method:: count() Return all current-filter matches, before pagination. The endpoint metadata needs the filtered total even after execution has applied page limits and offsets to its searchers. Retaining it here also keeps the value stable once this one-shot result iterator is consumed. .. py:function:: execute_query(adapter, entries, response_fields, unknown_response_fields, response_limit, response_offset, filter_ast = None, *, sort = None, debug = False) Execute a translated query across the adapter's sources. :param adapter: Backend adapter providing sources and schema. :param entries: Entry endpoints to query. :param response_fields: Recognized fields to return. :param unknown_response_fields: Unknown fields to return as null. :param response_limit: Maximum number of returned rows. :param response_offset: Number of matching rows to skip. :param filter_ast: Parsed filter, when one was requested. :param sort: Fields and directions to sort by. :param debug: Enable backend diagnostics. :return: Lazy results for the requested page. :raises httk.serve.optimade.model.errors.TranslatorError: If sorting across multiple sources is requested. .. py:function:: simple_property_handlers(entry_type, property_keys, property_fulltypes) Build a filter handler table for an entry type from a property-key map. Provides default handlers for standard ``id`` (matched against the :data:`~httk.store.query.protocols.ID_FIELD` field) and ``type`` (a constant equal to ``entry_type``). Entries in ``property_keys`` replace those defaults when their names overlap. For every property named in ``property_keys`` (which maps property names to backend field names), handlers are generated from the property's fulltype in ``property_fulltypes`` (default ``"string"``): string properties get comparison and stringmatching handlers; integer and float properties get a numeric comparison handler; ``list of ...`` properties get a HAS (set membership) handler. Every generated property also gets a ``known`` unknown handler. :param entry_type: The served entry type used by the constant ``type`` handler. :param property_keys: Mapping from served property names to backend field names. :param property_fulltypes: Fulltypes keyed by served property name. :return: A handler table keyed by served property name. .. py:class:: InMemoryStore(tables) Provide a store over dictionary rows. :param tables: Row lists keyed by table name. .. py:attribute:: tables .. py:method:: searcher(*, as_of = None) Create a searcher over this store's tables. :param as_of: Optional historic timestamp cutoff; unsupported here. :return: Fresh in-memory searcher. :raises ValueError: If a historic cutoff is requested. .. py:class:: PartialDimension Describe one list axis of a :class:`PartialValue`. ``length`` is the number of items along the axis (``None`` when unknown or entry-dependent and not declared). ``sliceable`` indicates whether the server can honour a slice request for this axis. :param name: Dimension name used in response metadata. :param length: Number of items, or ``None`` when unknown. :param sliceable: Whether the server accepts slices on this axis. .. py:attribute:: name :type: str .. py:attribute:: length :type: int | None :value: None .. py:attribute:: sliceable :type: bool :value: False .. py:class:: PartialValue Describe a property value provided lazily, one slice at a time. ``fetch`` takes a tuple of Python slices (one per dimension, with the usual *exclusive* stop) and returns the corresponding nested lists. :param dimensions: Axes describing the value. :param fetch: Slice retrieval operation. .. py:attribute:: dimensions :type: tuple[PartialDimension, Ellipsis] .. py:attribute:: fetch :type: collections.abc.Callable[[tuple[slice, Ellipsis]], Any] .. py:class:: QueryFunction Bases: :py:obj:`Protocol` The callback seam through which the request engine runs queries on a backend. .. py:class:: QueryResults Bases: :py:obj:`Protocol` The results of a query against a backend, as consumed by the entry endpoints. Iteration yields one :class:`~httk.serve.optimade.model.results.ResultRow` per entry; its ``values`` map OPTIMADE response-field names to values, and the ``id`` and ``type`` keys are always present. .. py:property:: more_data_available :type: bool Report whether another page is available. .. py:method:: count() Return the total number of matches before pagination. .. py:function:: adapter_from_providers(providers, **options) Build a :class:`~httk.serve.optimade.backend.adapter.BackendAdapter` serving the given entry providers. Every provider's :meth:`~httk.core.EntryProvider.entry_types` become served entry types (described by their :class:`~httk.core.EntryTypeDefinition`), its :meth:`~httk.core.EntryProvider.property_keys` name the served subset and drive both the filter handlers (via :func:`~httk.store.query.optimade_filters.simple_property_handlers`) and the response-field extractors, and its :meth:`~httk.core.EntryProvider.records` are loaded into an :class:`~httk.serve.optimade.backend.memory_store.InMemoryStore`. Every served property MUST be described by the entry type's definition (a custom property must therefore live in an :meth:`~httk.core.EntryTypeDefinition.extended` definition); a :class:`ValueError` names any offender. All served properties beyond ``id``/``type`` are marked default-response. Extra keyword ``options`` (e.g. ``sortable``, ``recognized_prefixes``) are forwarded to :func:`~httk.serve.optimade.schema.served.build_served_schema`; every served property is sortable-capable, since the provider's property-key map is passed through as the source's :attr:`~httk.serve.optimade.backend.adapter.EntrySource.sort_keys`. Declared relationships (:meth:`~httk.core.EntryProvider.relationships`) are fully auto-wired for serving *and* filtering: for each entry type with declared relationships, a synthetic ``__rel_`` id-list field is materialized on EVERY row of that entry type (an empty list when the row has no related entries of that type, so inverse set semantics are well-defined), and a ``'.id'`` entry built with :func:`~httk.store.query.optimade_filters.relationship_id_handler` is merged into the entry type's derived filter-handler table (never overwriting an entry already present, mirroring how :class:`~httk.serve.optimade.backend.adapter.BackendAdapter` respects explicitly supplied handler tables). ``.id HAS ...`` filters — and, through the related-property resolver of :func:`~httk.serve.optimade.backend.translation.translate_filter`, depth-1 relationship-property filters such as ``references.doi CONTAINS "10.1"`` — therefore work without any hand-wiring. A provider's reverse relationships (``EntryProvider.reverse_relationships()``) are consumed too: their target-keyed related entries are append-merged into each served target entry's relationships (never clobbering the forward entries), so a derived reverse edge is served on the entry it points at. :param providers: Generic entry providers supplying definitions, keys, records, and relationships. :param \*\*options: Schema options forwarded to :func:`~httk.serve.optimade.schema.served.build_served_schema`. :return: Fully wired in-memory backend adapter. :raises ValueError: If provider keys or served properties are invalid. .. py:function:: providers_from_registry() Return the registered entry-provider factories keyed by their registered name. Resolves each factory registered via :func:`httk.core.register_entry_provider` (through ``httk.registry.*`` self-registration) into a callable. Providers need data, so applications instantiate them: ``providers_from_registry()["atomistic-structures"](data)``. :return: Registered provider factories keyed by registry name. .. py:class:: StoredBackendAdapter Serve one data federation per OPTIMADE entry type. :param federations: Durable federations keyed by entry endpoint. :param schema: Schema describing the federations' served entries. .. py:attribute:: federations :type: collections.abc.Mapping[str, Any] .. py:attribute:: schema :type: httk.serve.optimade.schema.served.ServedSchema .. py:method:: snapshot_cutoff_ns(entry_type, now_ns) Return the resolution-aware snapshot cutoff for one entry type. .. py:method:: query_function() Return the callback that queries the configured federations. :return: Query callback consumed by the OPTIMADE request engine. .. py:function:: adapter_from_store(store, **options) Build a lazy OPTIMADE adapter from every described family in one store. Families declared without an entry-type definition are deliberately ignored. This lets application-specific records, such as DSP publication declarations, coexist with OPTIMADE records in one durable layout. :param store: Entry store whose configured layout is discovered. :param \*\*options: Schema options forwarded to :func:`adapter_from_stores`. :return: Lazy adapter over all configured OPTIMADE families. :raises TypeError: If ``store`` does not implement :class:`EntryStore`. :raises ValueError: If the store contains no OPTIMADE-described family. .. py:function:: adapter_from_stores(sources, **options) Build a lazy store-backed adapter from durable entry sources. Sources with the same exact logical family are federated under one entry endpoint. The data layer owns all source/backing traversal and global pagination; this adapter advertises the family's definition and turns only the returned page into OPTIMADE result rows. :param sources: Durable entry sources to federate by entry type. :param \*\*options: Schema options forwarded to :func:`~httk.serve.optimade.schema.served.build_served_schema`. :return: Lazy adapter over the supplied durable sources. :raises ValueError: If sources conflict or expose incomplete sort mappings. :raises TypeError: If a source is not a stored entry source. .. py:function:: translate_filter(filter_ast, entries, adapter, sort = None) Build one searcher per entry source, with the filter applied to each. Relationship-property filters (dotted identifiers over served entry types) are resolved through the adapter's related-property resolver (built by ``_related_property_resolver``), so filtering ``references.doi`` behaves exactly like filtering ``/references`` directly. :param filter_ast: Parsed filter, or ``None`` for an unfiltered query. :param entries: Entry endpoints to search. :param adapter: Backend adapter supplying sources and handlers. :param sort: Response fields and descending flags for sorting. :return: Source/searcher pairs with the filter and sort applied. :raises httk.serve.optimade.model.errors.TranslatorError: If the filter cannot be translated. .. py:function:: translate_filter_node(node, search_variable, entry, entry_info, handlers, recognized_prefixes, served_entries = ()) Translate one filter node against an OPTIMADE *entry-info* property mapping. An OPTIMADE-side adaptation of :func:`~httk.store.query.optimade_filters.translate_filter_ast`: ``entry_info`` maps property names to their property dictionaries (only their ``'fulltype'`` keys are read) rather than straight to fulltypes, ``served_entries`` names the relationship targets, and failures surface as :class:`~httk.serve.optimade.model.errors.TranslatorError` instead of the upstream neutral :class:`~httk.store.FilterTranslationError`. No related-property resolver is threaded through, so relationship-property filters other than ``.id HAS ...`` raise a not-implemented (501) error. Use :func:`translate_filter` (which builds the resolver from its adapter) for full relationship-property filtering. :param node: Filter node to translate. :param search_variable: Backend variable used by the expression. :param entry: Entry endpoint being filtered. :param entry_info: Simplified property metadata for the entry. :param handlers: Property handlers used for translation. :param recognized_prefixes: Property-definition prefixes accepted by the filter. :param served_entries: Entry types available as relationship targets. :return: Backend search expression. :raises httk.serve.optimade.model.errors.TranslatorError: If the filter cannot be translated.