How it works

httk-optimade implements an OPTIMADE server as a pipeline of small, independently testable layers. The package was ported from the OPTIMADE implementation in httk v1 and reorganized around two explicit seams: a web-framework seam and a storage-backend seam.

Request flow

ASGI request
  └─ runtime/asgi.py     — adapts the HTTP request to a RawRequest
       └─ engine/processing.py — process(): dispatches to an endpoint
            ├─ engine/validate.py    — validates URL, version, and query parameters
            ├─ filter/               — parses the OPTIMADE filter language to an AST
            ├─ query_function        — the backend seam (executes entry queries)
            └─ endpoints/            — builds the JSON:API reply documents

  1. Runtime (httk.optimade.runtime): a Starlette application with a single catch-all GET route. It builds a RawRequest (base URL, representation, query string) and renders the resulting EndpointResponse. All errors are converted to OPTIMADE JSON:API error documents.

  2. Engine (httk.optimade.engine): validate_optimade_request() resolves the endpoint (including versioned base URLs like /v1.0.0/structures), validates query parameters, and computes the response fields. process() routes the validated request to the matching endpoint reply generator, parsing the filter= parameter when present.

  3. Filter parsing (httk.optimade.filter): the OPTIMADE filter grammar (shipped as optimade_filter_grammar.ebnf) is parsed with a vendored LR(1) parser into a nested-tuple abstract syntax tree.

  4. Backend (httk.optimade.backend): the filter AST is translated into backend search expressions and executed. This is the storage seam.

The backend seam

process() never touches storage directly; it calls a QueryFunction callback. The standard implementation is provided by BackendAdapter:

  • Store/Searcher protocols (backend/protocols.py) describe the query interface a storage backend must implement (mirroring the httk v1 httk.db searcher API: variable(), add(), count(), set operations like has_any, literal string matching (contains/startswith/endswith, which carry no pattern syntax), and the comparison operators).

  • EntrySource pairs a queryable target (e.g. a table or type) with a mapping from OPTIMADE response fields to row extractors. An entry endpoint can be backed by several sources; results are concatenated and offset/limit are redistributed across them.

  • Field handlers (backend/handlers.py) translate filter operations on OPTIMADE properties into search expressions over backend fields. A backend supplies a handler table per entry type; simple_property_handlers() derives one generically from a property-key map and the entry’s property types. When a BackendAdapter is given no handlers, it derives them from its schema.

httk-optimade is a generic implementation of the OPTIMADE protocol: it carries no materials-science knowledge of its own. The served entry types, their properties, and their records are supplied from outside — see Entry providers. A backend can also plug in at the lower level by implementing the store protocols directly; the repository’s examples/demo_server/ shows a complete in-memory implementation serving several entry types.

Entry providers

For a worked usage guide with runnable examples, see Serving entry providers.

The materials-science mapping lives outside httk-optimade, behind the neutral httk.core.EntryProvider contract (defined in httk-core, so neither package depends on the other). A provider describes its entry types (as first-class httk.core.EntryTypeDefinition objects — vendored standards or definitions built with from_optimade/from_simple), states how each served property maps to a record key, and yields the records:

  • entry_types() → entry-type name to an EntryTypeDefinition;

  • property_keys(entry_type) → served-property to record-key map (at least id/type; every served name must be described by the definition);

  • records(entry_type) → an iterable of plain JSON-able record dicts.

adapter_from_providers([...]) (backend/providers.py) turns one or more providers into a fully wired BackendAdapter over an in-memory store: it builds the ServedSchema from the definitions (validating each served property against the definition), the filter handlers from the property keys, and the response-field extractors from the property keys, then loads the records.

The materials provider itself lives in httk-atomistic (httk.atomistic.structure_entries.StructureEntryProvider), which serves OPTIMADE structures — species, species_at_sites, lattice_vectors, cartesian_site_positions, … — from httk.atomistic.Structure objects. Providers self-register a factory via httk.core.register_entry_provider, so providers_from_registry() can enumerate the installed ones (applications instantiate them with their data).

OPTIMADE version support

The served API version is 1.3.0 (versioned base URLs /v1, /v1.3, /v1.3.0). Relative to OPTIMADE v1.0.0, the implementation includes:

  • boolean values (TRUE/FALSE) in the filter language (v1.2),

  • the v1.2 entry listing info format: top-level id/type and properties presented as OPTIMADE Property Definitions (the property-definition model and generator live in httk-core, httk.core.property_definitions),

  • extended meta information: implementation with source_url and issue_tracker, and the optional schema, database, and request_delay fields via OptimadeConfig,

  • the structures properties added in v1.2 (space-group symmetry fields) and v1.3 (fractional_site_positions, site_coordinate_span, optimization_type, wyckoff_positions, …) — recognized in filters and as response fields; they are served as null until a backend implements them,

  • sorting of entry listings (the sort query parameter),

  • relationships between entries and the include query parameter (compound documents with a top-level included field), with per-identifier relationship meta (description and the v1.3 role),

  • filtering on relationships: <type>.id HAS ... and depth-1 relationship-property filters such as references.doi CONTAINS "10.1" (resolved by a two-phase semi-join over the related entry type; each dotted filter node is resolved independently, matching the reference implementation’s semantics),

  • the references, files, and trajectories entry types,

  • per-property metadata (meta.property_metadata and the x-optimade-metadata-definition in property definitions),

  • the partial data protocol (JSON Lines format and the dimension_slices query parameter) with the compact list representation for trajectories,

  • the license, available_licenses, and available_licenses_for_entries base-info attributes, and the warnings, last_id, and links.describedby response fields via OptimadeConfig.

Optional parts of the specification that are not implemented: cross-source sort merging, filtering on relationship paths nested deeper than one level (references.structures.x), on relationship meta (.description/.role), and dotted LENGTH filters, the sparse JSON Lines layout, index meta-databases, transaction mechanisms, and rejection of unrecognized query parameters.

Serving additional entry types

The served entry types and their properties are described by a ServedSchema (schema/served.py), built with build_served_schema(). A backend registers extra entry types by passing them in and wiring an EntrySource for each:

  • build_served_schema(definitions, served) derives the endpoint/field tables from a mapping of entry type to EntryTypeDefinition and the per-entry list of served property names (defaulting to every described property). The trajectories entry type is generated by frame-wrapping the structures properties (schema/trajectories.py, which takes the structures EntryTypeDefinition) and turning the result into a definition via entry_type_definition_from_simple.

  • BackendAdapter(schema=..., sources={entry: (EntrySource(...),)}) binds each served entry type to a queryable target and its field extractors.

The examples/demo_server/ backend registers references, files, and trajectories this way alongside structures and calculations.

Large properties and slicing

Field extractors may return a PartialValue (backend/partial.py) instead of a concrete value for large, dimensioned properties (e.g. a trajectory’s cartesian_site_positions). In a normal reply such a value is served as null with a meta.partial_data_links entry pointing at partial_data/<type>/<id>/<property>, which streams the data in the JSON Lines format. A client may instead request an inline slice with the dimension_slices=name[start:stop:step] query parameter (single-entry endpoints only); note that stop is inclusive per the specification. Values that are constant across a dimension (e.g. nelements across a trajectory’s frames) are served as single-item constant compact lists, which is legal because those axes are declared compactable.

Differences from the httk v1 implementation

  • The stdlib httk.httkweb.webserver binding was replaced by a Starlette ASGI app + uvicorn development server (serve()), and create_asgi_app() supports production ASGI deployment.

  • serve() takes a BackendAdapter instead of a raw httk.db store.

  • The client-side validation/ subpackage (stale at OPTIMADE 0.9.5) was not ported; use the official optimade-validator instead.

  • Several latent bugs were fixed (query-string derivation, caller-supplied endpoints, offsets beyond the result set) — see the regression tests in tests/.

  • Timestamps in meta are now UTC.