# How it works

*httk-optimade* implements an OPTIMADE server as a pipeline of small,
independently testable layers. The package was ported from the OPTIMADE
implementation in httk v1 and reorganized around two explicit seams: a
web-framework seam and a storage-backend seam.

## Request flow

```
ASGI request
  └─ runtime/asgi.py     — adapts the HTTP request to a RawRequest
       └─ engine/processing.py — process(): dispatches to an endpoint
            ├─ engine/validate.py    — validates URL, version, and query parameters
            ├─ filter/               — parses the OPTIMADE filter language to an AST
            ├─ query_function        — the backend seam (executes entry queries)
            └─ endpoints/            — builds the JSON:API reply documents
```

1. **Runtime** (`httk.optimade.runtime`): a Starlette application with a single
   catch-all GET route. It builds a `RawRequest` (base URL, representation,
   query string) and renders the resulting `EndpointResponse`. All errors are
   converted to OPTIMADE JSON:API error documents.

2. **Engine** (`httk.optimade.engine`): `validate_optimade_request()` resolves
   the endpoint (including versioned base URLs like `/v1.0.0/structures`),
   validates query parameters, and computes the response fields.
   `process()` routes the validated request to the matching endpoint reply
   generator, parsing the `filter=` parameter when present.

3. **Filter parsing** (`httk.optimade.filter`): the OPTIMADE filter grammar
   (shipped as `optimade_filter_grammar.ebnf`) is parsed with a vendored LR(1)
   parser into a nested-tuple abstract syntax tree.

4. **Backend** (`httk.optimade.backend`): the filter AST is translated into
   backend search expressions and executed. This is the storage seam.

## The backend seam

`process()` never touches storage directly; it calls a `QueryFunction`
callback. The standard implementation is provided by `BackendAdapter`:

- **`Store`/`Searcher` protocols** (`backend/protocols.py`) describe the query
  interface a storage backend must implement (mirroring the httk v1
  `httk.db` searcher API: `variable()`, `add()`, `count()`, set operations
  like `has_any`, literal string matching (`contains`/`startswith`/`endswith`,
  which carry no pattern syntax), and the comparison operators).

- **`EntrySource`** pairs a queryable target (e.g. a table or type) with a
  mapping from OPTIMADE response fields to row extractors. An entry endpoint
  can be backed by several sources; results are concatenated and
  offset/limit are redistributed across them.

- **Field handlers** (`backend/handlers.py`) translate filter operations on
  OPTIMADE properties into search expressions over backend fields. A backend
  supplies a handler table per entry type; `simple_property_handlers()` derives
  one generically from a property-key map and the entry's property types. When a
  `BackendAdapter` is given no handlers, it derives them from its schema.

*httk-optimade* is a generic implementation of the OPTIMADE *protocol*: it
carries no materials-science knowledge of its own. The served entry types,
their properties, and their records are supplied from outside — see
[Entry providers](#entry-providers). A backend can also plug in at the lower
level by implementing the store protocols directly; the repository's
`examples/demo_server/` shows a complete in-memory implementation serving
several entry types.

(entry-providers)=
## Entry providers

For a worked usage guide with runnable examples, see
[Serving entry providers](serving_providers.md).

The materials-science mapping lives *outside* *httk-optimade*, behind the neutral
`httk.core.EntryProvider` contract (defined in *httk-core*, so neither package
depends on the other). A provider describes its entry types (as first-class
`httk.core.EntryTypeDefinition` objects — vendored standards or definitions built
with `from_optimade`/`from_simple`), states how each served property maps to a
record key, and yields the records:

- `entry_types()` → entry-type name to an `EntryTypeDefinition`;
- `property_keys(entry_type)` → served-property to record-key map (at least `id`/`type`;
  every served name must be described by the definition);
- `records(entry_type)` → an iterable of plain JSON-able record dicts.

`adapter_from_providers([...])` (`backend/providers.py`) turns one or more
providers into a fully wired `BackendAdapter` over an in-memory store: it builds
the `ServedSchema` from the definitions (validating each served property
against the definition), the filter handlers from the property keys, and the
response-field extractors from the property keys, then loads the records.

The materials provider itself lives in *httk-atomistic*
(`httk.atomistic.structure_entries.StructureEntryProvider`), which serves
OPTIMADE `structures` — `species`, `species_at_sites`, `lattice_vectors`,
`cartesian_site_positions`, ... — from `httk.atomistic.Structure` objects.
Providers self-register a factory via `httk.core.register_entry_provider`, so
`providers_from_registry()` can enumerate the installed ones (applications
instantiate them with their data).

## OPTIMADE version support

The served API version is **1.3.0** (versioned base URLs `/v1`, `/v1.3`, `/v1.3.0`).
Relative to OPTIMADE v1.0.0, the implementation includes:

- boolean values (`TRUE`/`FALSE`) in the filter language (v1.2),
- the v1.2 entry listing info format: top-level `id`/`type` and properties
  presented as OPTIMADE Property Definitions (the property-definition model and
  generator live in *httk-core*, `httk.core.property_definitions`),
- extended `meta` information: `implementation` with `source_url` and
  `issue_tracker`, and the optional `schema`, `database`, and `request_delay`
  fields via `OptimadeConfig`,
- the structures properties added in v1.2 (space-group symmetry fields) and
  v1.3 (`fractional_site_positions`, `site_coordinate_span`,
  `optimization_type`, `wyckoff_positions`, ...) — recognized in filters and
  as response fields; they are served as `null` until a backend implements
  them,
- sorting of entry listings (the `sort` query parameter),
- relationships between entries and the `include` query parameter (compound
  documents with a top-level `included` field), with per-identifier
  relationship `meta` (`description` and the v1.3 `role`),
- filtering on relationships: `<type>.id HAS ...` and depth-1
  relationship-property filters such as `references.doi CONTAINS "10.1"`
  (resolved by a two-phase semi-join over the related entry type; each dotted
  filter node is resolved independently, matching the reference
  implementation's semantics),
- the `references`, `files`, and `trajectories` entry types,
- per-property metadata (`meta.property_metadata` and the
  `x-optimade-metadata-definition` in property definitions),
- the partial data protocol (JSON Lines format and the `dimension_slices`
  query parameter) with the compact list representation for trajectories,
- the `license`, `available_licenses`, and `available_licenses_for_entries`
  base-info attributes, and the `warnings`, `last_id`, and `links.describedby`
  response fields via `OptimadeConfig`.

Optional parts of the specification that are not implemented: cross-source sort
merging, filtering on relationship paths nested deeper than one level
(`references.structures.x`), on relationship `meta`
(`.description`/`.role`), and dotted `LENGTH` filters, the sparse JSON Lines
layout, index meta-databases, transaction mechanisms, and rejection of
unrecognized query parameters.

## Serving additional entry types

The served entry types and their properties are described by a `ServedSchema`
(`schema/served.py`), built with `build_served_schema()`. A backend registers
extra entry types by passing them in and wiring an `EntrySource` for each:

- `build_served_schema(definitions, served)` derives the endpoint/field tables
  from a mapping of entry type to `EntryTypeDefinition` and the per-entry list of
  served property names (defaulting to every described property). The
  `trajectories` entry type is generated by frame-wrapping the `structures`
  properties (`schema/trajectories.py`, which takes the structures
  `EntryTypeDefinition`) and turning the result into a definition via
  `entry_type_definition_from_simple`.
- `BackendAdapter(schema=..., sources={entry: (EntrySource(...),)})` binds each
  served entry type to a queryable target and its field extractors.

The `examples/demo_server/` backend registers `references`, `files`, and
`trajectories` this way alongside `structures` and `calculations`.

## Large properties and slicing

Field extractors may return a `PartialValue` (`backend/partial.py`) instead of
a concrete value for large, dimensioned properties (e.g. a trajectory's
`cartesian_site_positions`). In a normal reply such a value is served as `null`
with a `meta.partial_data_links` entry pointing at
`partial_data/<type>/<id>/<property>`, which streams the data in the JSON Lines
format. A client may instead request an inline slice with the
`dimension_slices=name[start:stop:step]` query parameter (single-entry
endpoints only); note that `stop` is *inclusive* per the specification. Values
that are constant across a dimension (e.g. `nelements` across a trajectory's
frames) are served as single-item `constant` compact lists, which is legal
because those axes are declared `compactable`.

## Differences from the httk v1 implementation

- The stdlib `httk.httkweb.webserver` binding was replaced by a Starlette
  ASGI app + uvicorn development server (`serve()`), and
  `create_asgi_app()` supports production ASGI deployment.
- `serve()` takes a `BackendAdapter` instead of a raw `httk.db` store.
- The client-side `validation/` subpackage (stale at OPTIMADE 0.9.5) was not
  ported; use the official `optimade-validator` instead.
- Several latent bugs were fixed (query-string derivation, caller-supplied
  endpoints, offsets beyond the result set) — see the regression tests in
  `tests/`.
- Timestamps in `meta` are now UTC.