{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "md01",
   "metadata": {},
   "source": [
    "# Walking an OPTIMADE API in-process\n",
    "\n",
    "This notebook is the narrated version of `examples/query_in_process.py`; the\n",
    "code cells are that script, unchanged. It answers one question: *what does an\n",
    "httk-optimade server actually reply?*\n",
    "\n",
    "The temptation is to answer it by calling the query engine directly. That\n",
    "answer is incomplete, because the JSON:API envelope \u2014 `data`, `meta`,\n",
    "`links`, `included` \u2014 is built by the ASGI application, not by the engine. So\n",
    "we speak HTTP instead, but without a port: `create_asgi_app` returns an\n",
    "ordinary ASGI application, and starlette's `TestClient` drives it in this very\n",
    "process. Routing, query-string validation, error handling and response\n",
    "rendering all run; nothing touches a socket, and there is no server to start\n",
    "or stop.\n",
    "\n",
    "The cells are not executed when the documentation is built, so no output is\n",
    "shown below. Run the notebook (or `python examples/query_in_process.py`) to\n",
    "see the responses."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "md02",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Three imports carry the whole example: the two ready-made providers\n",
    "*httk-data* registers for the standard `files` and `calculations` entry types,\n",
    "`adapter_from_providers`/`create_asgi_app` from *httk-optimade*, and\n",
    "starlette's `TestClient`. No provider class has to be written to have\n",
    "something to query \u2014 see\n",
    "[Serving entry providers](../serving_providers.md) for how to write one."
   ]
  },
  {
   "cell_type": "code",
   "id": "code03",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import json\n",
    "from typing import Any\n",
    "\n",
    "from httk.core import RelatedEntry\n",
    "from httk.data.entry_providers import CalculationEntryProvider, FileEntryProvider\n",
    "from starlette.testclient import TestClient\n",
    "\n",
    "from httk.optimade import adapter_from_providers, create_asgi_app"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "md04",
   "metadata": {},
   "source": [
    "## The data\n",
    "\n",
    "Three files and two calculations, as plain mappings. `CALCULATION_FILES`\n",
    "declares which files each calculation consumed and produced; those\n",
    "`RelatedEntry` objects become the calculation's OPTIMADE **relationships**\n",
    "block, which is what makes the `HAS` filter and the `include=files` request\n",
    "further down meaningful."
   ]
  },
  {
   "cell_type": "code",
   "id": "code05",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "FILES = {\n",
    "    \"file-1\": {\n",
    "        \"name\": \"INCAR\",\n",
    "        \"url\": \"https://example.org/files/calc-1/INCAR\",\n",
    "        \"size\": 512,\n",
    "        \"media_type\": \"text/plain\",\n",
    "        \"last_modified\": \"2021-03-15T10:20:00Z\",\n",
    "    },\n",
    "    \"file-2\": {\n",
    "        \"name\": \"OUTCAR\",\n",
    "        \"url\": \"https://example.org/files/calc-1/OUTCAR\",\n",
    "        \"size\": 204800,\n",
    "        \"media_type\": \"text/plain\",\n",
    "        \"last_modified\": \"2021-03-15T10:25:00Z\",\n",
    "    },\n",
    "    \"file-3\": {\n",
    "        \"name\": \"trajectory.xyz\",\n",
    "        \"url\": \"https://example.org/files/calc-2/trajectory.xyz\",\n",
    "        \"size\": 1048576,\n",
    "        \"media_type\": \"chemical/x-xyz\",\n",
    "        \"last_modified\": \"2021-03-16T08:00:00Z\",\n",
    "    },\n",
    "}\n",
    "\n",
    "CALCULATIONS = {\n",
    "    \"calc-1\": {\"last_modified\": \"2021-03-15T10:30:00Z\"},\n",
    "    \"calc-2\": {\"last_modified\": \"2021-03-16T08:30:00Z\"},\n",
    "}\n",
    "\n",
    "# Which files each calculation consumed and produced. The provider serves these\n",
    "# as the calculation's OPTIMADE relationships block.\n",
    "CALCULATION_FILES = {\n",
    "    \"calc-1\": [RelatedEntry(\"files\", \"file-1\", role=\"input\"), RelatedEntry(\"files\", \"file-2\", role=\"output\")],\n",
    "    \"calc-2\": [RelatedEntry(\"files\", \"file-3\", role=\"output\")],\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "md06",
   "metadata": {},
   "source": [
    "## The application, and a client bound straight to it\n",
    "\n",
    "`adapter_from_providers` builds the backend adapter (schema, filter handlers\n",
    "and record store all derived from what the providers declare);\n",
    "`create_asgi_app` wraps it in the OPTIMADE HTTP API. `sortable=` is the one\n",
    "thing declared by hand: a property may only be used in `?sort=` if the server\n",
    "advertises it in `/v1/info/files`.\n",
    "\n",
    "`baseurl` is the URL the server uses when it builds absolute links (the\n",
    "`links.next` we follow later), so it must match the host the client talks to."
   ]
  },
  {
   "cell_type": "code",
   "id": "code07",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "def make_client() -> TestClient:\n",
    "    \"\"\"An HTTP client bound straight to the OPTIMADE app \u2014 no socket, no server process.\"\"\"\n",
    "    adapter = adapter_from_providers(\n",
    "        [FileEntryProvider(FILES), CalculationEntryProvider(CALCULATIONS, relationships=CALCULATION_FILES)],\n",
    "        # Sortable is a promise the server publishes in /v1/info/files; only\n",
    "        # declared properties may be used in ?sort=.\n",
    "        sortable={\"files\": [\"size\", \"name\"]},\n",
    "    )\n",
    "    # baseurl is what the server uses to build absolute URLs (links.next below),\n",
    "    # so it must match the host the client talks to.\n",
    "    app = create_asgi_app(adapter, baseurl=\"http://localhost/\")\n",
    "    return TestClient(app, base_url=\"http://localhost\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "md08",
   "metadata": {},
   "source": [
    "A one-line helper: issue a GET and print the response exactly as a client\n",
    "would receive it \u2014 envelope included, since half the point is to see the\n",
    "envelope."
   ]
  },
  {
   "cell_type": "code",
   "id": "code09",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "def show(client: TestClient, url: str, **params: Any) -> dict[str, Any]:\n",
    "    \"\"\"GET one OPTIMADE URL and print the response, JSON:API envelope included.\"\"\"\n",
    "    response = client.get(url, params=params)\n",
    "    query = \"?\" + \"&\".join(f\"{key}={value}\" for key, value in params.items()) if params else \"\"\n",
    "    print(\"\\n=== GET \" + url + query + \" -> \" + str(response.status_code))\n",
    "    payload: dict[str, Any] = response.json()\n",
    "    print(json.dumps(payload, indent=1, sort_keys=True))\n",
    "    return payload"
   ]
  },
  {
   "cell_type": "code",
   "id": "code10",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "client = make_client()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "md11",
   "metadata": {},
   "source": [
    "## 1. `/v1/info` \u2014 what is served at all\n",
    "\n",
    "Capability discovery comes first: everything else is only valid if this\n",
    "endpoint advertises it. The response is a single resource object\n",
    "(`data` is an object here, not an array) whose `attributes` give the\n",
    "`api_version`, the `available_api_versions` with their base URLs, the\n",
    "`available_endpoints`, and `entry_types_by_format` \u2014 the entry types this\n",
    "implementation serves, per response format.\n",
    "\n",
    "`meta` is present on every OPTIMADE response: API version, the implementation\n",
    "and provider identification, a timestamp, and `query.representation` echoing\n",
    "the request that produced it."
   ]
  },
  {
   "cell_type": "code",
   "id": "code12",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# 1. What does this implementation serve? Capability discovery comes first,\n",
    "#    because everything below is only valid if /v1/info advertises it.\n",
    "info = show(client, \"/v1/info\")\n",
    "print(\"\\nserved entry types:\", info[\"data\"][\"attributes\"][\"entry_types_by_format\"][\"json\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "md13",
   "metadata": {},
   "source": [
    "## 2. `/v1/files` \u2014 an entry listing\n",
    "\n",
    "Now a *listing*: `data` is an array of resource objects, each with `id`,\n",
    "`type` and an `attributes` object holding the properties.\n",
    "\n",
    "`response_fields` narrows `attributes` to the properties asked for. It is not\n",
    "an absolute filter \u2014 properties the standard requires a response to carry\n",
    "(`url` for `files`) come back regardless \u2014 but it is the way to keep a\n",
    "response small. Without it, every served property is returned, `null`s\n",
    "included.\n",
    "\n",
    "The listing's `meta` gains the counters a client needs: `data_available` (how\n",
    "many entries exist), `data_returned` (how many match), and\n",
    "`more_data_available`. `links.next` is `null` because everything fits in one\n",
    "page."
   ]
  },
  {
   "cell_type": "code",
   "id": "code14",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# 2. An entry listing. response_fields trims 'attributes' to what is asked\n",
    "#    for (plus the required ones, such as 'url' here); without it every\n",
    "#    served property is returned, nulls included.\n",
    "show(client, \"/v1/files\", response_fields=\"name,size,media_type\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "md15",
   "metadata": {},
   "source": [
    "## 3. `?filter=` \u2014 a numeric comparison\n",
    "\n",
    "`size` is described in the entry type definition as an integer, and that\n",
    "description alone is what makes numeric comparison operators available for it.\n",
    "No handler code was written for this filter; the provider only said what type\n",
    "the property has.\n",
    "\n",
    "In the response, `data_returned` drops to the number of matches while\n",
    "`data_available` stays at the full collection size \u2014 that pair is how a client\n",
    "tells \"filtered\" from \"empty database\"."
   ]
  },
  {
   "cell_type": "code",
   "id": "code16",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# 3. Filtering. 'size' is described as an integer, so numeric comparisons\n",
    "#    are offered for it; no handler code was written for this.\n",
    "show(client, \"/v1/files\", filter=\"size > 100000\", response_fields=\"name,size\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "md17",
   "metadata": {},
   "source": [
    "## 4. `HAS` \u2014 list membership\n",
    "\n",
    "`HAS` is OPTIMADE's membership test on a list-valued property. Here the list\n",
    "is the set of file ids related to each calculation, so\n",
    "`files.id HAS \"file-2\"` asks *which calculations touched that file* \u2014 and only\n",
    "`calc-1` did.\n",
    "\n",
    "The matched calculation's `relationships` object shows what the filter\n",
    "searched: a `files` entry whose `data` array names each related resource by\n",
    "`type` and `id`, with the declared `role` (`input`/`output`) carried in the\n",
    "per-identifier `meta`."
   ]
  },
  {
   "cell_type": "code",
   "id": "code18",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# 4. List membership. HAS tests a list-valued property; here the list is the\n",
    "#    set of file ids related to each calculation.\n",
    "show(client, \"/v1/calculations\", filter='files.id HAS \"file-2\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "md19",
   "metadata": {},
   "source": [
    "## 5. `?sort=` \u2014 ordering by a declared-sortable property\n",
    "\n",
    "A leading `-` sorts descending, so the largest file comes first. Sorting is\n",
    "only offered for properties declared sortable when the adapter was built\n",
    "(`sortable={\"files\": [\"size\", \"name\"]}` above, published in\n",
    "`/v1/info/files`); asking to sort on anything else is answered with an OPTIMADE\n",
    "error response, not silently ignored."
   ]
  },
  {
   "cell_type": "code",
   "id": "code20",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# 5. Sorting. A leading '-' sorts descending. Sorting on a property that was\n",
    "#    not declared sortable is an error response, not a silently ignored one.\n",
    "show(client, \"/v1/files\", sort=\"-size\", response_fields=\"name,size\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "md21",
   "metadata": {},
   "source": [
    "## 6. Pagination \u2014 follow `links.next`\n",
    "\n",
    "With `page_limit=2` the server returns the first two files, sets\n",
    "`meta.more_data_available` to `true`, and hands back a complete URL in\n",
    "`links.next`. The client's job is simply to keep following that link until the\n",
    "server stops providing one \u2014 never to construct the next page's URL itself,\n",
    "since the server is free to paginate by offset, by cursor, or by anything else\n",
    "behind that opaque link.\n",
    "\n",
    "Note the URL that comes back is absolute (built from the `baseurl` given to\n",
    "`create_asgi_app`) and carries the query parameters of the original request,\n",
    "so filters and `response_fields` survive across pages."
   ]
  },
  {
   "cell_type": "code",
   "id": "code22",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# 6. Pagination. Never construct the next page's URL yourself: follow\n",
    "#    links.next until the server stops handing one back.\n",
    "print(\"\\n--- following links.next ---\")\n",
    "page_url = \"/v1/files?page_limit=2&response_fields=name,size\"\n",
    "page_number = 0\n",
    "while page_url is not None:\n",
    "    page_number += 1\n",
    "    page = show(client, page_url)\n",
    "    print(f\"page {page_number}: \" + \", \".join(entry[\"id\"] for entry in page[\"data\"]))\n",
    "    page_url = page[\"links\"][\"next\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "md23",
   "metadata": {},
   "source": [
    "## 7. `?include=` \u2014 a compound document\n",
    "\n",
    "The `relationships` block only names related resources by `type` and `id`. A\n",
    "client that wants the resources themselves adds `include=files`, and the\n",
    "server returns them in full in a top-level `included` array \u2014 a JSON:API\n",
    "*compound document*: one request instead of one round trip per relationship.\n",
    "\n",
    "Each included resource appears once no matter how many entries reference it,\n",
    "and `included` is a sibling of `data`, not nested inside it."
   ]
  },
  {
   "cell_type": "code",
   "id": "code24",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# 7. A compound document. The related resources named in each calculation's\n",
    "#    relationships block are returned in full in the top-level 'included'.\n",
    "compound = show(client, \"/v1/calculations\", include=\"files\")\n",
    "print(\"\\nincluded resources:\", [(entry[\"type\"], entry[\"id\"]) for entry in compound[\"included\"]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "md25",
   "metadata": {},
   "source": [
    "## Where to go next\n",
    "\n",
    "* [Serving entry providers](../serving_providers.md) \u2014 writing the\n",
    "  `EntryProvider` that supplies entry types, properties and records.\n",
    "* `examples/in_memory_backend.py` \u2014 standing up a backend from plain dict rows\n",
    "  without writing a provider at all.\n",
    "* `examples/provider_server/` and `examples/demo_server/` \u2014 the same API served\n",
    "  over a real port, the latter also covering partial data and trajectories."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}