httk.core.dataset_loader

Attributes

DecodeObjectCallback

Callback invoked as (dict_obj, jsonld_url) that returns the value to use in place of

Classes

DatasetLoaderRecord	Read-only attribute and mapping view over a Mapping[str, Any].
DatasetMeta	Describe header metadata extracted from a structured JSON-LD document.
DatasetLoader	Lazy loader for httk dataset files, resolved only when data is first accessed.

Functions

write_dataset_sqlar(document, destination)

Write a structured JSON-LD dataset document as a deterministic sqlar archive.

Module Contents

type httk.core.dataset_loader.DecodeObjectCallback = Callable[[dict[str, Any], str], Any]

Callback invoked as (dict_obj, jsonld_url) that returns the value to use in place of dict_obj (return the input unchanged to decline).

class httk.core.dataset_loader.DatasetLoaderRecord(data)

Bases: collections.abc.Mapping[str, Any]

Read-only attribute and mapping view over a Mapping[str, Any].

Top-level keys are reachable both as attributes (record.name) and as items (record["name"]); the wrapped values are the plain parsed JSON and are not themselves wrapped. Supports iteration over keys, len(), in, and keys().

Parameters:

data (collections.abc.Mapping[str, Any]) – The parsed top-level object exposed by this view.

keys()

Return a dynamic view of the record’s top-level keys.

Returns:

The wrapped mapping’s keys view.

Return type:

collections.abc.KeysView[str]

class httk.core.dataset_loader.DatasetMeta

Describe header metadata extracted from a structured JSON-LD document.

Parameters:

• context – The raw document context.

• id – The document identifier, if present.

• type – The document type, if present.

• header – Remaining top-level header fields.

• dataset_ids – Dataset names mapped to their identifiers.

• fields – Dataset names mapped to their field property URLs.

context: dict[str, Any]

The raw @context object.

id: str | None

The document @id, or None if absent.

type_: str | None

The document @type, or None if absent (trailing underscore avoids the builtin type).

header: dict[str, Any]

All remaining top-level keys except data, indicies, and @-keys (titles, creator, license, provenance, …).

dataset_ids: dict[str, str]

Mapping of dataset name to its @id.

fields: dict[str, dict[str, str]]

Mapping of dataset name to a mapping of field name to its property URL.

httk.core.dataset_loader.write_dataset_sqlar(document, destination)

Write a structured JSON-LD dataset document as a deterministic sqlar archive.

Empty list datasets and empty dictionary records are unsupported because the sqlar member grammar has no representation for them; this function raises instead of losing data.

Parameters:

• document (dict[str, Any]) – Structured JSON-LD dataset document to archive.

• destination (str | pathlib.Path) – Destination filename ending in .sqlar.

Raises:

ValueError – If the destination or document cannot be represented by the sqlar format.

class httk.core.dataset_loader.DatasetLoader(identifier, source, decode_object=None, **hints)

Lazy loader for httk dataset files, resolved only when data is first accessed.

A DatasetLoader is a declare-time placeholder: constructing it records its arguments and performs no I/O. The source is read the first time data, meta, or index is accessed. Files are either plain JSON (any JSON value is exposed as data with meta/index set to None) or a structured JSON-LD document (with @context, header fields, data, and optional indicies) whose header is exposed via meta, datasets via data.<name>, and lookup indices via index.<name>.

Loaders that share an identifier deduplicate through a class-level registry: the first load wins, and later loaders reusing that identifier return the same result while their source and decode_object arguments are ignored. Keeping identifiers unique is the caller’s responsibility. Not thread-safe.

Format is resolved from the source name after stripping any compression suffix: a .json name (e.g. data.json or data.json.gz) is parsed as JSON; any other recognizable suffix raises ValueError; a source with no determinable name is treated as JSON. Compression is handled transparently by the stream layer, so .json.gz and similar load directly. A plain .sqlar file is an alternative structured JSON-LD representation. It contains header.json, optional indicies.json, scalar data/{D}.json members, and list members at data/{D}/{i:05d}.json or data/{D}/{i:05d}/{field}.json. Sqlar sources require a real filename because their immutable SQLite connection is retained for the lifetime of the cached load; they cannot be compressed, streamed, or loaded from content. Empty list datasets and empty dictionary records cannot be represented and are rejected by the writer; individual members are limited to 256 MiB. Sqlar-backed record/sequence/data views (and DatasetLoaderRecord) pickle by materializing to plain containers; live iterators over them are not picklable. A str/Path source is interpreted as a filename unless its scheme marks it as a URL (http, https, ftp, file); bare network URLs are refused at read time, so pass kind="url" or a urllib.request.Request. Pass kind="content" for literal content or kind="filename" to force a filename interpretation.

Example

symmetry_basics = DatasetLoader(“symmetry_basics”, “data/spacegroup_symbols.json”) spacegroups = symmetry_basics.data.spacegroups # first access triggers the load

Parameters:

• identifier (str) – The deduplication key for this load.

• source (httk.core.datastream.TextstreamLike) – The filename, URL-like stream, request, or literal content to read.

• decode_object (DecodeObjectCallback | None) – An optional callback for JSON-LD objects identified by context URLs.

• **hints (Any) – Stream interpretation hints such as kind.

property data: Any

Return the lazily loaded dataset value.

property meta: DatasetMeta | None

Return structured-document metadata, or None for plain JSON.

property index: DatasetLoaderRecord | None

Return structured-document lookup indices, or None when absent.