MongoDB storage in detail¶

httk.store.backend.mongo stores the same plain frozen dataclasses as the SQL layer, but uses MongoDB’s document model: one document per record, embedded child arrays, and links to referenced records in their own collections. It exposes the same neutral Store/Searcher protocols, entry-family dispatch, stored properties, continuation paging, and entry-provider surface as the SQL backend.

Choose MongoStore when MongoDB is already the operational data service, when document-shaped records and embedded children are a useful fit, or when the same store must be shared by applications that already speak MongoDB. Choose SqlStore when a relational deployment, SQL tooling, or SQL’s stronger transaction and constraint model is the better operational fit. The two backends share the storage concepts and neutral query vocabulary, but they do not hide the MongoDB-specific limits documented in Differences from the SQL backend.

Installing¶

The backend is optional:

python -m pip install "httk-store[mongodb]"

This installs pymongo>=4.6. Importing httk.store itself does not require PyMongo; importing a MongoDB backend name without the extra raises an ImportError naming httk-store[mongodb].

Connecting and choosing a deployment¶

Create a MongoDatabase from a MongoDB URI and give it a database name. A replica set is the recommended deployment, including a single-node replica set for a development or CI database:

from httk.store.backend.mongo import MongoDatabase, MongoStore

uri = "mongodb://127.0.0.1:27017/?replicaSet=httk2rs"
with MongoDatabase.connect(uri, database="materials", transactions="require") as database:
    store = MongoStore(database, entry_records={})
    # Use store.save(), store.fetch(), and store.searcher() here.

The transactions option has three values:

"auto" (the default) probes the server. A replica set enables multi-document transactions; a standalone server opens in degraded mode.
"require" refuses to open unless the hello response identifies a replica set. Use this when a torn multi-document write is unacceptable.
"never" explicitly selects degraded mode, even when the server is a replica set. This is useful for tests that exercise the no-transaction behavior.

In degraded mode MongoStore emits a warning and does not provide multi-document transaction atomicity. Writes are crash-safe at the individual document level and proceed bottom-up, so a crash can leave complete but unreachable orphan documents. A record cannot point to a missing referenced sid, but a record-family dispatch write can be left for a later repair. The transaction() context manager raises TransactionsUnavailableError in this mode; use a replica set and transactions="require" when explicit transactions are needed.

Entry IDs are reserved through unique ownership indexes before the parent document is written. Normal saves take ordinary writer leases and do not rescan the store. A failed non-transactional write may leave an orphan reservation; attempts to reuse that ID fail closed until store.fsck() reclaims it. A handled failure releases its writer lease, so ordinary fsck suffices. After a process is killed, store.fsck(force=True) may clear its stale lease only after you have verified that the writer is no longer running. Never force past a live writer.

MongoDatabase.connect() configures PyMongo with w="majority", journal=True, and readConcernLevel="majority". Explicit store transactions also use majority read and write concern with journaling. These defaults provide the intended durability behavior on a properly configured MongoDB deployment; they do not turn a standalone server into a multi-document transactional deployment.

Declaring records and opening a store¶

Record declarations are the same non-intrusive frozen-dataclass declarations described in Backend storage. The Mongo backend persists the logical entry-family declaration in its metadata collection. On the first open, pass entry_records; later opens validate the persisted declaration rather than silently changing it:

store = MongoStore(
    database,
    entry_records={StructureEntry: StructureRecord},
)

Beyond the declaration, reopen also verifies the same per-table schema fingerprint as SqlStore: a record class whose resolved on-disk shape or content identity changed since creation is rejected up front with StorageLayoutUpgradeRequiredError, whose diff names the offending collections ({"schema": {collection: {"expected", "actual"}}}). MongoStore(database, ..., upgrade=True) applies a purely additive change under the same rule as SqlStore — new tables plus new non-child, non-derived, IdentitySkip fields whose columns are all nullable. Documents are schemaless, so the apply is only the fingerprint re-stamp (done last, after every other check passes, since Mongo has no transaction to roll a bad open back); old documents read back with the new fields as None and keep their content_id. Non-additive or non-schema differences still raise, and a hint points at upgrade=True when the difference is exactly additive.

Application-private families can instead use the same explicit EntryFamilyDeclaration/EntryRecordDeclaration and entry_families= API as SqlStore. Such declarations bypass global discovery and must be supplied again on every reopen; see Vocabulary for the complete example and binding rules.

The document layout is backend-specific, while the vocabulary of entry families, records, content ids, sids, projections, and stored properties is shared with the SQL layer. See Vocabulary for those concepts and Declaring a storable class for the marker and schema rules. MongoDB sids are integers allocated from a reserved counters collection; they are local to a store and are never reused.

Storing and fetching¶

save() recursively stores a record graph and returns its integer sid. fetch() reconstructs the record, while fetch_by_content_id() and fetch_entry() provide content-addressed and entry-family access. MongoStore has no lazy-row machinery: the whole document is already in memory at read, so the fetch verbs accept the eager keyword for backend transparency with SqlStore but always return a fully materialized record (values and semantics are identical either way). The three StorageInfo.dedup policies are supported with the same content, value, and non-deduplicating meanings as SqlStore; identity-excluded metadata conflicts are still checked. Nested non-storable children are embedded in the owning document. Nested storable records are stored in their own collection and referenced by sid. For the shared save/fetch, projection, validation, and dedup semantics, see Storing and fetching.

An entry family with several backing record classes has a separate dispatch collection. Saving a configured backing makes its content identity discoverable through fetch_entry(), which returns the concrete backing record. In transaction mode the backing and dispatch writes share one transaction. In degraded mode the backing is written first, so a crash can temporarily make fetch_entry() report dispatch integrity failure; re-saving the record or running fsck repairs the main-role case.

Record replacement and lineages¶

MongoStore carries the same logical_id lineage identity and append-only replacement API as the SQL backend: store.replace(predecessor, obj) saves a logical successor sharing the predecessor’s lineage, store.history(obj) walks that lineage oldest-first, and store.searcher(only_latest=True) restricts root variables to each lineage’s latest document. The semantics — idempotent same-lineage replacement, EntryReplacementError on a cross-lineage dedup hit, and only_latest leaving reference/child scopes unfiltered — match SQL exactly; see Record replacement and lineages.

Alternatives¶

store.save(obj, alternative_of=<main id>, alternative_kind=<kind>) records a named alternative representation of a stored main, exactly as on the SQL backend: every parent document carries the alt_id group identity (a main’s own logical_id, copied by its alternatives) and an optional alt_kind (absent on mains). Alternatives copy the group main’s public id, hash with the group identity folded in so they never dedup onto the main, and own one lineage per (alt_id, alt_kind). store.searcher() defaults to only_main_alt=True, hiding alternatives; pass only_main_alt=False to reveal them. StoredEntryFederation serves them latest-only under composite <id>~<kind> ids (query(..., alternatives=True), fetch_alternative(entry_id, kind)), while the revision stream stays mains-only. A store written before this axis lacks alt_id and any alternative-touching write refuses it with a rebuild error.

Store timestamps¶

MongoStore enables store-managed timestamps by default:

store = MongoStore(
    database,
    entry_records={},
    store_timestamps=True,
    store_timestamp_resolution=1_000,  # nanoseconds per stored unit; default: 1,000 (microseconds)
)

The stored value is time.time_ns() // store_timestamp_resolution. The public query API accepts a canonical nanosecond integer or an RFC3339/ISO-8601 timezone-aware value and converts it to the store’s units. For example, this historic query returns rows present at T:

searcher = store.searcher()
record = searcher.variable(StructureRecord)
searcher.output(record, "record")
searcher.add(record.store_timestamp <= "2026-01-01T00:00:00Z")
rows = searcher.results(record=record)

The equivalent OPTIMADE filter is:

from httk.store.backend.mongo import optimade_filter_searcher

rows = optimade_filter_searcher(
    store, StructureRecord, '_httk_store_timestamp <= "2026-01-01T00:00:00Z"'
)

present at time T means exactly store_timestamp <= T. FIRST-STORED-WINS applies: a deduplication re-save does not replace the original timestamp, and promoting a dependency to a main row does not replace it. One timestamp is captured per save transaction and one per bulk batch, so all rows written by that unit share its value.

Before capture, the writer checks a process-local high-water mark. A clock regression smaller than 1 ms waits briefly when clock_regression_grace=True (the default); larger regressions, or a failed grace wait, raise StoreClockRegressionError. Set clock_regression_grace=False to skip the wait, or allow_clock_regression=True to disable the guard. The mark is per-process: reopening seeds it from stored rows, but it is not a cross-process clock-coordination protocol.

store.fsck() checks for timestamps beyond the current clock plus the allowed future slack. An administrative repair can clamp them:

store.fsck(repair=True, clamp_future_timestamps=True, known_types=(StructureRecord,))

Clamping is destructive to historic-query fidelity; inspect a non-repair fsck report and confirm the skew before using it.

The query stack also exposes as_of=T on stored-property federation query()/fetch() and on the general FederatedStore.searcher(). The serving layer accepts _httk_as_of and includes it in stable pagination plans. Stored federation is availability-first: a source with store_timestamps=False deliberately ignores the cutoff and serves that source’s current state; sources with timestamps enabled apply their own-resolution cutoff. Existing layouts do not require an enable/disable migration for reading this capability.

Roles, leases, and fsck¶

Every Mongo record document has a store-managed role:

main marks a top-level record or a record addressed by an entry dispatch.
dep marks a record reachable only as a dependency of another record.

This distinction lets MongoStore retain crash residue safely until an explicit integrity pass. store.fsck() takes an exclusive fsck lease, blocks writers for its duration, checks entry dispatches, repairs missing dispatches for main-role family records, marks records reachable from roots, and deletes unmarked dependency-role documents. It never creates a dispatch for a dependency-role backing. The return value is an immutable FsckSummary with per-collection examined, repaired, conflict, and deleted counts plus reported violations.

After reopening a store, pass record classes that were not discoverable from the current store declaration or session through known_types so fsck can attribute their ordinary collections safely:

summary = store.fsck(
    repair=True,
    collect_garbage=True,
    known_types=(StructureRecord, Author),
)

repair_conflicts=True allows fsck to delete invalid dispatch documents after reporting them. It is a repair choice, not the default. force=True is an administrative stale-lock override: use it only after verifying that the previous owner is dead. The lease protocol has no fencing. The clear_stale_lock() operation has the same administrative requirement.

Running fsck while other store processes remain open is discouraged. A live process can retain identity-cached instances of records that fsck swept; its next write observes the generation bump and clears those caches, but a cached read before then can be a silent stale read. Uncached fetches of swept sids raise KeyError, and sids are never reused.

Querying and paging¶

MongoStore.searcher() follows the same neutral query protocols and expression vocabulary as SqlStore: bind variables with variable(), add conditions with add(), declare outputs, and consume either portable SearchResult values or a named results() set. Reference paths, child set operations, stored-property plans, scalar projections, sorting, offsets, limits, and OPTIMADE filter wiring use the shared concepts documented in Searching and Neutral portable Store profile. Disconnected cartesian variables are outside MongoStore’s supported query profile.

search = store.searcher()
s = search.variable(StructureRecord)
search.add(s.spacegroup == 225)
search.add(s.symbols.has_only("O", "Ca", "Ti"))
results = search.results(structure=s, energy=s.energy)

for row in results:
    print(row.structure.formula, row.energy)

Mongo result sets provide len(), iteration, first(), one(), scalars(), and scalar column() access with the shared result exceptions. results() materializes the result rows needed by its consumer; a query with a client-verified predicate applies verification before count, offset, limit, or output consumption.

MongoResultSet.page() is the optional keyset-paging capability described by the neutral PageableResultSetLike protocol. It uses a live aggregation and an opaque continuation token, with an internal sid tie-breaker and explicit null ordering. The normal restrictions apply: one root variable, scalar root outputs for order keys, no add_sort(), nonzero offset, or query limit, and a page size of at most 10,000. Pages do not promise snapshot consistency across calls. See Continuation pages for the token and consumer contract.

Stored properties that use scaled_exact_equal() (or another predicate that needs exact client verification) have an important Mongo-specific cost. The client-side evaluator is authoritative over hydrated records. In the Phase 5 implementation, the server prefilter is the degenerate, empty, trivially conservative prefilter: every candidate backing is transferred to the client for exact evaluation. The result iterator over-fetches candidates and applies offsets, limits, counts, and page assembly only after verification. This has the same per-row evaluation asymptotics as SQL’s UDF full scan, plus candidate transfer cost; the approved epsilon-window prefilter remains a follow-up optimization.

Entry providers and federation¶

httk.store.backend.mongo.StoreEntryProvider serves configured entry families or concrete backing records through the neutral httk.core.EntryProvider contract. Family entries use the Mongo stored-property plan, and relationship links can be declared with the same provider-facing concepts as the SQL surface — the provider path serves the StrongLink provenance relationships in both directions, like SQL. Mongo federation, however, serves no relationships at all (see Differences below). Stored-federation membership uses the Mongo entry-store protocol and content identities; it does not require converting a Mongo store into a SQL store.

Differences from the SQL backend¶

The following are accepted residual divergences of the MongoDB design. They are operational behavior, not guarantees to be inferred from SQL parity.

Degraded dispatch crash window. Degraded mode admits a crash window where an entry record exists without its dispatch document; fetch_entry() raises, re-save or fsck repairs, and the mode is announced.
Non-transactional index and validator creation. Index and validator creation is not transactional. It is idempotent, additive, and synchronous before the first insert.
No transactions in degraded mode. transaction() raises in degraded mode.
Orphan documents. Unreachable orphan documents, for any dedup policy, can exist between a degraded-mode crash or dedup-discard and the next fsck. Degraded-mode compensation deletes nothing; fsck is the collector. SQL’s v2.3.0 degraded SQLite profile follows the same main/dependency role and fsck model; see the SQL permanentization section.
Client-verified exact predicates. scaled_exact_equal() and any client-verified predicate cost client-side verification and over-fetch. If the packet-level non-pageable fallback is taken, it requires separate maintainer sign-off. In the Phase 5 state, the server prefilter is the degenerate empty, trivially conservative one: such plans transfer every candidate backing to the client for exact evaluation. They have the same per-row evaluation asymptotics as SQL’s UDF full scan, but with candidate transfer cost. The epsilon window remains an approved follow-up optimization.
Fsck exclusion and stale-lock administration. fsck blocks all writes for its duration. Cross-process exclusion is an advisory lease handshake. force=True stale-lock override is an administrative assertion with no fencing.
Post-fsck cached instances. After an fsck in one process, another live process may serve identity-cached instances of swept orphans until its next write observes the generation bump. Cached-instance reads are potentially silent stale reads, with no signal to the caller. Uncached fetches of swept sids raise KeyError; aliasing never occurs because sids are not reused. Running fsck while other store processes are open is therefore discouraged.
BSON size ceiling. Records whose embedded document exceeds MongoDB’s BSON size ceiling (16 MB) are rejected with RecordTooLargeError; SQL has no such ceiling.
Fetched-object identity. Fetched-object identity-while-alive is a per-store implementation property, not a portable protocol guarantee.
String matching case. String matching is canonically case-sensitive; SQLite’s ASCII case-insensitive LIKE is the divergent backend.
Sharded clusters. Sharded clusters are untested and unsupported for now.
Relationships and relationship filtering. The Mongo StoreEntryProvider path serves the StrongLink provenance relationships in both directions, matching SQL. Mongo federation serves no relationships (its per-row relationships channel is empty), and there is no _httk_relationships relationship filtering on a Mongo-federated route.

Testing profiles¶

Mongo tests are opt-in: set HTTK_TEST_MONGODB_URI to a reachable MongoDB deployment. Without it, Mongo-specific tests and the Mongo parameter of the neutral backend suite are skipped. A replica set exercises transaction mode; tests also explicitly select transactions="never" where degraded behavior is under test.

The default test profile excludes tests marked extended and uses the normal fast coverage. The extended profile includes those tests and increases the seeded randomized fsck graph rounds. Run both tiers with the Mongo URI:

export HTTK_TEST_MONGODB_URI='mongodb://127.0.0.1:27127/?replicaSet=httk2rs'
python -m pytest tests/ -q
HTTK_TEST_PROFILE=extended python -m pytest tests/ -q -m ""

The extended knob is intentionally independent of the connection URI. It changes test depth, not MongoStore semantics. The repository’s default make ci remains Mongo-free; the dedicated CI job runs the live Mongo suite in both tiers.