Native LMDB Store

lmdb-lace provides a native durable store without resident Rust collections proportional to admitted records or posting members. It implements the existing store operation contract; runtimes continue to use lace-sans-io directly and own execution, publication, polling, and close.

Composition

core/crates/lmdb-lace is an unpublished workspace crate depending on lace-sans-io. It contains the complete synchronous backend, private Rust LMDB ownership, and builds one C engine from the crate’s engine/ submodule. Its only public type is NativeLmdbStore, with open, consuming close, offline compact, and the lace-sans-io StoreBackend implementation (Blocking execution). It is a runtime implementation surface, not application record access.

lace-rt, Tokio, FFI, and PyO3 each have a default-off lmdb feature enabling an optional dependency. lmdb-lace’s build script is the only list of supported targets, those in the profile below, and fails the build for any other target. There is no reverse dependency from lace-sans-io, separate binding crate, backend registry, or external publication gate. Disabled features expose no LMDB constructors, classes, or C symbols. Native C support does not put LMDB into the common WASM artifact.

The fixed profile is:

Readers, database count, and threading mode are not application options. LMDB 1.0.1 applies the configured map size on every open and persists only increases, so the arena fixes a root’s capacity instead: every opener maps exactly the recorded capacity. Windows charges commit memory for mapped views in proportion to their size, about 2 GiB per TiB for LMDB’s reserved view and 4 GiB per TiB for the arena. A 1 TiB root needs address space for both mappings; a kernel with a 39-bit user address space fails its open. The process tests ran on a physical Android 12 arm64 device; app-sandbox file and locking access and iOS devices still need verification. LMDB grows data.mdb as pages are written on every platform. Encryption, checksums, chunk remapping, VL32, previous-snapshot opening, and two-phase commit are never configured; open rejects any environment flag besides MDB_NOTLS. Unsupported host behavior remains a store-open failure, not a sizing fallback. Source provenance, the patch, and licenses reside in the crate’s README.md; its SAFETY.md owns the private wrapper’s lifetime and C-boundary argument.

Physical root and bootstrap

root/
  store-format
  arena-lock
  records
  lmdb/
    data.mdb
    lock.mdb

The exact final marker is lace-native-lmdb-store/3\n. Initialization uses lace-native-lmdb-store/3 initializing\n in the same permanent marker file. That file is the bootstrap-lock rendezvous and is never replaced or removed. Postings are written at admission and never rebuilt, so a change to the projected 020 record facts changes this marker, unlike the filesystem-shaped stores’ index generations. Foreign markers and unknown entries fail closed; this is not a filesystem-store or Linkspace import format.

Opening rejects inherited process state, creates missing directories, and takes the marker lock. Openers and initializers hold it briefly, so a held lock is retried for at most one second before the open fails Unavailable; a compaction holds it for its whole run. Only initialization synchronizes the physical ancestor chain, including after interrupted ancestor creation. Files, the store root, and its immediate parent require successful synchronization. Above that parent, the walk stops on PermissionDenied; that inaccessible ancestor and those above it are outside the directory-entry durability guarantee. Other errors still fail open. Windows has no unprivileged directory sync; there, synchronized files carry no independent directory-entry guarantee. Ordinary open of a final root performs no explicit file/directory sync and reads no admission records. Initialization:

1. validates the marker, component kinds, and permitted existing artifacts;
2. writes and syncs the initializing marker before creating authority files;
3. opens LMDB and atomically creates the complete named schema if absent;
4. verifies that all three databases are empty before initializing the arena;
5. initializes and syncs the fixed arena, then explicitly syncs LMDB;
6. synchronizes LMDB, root, and parent directory boundaries; and
7. writes and syncs the final marker and completes the root boundary again.

A strict initializing/final-marker prefix may resume only with valid empty initialization artifacts. An interrupted LMDB schema transaction leaves either no named databases or the complete set; a partial set is rejected, not filled in. An already-complete empty schema still receives explicit engine sync before marker publication: committing a clean transaction does not establish recovered file durability. No operation is exposed before successful finalization.

A final marker requires the arena and data file. Missing authority is corruption, not a request to recreate it. Normal schema open never uses MDB_CREATE. lock.mdb is operational lock state and may be recreated by LMDB. Open checks exact database flags and names, effective map/readers, configured database capacity, LMDB maximum key size, data-file size, and a valid arena capacity; the map must equal that capacity.

Ownership and trust

The root is trusted cooperating local storage on a filesystem providing coherent positional-write/read-only-mmap visibility, truthful file/directory sync, and working interprocess locks. Arbitrary application bytes are not LMDB page metadata. The C engine is not assumed to safely reject every directly corrupted engine file; Rust row checks run only after C returns rows. Direct modification, replacement, truncation, or uncoordinated LMDB access is unsupported. Process-kill tests establish process-crash behavior, not power-loss or failing-disk protection. On Apple, cooperating processes use the same OS user. Linux file locks, which the kernel releases with their owner, and Windows abandoned mutexes recover a dead lock owner; LMDB then repairs the lock state. Because the kernel, not the C library, holds Linux locks, processes built against glibc, musl, or bionic share a root concurrently. A dead process’s reader slot is reclaimed once that process has exited: through its released pid lock on POSIX, and by waiting on its process handle on Windows. Apple’s named semaphores do not recover dead lock owners while other environments remain open: a process killed while holding a writer or reader-table lock can block surviving users. All users of that root close or exit before reopening resets the locks. Lace adds no recovery beyond LMDB’s own.

Only one environment for a physical root may be independently opened in a process. Device/inode reservation (volume serial and file index on Windows) precedes LMDB open and persists through close. Relative, symlink, and whole-root aliases share that reservation; separately aliased data/lock components and unrelated same-process bindings are unsupported. Separate processes may open the same root. Same-process users share one Lace’s handles instead of independently opening its root.

The environment may move between runtime workers; every transaction and cursor is created, used, and dropped on one thread. Database handles borrow their environment and are checked against each transaction’s owner. Cursor results borrow transactions, never seek buffers. Writer mutation excludes all outstanding mapped reads. No snapshot survives a store operation. Opening briefly occupies a reader slot for its schema transaction and releases it on commit, so a freshly opened store holds none. After its first read the environment keeps that reader reset, holding its slot but pinning no pages, and renews it for the next read without taking LMDB’s reader-table lock. At most 256 processes that have read from one root can keep it open at once; a further process’s open or first read then fails as exhausted readers. A slot therefore reveals neither open stores nor records that outlive them. Forked children reject inherited operations before C or inherited synchronization; destruction skips inherited C state. Exec before reopening in the child. A Linux child shares the parent’s lock-file description until it execs or exits; a lock its parent held when dying stays held until then.

Record results use the shared native_record_arena owner in lace-sans-io::runtime_support. Narrow unsafe initialization, positional-write, and admitted-range construction retain the permanent-file and immutable-range obligations. The existing filesystem backend shares that implementation rather than a copied arena. Returned Records retain only the mapping owner and their range, and may outlive the LMDB transaction, store, and Lace. The mapping owner holds a shared lock on the root’s permanent arena-lock file for its lifetime (live-record exclusion); open creates the file when absent.

Schema

Integers use explicit big-endian encodings; no custom or native-integer comparators are installed.

Ids start at one and are contiguous. The last primary key is StorePosition; an empty primary is before-first. Arena ranges are contiguous from offset zero. Hash mappings and primary rows have exact reverse associations. Duplicate record admission does not change the id or position. Contained records are not separately admitted by storing an outer record.

Primitive keys are:

0x01 || name-length:u16be || name
0x02 || name-length:u16be || name || value-length:u16be || value
0x03 || target-length:u16be || target
0x04 || name-length:u16be || name || data-length:u16be || data

They index field presence, exact field (record kind is exact :Type), link target, and link name/data. There are no prefix postings. The maximum key is 131 bytes under the record field-line bound. Projection uses borrowed visitation and a fixed stack buffer, including structural/inherited fields. Repeated equal occurrences insert the same pair with NO_DUP_DATA; only that exact posting-pair KEYEXIST is accepted. There is no posting cache, heat policy, repair transaction, or stored containment index.

Transaction-scoped operations

Open is independent of the admitted record count: it checks only the fixed physical profile, named schema, and mapping sizes. It does not scan admissions, verify expected postings, or audit reverse mappings. Atomic store transactions establish those invariants; ordinary open trusts them. There is no validated prefix, cached committed end, consistency-audit mode, or incremental replay.

Each operation reads the last primary entry from its own coherent transaction. That single B-tree lookup supplies high-water and the committed arena end. Writers always allocate from this transactionally observed tip, including after an uncertain prior commit; no cached position permits overwriting admitted bytes. Only rows actually used by an operation undergo decoding, association, and range/glyph checks. No store-wide work is added to ordinary lookups or polling.

• CurrentStorePosition returns that transaction’s high-water.
• Resolve preserves request order and excludes valid ids above read_through.
• Exact loads return arena-backed records or authoritative missing outcomes; a missing primary at or below snapshot high-water is corruption.
• Find runs the shared lace-sans-io planner and read executor. For posting keys, the smallest posting by duplicate count drives an ascending bounded seek and every other key is an exact pair seek; a prefix narrows by its field’s presence posting, and a key with no posting yields no candidates. Queries without keys scan ids. Every candidate uses the complete record matcher; a candidate without a primary row is CorruptStore. Only output rows are collected. One further matching row determines complete.
• A finite read boundary above observed high-water is an error. Raw u64::MAX means the observed high-water. Zero-limit finds return empty-complete.

StoreRecords first checks its inputs in a read transaction. When every record is already admitted there, it returns their duplicate outcomes and that snapshot’s position without a writer: admissions are immutable. Otherwise one write transaction serializes allocation root-wide. It reads the committed tip, then writes new records in input order and inserts primary, hash, and posting rows without committing. Repeated input hashes see prior uncommitted mappings; no batch hash map is needed. After all new bytes are written, the arena is synced once, then LMDB commits once. The sole retained backend batch collection is the positional outcome vector. LMDB has its own transaction/dirty-page bookkeeping and may exhaust it before map capacity.

A pre-commit failure aborts all metadata. Unreferenced suffix bytes may be reused by a later writer after reading its own transaction’s committed tip. A commit error remains an error even if progress occurred. The owning Lace performs its existing mandatory position refresh and publishes only progress it can confirm. Neither store nor binding submits a machine response or publishes a wake itself.

Compaction

Compaction is the only path that removes records. It holds the store-format bootstrap lock for its whole duration, so another process’s open fails Unavailable once its bounded retry expires, and it requires an exclusive arena-lock, which fails while any other store or arena-backed record of the root is alive. After moving the arena it rebuilds all three databases from the compaction plan in batched write transactions; postings are rebuilt, not regenerated lazily. Ids and positions restart contiguously in the retained records’ previous order. The freed arena tail is punched; data.mdb does not shrink, and its emptied pages return to LMDB’s freelist for later admissions. Compaction files are valid root entries only on the compaction path. Retention, the move, the reload, and recovery are owned by store-compaction.md.

Errors and integration

Map/arena/id exhaustion and confirmed quota or no-space errors are CapacityExceeded; duplicate environment open, exhausted readers, and bootstrap-lock contention are Unavailable. Schema, row, association, and engine corruption errors map to CorruptStore. MDB_MAP_RESIZED also permanently retires the concrete store; there is no resizing path. It detects growth beyond the local map, not every change to a peer’s configured map size. Other engine, transaction, lock, commit, and OS errors are StoreFailed. Only lookup/cursor NOTFOUND in permitted positions means missing or exhaustion; query errors never trigger a late fallback.

Rust and Python constructors adopt with external advancement enabled. The store marks its execution blocking, so lace-rt uses its blocking workers and Tokio spawn_blocking. PyO3 releases the GIL around native work and close. Native C owns an opaque store and fills existing StoreWork results. All retain the runtime ownership, foreign work boundary, and existing position-polling mechanisms unchanged.