Portable File Store

The portable file store keeps admitted records durably in plain files, using only ordinary file reads, positioned writes, syncs and directory operations. The same code runs on hosts and on a small device’s FAT card through ESP-IDF’s VFS, holds no per-record state in memory, receives and sends records of any valid size in bounded pieces, and answers repeated queries from posting lists read a page at a time. It implements the semantic store contract of store-record-fact-access.md; this note owns its layout, commit order, index and recovery. It is not a filesystem-shaped store profile and shares no root with one.

Constructor and ownership

Lace::portable(root, PortableStoreFormat) in lace-rt and lace-tokio opens or creates a root and adopts position_may_advance_externally = false. The format sets the segment size and the initial index size and is used only when the root is created; reopening uses the values recorded in format.

One Lace owns a root until close or final owner drop. A second opener in the same process fails as Unavailable, by absolute root path. Where the platform has file locks, the owner also holds a kernel lock on lock, which excludes other processes and path aliases; where it has none (ESP-IDF), one process is assumed to own the card. Records the store returned stay readable after close: each retains its segment’s shared open handle, never reopened by path. A process-local weak registry by absolute segment path reuses that handle after cache eviction or a new adoption. FatFs forbids a second open of a file with a live writable handle. The registry retains no descriptor; its entry is removed only after the last owner’s descriptor closes. An opener racing that final drop waits for its close before opening again. This does not extend root ownership.

Layout

root/
  format          marker, segment bytes, initial index slots
  lock            lock rendezvous; holds no data
  head0, head1    alternating committed heads
  rows            fixed 64-byte rows, one per admitted record
  index           hash index
  seg00000 ...    record bytes, one file per segment
  postings.v2/    posting lists and their `dirty` marker
  staging/        one subdirectory per adoption

Names in the root are 8.3-compatible, so FAT without long names holds the records; FAT returns them in upper case and names read back compare without case. Posting-list paths below postings.v2/ need long names (ESP-IDF CONFIG_FATFS_LFN_HEAP); without them every list write fails, which disables lists for the session, and queries scan.

• Segments hold canonical record bytes appended in admission order. A segment’s size is fixed at creation, at least the 33,620,814-byte maximum record and below FAT’s 4 GiB file limit (default 256 MiB). A record never spans segments: one that does not fit starts the next segment. Segments grow by appending rather than preallocation, because extending a file on ESP-IDF’s FAT writes zeros over the whole extension.
• Rows use the native record-log row: hash text, byte offset and byte length. The offset is global: segment number × segment size + offset in the segment. StoreRecordId(i) is row i − 1; StorePosition is the committed row count.
• Heads hold a sequence number, the committed row count, the committed byte end and a BLAKE3 checksum. Head n is written to head(n mod 2).
• format holds lace-portable-file-store/1\n and the two sizes. Opening an absent or empty root, or one holding only an incomplete format, lock and empty head files, creates it: both empty head files, then format, then one directory sync. Any other root without a valid format fails as CorruptStore.

Commit

One StoreRecords call classifies duplicates against the index and within the batch, then commits its new records:

1. write every record’s bytes at the committed end, copying file-range records through one store-owned buffer of the runtime’s IO chunk size, and sync each segment written (and the directory when a segment lies past the committed end, since an earlier failed commit may have created it);
2. write the rows and sync rows;
3. write the next head over the older head file and sync it.

The synced head is the admission point. A failure before the head write leaves the committed state unchanged; the next commit overwrites the same bytes, rows and head file. A failed head write may still reach the disk, so the store refuses further commits with StoreFailed until it is reopened; reads continue at the previous head. Duplicate admission returns the existing id and writes nothing.

Recovery

Open selects the valid head with the highest sequence; a head whose checksum fails or that sits in the wrong file is invalid. A torn head write leaves the previous head valid. With no valid head the store is empty unless both head files were written, which no crash produces and which fails as CorruptStore. Bytes and rows beyond the chosen head are ignored and later overwritten. A rows file shorter than its head fails. Committed bytes are trusted: loads check the prefix’s glyph and Data-Length and the row’s hash, not record hashes or Marks.

Hash index

index is a 32-byte header and a power-of-two table of 8-byte slots: a row id in the low 40 bits and a 24-bit hash fingerprint. Lookup probes linearly from the hash’s home slot, reading 64 slots at a time, and confirms a fingerprint match against the row’s hash. The table stays at most half full, so exact lookup and duplicate checks take an expected one slot read and one row read.

Commits insert slots without syncing them. A checkpoint at open, close and drop syncs the slots before the header records how many rows they cover, so a covered row is always findable. Open inserts any rows past the covered count again, which is a no-op for slots already written. A header that is invalid or claims more rows than the head, or a table about to pass half full, is rebuilt by streaming rows into a fresh table of doubled size: the header is invalidated durably before slots are cleared.

Queries and loads

FindRecords runs through the shared planner, executor and matcher. An exact hash uses the index. Other queries use the posting lists of their keys that exist, intersected in increasing id order with the shared traversal (the shortest bounded list drives); with none they scan ids in their bounds. Every candidate costs its row and its record prefix in at most two bounded reads (1 KiB, then up to the maximum prefix). A loaded record is a file-range record over its segment.

Posting lists

Lists use the native filesystem store’s primitive keys, .ids paths and file format below postings.v2/: matching ids as little-endian u64 in increasing order, then a sentinel position through which every record was examined. Which keys get lists follows the native heat window and known-permanent selection. The store keeps no ids in memory, only each hot list’s id count and positions:

• a key that becomes hot gets its list built, or an existing one extended, in one pass over the records it has not examined, shared by every list made hot at once;
• every commit appends the new matching ids and sentinel to each hot list in one positioned write, and only advances the in-memory position of a hot list without matches;
• a query also uses lists of keys no longer hot, extending each to its read boundary first;
• a query reads each list a 4 KiB page at a time after two binary searches for its bounds, checking that ids increase and stay within the sentinel.

So a warm query reads its candidates’ records, not every record, and a commit costs one write per hot list it matches.

Lists are acceleration, not committed state, and their writes are not synced as they happen. Before its first list write a session creates postings.v2/dirty. Close and drop write the hot lists’ sentinels, sync every list written, then remove dirty. Open removes every list when dirty exists, since a crash may have torn any of them, removing dirty last. A list that is malformed at open (length, sentinel above the head, more ids than its sentinel) or missing is rebuilt; one found malformed while a query reads it is removed, and the query scans the rest of its range. A failed list write disables lists for the session and leaves dirty. Record truth that cannot be read during list work fails the operation.

The directory name is the index generation, as for the filesystem-shaped stores: a change to the projected 020 record facts names a new one, and open removes postings and every other postings.* directory, comparing names without case. The format marker changes only with record bytes or layout.

Staging

The store supplies staging under staging/ for connected reception, as runtime-provider-architecture.md defines. Each adoption stages into its own subdirectory, registered as live in the process while the staging handle or any staged record exists. Opening removes every other entry: residue of earlier processes, and of earlier adoptions whose records are gone. A file a live record may hold is never removed, which FAT requires.

Limits

• At most 100,000 segments and 2^40 − 1 rows; beyond them admission fails with CapacityExceeded.
• The segment cache retains at most four handles; a commit retains only its current destination, syncing and releasing that owner before acquiring the next. A cache miss reuses a live record’s handle, or briefly opens another segment before eviction; input and returned records can retain further handles. On ESP-IDF the FAT mount’s open-file limit must also cover rows, index and staging files.
• No compaction: records are never removed.
• Hot lists are at most the heat configuration’s known-permanent ceiling (64); each list a query reads holds one 4 KiB page.
• Power-loss safety on real cards is unverified; it relies on each sync making the written data and file sizes durable.