Lace-065 · ILTP Stream Encoding

Tags: transport, iltp, compression

Purpose

Stream encoding lets two ILTP endpoints compress each direction of a logical session with a format both support. Each direction is chosen independently from offers in the 060 session prefaces, without another message or round trip. It operates on ILTP bytes inside any channel protection, so no transport, endpoint grammar, or channel profile participates. A constrained link can use a small window while a server moving large Blobs uses a large one.

Stream encoding is a layer between the 060 item stream and the adopted IO. The encoder takes the decoded bytes of each accepted writer batch and the decoder yields decoded bytes to the 060 item decoder; items, their framing, caps, and meaning are unchanged. 060 defines the preface syntax carrying the offers and which items an encoded stream carries, and 050 owns the lifecycle steps that write an end mark.

Offers

Each 060 preface carries two optional token-list fields for its sender:

Decode names the encodings the sender decodes on its incoming direction. Each token fixes the decoder memory the sender commits. Order carries no meaning.
Encode names the encodings the sender is willing to produce on its outgoing direction, most preferred first.

identity is implicitly the last entry of both lists, from which 060 excludes it, so every pair agrees on at least one encoding; an absent field lists identity alone. A sender lists only tokens it implements. A receiver ignores syntactically valid tokens this specification does not define; this is the only extension point of the profile, and new tokens need no new ILTP version.

Offers are framing. They are not proof, plan, policy, or authority inputs, and a proof does not cover them.

Selection

For each direction from endpoint A to endpoint B:

encoding(A→B) = first token in A.Encode that B.Decode also lists, else identity

Each endpoint computes both directions from the two prefaces once it owns both. Selection compares token text, so both endpoints agree even when one does not know a token the other lists. Each direction’s encoding is fixed for its logical session; there is no in-session switch. A fresh preface pair selects again.

The receiver bounds what it must decode and the sender bounds what it must produce, so list order only breaks ties between encodings both accept; the sender, which owns its output stream, breaks them. The compression level, block types, and match search are sender-local choices. A decoder needs only the token.

Tokens

Token	Format	Maximum window	Dictionary
`identity`	none	–	–
`deflate12c`	raw deflate, RFC 1951	4096 bytes	preset history
`zstd20c`	magicless zstd frame, RFC 8878	2^20 bytes	raw content

A token fixes its format, maximum window, dictionary, and stream flags. Changing any of them requires a new token name.

`identity`

The bytes after the preface are the ILTP items themselves. identity has no flush marker and no end mark.

`deflate12c`

The direction carries one raw RFC 1951 stream, without zlib or gzip wrapping. The dictionary is preset history preceding the first decoded byte; a back-reference may reach into it like into earlier decoded bytes. A sender MUST NOT emit a distance greater than 4096 bytes, counting the dictionary as preceding history, or one reaching before the start of the dictionary. A receiver decodes with a 4096-byte window and MUST reject a reference it cannot resolve. It is not required to detect every distance beyond the window: zlib’s raw inflate resolves some against output it still holds. Decoder memory stays bounded by the window either way. Every block type is valid.

The encoding of each accepted writer batch ends with a sync flush: an empty stored block, ending 00 00 ff ff, after which every byte of the batch decodes without further input. The end mark is the block with BFINAL set. A sender writes it immediately after a sync flush as the empty fixed-Huffman block 03 00. The encoded stream ends with that block; the rest of its last byte is padding, and the following byte is raw.

`zstd20c`

The direction carries one RFC 8878 frame without its magic number. The frame header descriptor MUST be 00, so the frame has no single-segment flag, checksum, dictionary ID, or content size. It is followed by exactly one Window_Descriptor declaring at most 2^20 bytes. Another header descriptor is invalid, and a receiver MUST reject a declared window larger than 2^20 bytes. A sender MUST NOT reference beyond the declared window, counting the dictionary as preceding history, or before the start of the dictionary; a receiver MUST reject a reference it cannot resolve.

The dictionary is a raw content dictionary: its bytes precede the frame content as history, and entropy tables and repeat offsets start at their frame defaults. Block types and sizes follow RFC 8878 limits at the sender’s choice.

The encoding of each accepted writer batch ends with a block flush: every byte of the batch is in complete blocks. The end mark is the block with Last_Block set. A sender writes it as the empty raw block 01 00 00. The frame ends with that block, and the following byte is raw.

Dictionary

deflate12c and zstd20c prime every fresh encoder and decoder with one shared dictionary, again in every logical session. The dictionary is ILTP vocabulary for a short cold session: operand body and rule fragments, proof, policy announcement and answers, listing, symbol, response, change, request, and fence text. The most frequent items come last, where deflate12c’s window keeps them longest. It contains no hash other than the empty Blob’s, and no key, application field name or value, or Interlang variable name. It is at least 512 bytes, because libzstd loses long-range matches for a whole session with a raw content dictionary of 511 bytes or less. The preface is never encoded and contributes nothing to it.

The dictionary is these 830 bytes, listed as UTF-8 fragments concatenated in order. Each fragment is one double-quoted line; \n inside the quotes is LF (0A), \t is HT (09), and no other escape occurs. Lines beginning with # are labels and contribute no bytes:

# Operand body and the generic parts of compiled 040 rules.
"::policy-body\n🖧: B."
".H3\nData-Length: "
"\n\n"
"ExposeRecord("
") :- Have("
"), Field("
",'By','0','V."
".H3'), not Field("
",'By','1',_).\n"
"SelectAdvertisedCandidate("
") :- Advertised("
"), AdvertisedField("
"','0','"
"'), not AdvertisedField("
",'1',_), PeerConfidential("
"SelectStoredRecord("
# Proof: none, or a Seal over the empty Blob.
"::as unknown\n"
"::as\n🖧: S."
".H3\nBy: V."
".H3\nMark: "
"\n🖧: P."
".H3\nChallenge-side0: "
"\nChallenge-side1: "
"\nChannel-Binding: "
"\nProof-Profile: lace-iltp-peer-proof-v3\nProof-Role: side0\n"
"🖧: B.svyLzSM7ffc91i~XDbkMnuOsdjsw_6GrXpTSckqHlpO.H3\nData-Length: 0\n\n\n"
# Policy announcement and its answers.
"::policy B."
".H3\n::policy-known\n"
"::policy-missing\n"
# Listings, projected fields and symbols.
"\n::offers\nS."
".H3\n\tBy: V."
".H3\n\t"
": "
"\nP."
".H3\n\t:Length: "
"\n\t:Type: S\n\tData-Length: "
"\n\tMark: "
"\n\n::offers-unshared\n"
"::symbols\n"
" 1\n\n"
# Responses.
"::response\n🖧: S."
".H3\nBy: V."
".H3\nMark: "
"\n🖧: P."
".H3\n"
"\n🖧: B."
".H3\nData-Length: "
"\n\n"
# Later-round changes and requests.
"::offers-changed\n+ S."
".H3\n- S."
".H3\n\n"
"::want\nS."
".H3\nP."
".H3\n\n"
# Fence and an unchanged round.
"\n::round-end\n::offers-changed\n\n::want\n\n"

Its unkeyed BLAKE3 hash is:

eb56b8cfb9f118671f72c2b391e8a39fa2dfe57bd3af49613047851847e545fd

Encoded stream scope

Prefaces are always raw. A direction’s encoded stream begins with the byte after its raw preface and covers every item of that logical session: proofs, policy announcements and bodies, listings, requests, responses, and fences. A fresh logical session starts a fresh encoder and decoder primed from the dictionary; no history crosses a preface. A fresh preface follows the end mark raw; 060 rejects one decoded from a compressed encoding.

The encoder flushes at the end of every accepted writer batch and needs no other flush. It may emit output for a large batch, such as a 32 MiB Blob response, before the batch ends. A receiver relies on no flush point other than decoded item boundaries.

End mark and end of stream

A compressed direction ends with its end mark, written only at an item boundary after a flushed batch. The sender writes it before a fresh preface on re-prefacing, during bounded completion, and on cancellation of an open operation at fixed point when the writer accepts it immediately, as 050 defines. Terminal failure writes no end mark.

After an end mark, the decoder consumes no further encoded bytes. The next byte MUST begin a raw preface, or the direction ends with EOF; anything else is a peer protocol failure. If the decoded bytes are not at an item boundary when the end mark is consumed, the stream is truncated.

060 counts EOF as boundary EOF only at a decoded item boundary where the direction’s encoding is at its end:

before a preface and after an end mark, the raw direction is at its end;
identity is at its end at every item boundary; and
a compressed encoding is at its end once its end mark has been consumed.

Unlike identity, a compressed direction thereby distinguishes a graceful close from an abrupt one.

Limits and accounting

Every 060 hard cap, current-item accounting, and 050 operation budget applies to decoded ILTP bytes. The decoder produces decoded bytes only as the bounded reader requests them, and the token’s window and dictionary fix its memory, so expansion cannot grow buffering: output stalls under backpressure.

Encoded wire bytes are not counted separately. Blocks that decode to nothing cost their sender the same bandwidth and the receiver decode work linear in those bytes, without retained state; in prefacing, establishment, and active rounds the absolute phase deadline still bounds them.

Malformed encoded input, a window or dictionary violation, or an invalid frame header is a peer protocol failure. Local inability to encode is an Integration failure. Selected encodings and wire byte counts are diagnostics, not observations, policy facts, or protocol authority.

Example

Endpoint	`Decode`	`Encode`
server S	`deflate12c,zstd20c`	`zstd20c,deflate12c`
microcontroller node M	`deflate12c`	`deflate12c`
browser B	`deflate12c,zstd20c`	`deflate12c`

S→M and M→S select deflate12c. S→B selects zstd20c and B→S deflate12c, because B’s build decodes but does not encode zstd. Any endpoint paired with a build offering no encodings selects identity in both directions.

This fragment shows S (side0) and B (side1) at the end of a bounded session. Prefaces and the framing bytes named above are exact; the … payload bytes and block layout are illustrative, since they depend on the sender’s level and the dictionary:

S→B  🪢: iltp/0.13\nNonce: <N0>\n                       raw, 122 B
     Decode: deflate12c,zstd20c\nEncode: zstd20c,deflate12c\n\n
B→S  🪢: iltp/0.13\nNonce: <N1>\n                       raw, 114 B
     Decode: deflate12c,zstd20c\nEncode: deflate12c\n\n

S→B  00 50                        zstd20c frame header: window 2^20
     …                            complete blocks, flushed per batch
     01 00 00                     end mark: empty last raw block

B→S  …  00 00 ff ff               batch ending with a sync flush
     …  00 00 ff ff
     03 00                        end mark: empty BFINAL block

After its end mark, each direction continues in exactly one way:

… 01 00 00 | EOF                              boundary EOF at fixed point: clean
… 01 00 00 | f0 9f aa a2 3a 20 …              raw fresh preface: new logical session
… EOF before 01 00 00                         framing truncation: peer protocol failure