Conversation
BaboonComparator now classifies, per evolution step and per type, whether an older version's codec can decode newer-version data: IDENTICAL (byte-identical), PREFIX_ANY_MODE / PREFIX_COMPACT (appended-only fields; top-level framed UEBA reads), JSON_ADDITIVE (tolerant JSON readers), or nothing. Step tiers are composed into contiguous suffix runs (BaboonEvolution.typesForwardReadable) via an order-sensitive own-structure check plus a fixpoint over codec-relevant dependencies; emitted in baboon-meta.json as "forwardReadable". Scala + TypeScript backends emit the metadata as baboonForwardReadable (per-type constant + BaboonGenerated member + BaboonMetadata lookup), with end-to-end stub specs proving each tier against real cross-version blobs, including negative controls. Design: docs/drafts/20260911-0937-forward-compat-metadata.md. Note: ForwardCompatComparatorTest documents a pre-existing sameIn overclaim (deepSchemaRepr sorts flattened dep reprs, erasing member order inside deps; a reordered-enum host stays 'unmodified' although its UEBA bytes change). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
C#, Python, Rust, Kotlin (+KMP), Java, Dart and Swift now emit the baboonForwardReadable version->tier map as a per-type constant, a BaboonGenerated(-equivalent) member, and a forwardReadableVersions(typeId) registry lookup, mirroring the existing baboonSameInVersions surfaces. Hand-written runtime-test stubs implementing the extended interfaces are updated (C#/Kotlin/KMP/Java/Dart); Swift and Python use source-compatible defaults overridden by generated code; Rust emits the real table where it previously had only degenerate own-version sameIn data. docs/forward-compat.md documents tiers, the prefix-* client contract, and metadata locations. docs/ueba-format.md: corrected the enum wire description (single positional u8, not i32 of the discriminant; const values never hit the wire) — verified against the C#, Scala, TypeScript and Rust generators. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sameIn/unmodified overclaimed byte-identity: deepSchemaRepr sorted the flattened dependency repr lines per field (and sorted ADT branch reprs), erasing member/field/branch order inside dependencies and type-constructor argument order — while UEBA is positional (enum discriminants, ADT branch indices, field bytes, map K/V order). A host of a reordered dep enum, a reordered dep ADT, or a dep with swapped map arguments classified as 'unmodified', so sameIn ranges promised byte-identity the wire does not have. Fix: dependency reprs stay contiguous and internally ordered (determinism now comes from sorting the dependency IDs instead of the flattened lines), each field line carries its full type-ref rendering, and ADT branch order — the UEBA discriminant order — is preserved. Reproduced fail-first in ForwardCompatComparatorTest (EnumReorderHost / SumReorder / SumReorderHost / MapSwapHost fixtures), which now pins the corrected classification. Consequences: - Lockfiles persist deepId-derived signatures: Locks now carries a 'scheme' marker (2 = this hashing; absent/1 = legacy). A stale-scheme lockfile is incomparable — enforcement is skipped for that one run and the file is re-signed in place even under create-only (T156 matrix extended). - The M20 manual→sugared ADT rewrite is honestly classified: the sugared expansion reorders branches, shifting positional UEBA discriminants, so the step is deepModified with a derivable CopyAdtBranchByName conversion. PR-D's branch sort had masked this wire break; M20AdtEvolutionTest updated to assert the conversion-derivability guarantee instead of byte-identity. Also adds .jvmopts (-Xmx8g -Xss8m): sbt's 1024 MB launcher default OOMs the Scala.js parallel optimizer on many-core machines during 'sbt +test' (reproduced on a clean clone of main, pre-existing). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dReadPolicy — Scala/TS pilot Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… 9 backends Every runtime now carries BaboonTypeMeta.domainVersionReadableMin (five-field construction keeps readable-min = minCompat), writes/reads the JSON envelope key $rv (elided when equal to the effective $uv), exposes the generated per-type baboonMinReaderVersions (tier -> oldest reader) that feeds it, and resolves JSON codecs under a facade-level ForwardReadPolicy (Tolerant default: honour $rv for payloads newer than any registered version; Lossless: $uv only). UEBA resolution is unchanged (v1 envelope has no slot for the bound). Codegen: baboonMinReaderVersions emitted by all 9 DomainTreeTools / the Rust facade generator. Dart test fakes gain the new provider getter. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…acade meta tables
All nine binary envelope readers treated any hasMinCompat value other than 1 as
"elided", so an illegal flag byte (e.g. 0x02) was silently misparsed — the
min-compat string was consumed as the type identifier. codec-envelope.md §2.1
mandates rejection; readers now return None/null/undefined/Ok(None) for any
value other than 0x00/0x01. Reproduced fail-first in the Scala and TypeScript
stub suites (BaboonTypeMetaCodecSpec, TypeMetaFlagByte.test.ts), verified in
Python directly.
The TypeScript and Rust generated facade metadata registries emitted degenerate
own-version tables (sameInVersions = [own], forward = {own: identical}); they
now carry the real per-type sameIn / forward-readable tables like the other
backends (unknown type ids resolve to empty rather than fabricated data).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…TextTree The TypeScript and Rust domain-facade generators and the GraphQL SDL translator assembled generated code with StringBuilder appends and shipped it through TextTree.verbatim / OutputFile(String). Every other emitter in the compiler builds a TextTree; these three now do too (q-interpolated templates, joinN/joinNN, shift/trim), with per-version, registration and type-definition pieces as composable subtrees instead of imperative append sequences. GraphQL doc comments and the BaboonAny scalar description are interpolated as verbatim nodes: plain String interpolation is escape-processed at render time and stripMargin would strip `|` margins from user text. Generated output is unchanged: schema.graphql files are byte-identical for the whole test model directory, and the 36 TS + 36 Rust facades differ only in the trailing newline, which now matches every other TextTree-rendered file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…aders decode newer payloads with their newest codec The v1 binary envelope has one bound slot, domainVersionMinCompat, and its layout is frozen (a new field, a flag bit or a trailer would all be a format change). Forward reads for UEBA therefore ride on the VALUE of that slot, chosen by the writer through ForwardWritePolicy on BaboonCodecContext, next to the index mode the bound depends on: - Strict (default): the byte-identical bound, exactly as before. - Tolerant: the prefix-read bound from baboonMinReaderVersions for the payload's index mode (prefix-compact for compact, prefix-any-mode for indexed). Equal to the Strict bound when no prefix relationship exists, so such envelopes stay byte-identical. Readers in all nine runtimes now trust the bound: a payload from a version newer than every registered one is decoded with the reader's newest codec as soon as the bound reaches a registered version, instead of with the bound version's codec via the sameIn scan. Readability is monotone along the chain, so the newest codec is correct under both bound semantics and loses the fewest fields. Reproduced fail-first over a new three-version shared fixture (fwd-e2e-chain-ok): a reader registering 1.0.0 and 2.0.0 decoded a 3.0.0 envelope bound at 1.0.0 as ChainAppend(1) and dropped the 2.0.0 field. The cost is documented in codec-envelope.md §2.1.2 and forward-compat.md: a binary reader cannot tell a Tolerant envelope from a byte-identical one, so ForwardReadPolicy.Lossless has no effect on binary reads and re-encoding intermediaries must run at the writer's version or newer. The Rust runtime crossed the JVM 64KB embedded-constant limit with this change; BaboonTypeMeta and its wire codec moved to baboon_type_meta.rs and are re-exported from baboon_codecs_facade.rs so every existing path still resolves (CLAUDE.md records the symptom and the fix). E2E coverage: ForwardCompatBinEnvelopeSpec (Scala stub) and ForwardCompatBinEnvelope.test.ts (TS stub) — Strict unchanged/refused, Tolerant compact decoded by an old reader, json-additive/identical/grown-enum envelopes identical to Strict, indexed context not lowering a var-len append, and the three-version chain decoded with the mid reader's newest codec. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…mbination Adds a "Worked examples: what changed on the wire" section: the three knobs (writer ForwardWritePolicy, index mode, reader ForwardReadPolicy) and the reader-rule change; per-type writer bounds for the fwd-e2e fixtures; the real JSON envelopes with both reader policies' outcomes; annotated UEBA bytes for Strict vs Tolerant; the full value x context matrix as seen by a 1.0.0-only reader; the three-version chain before/after the newest-codec rule; and a summary matrix. All envelopes and outcomes were captured from the generated Scala stub. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…very mode Reworks the worked-examples section around the fixture types themselves: each of FwdAppendVar, FwdMidInsert, FwdStable, FwdEnumHost and the three-version ChainAppend is shown as its .baboon source across versions, its per-tier bounds, the JSON envelope with both reader policies' outcomes, and the UEBA envelope under Strict/Tolerant x compact/indexed with annotated bytes and the older reader's result. All envelopes and outcomes were captured from the generated Scala stub. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…rs accept v1 and v2, writers default to v1
The v1 binary envelope has one bound slot, so a v1 reader cannot tell a
byte-identical read from a prefix read. metaVersion 2 is the JSON envelope's
field set in binary form:
02 | domainId | domainVersion | flags | [minCompat] | [readableMin] | typeId | payload
flags bit 0: minCompat follows (elided when == domainVersion); bit 1:
readableMin follows (elided when == effective minCompat); any other bit is
rejected. minCompat is always the byte-identical bound and readableMin the
prefix bound for the payload's index mode, so ForwardWritePolicy is
irrelevant under v2 and the reader's ForwardReadPolicy applies to binary
exactly as it does to JSON: Tolerant decodes with the newest codec once
readableMin reaches a registered version, Lossless requires minCompat to.
The layout is selected per encode through BaboonCodecContext.envelopeVersion
(V1 default, V2 opt-in) in all ten runtime directories; the fully specified
context constructor gained the parameter. writeBin dispatches on
meta.metaVersion and fails fast on anything else; readMeta accepts 1 and 2 and
rejects other versions and unknown v2 flag bits. Binary getCodec now passes
the reader's policy through; v1 metas carry readableMin == minCompat, so their
behaviour is unchanged.
Reproduced fail-first in the Scala and TypeScript stubs: a hand-assembled v2
envelope was rejected by readMeta ("v2 envelope must be readable"). New specs
BinEnvelopeV2Spec / BinEnvelopeV2.test.ts cover the reader, the byte-identical
default, per-type flags and bounds in both index modes, round trips through
the writer and old readers under each policy, the three-version chain with
Lossless now enforceable for binary, and rejection of unknown flag bits and
metaVersions.
baboon_runtime.swift crossed the JVM 64KB embedded-constant limit;
BaboonVersion/BaboonDomainVersion/BaboonTypeMeta/BaboonTypeMetaCodec moved to
baboon_type_meta.swift in the same module (translator emits it; CLAUDE.md
records the pattern).
Spec: codec-envelope.md §1, §2.1.3 (new), §3 (byte 2 active), §5, §6.
forward-compat.md: v2 section, knobs, per-type v2 examples, summary matrix.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Adds the captured envelope-v2 bytes for FwdMidInsert and FwdEnumHost (flags 0: the v1 bytes with 02 in front) so the per-type walk-through covers v2 for all five fixture types, and explains how the v2 blocks relate to the shared v1 header. All v2 examples were re-captured from the generated Scala stub and match the bytes asserted by BinEnvelopeV2Spec. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… sameIn dyn and Python multi-version facade A byte-for-byte check of the binary envelope across backends (same values, same contexts) found two asymmetries the per-language suites had not: - Rust: the generated `baboon_same_in_versions_dyn` returned `[own version]` for every type while the Meta struct carried the real sameIn table. The envelope's byte-identical bound is the head of that run, so Rust-written envelopes of unchanged types (v1 and v2) elided the bound every other backend publishes — an older reader in any language refused them. The dyn impl now emits the real run (RsBaboonTranslator). - Python: `_register_version` sorted by `v.version.version`, an attribute that does not exist, so no domain facade with more than one version could be constructed (no runtime test built one). Sort by `v.version`. Every backend that had no envelope test — C#, Kotlin, KMP, Java, Python, Dart, Swift, Rust — gains a golden-bytes suite asserting the exact sequences from docs/forward-compat.md (v1 Strict, v1 Tolerant, v2 compact and indexed for FwdAppendVar; FwdStable v1 and v2; FwdEnumHost v2; the three-version chain v2), the v1/Strict default of the built-in contexts, and a v2 round trip through the writer's own facade. Scala and TypeScript already assert the same sequences structurally. Swift: test/sw-stub/Package.swift is hand-written and had no targets for the fwde2e.fwd / fwde2e.chain fixtures, so the Swift lane never compiled or tested them (SwiftPM only warns about unowned Sources/ dirs). Targets, per-module test targets and the RuntimeTests dependencies are added. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Adds §2.1.4 to codec-envelope.md, the v2 counterpart of the v1 conformance block: annotated canonical bytes for FwdAppendVar (flags bit 1, readableMin present) and FwdStable (flags bit 0, minCompat present), byte counts against the v1 form, and pointers to the golden-bytes suites that pin them in every backend. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
forward-compat.md gains a "Known gaps" section: no runtime e2e for prefix-any-mode (and why), non-uniform missing-tier handling across runtimes (and why it is unreachable for generated types), and the timestamp kind-byte round-trip flake tracked as #91. CLAUDE.md records that test/sw-stub/Package.swift is hand-written and silently skips fixture domains until their targets are added. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ral fallback in the converter round-trip test Missing-tier handling: Python, Swift and Rust fell back to the byte-identical bound when baboonMinReaderVersions lacked the json-additive or prefix tier, because their base types gave the member a default; the other seven runtimes failed fast. The defaults are removed (abstract property / no protocol-extension default / required trait method), the writers raise an encoder failure, the Rust facade's type-meta construction returns Result instead of a bare value, and the hand-written test fakes in the Swift and Rust stubs now provide the four tiers. Fail-first tests: test_min_reader_tiers_fail_fast.py, MinReaderTiersFailFastTests.swift, min_reader_tiers_fail_fast_tests.rs. Timestamp kind byte (#91): the C# RpDateTime intentionally carries its DateTimeKind on the wire and the Scala/JVM side is left unchanged, so a tso/tsu is not byte-stable across writers by design. RTCodecTest therefore falls back to a structural, equal-length comparison when the re-encoded bytes differ, which is the property the round trip actually promises. Python wrote 2 (Local) for every non-zero offset; it now writes 0 like the other non-.NET runtimes (test_timestamp_kind_byte.py). docs/ueba-format.md defines the byte. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Address the backend/runtime review across all targets with shared semantic plans, backend-local scalar and service emitters, and runtime-owned envelope and MCP helpers. Fix renamed ADT branch evolution and source/target conversion validation, preserving custom validator contracts. Add behavioral regressions, compatibility coverage, and the implementation ledger. Verified the full 205-action CI matrix, 200 serialization checks, and 162 RPC checks; repeated clean JVM/JS compilation, compiler tests, and native build before commit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.