feat(avro): derived whole-record builder — recordBuilder[A], recursive sub-record levels (#95) - #116
Merged
Merged
Conversation
…e sub-record levels (#95) The #95 filer's production path is a hand-built leaf-by-leaf .put builder: fast (744-768 B/op on a 66-leaf nested ClickInfo) but one hand-maintained line per leaf. The positional builder over held per-field leaf codecs removes the boilerplate but re-enters Codec[Sub].encode for every nested sub-record — the filer implemented it, measured 7.5x time / 16.9x allocation against their hand-built path, and rejected it: it strips only the outermost shell of vulcan's composition. recordBuilder[A] derives the hand-built shape at compile time instead: - the macro walks A's case fields into a plain runtime RecordShape IR (one assembly call emitted; no staged code); - construction resolves every case field's schema slot by NAME via a new all-or-nothing AvroWalk.recordSlots rung, and validates every arm against the schema it writes into; - toRecord is pure positional puts: primitives direct, nested case classes RECURSE into sub-record levels (self-recursive types resolve through the runtime level chain), None puts null exactly as vulcan's OptionCodec, everything else through the field type's own vulcan.Codec — summoned at the derivation site, so a missing leaf codec is a compile error naming the field. Construction is total: every refusal (unresolved case field, two case fields claiming one column, arm/schema disagreement, non-record schema) comes back as the Exception half of Exception | WholeRecordBuilder[A], naming the field and the record. The one documented difference from codec.encode: schema-only (computed/derived) columns keep their in-record default — the hand-built .put contract, round-trip-safe through the codec's decode. Doctrine pass on the vulcan surface per review: no naked throws. AvroVulcan.codec is two forms — codec(schema) (total; schema in hand) and codec[A]: Either[Exception, AvroCodec[A]] — schemaOf deleted; the opt-in given stays the one documented eager-failure site. Measured (benchmarks ClickRecordBench, -prof gc, the filer's shape): derived 744 B/op / hand-built 768 / rejected positional 23,912 / full codec 28,408 — the allocation gate is hand-built-equal, and ns/op stays sub-microsecond through codecPrism[...].record.reverseGet (asAvroCodec wiring, +0.001 B/op). Gates: avroIntegration/test 235 passed (21 new across the builder spec, macro-error catalogue and the bridge spec), root aggregate compile, scalafmt on touched modules, .github/bench tooling unittests.
Contributor
|
🚀 Cloudflare Pages preview for https://fafb21ab.cats-eo-docs.pages.dev Branch alias: https://feat-whole-record-builder-95.cats-eo-docs.pages.dev Built from commit |
…ual gate missed CI (PR #116) caught what my per-module 'scalafmt' run didn't cover: the Test-config sources and parts of the macro file were unformatted. Full 'scalafmtAll ; scalafmtCheckAll' is green; avroIntegration/test re-run: 235 passed / 0 failed.
Contributor
Benchmark A/BAllocation (B/op) — authoritative
433 more benchmarks
Timing (ns/op) — directional only, same-VM but shared runner
New (head only, not diffed): base_sha: |
…re-commit hook flagged benchmarks sits outside the root aggregate, so scalafmtAll never sees it: ClickRecordBench's later edits drifted. scalafixAll --check also ordered the new files' import blocks. Both surfaced by the pre-commit hook on this commit.
…not save-once setup-java's 'cache: sbt' preset only SAVES on a primary-key miss, so dependencies fetched after a key was minted are re-downloaded from Central on every later run — and one flaky fetch fails the job. That is the scalafix-cli_3.8.4:0.14.7 failure on this PR: the artifact was on Central, main's identical step resolved it green, but the PR runners had to re-fetch it and hit a blip. Twice. Replace it with actions/cache@v4 over ~/.cache/coursier, ~/.ivy2/cache and ~/.sbt, keyed on the build definition with a broad restore-keys prefix, in ci.yml (via the generator — githubWorkflowJobSetup now strips the setup-java cache param, injects the cache step after each setup-java leg, and rewires the 'sbt update' warmups to the cache step's cache-hit), plus the five hand-written workflows. actions/cache saves on every non-exact-key run, so late-fetched dependencies enter the cache chain for every subsequent run instead of fossilising with the key.
…he hook doesn't The pre-commit hook runs scalafmtCheckAll/benchmarks/scalafmtCheck but not scalafmtSbtCheck, so the unformatted build.sbt edit passed locally and failed in CI's 'Check formatting' step.
…ting' The hook checked .scala/.md formatting and benchmarks/, but not the build's *.sbt files — an unformatted build.sbt edit passed the hook and failed CI's 'Check formatting' step (caught on PR #116). CI runs scalafmtSbtCheck right after scalafmtCheckAll; the hook now does too.
kryptt
added a commit
that referenced
this pull request
Sep 23, 2026
Minor release on the 0.x line. Two big changes since 0.16.0: the avro derived whole-record builder (#95/#116) and the exception audit's refactors (circe's write walk, jsoniter's string-path constructors). Both jsoniter's `fromPath` forms and avro's `AvroVulcan.codec[A]` change a public return type, so the 0.17.0 section leads with the recompile warning the release-notes convention asks for. - build.sbt: tlBaseVersion 0.16 -> 0.17. The tag still derives the published version; this is what gates which tags the build accepts and what the snapshot version reads as. - CHANGELOG.md: `## [0.17.0] - 2026-09-23` cut out of Unreleased, link refs updated. Added the leading recompile warning and the `AvroVulcan.codec` migration bullet the section was missing -- that using-form's return type change is breaking, not only the gain the Added bullet describes. - mima.sbt: the breaking history's `next:` slot is now `0.17`, carrying the avro entry beside jsoniter's. The `Affine` covariance entry that sat in `next` actually shipped in 0.13.0 (CHANGELOG line 430), so it moves to a 0.13 line instead of being attributed to a future release. Verified before tagging: `sbt githubWorkflowGenerate` leaves ci.yml byte-identical, scalafmt check + scalafmtSbtCheck + benchmarks/scalafmtCheck + githubWorkflowCheck all pass, and root-aggregate `sbt test` is green on JDK 21 (kyo needs 25 and leaves the aggregate there, as documented; CI's 25 lane covers it).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ref #95 — the filer's report, addressed.
The problem
The #95 filer's production hot path builds a fresh
GenericData.Recordfrom their ~66-fieldClickInfo(leaves in 5 nested sub-records: Geo 8, UserAgentInfo 9, PostClick 11, Ivt 20, MavenEntities 10). Their report:vulcan.Codec[A].encode— vulcan's per-field composition per level, ~384–468 B/field;.put(pos, value)builder — ~0.5 µs / 1128 B/op, but one hand-maintained line per leaf (the complaint);subCodec.encode.What this adds
AvroVulcan.recordBuilder[A]: Exception | WholeRecordBuilder[A]— the hand-built shape, derived:A's case fields at expansion into a plain runtimeRecordShapeIR and emits ONE assembly call (no staged code), so the assembly is ordinary testable Scala;AvroWalk.recordSlots, same two rungs astotalNominalIndex, inverted verdict: total + injective or a named failure) — and validates every arm against the schema it writes into; construction is total, never throws;toRecordis pure positional puts: primitives direct, nested case classes RECURSE into sub-record levels (self-recursive case classes resolve through the runtime level chain, so they terminate),Noneputs null exactly as vulcan'sOptionCodec, and everything else (enums, bytes, logical types, collections, sums, value classes) goes through the field type's ownvulcan.Codec— summoned at the derivation site, so a missing leaf codec is a compile error naming the field;WholeRecordBuilder.asAvroCodecinstalls it as anAvroCodec.encodein one line, keepingcodecPrism[...].record.reverseGetand drilled.fieldreads on the fast path;codec.encode: schema-only (computed/derived) columns keep their in-record default — the hand-built.putcontract, round-trip-safe through the codec's own decode; a required one fails at write time, as the hand-built builder does.Measured (
benchmarksClickRecordBench, the filer's shape,-prof gc; B/op gates, ns/op advises)record.reverseGetThe allocation gate is hand-built-equal; the rejected approach sits 32x above it on the same shape.
Doctrine pass (review)
No naked throws on any construction path:
recordSlotsreturnsException | Array[Int];derive/buildLevel/planFor/unwrapNullable/climbthread failures up as returned Exceptions with per-refusal message factories;AvroVulcan.codecis two forms —codec(schema)(total; schema in hand) andcodec[A]: Either[Exception, AvroCodec[A]]— withschemaOfdeleted; the opt-in given remains the one documented eager-failure site (a summon must produce a codec).encodestill throws on codec errors perAvroCodec's total-encode trait contract.Tests & docs
WholeRecordBuilderSpec(14 examples): fidelity vs codec encode (deep, every nested level), byte-identical wire bytes, round-trips through the codec's decode (Some/None), the computed-column contract, reordered codec field lists + snake_case columns (name resolution),codecPrismwiring, the recursive-case knot, six construction refusals (rename, Option/non-nullable, case-class/non-record, LONG/INT, normalisation ambiguity, slot collision).WholeRecordBuilderMacroErrorSpec(2): a missing leaf codec names the field; a non-case-class root refuses at compile time.AvroVulcanSpecupdated for the two-form bridge (plus the explicit-schema form).avroIntegration/test: 235 passed / 0 failed. Root aggregate compiles. scalafmt applied to touched modules;.github/benchtooling unittests pass withClickRecordBenchregistered.site/docs/integrations/avro.mdvulcan section rewritten for the two-form bridge + the builder; CHANGELOG entry under Unreleased.