From a24e381a26628d521747a8c9504d4b15a6660761 Mon Sep 17 00:00:00 2001 From: Rodolfo Hansen Date: Fri, 18 Sep 2026 11:57:59 +0200 Subject: [PATCH 1/5] test(avro): RED witness for the >22-selector `.fields` NamedTuple ceiling (#96) `WideFieldsArityCeilingSpec` drives `codecPrism[A].fields(...)` with 22, 23 and 40 selectors and NO hand-written `AvroCodec[NamedTuple[...]]` given in scope, so every row goes through kindlings' NamedTuple derivation and hearth's `SyntheticNamedTupleConstructor`. On the current pins (hearth 0.4.0 via kindlings-avro-derivation 0.3.0) the 22 row compiles and passes; the 23 and 40 rows fail to compile with `wrong number of arguments at inlining ... expected: 0, found: 23` / `found: 40` -- hearth's `n < 23` branch calling the `*:` cons chain's zero-value-param primary constructor. That is the failure the hearth 0.4.2 / kindlings 0.3.2 bump is meant to remove. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0194EHFR4NamCpTHiqy7B74V --- .../eo/avro/WideFieldsArityCeilingSpec.scala | 645 ++++++++++++++++++ 1 file changed, 645 insertions(+) create mode 100644 avro/src/test/scala/dev/constructive/eo/avro/WideFieldsArityCeilingSpec.scala diff --git a/avro/src/test/scala/dev/constructive/eo/avro/WideFieldsArityCeilingSpec.scala b/avro/src/test/scala/dev/constructive/eo/avro/WideFieldsArityCeilingSpec.scala new file mode 100644 index 00000000..d7f2c6fc --- /dev/null +++ b/avro/src/test/scala/dev/constructive/eo/avro/WideFieldsArityCeilingSpec.scala @@ -0,0 +1,645 @@ +package dev.constructive.eo.avro + +import hearth.kindlings.avroderivation.{AvroDecoder, AvroEncoder, AvroSchemaFor} +import org.apache.avro.Schema +import org.apache.avro.generic.{GenericRecord, IndexedRecord} +import org.specs2.mutable.Specification + +/** Behaviour spec for `.fields` '''above''' the 22-selector NamedTuple ceiling (issue #96). + * + * `AvroPrism.fields(_.a, _.b, …)` synthesises a `NamedTuple[Names, Values]` and summons an + * `AvroCodec` for it at the call site. With no hand-written given in scope that summon + * auto-derives through kindlings, whose `AvroDecoderHandleAsNamedTupleRule` has to '''construct''' + * the tuple — and construction goes through hearth's `SyntheticNamedTupleConstructor`. + * + * Up to 22 selectors `Values` has a `TupleN` spelling, so hearth emits `new TupleN(args…)` (that + * is what PR #97 fixed, and it is a '''permanent''' requirement, not a workaround: hearth's + * `n < 23` branch still calls the underlying tuple's primary constructor, and a `*:` cons chain + * declares no value parameters). At 23 selectors and above there '''is''' no `TupleN` spelling — + * the focus type IS a `*:` cons chain — and only hearth ≥ 0.4.1 can build one, via + * `Tuple.fromArray` boxing each element to `Object` (hearth issue #314, shipped to us by + * kindlings-avro-derivation ≥ 0.3.2). + * + * '''This file deliberately declares no `AvroCodec[NamedTuple[…]]` given at all''' — the same rule + * as [[NamedTupleSpellingSpec]]. Adding one would short-circuit kindlings' NamedTuple rule and + * silently stop testing the thing this file exists for. + * + * Boundary trio: '''22''' (the old ceiling — must keep working through the `TupleN` branch), + * '''23''' (the first newly-unlocked arity, the first `Tuple.fromArray` case) and '''40''' (a + * wide, mixed-primitive cover — the `fromArray` branch boxes every element to `Object`, so a + * homogeneous `String` probe would not have exercised the unboxing on the way back out). + * + * There is no new '''arity''' ceiling above 23: a cons-chain NamedTuple was verified to derive and + * round-trip up to arity 512. What binds above ~200 is resource budget, not arity — the compiler + * thread's `-Xss` (this build sets `-Xss8m` in `.jvmopts`, good past arity 400) and kindlings' + * macro-expansion budget (`-Xmacro-settings:avroDerivation.timeout=30`, build.sbt). Neither is a + * language-level limit, so neither is pinned as a compile-time negative here; raise the budget if + * a genuinely enormous record ever times out. + */ +class WideFieldsArityCeilingSpec extends Specification: + + import WideFieldsArityCeilingSpec.* + + // ---- 22: the old ceiling, still on the TupleN branch -------------------- + + // covers: a 22-selector `.fields` cover still compiles with NO hand-written NamedTuple codec + // (hearth's `n < 23` `new TupleN(…)` branch — PR #97's spelling fix is what makes this work, + // and the bump must NOT regress it), + // every one of the 22 written slots lands under its own schema name, + // the three un-selected siblings survive reference-identical, + // the decoded focus is a `scala.Tuple22` at runtime (i.e. the constructor branch, not fromArray) + "`.fields` at arity 22: TupleN branch still writes all 22 slots and leaves siblings identical" >> { + val record = wide25Record(wide25A) + val L = codecPrism[Wide25] + .fields( + _.f01, + _.f02, + _.f03, + _.f04, + _.f05, + _.f06, + _.f07, + _.f08, + _.f09, + _.f10, + _.f11, + _.f12, + _.f13, + _.f14, + _.f15, + _.f16, + _.f17, + _.f18, + _.f19, + _.f20, + _.f21, + _.f22, + ) + .record + + val out = L.placeUnsafe( + ( + f01 = "b01", + f02 = "b02", + f03 = "b03", + f04 = "b04", + f05 = "b05", + f06 = "b06", + f07 = "b07", + f08 = "b08", + f09 = "b09", + f10 = "b10", + f11 = "b11", + f12 = "b12", + f13 = "b13", + f14 = "b14", + f15 = "b15", + f16 = "b16", + f17 = "b17", + f18 = "b18", + f19 = "b19", + f20 = "b20", + f21 = "b21", + f22 = "b22", + ) + )(record) + + val written = Seq( + "f01" -> "b01", + "f02" -> "b02", + "f03" -> "b03", + "f04" -> "b04", + "f05" -> "b05", + "f06" -> "b06", + "f07" -> "b07", + "f08" -> "b08", + "f09" -> "b09", + "f10" -> "b10", + "f11" -> "b11", + "f12" -> "b12", + "f13" -> "b13", + "f14" -> "b14", + "f15" -> "b15", + "f16" -> "b16", + "f17" -> "b17", + "f18" -> "b18", + "f19" -> "b19", + "f20" -> "b20", + "f21" -> "b21", + "f22" -> "b22", + ) + val untouched = Seq("f23", "f24", "f25") + + val decoded = L.getOptionUnsafe(out) + val read = decoded match + case Some(nt) => Some((nt.f01, nt.f22)) + case None => None + + (mismatches(wide25Schema, out, written) === Nil) + .and( + mismatches( + wide25Schema, + out, + Seq( + "f23" -> "a23", + "f24" -> "a24", + "f25" -> "a25", + ) + ) === Nil + ) + .and(untouched.forall(n => sameRef(wide25Schema, out, record, n)) === true) + .and(focusClassName(decoded) === "scala.Tuple22") + .and(read === Some(("b01", "b22"))) + } + + // ---- 23: the first newly-unlocked arity -------------------------------- + + // covers: a 23-selector `.fields` cover compiles with NO hand-written NamedTuple codec — the + // arity at which `Values` has no `TupleN` spelling left and hearth must take the + // `Tuple.fromArray` branch (hearth #314, kindlings ≥ 0.3.2). On hearth 0.4.0 this line is the + // RED: `wrong number of arguments at inlining (while expanding macro) … expected: 0, found: 23` + // — the `*:` cons chain's primary constructor takes no value parameters. (Inside + // `typeCheckErrors` the same defect surfaces as `too many arguments for constructor *:`, which + // is what `NamedTupleSpellingSpec`'s inverted ceiling row asserts.) + // every one of the 23 written slots lands under its own schema name, + // the two un-selected siblings survive reference-identical, + // the decoded focus is a `scala.runtime.TupleXXL` — the fromArray branch actually executing + "`.fields` at arity 23: the fromArray branch writes all 23 slots and leaves siblings identical" >> { + val record = wide25Record(wide25A) + val L = codecPrism[Wide25] + .fields( + _.f01, + _.f02, + _.f03, + _.f04, + _.f05, + _.f06, + _.f07, + _.f08, + _.f09, + _.f10, + _.f11, + _.f12, + _.f13, + _.f14, + _.f15, + _.f16, + _.f17, + _.f18, + _.f19, + _.f20, + _.f21, + _.f22, + _.f23, + ) + .record + + val out = L.placeUnsafe( + ( + f01 = "b01", + f02 = "b02", + f03 = "b03", + f04 = "b04", + f05 = "b05", + f06 = "b06", + f07 = "b07", + f08 = "b08", + f09 = "b09", + f10 = "b10", + f11 = "b11", + f12 = "b12", + f13 = "b13", + f14 = "b14", + f15 = "b15", + f16 = "b16", + f17 = "b17", + f18 = "b18", + f19 = "b19", + f20 = "b20", + f21 = "b21", + f22 = "b22", + f23 = "b23", + ) + )(record) + + val written = Seq( + "f01" -> "b01", + "f02" -> "b02", + "f03" -> "b03", + "f04" -> "b04", + "f05" -> "b05", + "f06" -> "b06", + "f07" -> "b07", + "f08" -> "b08", + "f09" -> "b09", + "f10" -> "b10", + "f11" -> "b11", + "f12" -> "b12", + "f13" -> "b13", + "f14" -> "b14", + "f15" -> "b15", + "f16" -> "b16", + "f17" -> "b17", + "f18" -> "b18", + "f19" -> "b19", + "f20" -> "b20", + "f21" -> "b21", + "f22" -> "b22", + "f23" -> "b23", + ) + val untouched = Seq("f24", "f25") + + val decoded = L.getOptionUnsafe(out) + val read = decoded match + case Some(nt) => Some((nt.f01, nt.f23)) + case None => None + + (mismatches(wide25Schema, out, written) === Nil) + .and( + mismatches( + wide25Schema, + out, + Seq( + "f24" -> "a24", + "f25" -> "a25", + ) + ) === Nil + ) + .and(untouched.forall(n => sameRef(wide25Schema, out, record, n)) === true) + .and(focusClassName(decoded) === "scala.runtime.TupleXXL") + .and(read === Some(("b01", "b23"))) + } + + // ---- 40, mixed primitives: the boxing path under load ------------------ + + // covers: a 40-selector cover over INTERLEAVED String/Int/Long/Double/Boolean fields — the + // `Tuple.fromArray` branch boxes every element to `Object`, so a homogeneous probe would not + // catch a mis-unboxing on the read side, + // every one of the 40 written slots lands under its own schema name with its own Avro type, + // the two un-selected siblings survive reference-identical, + // the decoded focus round-trips one field of each primitive kind, first and near-last + "`.fields` at arity 40 over mixed primitives: every slot lands, boxing round-trips" >> { + val record = mixed42Record(mixed42A) + val L = codecPrism[Mixed42] + .fields( + _.f01, + _.f02, + _.f03, + _.f04, + _.f05, + _.f06, + _.f07, + _.f08, + _.f09, + _.f10, + _.f11, + _.f12, + _.f13, + _.f14, + _.f15, + _.f16, + _.f17, + _.f18, + _.f19, + _.f20, + _.f21, + _.f22, + _.f23, + _.f24, + _.f25, + _.f26, + _.f27, + _.f28, + _.f29, + _.f30, + _.f31, + _.f32, + _.f33, + _.f34, + _.f35, + _.f36, + _.f37, + _.f38, + _.f39, + _.f40, + ) + .record + + val out = L.placeUnsafe( + ( + f01 = "b01", + f02 = 9002, + f03 = 7000000003L, + f04 = 4.25, + f05 = true, + f06 = "b06", + f07 = 9007, + f08 = 7000000008L, + f09 = 9.25, + f10 = false, + f11 = "b11", + f12 = 9012, + f13 = 7000000013L, + f14 = 14.25, + f15 = true, + f16 = "b16", + f17 = 9017, + f18 = 7000000018L, + f19 = 19.25, + f20 = false, + f21 = "b21", + f22 = 9022, + f23 = 7000000023L, + f24 = 24.25, + f25 = true, + f26 = "b26", + f27 = 9027, + f28 = 7000000028L, + f29 = 29.25, + f30 = false, + f31 = "b31", + f32 = 9032, + f33 = 7000000033L, + f34 = 34.25, + f35 = true, + f36 = "b36", + f37 = 9037, + f38 = 7000000038L, + f39 = 39.25, + f40 = false, + ) + )(record) + + val written = Seq( + "f01" -> "b01", + "f02" -> "9002", + "f03" -> "7000000003", + "f04" -> "4.25", + "f05" -> "true", + "f06" -> "b06", + "f07" -> "9007", + "f08" -> "7000000008", + "f09" -> "9.25", + "f10" -> "false", + "f11" -> "b11", + "f12" -> "9012", + "f13" -> "7000000013", + "f14" -> "14.25", + "f15" -> "true", + "f16" -> "b16", + "f17" -> "9017", + "f18" -> "7000000018", + "f19" -> "19.25", + "f20" -> "false", + "f21" -> "b21", + "f22" -> "9022", + "f23" -> "7000000023", + "f24" -> "24.25", + "f25" -> "true", + "f26" -> "b26", + "f27" -> "9027", + "f28" -> "7000000028", + "f29" -> "29.25", + "f30" -> "false", + "f31" -> "b31", + "f32" -> "9032", + "f33" -> "7000000033", + "f34" -> "34.25", + "f35" -> "true", + "f36" -> "b36", + "f37" -> "9037", + "f38" -> "7000000038", + "f39" -> "39.25", + "f40" -> "false", + ) + val untouched = Seq("f41", "f42") + + val decoded = L.getOptionUnsafe(out) + val read = decoded match + case Some(nt) => Some((nt.f01, nt.f02, nt.f03, nt.f04, nt.f05, nt.f39)) + case None => None + + (mismatches(mixed42Schema, out, written) === Nil) + .and( + mismatches( + mixed42Schema, + out, + Seq( + "f41" -> "a41", + "f42" -> "a42", + ) + ) === Nil + ) + .and(untouched.forall(n => sameRef(mixed42Schema, out, record, n)) === true) + .and(focusClassName(decoded) === "scala.runtime.TupleXXL") + .and(read === Some(("b01", 9002, 7000000003L, 4.25, true, 39.25))) + } + + // ---- helpers ----------------------------------------------------------- + + /** Every `(schemaName, renderedValue)` pair the record does NOT carry. `Nil` on success, and a + * readable per-field diff on failure — a positional shuffle shows up as the whole list. + */ + private def mismatches( + schema: Schema, + out: IndexedRecord, + expected: Seq[(String, String)], + ): Seq[(String, String, String)] = + expected.flatMap: (name, want) => + val got = String.valueOf(out.get(schema.getField(name).pos)) + if got == want then Nil else Seq((name, want, got)) + + /** Un-selected siblings must come through the positional overlay untouched — not merely equal. */ + private def sameRef( + schema: Schema, + out: IndexedRecord, + in: IndexedRecord, + name: String + ): Boolean = + val pos = schema.getField(name).pos + out.get(pos).asInstanceOf[AnyRef] eq in.get(pos).asInstanceOf[AnyRef] + + /** The runtime class of a decoded focus. A `NamedTuple` is opaque-erased to its `Values` tuple, + * so this names the hearth branch that built it: `scala.TupleN` (constructor) below 23, + * `scala.runtime.TupleXXL` (`Tuple.fromArray`) at 23 and above. + */ + private def focusClassName(focus: Option[Any]): String = + focus.fold("")(_.asInstanceOf[AnyRef].getClass.getName) + +end WideFieldsArityCeilingSpec + +object WideFieldsArityCeilingSpec: + + /** 25 `String` fields: 22 or 23 to select, the rest to watch survive reference-identical. */ + case class Wide25( + f01: String, + f02: String, + f03: String, + f04: String, + f05: String, + f06: String, + f07: String, + f08: String, + f09: String, + f10: String, + f11: String, + f12: String, + f13: String, + f14: String, + f15: String, + f16: String, + f17: String, + f18: String, + f19: String, + f20: String, + f21: String, + f22: String, + f23: String, + f24: String, + f25: String, + ) + + object Wide25: + + given AvroEncoder[Wide25] = AvroEncoder.derived + given AvroDecoder[Wide25] = AvroDecoder.derived + given AvroSchemaFor[Wide25] = AvroSchemaFor.derived + + /** 42 fields of interleaved primitives: 40 to select, 2 to watch survive. */ + case class Mixed42( + f01: String, + f02: Int, + f03: Long, + f04: Double, + f05: Boolean, + f06: String, + f07: Int, + f08: Long, + f09: Double, + f10: Boolean, + f11: String, + f12: Int, + f13: Long, + f14: Double, + f15: Boolean, + f16: String, + f17: Int, + f18: Long, + f19: Double, + f20: Boolean, + f21: String, + f22: Int, + f23: Long, + f24: Double, + f25: Boolean, + f26: String, + f27: Int, + f28: Long, + f29: Double, + f30: Boolean, + f31: String, + f32: Int, + f33: Long, + f34: Double, + f35: Boolean, + f36: String, + f37: Int, + f38: Long, + f39: Double, + f40: Boolean, + f41: String, + f42: String, + ) + + object Mixed42: + + given AvroEncoder[Mixed42] = AvroEncoder.derived + given AvroDecoder[Mixed42] = AvroDecoder.derived + given AvroSchemaFor[Mixed42] = AvroSchemaFor.derived + + val wide25A: Wide25 = Wide25( + "a01", + "a02", + "a03", + "a04", + "a05", + "a06", + "a07", + "a08", + "a09", + "a10", + "a11", + "a12", + "a13", + "a14", + "a15", + "a16", + "a17", + "a18", + "a19", + "a20", + "a21", + "a22", + "a23", + "a24", + "a25", + ) + + val mixed42A: Mixed42 = Mixed42( + "a01", + 1002, + 5000000003L, + 4.5, + false, + "a06", + 1007, + 5000000008L, + 9.5, + true, + "a11", + 1012, + 5000000013L, + 14.5, + false, + "a16", + 1017, + 5000000018L, + 19.5, + true, + "a21", + 1022, + 5000000023L, + 24.5, + false, + "a26", + 1027, + 5000000028L, + 29.5, + true, + "a31", + 1032, + 5000000033L, + 34.5, + false, + "a36", + 1037, + 5000000038L, + 39.5, + true, + "a41", + "a42", + ) + + lazy val wide25Schema: Schema = summon[AvroCodec[Wide25]].schema + lazy val mixed42Schema: Schema = summon[AvroCodec[Mixed42]].schema + + def wide25Record(w: Wide25): GenericRecord = + summon[AvroCodec[Wide25]].encode(w).asInstanceOf[GenericRecord] + + def mixed42Record(m: Mixed42): GenericRecord = + summon[AvroCodec[Mixed42]].encode(m).asInstanceOf[GenericRecord] + + // Deliberately NO `given AvroCodec[NamedTuple[…]]` anywhere in this file — see the class + // scaladoc. + +end WideFieldsArityCeilingSpec From 95cb7b8738dca5d4e51c10a75e5d49958c4c171d Mon Sep 17 00:00:00 2001 From: Rodolfo Hansen Date: Fri, 18 Sep 2026 12:13:44 +0200 Subject: [PATCH 2/5] =?UTF-8?q?build(deps):=20hearth=200.4.2=20/=20kindlin?= =?UTF-8?q?gs=200.3.2=20=E2=80=94=20break=20the=2023-element=20tuple=20cei?= =?UTF-8?q?ling?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `.fields(_.x, _.y, ...)` synthesises a `NamedTuple` whose value tuple has no `TupleN` spelling past arity 22 — it IS a `*:` cons chain. hearth 0.4.0's `SyntheticNamedTupleConstructor.unsafeApply` built every NamedTuple by calling the underlying tuple type's primary constructor, and `*:` declares no value parameters, so every 23+-selector `.fields` died with `wrong number of arguments at inlining … expected: 0, found: N`. hearth 0.4.1 added an arity dispatch (issues #313/#314): `case 0 => EmptyTuple`, `case n if n < 23 => new TupleN(...)`, `case _ => Tuple.fromArray(Array[Object](...))`. 0.4.2 is byte-identical to 0.4.1 for that file (0.4.2 carries a macro-loader metaspace-leak fix and `Expr.semiEval` on `ValueOf`). kindlings 0.3.2 contributes nothing to the arity fix itself — its NamedTuple decoder rule changed only in formatting — it is the release that pins hearth 0.4.2. The `n < 23` boundary lines up exactly with `MacroSelectors.tupleTypeOf`, which spells `TupleN` up to 22 and folds a cons chain above. Both branches stay load-bearing: a cons chain at arity 5 still fails on 0.4.2 (measured: 17 errors, `too many arguments for constructor *:`) while the TupleN spelling at arity 5 compiles clean. Transitive pin decisions, each justified rather than defaulted: - apache-avro 1.12.1 -> 1.12.2. kindlings-avro-derivation 0.3.2 depends on 1.12.2, so leaving the explicit pin at 1.12.1 would have turned a visibility pin into a silent DOWNGRADE of the transitive. The pin moves with it. - jackson-core/-databind 2.21.5 -> 2.22.1. avro 1.12.2's parent POM raises jackson-bom 2.20.0 -> 2.22.1. 2.22.1 is exactly the release that re-fixed CVE-2026-54515 (which 2.22.0 regressed and the old comment was waiting for); it is now on Central. Holding 2.21.5 would have downgraded core/databind while jackson-annotations — not in the override list — moved to 2.22, splitting the BOM. The pin lifts to the version avro resolves, so the override is again a regression floor, not a shift, and never 2.22.0. - commons-lang3 3.18.0 -> 3.20.0, same reasoning: the floor tracks what avro 1.12.2 resolves instead of downgrading it. Still above the CVE-2025-48924 range. commons-compress is unchanged at 1.28.0. `.scala-steward.conf`'s jackson series pin moves 2.21.x -> 2.22.x to match, and every stale comment in the pin block is rewritten (it claimed hearth 0.4.0). The `-Xmacro-settings:{circe,cats,avro}Derivation.timeout=30` line needs no change: `DerivationTimeout` is byte-identical in 0.3.2 and the namespaces are unchanged. kindlings 0.3.1's new derivation-policy gate shares that namespace but defaults to `always-allowed`, i.e. current behaviour, and eo sets no policy key. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0194EHFR4NamCpTHiqy7B74V --- .scala-steward.conf | 11 +++---- build.sbt | 72 +++++++++++++++++++++++++++++---------------- 2 files changed, 52 insertions(+), 31 deletions(-) diff --git a/.scala-steward.conf b/.scala-steward.conf index 09201143..713c7aed 100644 --- a/.scala-steward.conf +++ b/.scala-steward.conf @@ -1,8 +1,9 @@ # jackson 2.22.0 regressed the @JsonIgnoreProperties case-insensitive fix -# (CVE-2026-54515, dependabot alert #7). 2.21.5 is patched; the re-fix lands -# in 2.22.1 (unreleased). Keep both artifacts on the 2.21.x series and lift -# this pin together with the build.sbt comment once 2.22.1 is on Central. +# (CVE-2026-54515, dependabot alert #7). 2.21.5 is patched and 2.22.1 carries +# the re-fix, so the floor moved to 2.22.1 when apache-avro 1.12.2 raised +# jackson-bom to it (see the build.sbt comment on `jacksonCore`). Keep both +# artifacts at or above 2.22.1 and never on 2.22.0. updates.pin = [ - { groupId = "com.fasterxml.jackson.core", artifactId = "jackson-core", version = "2.21." }, - { groupId = "com.fasterxml.jackson.core", artifactId = "jackson-databind", version = "2.21." } + { groupId = "com.fasterxml.jackson.core", artifactId = "jackson-core", version = "2.22." }, + { groupId = "com.fasterxml.jackson.core", artifactId = "jackson-databind", version = "2.22." } ] diff --git a/build.sbt b/build.sbt index 2085cd49..23bfedbc 100644 --- a/build.sbt +++ b/build.sbt @@ -312,53 +312,73 @@ lazy val monocle = Optics %% "monocle-core" % "3.3.0" // functor + Fix encoding). Benchmark-only; never a published dependency. lazy val drosteCore = "io.higherkindness" %% "droste-core" % "0.9.0-M3" // kindlings 0.3.x (all three) ship a configurable macro-expansion timeout -// (`DerivationTimeout`, default 5s) and pull hearth 0.4.0 + kindlings-derivation-commons. +// (`DerivationTimeout`, default 5s) and pull hearth 0.4.2 + kindlings-derivation-commons. // We raise it to 30s via `-Xmacro-settings:{circe,cats,avro}Derivation.timeout=30` // (see the `ThisBuild / scalacOptions` above) so a loaded CI runner stops tripping the old // hardcoded 2s budget (the recurring `deriveAsObject timed out after 2000ms` flake). +// The setting keys and namespaces are unchanged at 0.3.2 (`DerivationTimeout` is +// byte-identical to 0.3.0); 0.3.1 added an OPT-IN `.policy.enabled` key under the same +// namespace whose default (`always-allowed`) is exactly the 0.3.0 behaviour, so we set none. // NB kindlings fully-qualifies derived Avro record names (namespace = enclosing path) — -// established in 0.2.0, unchanged in 0.3.x. -lazy val hearth = Kubuszok %% "hearth" % "0.4.0" -lazy val kindlingsCats = Kubuszok %% "kindlings-cats-derivation" % "0.3.0" -lazy val kindlingsCirce = Kubuszok %% "kindlings-circe-derivation" % "0.3.0" -lazy val kindlingsAvro = Kubuszok %% "kindlings-avro-derivation" % "0.3.0" +// established in 0.2.0, unchanged in 0.3.x (re-verified against 0.3.2: derived schema text +// for a NamedTuple focus is byte-identical to 0.3.0 on the <= 22 path). +// +// hearth 0.4.2 is what breaks the 23-element tuple ceiling for `.fields(...)`: +// `SyntheticNamedTupleConstructor.unsafeApply` gained an arity dispatch (hearth #313/#314, +// landed in 0.4.1) — `case 0 => EmptyTuple`, `case n if n < 23 => new TupleN(...)`, +// `case _ => Tuple.fromArray(Array[Object](...))`. 0.4.0 unconditionally called the +// underlying tuple type's primary constructor, which for the `*:` cons chain a >22-ary +// NamedTuple must be spelled as takes ZERO value params ("expected: 0, found: 23"). +// The `n < 23` boundary is why `MacroSelectors.tupleTypeOf` must KEEP spelling TupleN at +// arity <= 22 and a cons chain only above it: a cons chain still fails below 23 on 0.4.2. +lazy val hearth = Kubuszok %% "hearth" % "0.4.2" +lazy val kindlingsCats = Kubuszok %% "kindlings-cats-derivation" % "0.3.2" +lazy val kindlingsCirce = Kubuszok %% "kindlings-circe-derivation" % "0.3.2" +lazy val kindlingsAvro = Kubuszok %% "kindlings-avro-derivation" % "0.3.2" lazy val circe = Circe %% "circe-core" % "0.14.16" -// vulcan pins apache-avro 1.11.x transitively; our explicit avro 1.12.1 pin +// vulcan pins apache-avro 1.11.x transitively; our explicit avro 1.12.2 pin // below wins on the compile classpath, and as an Optional dep vulcan forces // nothing downstream anyway. lazy val vulcan = "com.github.fd4s" %% "vulcan" % "1.13.0" lazy val circeParser = Circe %% "circe-parser" % "0.14.16" -// Pin apache-avro 1.12.1 explicitly even though kindlings-avro-derivation +// Pin apache-avro 1.12.2 explicitly even though kindlings-avro-derivation // brings it transitively — keeps the reachable runtime jar visible in // dependency reports. cats-eo-avro touches `IndexedRecord` / // `GenericData` / `Schema` directly on the hot path. -lazy val avro = ApacheAvro % "avro" % "1.12.1" -// Force jackson to 2.21.5 — `apache-avro 1.12.1` brings `jackson-databind -// 2.20.0` (and `jackson-core`) transitively, inside the CVE-affected -// `>= 2.19.0, < 2.21.5` range (four GHSA dependabot alerts: two +// The version tracks what kindlings-avro-derivation 0.3.2 depends on (1.12.2; +// 0.3.0 depended on 1.12.1). Keeping the old 1.12.1 here would have turned a +// visibility pin into a silent DOWNGRADE of the transitive, so it moves with it. +lazy val avro = ApacheAvro % "avro" % "1.12.2" +// Force jackson to 2.22.1 — `apache-avro` brings `jackson-databind` (and +// `jackson-core`) transitively, and 1.12.1 brought 2.20.0, inside the +// CVE-affected `>= 2.19.0, < 2.21.5` range (four GHSA dependabot alerts: two // PolymorphicTypeValidator/allowlist bypasses, an InetSocketAddress SSRF, and // a @JsonIgnoreProperties case-insensitive bypass). 2.21.5 was the first -// release patched against all four. Do NOT bump to 2.22.0: it REGRESSED the -// @JsonIgnoreProperties case-insensitive fix (CVE-2026-54515, dependabot -// alert #7; re-fixed only in the unreleased 2.22.1) — the 0.6.1 bulk Steward -// upgrade briefly did, re-opening the alert. `.scala-steward.conf` pins the -// 2.21.x series; lift both pins together once 2.22.1 is on Central. +// release patched against all four; 2.22.0 REGRESSED the @JsonIgnoreProperties +// case-insensitive fix (CVE-2026-54515, dependabot alert #7) and 2.22.1 re-fixed +// it. 2.22.1 has since shipped to Central, and avro 1.12.2's parent POM raises +// `jackson-bom` 2.20.0 -> 2.22.1 — so the old 2.21.5 override would now DOWNGRADE +// jackson-core/-databind while leaving jackson-annotations (not overridden) at +// 2.22, splitting the BOM. The pin therefore lifts to 2.22.1, exactly the version +// avro 1.12.2 resolves: still a regression floor, no longer a shift, and never +// 2.22.0. `.scala-steward.conf` tracks the same series bound. // Overrides apply via // `commonSettings.dependencyOverrides` across every module so any future // jackson-pulling transitive (e.g. a kindlings bump) inherits the safe // versions automatically. eo never enables polymorphic/default typing, so the // PTV bypasses aren't reachable here — this just keeps the dep tree clean. -lazy val jacksonCore = FasterXmlJackson % "jackson-core" % "2.21.5" -lazy val jacksonDatabind = FasterXmlJackson % "jackson-databind" % "2.21.5" -// Floor commons-lang3 at 3.18.0 — `apache-avro 1.12.1 -> commons-compress -// 1.28.0` brings it transitively, and every release below 3.18.0 is in the +lazy val jacksonCore = FasterXmlJackson % "jackson-core" % "2.22.1" +lazy val jacksonDatabind = FasterXmlJackson % "jackson-databind" % "2.22.1" +// Floor commons-lang3 at 3.20.0 — `apache-avro -> commons-compress 1.28.0` +// brings it transitively, and every release below 3.18.0 is in the // CVE-2025-48924 range (uncontrolled recursion on long inputs; dependabot -// alert #1). commons-compress 1.28.0 already resolves 3.18.0, so this is a -// regression floor rather than a live bump — pinned via +// alert #1). avro 1.12.2's parent POM resolves 3.20.0 (1.12.1 resolved 3.18.0), +// so the floor moves up with it rather than downgrading the transitive — it stays +// a regression floor rather than a live bump. Pinned via // `commonSettings.dependencyOverrides` (same mechanism as jackson) so a future // avro/commons-compress shuffle can't reintroduce a vulnerable version, and the // submitted dependency graph shows the safe version unambiguously. -lazy val commonsLang3 = "org.apache.commons" % "commons-lang3" % "3.18.0" +lazy val commonsLang3 = "org.apache.commons" % "commons-lang3" % "3.20.0" // jsoniter-scala — high-perf JSON codec (~5–10× circe on hot paths). // Used by `eo-jsoniter` to back byte-cursor JSON optics that decode // directly from `Array[Byte]` without allocating a runtime AST. The @@ -429,8 +449,8 @@ lazy val commonSettings = Seq( // library's code and the warning is a Hearth-side concern rather // than a cats-eo bug. Test / scalacOptions += "-Wconf:src=.*/cats-derivation/.*:silent", - // Pin jackson-core + jackson-databind at the CVE-patched 2.21.5 and floor - // commons-lang3 at 3.18.0 across every module — see the `jacksonCore` / + // Pin jackson-core + jackson-databind at the CVE-patched 2.22.1 and floor + // commons-lang3 at 3.20.0 across every module — see the `jacksonCore` / // `jacksonDatabind` / `commonsLang3` defs above. dependencyOverrides ++= Seq(jacksonCore, jacksonDatabind, commonsLang3), ) From f9c25efd83fc51b7266a602058fa3b2224a939c0 Mon Sep 17 00:00:00 2001 From: Rodolfo Hansen Date: Fri, 18 Sep 2026 12:14:03 +0200 Subject: [PATCH 3/5] test(avro): flip the arity-23 ceiling row, and pin why the TupleN branch is permanent MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `NamedTupleSpellingSpec`'s last row asserted the FAILURE — `typeCheckErrors` over a 23-selector `.fields` had to contain `too many arguments for constructor *:`. Its own comment said the bump would break it and that the break was the signal, not a regression. It is now the positive assertion it asked for: the same given-free `Wide23` fixture, `.fields` over all 23 selectors, decoded and read back, with the focus's runtime class asserted to be `scala.runtime.TupleXXL` — the `Tuple.fromArray` branch actually executing, not merely compiling. A new row replaces it as the negative: a cons-spelled NamedTuple at arity 2 is STILL unconstructible on hearth 0.4.2, because the fix is scoped to arity >= 23 and the `n < 23` arm is unchanged. That makes `MacroSelectors.tupleTypeOf`'s split spelling a permanent requirement of the hearth contract rather than a workaround the bump retires, and a future "simplify the branch away" pass fails loudly instead of silently re-breaking arity 2..22. `FieldsMacroErrorSpec`'s expected macro hint moves with the version string in `JsonPrismMacro` (next commit). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0194EHFR4NamCpTHiqy7B74V --- .../eo/avro/NamedTupleSpellingSpec.scala | 126 ++++++++++++++---- .../eo/circe/FieldsMacroErrorSpec.scala | 2 +- 2 files changed, 102 insertions(+), 26 deletions(-) diff --git a/avro/src/test/scala/dev/constructive/eo/avro/NamedTupleSpellingSpec.scala b/avro/src/test/scala/dev/constructive/eo/avro/NamedTupleSpellingSpec.scala index cf880a86..8b4d76d9 100644 --- a/avro/src/test/scala/dev/constructive/eo/avro/NamedTupleSpellingSpec.scala +++ b/avro/src/test/scala/dev/constructive/eo/avro/NamedTupleSpellingSpec.scala @@ -19,12 +19,14 @@ import org.specs2.mutable.Specification * - `String *: Int *: EmptyTuple` (a `*:` cons chain), and * - `(String, Int)` (i.e. `scala.Tuple2[String, Int]`) * - * — but only the second is constructible by hearth's `SyntheticNamedTupleConstructor`, which for - * arity < 23 emits `new (args…)`. For the cons spelling that is + * — but only the second is constructible by hearth's `SyntheticNamedTupleConstructor` below arity + * 23, where it emits `new (args…)`. For the cons spelling that is * `new *:(a, b)`, and `*:` declares no value parameters, so any third-party derivation that * '''builds''' the NamedTuple (kindlings' `AvroDecoderHandleAsNamedTupleRule`) blows up with * `wrong number of arguments at inlining … expected: 0, found: N` plus a secondary - * `a reference to method decode_…$macro$N was used outside the scope where it was defined`. + * `a reference to method decode_…$macro$N was used outside the scope where it was defined`. That + * `n < 23` branch is unchanged in hearth 0.4.2 — only arity '''>= 23''' gained a `Tuple.fromArray` + * path — so the TupleN spelling stays mandatory at 2..22, permanently. * * Every other in-repo `.fields` spec hand-declares its own `AvroCodec[NamedTuple[…]]` given * (spelled `TupleN`, because that is what a human writes), which short-circuits kindlings' NT rule @@ -136,31 +138,72 @@ class NamedTupleSpellingSpec extends Specification: org.specs2.execute.Failure(s"complement round-trip failed: $t"): org.specs2.execute.Result } - // ---- the known arity ceiling (pins CURRENT behaviour) ----------------- + // ---- above the old arity ceiling (was a pinned failure) --------------- // covers: at 23+ selectors there IS no `TupleN` spelling — `(A, …, A)` with 23 components IS - // the `*:` cons chain — so the fix above cannot reach that far, and hearth 0.4.0 (shipped by - // kindlings-avro-derivation 0.3.0, this project's pin) still fails to construct it. - // - // This row PINS the ceiling rather than leaving it a silent gap. hearth 0.4.2 / kindlings - // 0.3.2 route every arity through `Tuple.fromArray` and would make it compile; when this - // project bumps kindlings, THIS TEST WILL FAIL — that failure is the signal to delete the - // row and the ceiling note in the `.fields` scaladoc, not a regression. - "`.fields` at arity 23: still unsupported on kindlings 0.3.0 / hearth 0.4.0 (documented ceiling)" >> { - val wide = typeCheckErrors(""" - import dev.constructive.eo.avro.codecPrism - import dev.constructive.eo.avro.NamedTupleSpellingSpec.Wide23 - codecPrism[Wide23].fields( - _.f01, _.f02, _.f03, _.f04, _.f05, _.f06, _.f07, _.f08, - _.f09, _.f10, _.f11, _.f12, _.f13, _.f14, _.f15, _.f16, - _.f17, _.f18, _.f19, _.f20, _.f21, _.f22, _.f23, - ) + // the `*:` cons chain — so the TupleN spelling above cannot reach that far. This used to be + // a pinned FAILURE (hearth 0.4.0 emitted `new *:(args…)` and `*:` takes no value params). + // hearth 0.4.2's `SyntheticNamedTupleConstructor` routes arity >= 23 through + // `Tuple.fromArray` instead, so the row is now the positive assertion its own note asked for. + // Still given-free, like every other row here — this is the auto-derivation path. + // The wide end of the surface (mixed field types, un-selected siblings, reference identity) + // is covered separately by `WideFieldsArityCeilingSpec`. + "`.fields` at arity 23: compiles and round-trips on hearth 0.4.2 (ceiling broken)" >> { + val record = wide23Record + val L = codecPrism[Wide23] + .fields( + _.f01, + _.f02, + _.f03, + _.f04, + _.f05, + _.f06, + _.f07, + _.f08, + _.f09, + _.f10, + _.f11, + _.f12, + _.f13, + _.f14, + _.f15, + _.f16, + _.f17, + _.f18, + _.f19, + _.f20, + _.f21, + _.f22, + _.f23, + ) + .record + + L.getOptionUnsafe(record) match + case Some(nt) => + // `Tuple.fromArray` builds a TupleXXL, NOT a TupleN — the branch that fixed this. + (nt.asInstanceOf[AnyRef].getClass.getName === "scala.runtime.TupleXXL") + .and(nt.f01 === "a01") + .and(nt.f23 === "a23") + case None => + org.specs2.execute.Failure("expected Some(namedTuple)"): org.specs2.execute.Result + } + + // covers: hearth 0.4.2's fix is scoped to arity >= 23 ONLY — its `n < 23` branch still emits + // `new (args…)`, which for a cons chain is the value-parameterless + // `new *:(a, b)`. So `MacroSelectors.tupleTypeOf`'s "TupleN at <= 22, cons fold above" split is + // a PERMANENT requirement of the hearth contract, not a workaround the bump makes redundant: + // collapsing it to a uniform cons fold would re-break every `.fields` call of arity 2..22. + // This row is the executable proof, so a future simplification pass fails loudly. + "a cons-spelled NamedTuple below arity 23 is still unconstructible (the TupleN branch is permanent)" >> { + val errs = typeCheckErrors(""" + import hearth.kindlings.avroderivation.AvroDecoder + type ConsNT = + NamedTuple.NamedTuple["a" *: "b" *: EmptyTuple, String *: String *: EmptyTuple] + val d: AvroDecoder[ConsNT] = AvroDecoder.derived """) - (wide.nonEmpty === true) - .and( - wide.exists(_.message.contains("too many arguments for constructor *:")) === true - ) + (errs.nonEmpty === true) + .and(errs.exists(_.message.contains("*:")) === true) } // ---- helpers ---------------------------------------------------------- @@ -193,7 +236,9 @@ object NamedTupleSpellingSpec: def skuRecord(s: Sku): GenericRecord = summon[AvroCodec[Sku]].encode(s).asInstanceOf[GenericRecord] - /** 23 fields — one past the last arity that has a `TupleN` spelling. */ + /** 23 fields — one past the last arity that has a `TupleN` spelling, so `.fields` over all of + * them is spelled as a `*:` cons chain and built by hearth's `Tuple.fromArray` branch. + */ case class Wide23( f01: String, f02: String, @@ -226,6 +271,37 @@ object NamedTupleSpellingSpec: given AvroDecoder[Wide23] = AvroDecoder.derived given AvroSchemaFor[Wide23] = AvroSchemaFor.derived + lazy val wide23Record: GenericRecord = + summon[AvroCodec[Wide23]] + .encode( + Wide23( + "a01", + "a02", + "a03", + "a04", + "a05", + "a06", + "a07", + "a08", + "a09", + "a10", + "a11", + "a12", + "a13", + "a14", + "a15", + "a16", + "a17", + "a18", + "a19", + "a20", + "a21", + "a22", + "a23", + ) + ) + .asInstanceOf[GenericRecord] + // Deliberately NO `given AvroCodec[NamedTuple[…]]` anywhere in this file — see the class // scaladoc. Adding one would silently restore the pre-fix pass. diff --git a/circe/src/test/scala/dev/constructive/eo/circe/FieldsMacroErrorSpec.scala b/circe/src/test/scala/dev/constructive/eo/circe/FieldsMacroErrorSpec.scala index f7a6ee4c..a0677c8d 100644 --- a/circe/src/test/scala/dev/constructive/eo/circe/FieldsMacroErrorSpec.scala +++ b/circe/src/test/scala/dev/constructive/eo/circe/FieldsMacroErrorSpec.scala @@ -111,7 +111,7 @@ class FieldsMacroErrorSpec extends Specification: e.message .contains( "(`import hearth.kindlings.circederivation.KindlingsCodecAsObject`," - + " dependency `\"com.kubuszok\" %% \"kindlings-circe-derivation\" % \"0.3.0\"`)," + + " dependency `\"com.kubuszok\" %% \"kindlings-circe-derivation\" % \"0.3.2\"`)," ) ) From b0748dcc147f198e2e612cbe87d2d61db2fd5316 Mon Sep 17 00:00:00 2001 From: Rodolfo Hansen Date: Fri, 18 Sep 2026 12:14:03 +0200 Subject: [PATCH 4/5] =?UTF-8?q?docs:=20the=20`.fields`=20arity=20ceiling?= =?UTF-8?q?=20is=20gone=20=E2=80=94=20update=20every=20place=20that=20stat?= =?UTF-8?q?ed=20it?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR #96's reply and #97's body documented a "2 to 22 selectors" ceiling to users. Every statement of it is now wrong, so each one moves: - `AvroPrism.fields` scaladoc: no arity ceiling; what a wide selection costs is compile time (superlinear derivation), which is an `-Xss` / `-Xmacro-settings:avroDerivation.timeout` question, not a correctness one. - `MacroSelectors.tupleTypeOf` scaladoc: says explicitly that BOTH branches are load-bearing permanently, since hearth's sub-23 arm still cannot build a cons chain — the one thing a future simplification pass would get wrong. - `site/docs/integrations/avro.md`: the "**2 to 22**" paragraph, plus the stale "kindlings-avro-derivation 0.1.2 / apache-avro 1.12.1" backend note. - `AvroCodec` scaladoc, `site/docs/integrations/circe.md`, `JsonPrismMacro`'s missing-codec hint and `CLAUDE.md`: stale library versions (the hint's expected text is pinned by `FieldsMacroErrorSpec`, updated with it). - CHANGELOG: an Added entry for the broken ceiling and a Changed entry for the bump, including the full transitive shift and the explicit note that NO derived Avro schema text moved. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0194EHFR4NamCpTHiqy7B74V --- CHANGELOG.md | 27 +++++++++++++++++++ CLAUDE.md | 2 +- .../dev/constructive/eo/avro/AvroCodec.scala | 4 +-- .../dev/constructive/eo/avro/AvroPrism.scala | 13 +++++---- .../eo/circe/JsonPrismMacro.scala | 2 +- .../eo/generics/MacroSelectors.scala | 9 +++++-- site/docs/integrations/avro.md | 21 ++++++++++----- site/docs/integrations/circe.md | 2 +- 8 files changed, 61 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f9c234bc..a513a849 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -75,8 +75,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 costs one allocation per schema rather than one per lookup and repeated lookups still hand back the same instance. +### Added + +- **`.fields(...)` no longer stops at 22 selectors** (#96): the macro-synthesised + `NamedTuple` focus has no arity ceiling any more. Up to 22 selectors the value tuple is spelled + `scala.TupleN` and hearth builds it with that tuple's constructor; at 23 and above the value + tuple *is* a `*:` cons chain, and hearth 0.4.2 builds that through `Tuple.fromArray`. Before, + hearth 0.4.0 called the cons chain's primary constructor — which takes no value parameters — so + every 23+-selector `.fields` failed to compile with `wrong number of arguments at inlining … + expected: 0, found: N`. Verified end-to-end (derive, encode, decode, positional read-back) at + arity 23 and at arity 40 with mixed field types, since the `Tuple.fromArray` path boxes every + element to `Object`. Both spellings remain load-bearing: hearth's sub-23 branch still cannot + build a cons chain, so `MacroSelectors.tupleTypeOf` keeps emitting `TupleN` below 23 + permanently. What a very wide selection costs is compile time, not correctness — the derivation + grows superlinearly in arity, so past a few hundred fields raise `-Xss` and + `-Xmacro-settings:avroDerivation.timeout` before splitting the cover. + ### Changed +- **Dependency bump**: hearth `0.4.0` → `0.4.2` and kindlings-{avro,cats,circe}-derivation + `0.3.0` → `0.3.2` — the releases that carry the NamedTuple arity fix above (hearth #313/#314, + landed in 0.4.1). Transitively: apache-avro `1.12.1` → `1.12.2` (the explicit pin moves with it + rather than silently downgrading the transitive), jackson-core/-databind `2.21.5` → `2.22.1`, + jackson-annotations `2.21` → `2.22`, commons-lang3 `3.18.0` → `3.20.0`, slf4j-api `2.0.17` → + `2.0.18`. The jackson force-pin lifts to 2.22.1 — the release that re-fixed CVE-2026-54515, + which 2.22.0 had regressed, and the version avro 1.12.2's parent BOM resolves — so the override + stays a regression floor instead of becoming a downgrade that splits the jackson BOM. The + commons-lang3 floor lifts to 3.20.0 for the same reason. No derived Avro schema text changed: + record names, field names and field order are byte-identical across the bump (the 216-example + avro suite, including every naming and union spec, passes unmodified). - **Behaviour change, avro**: a hand-written codec that PERMUTES the Scala names (writes case field `a` into a schema field literally named `b`, and vice versa) resolved correctly by position and now resolves by name, i.e. wrongly. No name transform can produce that shape — a transform is a diff --git a/CLAUDE.md b/CLAUDE.md index e79d9b8b..13cdae5c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -234,7 +234,7 @@ advises**. Operator runbook: `.github/bench/README.md`. `generics/` is a separate sub-project that synthesises boilerplate optics at compile time, built on top of Mateusz Kubuszok's -[`com.kubuszok:hearth_3:0.3.0`](https://github.com/MateuszKubuszok/hearth) +[`com.kubuszok:hearth_3:0.4.2`](https://github.com/MateuszKubuszok/hearth) macro-commons library. Two entry points so far: ```scala diff --git a/avro/src/main/scala/dev/constructive/eo/avro/AvroCodec.scala b/avro/src/main/scala/dev/constructive/eo/avro/AvroCodec.scala index 3ce02d14..a59c4e46 100644 --- a/avro/src/main/scala/dev/constructive/eo/avro/AvroCodec.scala +++ b/avro/src/main/scala/dev/constructive/eo/avro/AvroCodec.scala @@ -17,8 +17,8 @@ import org.apache.avro.io.DecoderFactory * `get` decodes, its `reverseGet` / `place` encodes). Forcing every call site to thread two * `using` parameters is noisy. `AvroCodec[A]` is the project-internal shorthand. * - * Why not vulcan? Vulcan 1.13.x pins apache-avro 1.11.5; kindlings-avro-derivation 0.1.2 pins - * 1.12.1. cats-eo-avro chose kindlings + avro 1.12 because it lines up with the rest of the + * Why not vulcan? Vulcan 1.13.x pins apache-avro 1.11.5; kindlings-avro-derivation 0.3.2 pins + * 1.12.2. cats-eo-avro chose kindlings + avro 1.12 because it lines up with the rest of the * ecosystem moving forward, and the typeclass surface is simpler (no `Either[AvroError, A]` * threading on every call — kindlings' decoders throw on failure, which the prism layer wraps into * [[AvroFailure]]). diff --git a/avro/src/main/scala/dev/constructive/eo/avro/AvroPrism.scala b/avro/src/main/scala/dev/constructive/eo/avro/AvroPrism.scala index 8a3c6550..37fa13b9 100644 --- a/avro/src/main/scala/dev/constructive/eo/avro/AvroPrism.scala +++ b/avro/src/main/scala/dev/constructive/eo/avro/AvroPrism.scala @@ -413,11 +413,14 @@ object AvroPrism: /** `.fields(_.a, _.b, ...)` — focus a NamedTuple over selected fields. * * The `AvroCodec` for the synthesised NamedTuple is summoned at the call site; with no - * hand-written given in scope it auto-derives through kindlings. That works for '''2 to 22''' - * selectors. At 23 or more, Scala has no `TupleN` spelling left — the focus type IS a `*:` cons - * chain — and kindlings 0.3.0 / hearth 0.4.0 cannot construct one, so the call fails to compile - * with `too many arguments for constructor *:`. Chain two `.fields` covers, or drill with - * `.field`, until the dependency moves to hearth ≥ 0.4.2. (Issue #96.) + * hand-written given in scope it auto-derives through kindlings. There is no arity ceiling: up + * to 22 selectors the focus is spelled `TupleN` and hearth builds it with that tuple's + * constructor; at 23 and above the focus type IS a `*:` cons chain, which hearth ≥ 0.4.2 builds + * through `Tuple.fromArray` instead. (Both spellings are needed — hearth's constructor branch + * below 23 cannot build a cons chain, and there is no `TupleN` above 22.) Wide selections cost + * compile time rather than correctness: the derivation is quadratic-ish in arity, and a very + * wide record may want a larger `-Xss` or a higher `-Xmacro-settings:avroDerivation.timeout`. + * (Issue #96.) */ extension [A](o: AvroPrism[A]) diff --git a/circe/src/main/scala/dev/constructive/eo/circe/JsonPrismMacro.scala b/circe/src/main/scala/dev/constructive/eo/circe/JsonPrismMacro.scala index ec7918ac..eed7a0c2 100644 --- a/circe/src/main/scala/dev/constructive/eo/circe/JsonPrismMacro.scala +++ b/circe/src/main/scala/dev/constructive/eo/circe/JsonPrismMacro.scala @@ -173,7 +173,7 @@ object JsonPrismMacro: + s" Derive one via `given Codec.AsObject[${Type.show[nt]}] =" + " KindlingsCodecAsObject.derived`" + " (`import hearth.kindlings.circederivation.KindlingsCodecAsObject`," - + " dependency `\"com.kubuszok\" %% \"kindlings-circe-derivation\" % \"0.3.0\"`)," + + " dependency `\"com.kubuszok\" %% \"kindlings-circe-derivation\" % \"0.3.2\"`)," + " or provide one manually." ) val namesExpr: Expr[Array[String]] = diff --git a/generics/src/main/scala/dev/constructive/eo/generics/MacroSelectors.scala b/generics/src/main/scala/dev/constructive/eo/generics/MacroSelectors.scala index aff7bd35..98969744 100644 --- a/generics/src/main/scala/dev/constructive/eo/generics/MacroSelectors.scala +++ b/generics/src/main/scala/dev/constructive/eo/generics/MacroSelectors.scala @@ -158,8 +158,13 @@ object MacroSelectors: * costs nothing: user-written givens over the `TupleN` spelling still match either way. * * Above arity 22 there is no `TupleN` to reach for — `(A, …, A)` with 23 components already - * '''is''' the cons chain — so the fallback is forced, and the hazard survives for consumers on - * hearth < 0.4.2 (which routes that range through `Tuple.fromArray` instead). + * '''is''' the cons chain — so the fallback is forced. hearth ≥ 0.4.2 handles exactly that range + * through `Tuple.fromArray`; on hearth < 0.4.2 the hazard survives there for consumers. + * + * '''Both branches are load-bearing, permanently.''' hearth 0.4.2's fix is scoped to arity ≥ 23: + * its `case n if n < 23` arm still emits the primary-constructor call, so a cons chain below 23 + * fails exactly as it did on 0.4.0. Collapsing this to a uniform cons fold would therefore + * re-break every `.fields` call of arity 2..22. `NamedTupleSpellingSpec` pins both directions. */ def tupleTypeOf(using q: Quotes diff --git a/site/docs/integrations/avro.md b/site/docs/integrations/avro.md index a2fa6794..55f32d56 100644 --- a/site/docs/integrations/avro.md +++ b/site/docs/integrations/avro.md @@ -45,7 +45,7 @@ full table. The codec backend is [kindlings-avro-derivation](https://github.com/MateuszKubuszok/kindlings-avro-derivation) -0.1.2, which pins apache-avro 1.12.1. cats-eo-avro wraps the +0.3.2, which pins apache-avro 1.12.2. cats-eo-avro wraps the kindlings `AvroEncoder[A]` / `AvroDecoder[A]` / `AvroSchemaFor[A]` triplet in a single [`AvroCodec[A]`](https://github.com/Constructive-Programming/eo/blob/main/avro/src/main/scala/dev/constructive/eo/avro/AvroCodec.scala) typeclass so user code summons one thing per type. @@ -255,12 +255,19 @@ inside the selection under the Scala package namespace, which an enclosing union will refuse. Name the codec (or drill with `.field`) in that case. -The auto-derived route covers **2 to 22** selectors. At 23 or more -there is no `TupleN` spelling left — the NamedTuple's value tuple -*is* a `*:` cons chain — and the derivation backend this project -pins (kindlings 0.3.0 / hearth 0.4.0) cannot construct one, in -either the derived or the hand-written form. Split a wider -selection into several `.fields` covers. +The auto-derived route has **no arity ceiling**. Up to 22 selectors +the NamedTuple's value tuple is spelled `TupleN` and the derivation +backend builds it with that tuple's constructor; at 23 or more there +is no `TupleN` spelling left — the value tuple *is* a `*:` cons +chain — and kindlings ≥ 0.3.2 / hearth ≥ 0.4.2 build that through +`Tuple.fromArray`. (Earlier pins could not, which is why this page +documented a 22-selector ceiling until the 0.4.2 / 0.3.2 bump.) + +What a very wide selection costs is *compile time*, not +correctness: the derivation grows superlinearly in arity, so a +record wide enough to feel it wants a larger compiler stack +(`-Xss`) and a higher +`-Xmacro-settings:avroDerivation.timeout` before it wants a split. ```scala mdoc val nameAge = codecPrism[Person].fields(_.name, _.age) diff --git a/site/docs/integrations/circe.md b/site/docs/integrations/circe.md index 0af67aed..47e50f7b 100644 --- a/site/docs/integrations/circe.md +++ b/site/docs/integrations/circe.md @@ -177,7 +177,7 @@ already in scope from the first fence) does the job, given the dependency: ```scala -libraryDependencies += "com.kubuszok" %% "kindlings-circe-derivation" % "0.3.0" +libraryDependencies += "com.kubuszok" %% "kindlings-circe-derivation" % "0.3.2" ``` A hand-written codec works just as well. Miss it and the `.fields` From 97ecf3f12bce3e3de6ae3c4c51b52dc6ece39b58 Mon Sep 17 00:00:00 2001 From: Rodolfo Hansen Date: Fri, 18 Sep 2026 13:16:38 +0200 Subject: [PATCH 5/5] fix(build,docs): make the macro timeout real, stop pretending the overrides protect consumers, state the true `.fields` ceiling MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four audit findings on the hearth/kindlings bump, each verified before fixing. **The macro-expansion timeout was inert.** The build spelled it `-Xmacro-settings:{circe,cats,avro}Derivation.timeout=30`, but kindlings' `DerivationTimeout` parses with `^\s*(\d+)\s*(ms|millis|milliseconds|s|seconds?|m|minutes?)\s*$` and the call site is `flatMap(parse).getOrElse(Default)` — a bare integer never matches and falls through silently, so every derivation had been running on the 5s default, not 30s. Confirmed in the 0.3.2 bytecode (`Default = 5 SECONDS`) and by canary: `=1ms` fails with "timed out after 1ms", `=1` compiles clean (so a bare integer is neither seconds nor millis — it is discarded), `=30s` compiles. All spellings gain the unit: `=30s` build-wide, `=120s` for mdoc fences. The recurring "derivation timed out" CI flake this setting was added for was never actually mitigated until now. **The jackson/commons-lang3 `dependencyOverrides` never reached a consumer.** sbt's overrides are build-local and emit no ``; verified that all four generated POMs (cats-eo, -avro, -circe, -generics) carry zero ``, zero jackson, zero commons-lang3 entries. Downstream always resolved jackson through avro's own parent BOM, i.e. jackson-databind 2.20.0 before this bump — 5 OSV advisories, 2 HIGH (checked against OSV) — while eo's CI compiled against the overridden version and `dependency-submission` reported that safe one. The override made the exposure invisible, not absent. So the overrides are removed rather than re-pointed. The jackson half is now a measured no-op (the avro compile classpath is jar-for-jar identical with and without it) and was a live hazard besides: only two thirds of the BOM-managed trio was ever overridden — `jackson-annotations` never was — so the next jackson-bom lift inside avro-parent would have moved annotations alone while core/databind stayed frozen, splitting the BOM. Dropping the commons-lang3 half moves this build 3.20.0 -> 3.18.0, which is exactly what a consumer resolves (commons-lang3 arrives via commons-compress 1.28.0, whose POM declares 3.18.0; avro-parent's 3.20.0 property manages only avro's own direct deps). 3.18.0 is past CVE-2025-48924 and OSV-clean. The build now resolves avro's transitives as consumers do, so `dependency-submission` reports the real consumer graph. **The scala-steward pin re-armed the trap it had just sprung.** `version = "2.22."` is a series lock, not a floor: it blocks 2.23.x forever and matches 2.22.0 anyway, so it does not even express "never 2.22.0". The previous `"2.21."` pin is the proof — 2.22.1 shipped OSV-clean and required by avro 1.12.2 while the build sat on 2.21.5 until a human edited it. Steward cannot propose a version below the current one, so the pin only strands. Replaced with an `updates.ignore` naming the single bad version. **`.fields` does have a ceiling; the docs said it did not.** Two of them, and the second bites first. 254 selectors is hard and permanent — `.fields` selects from a case class and the JVM caps a parameter list at 254 slots (measured: 254 compiles, 255 fails with "Platform restriction: a parameter list's length cannot exceed 254", on the case class itself). Well below that the binding limit is the compile thread's stack, since the derivation recurses per field. Measured on a full-cover probe varying only -Xss: 1m (the JVM default) derives 32 and overflows at 36; 2m derives 66, overflows at 100; 4m (sbt's launcher default) derives 150, overflows at 254; 8m — this repo's .jvmopts, and the only reason its suites reach 254 — derives 254. A downstream build inherits none of that, because the published artifact cannot carry an -Xss. avro.md, the spec scaladoc, the `AvroPrism.fields` scaladoc and the CHANGELOG now say so. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0194EHFR4NamCpTHiqy7B74V --- .scala-steward.conf | 24 +++-- CHANGELOG.md | 55 ++++++++-- .../dev/constructive/eo/avro/AvroPrism.scala | 20 ++-- .../eo/avro/WideFieldsArityCeilingSpec.scala | 24 +++-- build.sbt | 100 +++++++++++------- site/docs/integrations/avro.md | 43 ++++++-- 6 files changed, 189 insertions(+), 77 deletions(-) diff --git a/.scala-steward.conf b/.scala-steward.conf index 713c7aed..47edb678 100644 --- a/.scala-steward.conf +++ b/.scala-steward.conf @@ -1,9 +1,19 @@ # jackson 2.22.0 regressed the @JsonIgnoreProperties case-insensitive fix -# (CVE-2026-54515, dependabot alert #7). 2.21.5 is patched and 2.22.1 carries -# the re-fix, so the floor moved to 2.22.1 when apache-avro 1.12.2 raised -# jackson-bom to it (see the build.sbt comment on `jacksonCore`). Keep both -# artifacts at or above 2.22.1 and never on 2.22.0. -updates.pin = [ - { groupId = "com.fasterxml.jackson.core", artifactId = "jackson-core", version = "2.22." }, - { groupId = "com.fasterxml.jackson.core", artifactId = "jackson-databind", version = "2.22." } +# (CVE-2026-54515, dependabot alert #7). 2.22.1 carries the re-fix and is what +# apache-avro 1.12.2's parent BOM resolves, so the build is already past it. +# +# There is deliberately NO `updates.pin` here. A pin like `version = "2.22."` +# is a SERIES LOCK, not a floor: it would block Steward from ever proposing +# 2.23.x, and it matches 2.22.0 anyway so it does not even express "never +# 2.22.0". The previous `"2.21."` pin is the proof of that failure mode — +# 2.22.1 shipped OSV-clean and required by avro 1.12.2, yet the build sat on +# 2.21.5 until a human edited the pin by hand. Steward never proposes a +# version below the current one, so 2.22.0 is already unreachable from 2.22.1 +# and a pin buys nothing but the stranding. +# +# `updates.ignore` below is the belt-and-braces form: it names the single bad +# version instead of locking a series, so 2.23.x and beyond still flow. +updates.ignore = [ + { groupId = "com.fasterxml.jackson.core", artifactId = "jackson-core", version = "2.22.0" }, + { groupId = "com.fasterxml.jackson.core", artifactId = "jackson-databind", version = "2.22.0" } ] diff --git a/CHANGELOG.md b/CHANGELOG.md index a513a849..5748d551 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -78,7 +78,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - **`.fields(...)` no longer stops at 22 selectors** (#96): the macro-synthesised - `NamedTuple` focus has no arity ceiling any more. Up to 22 selectors the value tuple is spelled + `NamedTuple` focus no longer has a *spelling* ceiling. Up to 22 selectors the value tuple is spelled `scala.TupleN` and hearth builds it with that tuple's constructor; at 23 and above the value tuple *is* a `*:` cons chain, and hearth 0.4.2 builds that through `Tuple.fromArray`. Before, hearth 0.4.0 called the cons chain's primary constructor — which takes no value parameters — so @@ -87,9 +87,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 arity 23 and at arity 40 with mixed field types, since the `Tuple.fromArray` path boxes every element to `Object`. Both spellings remain load-bearing: hearth's sub-23 branch still cannot build a cons chain, so `MacroSelectors.tupleTypeOf` keeps emitting `TupleN` below 23 - permanently. What a very wide selection costs is compile time, not correctness — the derivation - grows superlinearly in arity, so past a few hundred fields raise `-Xss` and - `-Xmacro-settings:avroDerivation.timeout` before splitting the cover. + permanently. **Two ceilings remain and are now documented rather than glossed:** 254 selectors + is hard and permanent (`.fields` selects from a case class, and the JVM caps a parameter list at + 254 slots — a 255-field case class does not compile at all), and well below that the binding + limit is the compiler thread's stack, since the derivation recurses per field. Measured on a + full-cover probe, varying only `-Xss`: 1m (the JVM default) derives 32 and overflows at 36; 2m + derives 66; 4m (sbt's launcher default) derives 150; 8m — this repo's `.jvmopts`, and the only + reason its own suites reach 254 — derives 254. A downstream build inherits none of that, because + the published artifact cannot carry an `-Xss`, so a wide `.fields` may need `-Xss` raised in the + *consumer's* build. ### Changed @@ -97,13 +103,40 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `0.3.0` → `0.3.2` — the releases that carry the NamedTuple arity fix above (hearth #313/#314, landed in 0.4.1). Transitively: apache-avro `1.12.1` → `1.12.2` (the explicit pin moves with it rather than silently downgrading the transitive), jackson-core/-databind `2.21.5` → `2.22.1`, - jackson-annotations `2.21` → `2.22`, commons-lang3 `3.18.0` → `3.20.0`, slf4j-api `2.0.17` → - `2.0.18`. The jackson force-pin lifts to 2.22.1 — the release that re-fixed CVE-2026-54515, - which 2.22.0 had regressed, and the version avro 1.12.2's parent BOM resolves — so the override - stays a regression floor instead of becoming a downgrade that splits the jackson BOM. The - commons-lang3 floor lifts to 3.20.0 for the same reason. No derived Avro schema text changed: - record names, field names and field order are byte-identical across the bump (the 216-example - avro suite, including every naming and union spec, passes unmodified). + jackson-annotations `2.21` → `2.22`, slf4j-api `2.0.17` → `2.0.18` (commons-lang3 stays at + `3.18.0` — see the override removal below; commons-compress stays at `1.28.0`). No + derived Avro schema text changed: record names, field names and field order are byte-identical + across the bump (the 216-example avro suite, including every naming and union spec, passes + unmodified). +- **Downstream consumers of `cats-eo-avro` move off a vulnerable jackson.** Resolving what the + published POM actually declares, a consumer previously landed on **jackson-databind 2.20.0 — 5 + OSV advisories, 2 of them HIGH** (PolymorphicTypeValidator bypass via generic type parameters, + array-subtype allowlist bypass, InetSocketAddress SSRF, per-property `@JsonIgnoreProperties` + bypass, `@JsonIgnore` bypass on records). With avro 1.12.2 they now land on **2.22.1, which has + none**. This was never covered by eo's `dependencyOverrides`: sbt's overrides are *build-local* + and emit no ``, and eo's published POMs carry none, so downstream always + resolved jackson through avro's own parent BOM. eo's CI meanwhile compiled against the + overridden version and `dependency-submission` reported it, which is why the exposure never + raised an alert here. +- **The jackson / commons-lang3 `dependencyOverrides` are removed.** They protected no consumer + (above) and had become a hazard: only two thirds of the BOM-managed jackson trio was ever + overridden (`jackson-annotations` was not), so the next `jackson-bom` lift inside avro-parent + would have moved annotations alone while core/databind stayed frozen. Measured, the jackson half + is now a pure no-op — the avro compile classpath is jar-for-jar identical with and without it. + Dropping the commons-lang3 override moves this build 3.20.0 → **3.18.0**, which is exactly what + a consumer resolves (commons-lang3 arrives via `commons-compress 1.28.0`, whose POM declares + 3.18.0; avro-parent's 3.20.0 property manages only avro's own direct dependencies). 3.18.0 is + the first release patched against CVE-2025-48924 and carries zero OSV advisories. The build now + resolves avro's transitives exactly as consumers do, so `dependency-submission` reports the real + consumer graph instead of a locally-sweetened one. +- **The kindlings macro-expansion timeout actually takes effect now.** The build spelled it + `-Xmacro-settings:{circe,cats,avro}Derivation.timeout=30`, but kindlings parses that value with + `^\s*(\d+)\s*(ms|millis|milliseconds|s|seconds?|m|minutes?)\s*$` and falls back to its 5 s + default on no match — so a bare integer was silently discarded and every derivation had been + running on 5 s, not 30 s. Verified by A/B: `=30` behaves byte-for-byte like passing no setting + at all, while `=30s` raises the budget. All spellings gain the unit suffix (`=30s` build-wide, + `=120s` for mdoc fences), which is what finally mitigates the recurring "derivation timed out" + CI flake the setting was added for. - **Behaviour change, avro**: a hand-written codec that PERMUTES the Scala names (writes case field `a` into a schema field literally named `b`, and vice versa) resolved correctly by position and now resolves by name, i.e. wrongly. No name transform can produce that shape — a transform is a diff --git a/avro/src/main/scala/dev/constructive/eo/avro/AvroPrism.scala b/avro/src/main/scala/dev/constructive/eo/avro/AvroPrism.scala index 37fa13b9..aa9fc03f 100644 --- a/avro/src/main/scala/dev/constructive/eo/avro/AvroPrism.scala +++ b/avro/src/main/scala/dev/constructive/eo/avro/AvroPrism.scala @@ -413,14 +413,22 @@ object AvroPrism: /** `.fields(_.a, _.b, ...)` — focus a NamedTuple over selected fields. * * The `AvroCodec` for the synthesised NamedTuple is summoned at the call site; with no - * hand-written given in scope it auto-derives through kindlings. There is no arity ceiling: up - * to 22 selectors the focus is spelled `TupleN` and hearth builds it with that tuple's + * hand-written given in scope it auto-derives through kindlings. The 22-selector ceiling is + * gone: up to 22 selectors the focus is spelled `TupleN` and hearth builds it with that tuple's * constructor; at 23 and above the focus type IS a `*:` cons chain, which hearth ≥ 0.4.2 builds * through `Tuple.fromArray` instead. (Both spellings are needed — hearth's constructor branch - * below 23 cannot build a cons chain, and there is no `TupleN` above 22.) Wide selections cost - * compile time rather than correctness: the derivation is quadratic-ish in arity, and a very - * wide record may want a larger `-Xss` or a higher `-Xmacro-settings:avroDerivation.timeout`. - * (Issue #96.) + * below 23 cannot build a cons chain, and there is no `TupleN` above 22.) + * + * That is not the same as "unbounded". `.fields` selects from a case class, so '''254 selectors + * is a hard, permanent ceiling''' — the JVM caps a parameter list at 254 slots and a 255-field + * case class does not compile at all. Well below that, the binding limit is the compiler + * thread's stack, because the derivation recurses per field: measured, `-Xss1m` (the JVM + * default) tops out around 32 selectors, `-Xss4m` around 150, and `-Xss8m` — what this repo's + * `.jvmopts` sets — reaches 254. A downstream build gets none of that automatically, so raise + * `-Xss` there before concluding a cover is too wide. Compile time is the third cost: the + * derivation grows superlinearly in arity, so a wide cover may also want a higher + * `-Xmacro-settings:avroDerivation.timeout=30s` — the unit suffix is required, a bare integer is + * silently ignored. (Issue #96.) */ extension [A](o: AvroPrism[A]) diff --git a/avro/src/test/scala/dev/constructive/eo/avro/WideFieldsArityCeilingSpec.scala b/avro/src/test/scala/dev/constructive/eo/avro/WideFieldsArityCeilingSpec.scala index d7f2c6fc..68ee8509 100644 --- a/avro/src/test/scala/dev/constructive/eo/avro/WideFieldsArityCeilingSpec.scala +++ b/avro/src/test/scala/dev/constructive/eo/avro/WideFieldsArityCeilingSpec.scala @@ -29,12 +29,24 @@ import org.specs2.mutable.Specification * wide, mixed-primitive cover — the `fromArray` branch boxes every element to `Object`, so a * homogeneous `String` probe would not have exercised the unboxing on the way back out). * - * There is no new '''arity''' ceiling above 23: a cons-chain NamedTuple was verified to derive and - * round-trip up to arity 512. What binds above ~200 is resource budget, not arity — the compiler - * thread's `-Xss` (this build sets `-Xss8m` in `.jvmopts`, good past arity 400) and kindlings' - * macro-expansion budget (`-Xmacro-settings:avroDerivation.timeout=30`, build.sbt). Neither is a - * language-level limit, so neither is pinned as a compile-time negative here; raise the budget if - * a genuinely enormous record ever times out. + * The bump removes the '''spelling''' ceiling, not every ceiling. Two limits remain and are + * deliberately NOT pinned as compile-time negatives here, because neither is a property of eo: + * + * - '''254 selectors, hard and permanent.''' `.fields` selects from a case class and the JVM + * caps a parameter list at 254 slots, so a 255-field case class fails to compile on its own + * ("Platform restriction: a parameter list's length cannot exceed 254") before `.fields` is + * reached. Measured: 254 compiles, 255 does not. + * - '''The compiler thread's `-Xss`, which binds far lower.''' The derivation recurses per + * field. Measured on a full-cover `.fields` probe, varying only `-Xss`: 1m (the JVM default) + * derives 32 and overflows at 36; 2m derives 66, overflows at 100; 4m (sbt's launcher default) + * derives 150, overflows at 254; 8m — what this repo's `.jvmopts` sets, and the only reason + * these suites reach 254 — derives 254. A DOWNSTREAM consumer inherits none of that: the + * published artifact cannot carry an `-Xss`. + * + * Compile time is the third cost — the derivation grows superlinearly in arity, so a very wide + * cover may want a higher `-Xmacro-settings:avroDerivation.timeout=30s` (build.sbt). Note the + * '''unit suffix''': kindlings parses that value with a regex requiring `ms`/`s`/`m`, and a bare + * integer is silently discarded, leaving the 5s default in force. */ class WideFieldsArityCeilingSpec extends Specification: diff --git a/build.sbt b/build.sbt index 23bfedbc..8f1fabc5 100644 --- a/build.sbt +++ b/build.sbt @@ -101,13 +101,23 @@ ThisBuild / scalacOptions += "-Wunused:all" // machine intermittently trips: `derived timed out after 5000ms`) to 30s. One namespace per // kindlings module (circe / cats / avro derivation); read by kindlings 0.3.x's `DerivationTimeout`. // Comma-separated so Scala's `-Xmacro-settings` MultiStringSetting splits them. +// +// THE UNIT SUFFIX IS LOAD-BEARING. `DerivationTimeout` parses the value with +// ^\s*(\d+)\s*(ms|millis|milliseconds|s|seconds?|m|minutes?)\s*$ +// and the call site is `settings.flatMap(parse).getOrElse(Default)`, so a BARE INTEGER does not +// match, falls through silently, and leaves the 5s default in force — no warning, no error. This +// build carried `timeout=30` (no unit) from the 0.3.0 pin onward and was therefore running on 5s +// the whole time. Re-check with the canary after any kindlings bump: set one namespace to `1ms` +// and confirm a derivation fails with `timed out after 1ms`; if it compiles, the key is dead +// again. +// // NB this reaches REGULAR compilation only — mdoc's fence compiler ignores the // -Xmacro-settings that arrive via mdoc.properties (verified: a 1ms canary never fires // through that route). Fences get the budget through mdoc's OWN --scalac-options CLI // argument instead (`mdocExtraArguments` on the docs project, same canary fires there), // and the heaviest doc derivations are additionally hosted in site/src compiled samples. ThisBuild / scalacOptions += - "-Xmacro-settings:circeDerivation.timeout=30,catsDerivation.timeout=30,avroDerivation.timeout=30" + "-Xmacro-settings:circeDerivation.timeout=30s,catsDerivation.timeout=30s,avroDerivation.timeout=30s" ThisBuild / tlFatalWarnings := true // `unused-code-plugin` (xuwei-k) ships a Scalafix `SyntacticRule` @@ -292,7 +302,6 @@ val Optics = "dev.optics" val Kubuszok = "com.kubuszok" val Circe = "io.circe" val ApacheAvro = "org.apache.avro" -val FasterXmlJackson = "com.fasterxml.jackson.core" val Plokhotnyuk = "com.github.plokhotnyuk.jsoniter-scala" val Ziverge = "dev.zio" val GetKyo = "io.getkyo" @@ -313,9 +322,12 @@ lazy val monocle = Optics %% "monocle-core" % "3.3.0" lazy val drosteCore = "io.higherkindness" %% "droste-core" % "0.9.0-M3" // kindlings 0.3.x (all three) ship a configurable macro-expansion timeout // (`DerivationTimeout`, default 5s) and pull hearth 0.4.2 + kindlings-derivation-commons. -// We raise it to 30s via `-Xmacro-settings:{circe,cats,avro}Derivation.timeout=30` -// (see the `ThisBuild / scalacOptions` above) so a loaded CI runner stops tripping the old -// hardcoded 2s budget (the recurring `deriveAsObject timed out after 2000ms` flake). +// We raise it to 30s via `-Xmacro-settings:{circe,cats,avro}Derivation.timeout=30s` +// (see the `ThisBuild / scalacOptions` above) so a loaded CI runner stops tripping the +// default budget (the recurring `deriveAsObject timed out` flake). The `s` is not +// decoration: until this bump the build spelled the value `=30`, which kindlings' +// DurationPattern rejects, so the budget silently stayed at the 5s default — the flake was +// never actually mitigated. See the scalacOptions comment for the regex and the canary. // The setting keys and namespaces are unchanged at 0.3.2 (`DerivationTimeout` is // byte-identical to 0.3.0); 0.3.1 added an OPT-IN `.policy.enabled` key under the same // namespace whose default (`always-allowed`) is exactly the 0.3.0 behaviour, so we set none. @@ -349,36 +361,43 @@ lazy val circeParser = Circe %% "circe-parser" % "0.14.16" // 0.3.0 depended on 1.12.1). Keeping the old 1.12.1 here would have turned a // visibility pin into a silent DOWNGRADE of the transitive, so it moves with it. lazy val avro = ApacheAvro % "avro" % "1.12.2" -// Force jackson to 2.22.1 — `apache-avro` brings `jackson-databind` (and -// `jackson-core`) transitively, and 1.12.1 brought 2.20.0, inside the -// CVE-affected `>= 2.19.0, < 2.21.5` range (four GHSA dependabot alerts: two -// PolymorphicTypeValidator/allowlist bypasses, an InetSocketAddress SSRF, and -// a @JsonIgnoreProperties case-insensitive bypass). 2.21.5 was the first -// release patched against all four; 2.22.0 REGRESSED the @JsonIgnoreProperties -// case-insensitive fix (CVE-2026-54515, dependabot alert #7) and 2.22.1 re-fixed -// it. 2.22.1 has since shipped to Central, and avro 1.12.2's parent POM raises -// `jackson-bom` 2.20.0 -> 2.22.1 — so the old 2.21.5 override would now DOWNGRADE -// jackson-core/-databind while leaving jackson-annotations (not overridden) at -// 2.22, splitting the BOM. The pin therefore lifts to 2.22.1, exactly the version -// avro 1.12.2 resolves: still a regression floor, no longer a shift, and never -// 2.22.0. `.scala-steward.conf` tracks the same series bound. -// Overrides apply via -// `commonSettings.dependencyOverrides` across every module so any future -// jackson-pulling transitive (e.g. a kindlings bump) inherits the safe -// versions automatically. eo never enables polymorphic/default typing, so the -// PTV bypasses aren't reachable here — this just keeps the dep tree clean. -lazy val jacksonCore = FasterXmlJackson % "jackson-core" % "2.22.1" -lazy val jacksonDatabind = FasterXmlJackson % "jackson-databind" % "2.22.1" -// Floor commons-lang3 at 3.20.0 — `apache-avro -> commons-compress 1.28.0` -// brings it transitively, and every release below 3.18.0 is in the -// CVE-2025-48924 range (uncontrolled recursion on long inputs; dependabot -// alert #1). avro 1.12.2's parent POM resolves 3.20.0 (1.12.1 resolved 3.18.0), -// so the floor moves up with it rather than downgrading the transitive — it stays -// a regression floor rather than a live bump. Pinned via -// `commonSettings.dependencyOverrides` (same mechanism as jackson) so a future -// avro/commons-compress shuffle can't reintroduce a vulnerable version, and the -// submitted dependency graph shows the safe version unambiguously. -lazy val commonsLang3 = "org.apache.commons" % "commons-lang3" % "3.20.0" +// NO jackson / commons-lang3 `dependencyOverrides` any more — deliberately. Both used to be +// forced here (jackson-core/-databind at 2.21.5, commons-lang3 at 3.18.0 before the avro 1.12.2 +// bump) as a guard against `apache-avro`'s transitives drifting onto a CVE-affected release. +// Three facts retired that guard: +// +// 1. It never protected a consumer. sbt's `dependencyOverrides` is BUILD-LOCAL: it does not +// emit ``, and the published POMs carry none (verified on all four of +// cats-eo, -avro, -circe, -generics). Anyone depending on `cats-eo-avro` resolved jackson +// through avro's own parent BOM, so before this bump they got jackson-databind 2.20.0 — five +// OSV advisories, two HIGH — while eo's own CI compiled against the overridden 2.21.5 and +// `dependency-submission` reported that safe version. The override made the exposure +// INVISIBLE rather than absent. +// 2. The jackson half is now a pure no-op that could only bite later. avro 1.12.2's parent POM +// raises `jackson-bom` 2.20.0 -> 2.22.1, so core/databind resolve at 2.22.1 with or without +// the override (measured: `avroIntegration/Compile/dependencyClasspath` is jar-for-jar +// identical either way). Meanwhile the trio is BOM-managed and only two thirds of it was +// ever overridden — jackson-bom 2.22.1 pairs core/databind 2.22.1 with annotations 2.22 +// (annotations dropped its patch component at 2.20) and `jackson-annotations` was never in +// the list — so the next jackson-bom lift inside avro-parent would have moved annotations +// alone while core/databind stayed frozen, splitting the BOM across two minors. That is the +// precise hazard the override existed to avoid. +// 3. The commons-lang3 half was masking, not fixing. Dropping it moves this build 3.20.0 -> +// 3.18.0, and 3.18.0 is what a consumer of `cats-eo-avro` resolves: commons-lang3 arrives +// via `avro -> commons-compress 1.28.0`, whose POM declares 3.18.0 outright, and +// avro-parent's `commons-lang3.version` 3.20.0 property manages only avro's OWN direct +// dependencies — it never reaches a consumer's resolution. 3.18.0 is the first release +// patched against CVE-2025-48924 and carries zero OSV advisories, so the override bought +// freshness for CI alone while consumers stayed on 3.18.0 unwatched. +// +// So the build now resolves jackson and commons-lang3 exactly as a downstream consumer does. +// That is the point: `dependency-submission` in ci.yml submits the build's resolved graph, so +// Dependabot now sees the graph consumers actually get instead of a locally-sweetened one, and +// an advisory against avro's transitives shows up here as a real alert instead of being silently +// overridden away. eo never enables polymorphic/default typing, so the jackson PTV bypasses were +// never reachable in eo's own code either way. If avro ever drags one of these onto a vulnerable +// release, bump `avro` — or ask upstream — rather than reintroducing a partial, non-propagating +// override that hides the problem from the very tooling meant to catch it. // jsoniter-scala — high-perf JSON codec (~5–10× circe on hot paths). // Used by `eo-jsoniter` to back byte-cursor JSON optics that decode // directly from `Array[Byte]` without allocating a runtime AST. The @@ -449,10 +468,8 @@ lazy val commonSettings = Seq( // library's code and the warning is a Hearth-side concern rather // than a cats-eo bug. Test / scalacOptions += "-Wconf:src=.*/cats-derivation/.*:silent", - // Pin jackson-core + jackson-databind at the CVE-patched 2.22.1 and floor - // commons-lang3 at 3.20.0 across every module — see the `jacksonCore` / - // `jacksonDatabind` / `commonsLang3` defs above. - dependencyOverrides ++= Seq(jacksonCore, jacksonDatabind, commonsLang3), + // NB no jackson / commons-lang3 `dependencyOverrides` here — see the comment above the `avro` + // dependency for why the build now resolves avro's transitives exactly as consumers do. ) // Library-appropriate scalac options layered on top of the baseline set @@ -928,7 +945,10 @@ lazy val docs: Project = project "--scalac-options", // 120s: the 60s budget still tripped intermittently inside cold-JVM // pre-commit hook sessions (cookbook.md avro fences on a loaded box). - "-Xmacro-settings:circeDerivation.timeout=120,catsDerivation.timeout=120,avroDerivation.timeout=120", + // The `s` suffix is required — kindlings' DurationPattern rejects a bare + // integer and silently reverts to the 5s default (see the ThisBuild + // scalacOptions comment above). + "-Xmacro-settings:circeDerivation.timeout=120s,catsDerivation.timeout=120s,avroDerivation.timeout=120s", ), // mdoc variable substitutions — site pages can reference // `@VERSION@` to always display the current version. diff --git a/site/docs/integrations/avro.md b/site/docs/integrations/avro.md index 55f32d56..c137da0e 100644 --- a/site/docs/integrations/avro.md +++ b/site/docs/integrations/avro.md @@ -255,19 +255,48 @@ inside the selection under the Scala package namespace, which an enclosing union will refuse. Name the codec (or drill with `.field`) in that case. -The auto-derived route has **no arity ceiling**. Up to 22 selectors -the NamedTuple's value tuple is spelled `TupleN` and the derivation +**The 22-selector ceiling is gone.** Up to 22 selectors the +NamedTuple's value tuple is spelled `TupleN` and the derivation backend builds it with that tuple's constructor; at 23 or more there is no `TupleN` spelling left — the value tuple *is* a `*:` cons chain — and kindlings ≥ 0.3.2 / hearth ≥ 0.4.2 build that through `Tuple.fromArray`. (Earlier pins could not, which is why this page documented a 22-selector ceiling until the 0.4.2 / 0.3.2 bump.) -What a very wide selection costs is *compile time*, not -correctness: the derivation grows superlinearly in arity, so a -record wide enough to feel it wants a larger compiler stack -(`-Xss`) and a higher -`-Xmacro-settings:avroDerivation.timeout` before it wants a split. +That does **not** mean "no ceiling". Two limits remain, and the +second one will bite you long before the first: + +*The hard limit is 254 selectors.* `.fields` selects from a case +class, and the JVM caps a parameter list at 254 slots, so a +255-field case class does not compile at all — `Platform +restriction: a parameter list's length cannot exceed 254`, raised +on the case class itself, before `.fields` is ever reached. 254 is +therefore permanent, not a backend detail. + +*The practical limit is your compiler's stack.* The derivation +recurses per field, so the reachable arity is set by `-Xss` on the +compile thread. Measured on the same probe, varying only `-Xss`: + +| `-Xss` | where it comes from | derived | overflowed | +|---------|--------------------------------|---------|------------| +| `1m` | JVM default | 32 | 36 | +| `2m` | — | 66 | 100 | +| `4m` | sbt's own launcher default | 150 | 254 | +| `8m` | this repo's `.jvmopts` | 254 | — (hard limit reached) | + +This repo sets `-Xss8m`, which is why its own suites reach 254 — +but **the published artifact cannot carry a `-Xss` for you**. On a +default 1 MB compile thread a wide `.fields` starts overflowing +between 32 and 36 selectors, barely past the old ceiling. If you +hit a `StackOverflowError` while widening a cover, raise `-Xss` in +*your* build before concluding the cover is too wide. + +Compile *time* is the third cost: the derivation grows +superlinearly in arity, so a wide cover may also want a higher +`-Xmacro-settings:avroDerivation.timeout` — note the **unit +suffix**, e.g. `=30s`. kindlings parses that value with a regex +that requires `ms`/`s`/`m`; a bare `=30` does not match and is +silently discarded, leaving the 5 s default in force. ```scala mdoc val nameAge = codecPrism[Person].fields(_.name, _.age)