avro: kindlings-derived AvroCodec encode is ~1.7× slower and allocates ~9–18% more than writing the same record straight to a BinaryEncoder

[luismartinez-adtechnacity]

Context

We replaced a vulcan Codec.toBinary on a hot produce path with two alternatives, both producing byte-identical Avro binary (verified by property tests against the vulcan bytes):

1. Hand-written: fields written directly to an org.apache.avro.io.BinaryEncoder in schema order (writeIndex / writeString / writeLong …). Union payloads with ~18 branches still go through cached GenericDatumWriters.
2. eo / kindlings: AvroEncoder.derived (via AvroCodec.derived) builds the Avro generic record, and a plain Avro GenericDatumWriter serializes it under the registered writer schema. Custom leaf encoders handle micros instants and map key order. A nested sub-record is spliced in with AvroPrism.graftBytes.

Both replace the same vulcan path. The record is a ~15-field top level with a nested ~55-field all-Option metrics record, an 18-branch union, and a ~20-field nested record that is either spliced (graft) or null.

Numbers (JMH, -prof gc, 1 fork, 5 warmup + 8 measurement iterations, µs/op and B/op)

Route	vulcan	hand-written	eo / kindlings
with graft splice	6.85 µs, 22,272 B	1.05 µs, 6,904 B	1.81 µs, 7,528 B
without graft	3.26 µs, 12,328 B	1.01 µs, 3,856 B	1.40 µs, 4,544 B

The eo route is a big win over vulcan (3.8× / 2.3×), but it's about 1.4–1.7× slower than the hand-written writer and allocates 9–18% more. Caveat: the hand-written and eo numbers come from separate runs on the same machine; allocation should be reliable, timings indicative.

Hypothesis

The gap is the intermediate generic-record tree. AvroEncoder.derived materializes a GenericData.Record (plus boxed leaves / java.util collections) for every nested record, which GenericDatumWriter then walks with per-field schema dispatch. The hand-written path streams straight to the encoder.

Questions / possible directions

• Could kindlings/eo derive a streaming encoder (A => BinaryEncoder => Unit) that writes fields in schema order without building the generic tree, keeping AvroCodec[A] as the typeclass? That's roughly the encode-side counterpart of what No efficient whole-record construction primitive (build-from-scratch beats AvroRecordPrism) #95 addressed for construction.
• If not, is there a recommended pattern for writing a derived record directly to bytes (e.g. an AvroCodec method returning Array[Byte] that skips GenericData.Record)?
• Is the extra allocation expected from the derived encoders (boxing Option / numeric leaves), or is it avoidable?

Versions: cats-eo-avro 0.14.0, kindlings-avro-derivation 0.3.2, hearth 0.4.2, avro 1.12.1, Scala 3.9.0, JDK 25 (Corretto).

Happy to share a reduced JMH reproducer if useful.

[kryptt]

Thanks for the detailed writeup — reproduced the shape (scaled down: 15-field top level, 12-field all-Option nested record, 18-branch union, byte-identical hand-written vs eo output, same as your methodology) and decomposed the route. Your hypothesis is half right, and the wrong half was ours to fix.

Attribution on a 245 B record

Route	ns/op	B/op
kindlings encode(a) → Any (tree build only)	532	664
datum → bytes, fresh ByteArrayOutputStream + GenericDatumWriter + BufferedBinaryEncoder	1,599	2,928
datum → bytes, reused buffer/encoder/writer	1,311	392
full eo route (AvroCodec.encodeValue) before the fix	1,670	3,456
hand-written fields straight to a reused BinaryEncoder	761	397

• The time gap over hand-written really is the intermediate generic-record tree plus GenericDatumWriter dispatch (casts, positional reads, resolveUnion per union field). That lives in kindlings' derivation and avro's datum model.
• The allocation gap, however, was mostly our plumbing: writeDatum allocated a fresh ByteArrayOutputStream (32-byte start — so a multi-KB payload reallocates and copies a dozen times on the way up), a fresh writer, and a fresh encoder per call. Reusing exactly those three takes a write from 2,928 B/op to 392 B/op.

Fix: #120 makes writeDatum write through per-thread reused plumbing (mirror of what the decode side already did via cached readers/decoders). Full eo route drops 3,456 → 904 B/op on the fixture, wire bytes unchanged (exact-bytes specs + new retention/abort/concurrency pins). Your numbers should land close to hand-written on the B/op axis once you run against the branch; the residual ns/op delta is the tree + dispatch cost.

Answers to your questions:

1. Streaming A => BinaryEncoder => Unit: yes, that's the encode-side twin of No efficient whole-record construction primitive (build-from-scratch beats AvroRecordPrism) #95's whole-record-builder question — but it has to come from kindlings: their AvroEncoder[A] is A => Any, so no shim on the eo side can skip materializing the datum. Upstream derivation request; if it lands, wiring an AvroCodec method to it is straightforward.
2. Skipping GenericData.Record today: the byte-carried optics are the zero-tree path — AvroPrism writes re-encode only the focused span, and graftBytes splices pre-encoded payloads with no datum at all (your nested-record splice case is exactly where this applies).
3. Boxing: boxed numeric/Option leaves and one Object[] + GenericData.Record per nested record are inherent to going through the datum model. The buffer/encoder churn on top was not — that's fixed.

If you get a chance to run your reduced JMH reproducer against #120's branch (perf/avro-write-plumbing-119), I'd very much like the before/after gc.alloc.rate.norm — your payload (~7 KB) sits right where the ByteArrayOutputStream growth chain was worst, so I expect the effect to show more strongly than in our scaled-down fixture. Will cut a release with this soon.

[luismartinez-adtechnacity]

Thanks for the quick turnaround on #120. Numbers against it below.

One correction first: our record is not ~7 KB on the wire. The 7,528 in the issue table was B/op. The real record encodes to 1,377 B (429 B with the nested record null). So I added an inflated variant (6,713 B, a larger map field) to reach the size you were interested in.

Setup:

• perf(avro): reuse write plumbing in writeDatum — attribution + fix for #119 #120's source diff (git diff v0.17.0 a9b9c2b -- avro/src, which is only AvroBinaryCursor.scala plus the spec) applied cleanly onto v0.16.0, published locally. Our build is on 0.16.0 and 0.17.0's AvroVulcan.codec signature change touches many call sites, so this keeps the A/B down to the perf(avro): reuse write plumbing in writeDatum — attribution + fix for #119 #120 change alone. AvroWriteCorrectnessSpec passes on that backport.
• JMH -prof gc, 1 fork, 5 warmup + 8 measurement iterations, same machine, configs alternated over several rounds. The machine was shared and loaded, so B/op is solid (stable to about ±150 B across rounds) and µs/op is indicative.
• Route: AvroCodec.encodeValue on a vulcan-bridged codec: a ~15-field top level, a nested all-Option ~55-field record, an 18-branch union, and a ~20-field nested record (present or null). Output checked byte-identical to vulcan.

Payload	0.16.0	+ #120	Δ B/op
429 B	3.71 ± 0.23 µs, 12,320 B/op	3.33 ± 0.07 µs, 9,600 B/op	−2.7 KB
1,377 B	8.01 ± 0.73 µs, 27,100 B/op	7.91 ± 0.60 µs, 23,420 B/op	−3.7 KB
6,713 B	35.5 ± 25 µs, 127,976 B/op	25.1 ± 0.2 µs, 111,440 B/op	−16.5 KB

Observations:

• The saving matches the plumbing exactly. Under the BufferedBinaryEncoder's 2 KB buffer the stream only sees one write at flush, so it resizes once. The saving there is the encoder buffer, the writer and one resize. Past 2 KB the doubling chain kicks in and the saving scales with it.
• The rest of this route's allocation is datum-tree construction (vulcan's encode side, in these numbers), not eo's write.
• Our hot path splices bytes rather than writing datums. It encodes with its own reused GenericDatumWriter and splices the nested record with graftBytes, so it never hits writeDatum and perf(avro): reuse write plumbing in writeDatum — attribution + fix for #119 #120 is neutral there (unchanged within noise, as expected). The splice path costs 7.5 KB/op and ~2 µs against vulcan's ~27 KB/op.

LGTM from our side.

[kryptt]

Follow-up: v0.18.0 is out (https://github.com/Constructive-Programming/eo/releases/tag/v0.18.0) and on Maven Central, so dev.constructive:cats-eo-avro_3:0.18.0 is available now. It's a drop-in for 0.17.x — the write plumbing is fully internal (private[avro]), no public signature moved, no wire-format change.

To pick up the #119 fix, just bump to 0.18.0 and re-run your reproducer. The one number worth comparing is gc.alloc.rate.norm (B/op) on the with-graft case — that's where the old fresh-ByteArrayOutputStream-per-call churn hurt most, and where you should see the biggest drop; the without-graft case should also land closer to your hand-written baseline. Expect the residual ns/op gap to hand-written to persist (that's the generic-record tree + GenericDatumWriter dispatch — the streaming-encoder piece, not this release).

If the allocation gap is closed on your side, happy to have #119 confirmed and closed; if any of it survives at your ~7 KB payload size, reopen/extend here and I'll dig into the remainder.

[luismartinez-adtechnacity]

Re-ran on 0.18.0 (same JMH config, -prof gc, byte-identical output asserted against vulcan):

Payload	0.16.0	0.18.0	Δ B/op
429 B	12,320 B/op	9,584 B/op	−2.7 KB
1,377 B	27,100 B/op	23,800 B/op	−3.3 KB
6,713 B	127,976 B/op	111,400 B/op	−16.6 KB

These match the #120 backport numbers exactly, so the release carries the fix as advertised. Our graft/splice path (graftBytes plus our own reused GenericDatumWriter, ConfluentWire.attach) is unchanged between 0.16.0 and 0.18.0 on the same commit, as expected since it never goes through writeDatum. The rest of the encodeValue gap to hand-written is the datum-tree build and GenericDatumWriter dispatch, as you said. Upgraded cleanly across the 0.17 AvroVulcan.codec change. Happy to close. Thanks!