Skip to content

Optimize protobuf encoding and add CI performance reports - #4

Merged
Fyzu merged 2 commits into
mainfrom
agent/protobuf-java21-performance
Sep 6, 2026
Merged

Optimize protobuf encoding and add CI performance reports#4
Fyzu merged 2 commits into
mainfrom
agent/protobuf-java21-performance

Conversation

@Fyzu

@Fyzu Fyzu commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Protobuf WKT encoding performs avoidable reflective reads, and numeric map keys allocate temporary strings. This PR uses concrete WKT getters and direct quoted numeric-key writes, fixes codec edge cases, and adds repeatable CI comparisons so future PRs and default-branch pushes report their performance impact.

Encoding changes

  • Use concrete Timestamp/Duration/Struct/Value/ListValue getters with DynamicMessage fallback, one exact-size uint64 buffer, cached descriptors/WKT classification, and presence checks before reflective reads.
  • Fix deprecated fields omitted by typed encoding, unsigned map keys printed as signed values, and custom JSON names producing invalid JSON or uncompilable generated codecs.
  • Preserve numeric/boolean map-key spelling under writer settings; add regression fixtures and a 30-case codegen/typed/reflection × UTF-16/UTF-8 JMH matrix.

The full same-compiler CI comparison on Java 21/25 measured Struct throughput +95–116%, codegen/typed maps +12–29%, and reflection Timestamp +25–29%. Struct allocations fell about 44%; map allocations fell 26–39% across all paths. Of 60 comparisons, 31 show improvement signals, 29 are inconclusive, and none show regression signals or allocation growth alerts. These are per-workload advisory signals, not an overall application speedup. Full CI results and raw artifacts supersede the earlier mixed-compiler follow-up; the historical local reports and compact CSV evidence remain in docs/performance-port-2026-09-05.md and docs/performance-results/.

Performance CI

  • Compare exact PR base/head SHAs and pre-/post-push SHAs on main, using stock JMH on Java 21 and 25.
  • Clean-build both revisions with the same compiler, shared candidate benchmark inputs, and isolated Maven repositories; measure sequentially on the same runner, alternating variant order.
  • Retain raw samples, environment, source identities, build logs and Markdown reports for 90 days. Job summaries are available on every run.
  • A separate trusted workflow_run publisher validates JSON artifacts, adds commit checks, and updates one PR comment while rejecting stale results. It never executes PR/artifact code with write credentials.
  • Throughput alerts require ≥10% change and separated JMH intervals and remain advisory. The 14 allocation budgets stay enforced; missing/nonfinite GC measurements now fail and allocation reports are uploaded.

The publisher starts automatically once its workflow exists on main. This introducing PR already gets measurement job summaries and artifacts. No external account, JMH fork, branch-protection change, or paid runner is required. docs/performance-ci.md compares GitHub-native reporting, dedicated runners, CodSpeed, Bencher, and github-action-benchmark and explains protocol/limits.

Validation

  • Encoding: clean Maven verification on Corretto 21.0.11 and 25.0.4; 496 tests passed on each with -Werror.
  • Current commit passed build, all 14 allocation checks, and the official protobuf conformance suite: CI run.
  • CI tooling: 15 tests cover measurement validation, thresholds, invalid allocation data, artifact bounds/identity, token redirects, and stale/missing PR reports.
  • End-to-end local smoke run clean-built main and candidate and successfully measured three cases with identical JDK/input sources; smoke output is explicitly excluded from performance evidence.
  • Actionlint, shell syntax checks, Spotless, and whitespace checks passed.
  • Full Java 21/25 CI comparisons passed on this revision: 30 cases per JVM, all 60 summaries independently reconstructed from raw JMH result files and validated. Performance run.

Context reuse, packed field names, primitive-list rewrites, and CodSpeed infrastructure remain deferred.

Fyzu and others added 2 commits September 5, 2026 23:29
Port measured WKT, numeric map-key, uint64 buffer, and descriptor caching improvements. Fix deprecated fields and escaped custom names across codecs, preserve map-key behavior under writer settings, and add regression tests, JMH coverage, allocation budgets, and performance evidence.
Build base and candidate with the same JDK and shared benchmark inputs on Java 21 and 25. Keep throughput signals advisory, enforce complete allocation data, and publish validated reports from a separate trusted workflow.
@Fyzu Fyzu changed the title Optimize protobuf WKT and map encoding; fix codec edge cases Optimize protobuf encoding and add CI performance reports Sep 6, 2026
@Fyzu

Fyzu commented Sep 6, 2026

Copy link
Copy Markdown
Contributor Author

First full CI performance report

Performance run: passed on Java 21 and 25. All 60 base/head comparisons were validated against the raw JMH results. Each JVM comparison clean-built both revisions with the same compiler and benchmark inputs, then measured both on one runner.

JVM Improvement signals Regression signals Inconclusive Allocation growth alerts
Java 21 15 0 15 0
Java 25 16 0 14 0

Observed changes across the two within-JVM comparisons:

  • Struct: throughput +95–116%, with allocation down 44%.
  • Codegen/typed maps: throughput +12–29%, with allocation down 29–39%. Reflection maps allocate 26–34% less, while timing stays inconclusive under the alert rule.
  • Reflection Timestamp: throughput +25–29%, with allocation down 18–19%. Typed Timestamp improves +40–45% on Java 25; its Java 21 timing remains inconclusive despite higher measured means.
  • Simple-message timings remain inconclusive. There are no throughput regression signals or relative allocation growth alerts in this run.

These are advisory signals, not a universal speedup or a proof that smaller regressions are absent. A timing signal requires at least 10% change and non-overlapping JMH 99.9% intervals. The existing 14 absolute allocation budgets, build, and official conformance job also passed on this commit.

Both jobs used two forks, 3 × 1s warmup, 4 × 1s measurement, one thread, a 256 MiB heap, and the GC profiler. Java 21 ran on AMD EPYC 9V74; Java 25 ran on AMD EPYC 7763. Each base/head pair shared its runner; absolute throughput must not be compared between JVM jobs. Raw results, environment, build logs, and reports are available in the workflow artifacts for 90 days.

This first report is posted manually because GitHub starts the automatic publisher only after its workflow exists on main. After merge, it will update one report comment per PR and attach a report check to PR/default-branch commits.

Java 21: all 30 comparisons

Java 21 performance

Base: cb106d5 → candidate: 2facf0a
Shared benchmark source: 0a6e1acc0ab8a278090e2943d1fc820ac55b122c

Throughput alerts are advisory. Existing allocation budgets are enforced separately.
A timing signal needs at least 10% change and separated JMH 99.9% intervals; otherwise it is inconclusive.
Allocation alerts need both >5% and >16 B/op growth (or >16 B/op from zero).

Benchmark Base ops/s Candidate ops/s Change Timing B/op base → candidate Allocation
simpleCodegenUtf16 10,370,842 ±341,682 10,414,523 ±170,627 +0.4% inconclusive 295.5 → 295.5 within alert threshold
simpleCodegenUtf8 10,241,226 ±72,012 10,207,074 ±266,454 -0.3% inconclusive 271.5 → 271.5 within alert threshold
simpleTypedUtf16 7,943,958 ±39,171 7,766,646 ±414,158 -2.2% inconclusive 295.5 → 295.5 within alert threshold
simpleTypedUtf8 8,363,583 ±181,199 8,390,696 ±178,066 +0.3% inconclusive 271.5 → 271.5 within alert threshold
simpleReflectionUtf16 4,079,940 ±87,313 4,104,177 ±89,500 +0.6% inconclusive 340.0 → 340.0 within alert threshold
simpleReflectionUtf8 4,014,666 ±178,880 4,127,033 ±24,504 +2.8% inconclusive 316.0 → 316.0 within alert threshold
complexCodegenUtf16 698,043 ±12,166 710,305 ±30,278 +1.8% inconclusive 1529.9 → 1481.9 within alert threshold
complexCodegenUtf8 692,663 ±15,013 709,149 ±19,545 +2.4% inconclusive 1505.9 → 1457.9 within alert threshold
complexTypedUtf16 586,471 ±9,004 638,543 ±13,273 +8.9% inconclusive 1393.9 → 1289.9 within alert threshold
complexTypedUtf8 611,716 ±31,875 687,931 ±5,818 +12.5% improvement signal 1369.9 → 1265.9 within alert threshold
complexReflectionUtf16 321,765 ±19,919 365,878 ±8,730 +13.7% improvement signal 1809.9 → 1353.9 within alert threshold
complexReflectionUtf8 334,000 ±5,222 389,007 ±25,672 +16.5% improvement signal 1785.9 → 1329.9 within alert threshold
mapCodegenUtf16 76,413 ±3,349 87,047 ±5,906 +13.9% improvement signal 8241.2 → 5025.3 within alert threshold
mapCodegenUtf8 67,909 ±3,733 82,759 ±5,461 +21.9% improvement signal 8217.3 → 5001.2 within alert threshold
mapTypedUtf16 56,780 ±520 64,529 ±598 +13.6% improvement signal 7586.5 → 4805.6 within alert threshold
mapTypedUtf8 58,561 ±3,755 65,371 ±2,249 +11.6% improvement signal 7478.5 → 4781.6 within alert threshold
mapReflectionUtf16 41,492 ±2,788 45,309 ±1,391 +9.2% inconclusive 11474.2 → 7634.2 within alert threshold
mapReflectionUtf8 42,588 ±2,987 46,605 ±2,715 +9.4% inconclusive 11450.0 → 7610.2 within alert threshold
structCodegenUtf16 331,791 ±2,023 694,107 ±16,765 +109.2% improvement signal 1304.9 → 735.7 within alert threshold
structCodegenUtf8 328,503 ±2,508 642,553 ±14,084 +95.6% improvement signal 1280.9 → 711.7 within alert threshold
structTypedUtf16 328,649 ±1,653 693,528 ±6,308 +111.0% improvement signal 1304.9 → 735.7 within alert threshold
structTypedUtf8 313,182 ±22,329 649,290 ±8,114 +107.3% improvement signal 1280.9 → 711.7 within alert threshold
structReflectionUtf16 312,735 ±3,067 664,742 ±17,004 +112.6% improvement signal 1304.9 → 735.7 within alert threshold
structReflectionUtf8 309,704 ±6,502 631,736 ±5,040 +104.0% improvement signal 1280.9 → 711.7 within alert threshold
timestampCodegenUtf16 3,851,822 ±217,874 3,950,015 ±44,748 +2.5% inconclusive 464.0 → 464.0 within alert threshold
timestampCodegenUtf8 3,381,086 ±640,889 4,027,935 ±410,233 +19.1% inconclusive 440.0 → 440.0 within alert threshold
timestampTypedUtf16 2,038,832 ±471,660 2,784,428 ±313,894 +36.6% inconclusive 568.0 → 464.0 within alert threshold
timestampTypedUtf8 2,504,749 ±12,815 3,011,387 ±966,249 +20.2% inconclusive 544.0 → 440.0 within alert threshold
timestampReflectionUtf16 1,970,770 ±16,752 2,475,058 ±68,447 +25.6% improvement signal 568.0 → 464.0 within alert threshold
timestampReflectionUtf8 2,010,862 ±12,434 2,597,959 ±15,150 +29.2% improvement signal 544.0 → 440.0 within alert threshold
Java 25: all 30 comparisons

Java 25 performance

Base: cb106d5 → candidate: 2facf0a
Shared benchmark source: 0a6e1acc0ab8a278090e2943d1fc820ac55b122c

Throughput alerts are advisory. Existing allocation budgets are enforced separately.
A timing signal needs at least 10% change and separated JMH 99.9% intervals; otherwise it is inconclusive.
Allocation alerts need both >5% and >16 B/op growth (or >16 B/op from zero).

Benchmark Base ops/s Candidate ops/s Change Timing B/op base → candidate Allocation
simpleCodegenUtf16 10,578,497 ±280,965 10,458,581 ±184,067 -1.1% inconclusive 295.5 → 295.5 within alert threshold
simpleCodegenUtf8 11,291,915 ±374,536 10,947,881 ±210,398 -3.0% inconclusive 271.5 → 271.5 within alert threshold
simpleTypedUtf16 7,363,052 ±463,955 7,468,600 ±220,017 +1.4% inconclusive 295.5 → 295.5 within alert threshold
simpleTypedUtf8 7,509,394 ±162,979 7,217,980 ±969,836 -3.9% inconclusive 271.5 → 271.5 within alert threshold
simpleReflectionUtf16 3,812,740 ±138,725 3,834,762 ±88,143 +0.6% inconclusive 340.0 → 340.0 within alert threshold
simpleReflectionUtf8 3,958,051 ±16,564 3,878,738 ±75,244 -2.0% inconclusive 316.0 → 316.0 within alert threshold
complexCodegenUtf16 705,035 ±2,513 711,964 ±9,761 +1.0% inconclusive 1529.9 → 1481.9 within alert threshold
complexCodegenUtf8 695,071 ±2,171 702,646 ±8,148 +1.1% inconclusive 1505.9 → 1457.9 within alert threshold
complexTypedUtf16 600,122 ±5,221 645,933 ±34,480 +7.6% inconclusive 1393.9 → 1289.9 within alert threshold
complexTypedUtf8 606,470 ±3,308 652,121 ±45,106 +7.5% inconclusive 1369.9 → 1265.9 within alert threshold
complexReflectionUtf16 340,317 ±3,584 391,090 ±7,457 +14.9% improvement signal 1785.9 → 1353.9 within alert threshold
complexReflectionUtf8 338,562 ±7,945 397,554 ±17,700 +17.4% improvement signal 1761.9 → 1329.9 within alert threshold
mapCodegenUtf16 97,083 ±3,129 120,789 ±3,934 +24.4% improvement signal 6719.4 → 4589.5 within alert threshold
mapCodegenUtf8 95,506 ±7,868 123,368 ±9,674 +29.2% improvement signal 6575.4 → 4565.5 within alert threshold
mapTypedUtf16 71,887 ±993 82,843 ±3,198 +15.2% improvement signal 6815.4 → 4805.5 within alert threshold
mapTypedUtf8 73,203 ±1,630 82,443 ±2,778 +12.6% improvement signal 6875.5 → 4781.5 within alert threshold
mapReflectionUtf16 45,740 ±1,595 48,694 ±2,462 +6.5% inconclusive 10268.1 → 7634.2 within alert threshold
mapReflectionUtf8 48,927 ±7,088 51,296 ±1,837 +4.8% inconclusive 10243.9 → 7610.2 within alert threshold
structCodegenUtf16 346,039 ±20,824 748,367 ±35,609 +116.3% improvement signal 1304.9 → 735.7 within alert threshold
structCodegenUtf8 351,087 ±4,581 686,471 ±37,699 +95.5% improvement signal 1280.9 → 711.7 within alert threshold
structTypedUtf16 344,243 ±11,440 738,578 ±4,930 +114.6% improvement signal 1304.9 → 735.7 within alert threshold
structTypedUtf8 345,069 ±7,138 701,693 ±1,833 +103.3% improvement signal 1280.9 → 711.7 within alert threshold
structReflectionUtf16 339,522 ±939 708,234 ±33,833 +108.6% improvement signal 1304.9 → 735.7 within alert threshold
structReflectionUtf8 327,948 ±12,777 667,536 ±12,051 +103.5% improvement signal 1280.9 → 711.7 within alert threshold
timestampCodegenUtf16 4,133,560 ±15,560 4,135,420 ±9,009 +0.0% inconclusive 464.0 → 464.0 within alert threshold
timestampCodegenUtf8 4,799,459 ±106,949 4,803,623 ±11,117 +0.1% inconclusive 440.0 → 440.0 within alert threshold
timestampTypedUtf16 2,492,100 ±42,363 3,483,652 ±17,731 +39.8% improvement signal 568.0 → 464.0 within alert threshold
timestampTypedUtf8 2,728,228 ±133,774 3,943,016 ±24,407 +44.5% improvement signal 544.0 → 440.0 within alert threshold
timestampReflectionUtf16 1,958,363 ±234,616 2,454,709 ±82,159 +25.3% improvement signal 568.0 → 464.0 within alert threshold
timestampReflectionUtf8 2,188,362 ±20,057 2,814,535 ±60,239 +28.6% improvement signal 544.0 → 440.0 within alert threshold

@Fyzu
Fyzu merged commit 7d67995 into main Sep 6, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant