Skip to content

fix: bound the number of entries in a compression table advertisement - #3510

Merged
pjfanning merged 1 commit into
apache:mainfrom
pjfanning:bound-compression-advert
Sep 3, 2026
Merged

fix: bound the number of entries in a compression table advertisement#3510
pjfanning merged 1 commit into
apache:mainfrom
pjfanning:bound-compression-advert

Conversation

@pjfanning

Copy link
Copy Markdown
Member

Motivation

deserializeCompressionAdvertisement (ArteryMessageSerializer:179-192) resolves every key in
the advertised table — for actor refs that means resolveActorRef, which parses a path and
populates the per-thread resolve cache. The key list was unbounded, so the only limit was the
transport frame size.

Measured on this branch:

   256 entries ->     9009 wire bytes,  32 ms
  1000 entries ->    35793 wire bytes,  37 ms
 10000 entries ->   368793 wire bytes, 150 ms
 50000 entries ->  1922409 wire bytes, 622 ms

256 is what a peer legitimately advertises (compression.actor-refs.max defaults to 256), and
9 KB. The default maximum-frame-size of 256 KiB allows roughly 7000 entries, so the reachable
worst case is around 100 ms of CPU on the inbound control stream per message, repeatable.

Modification

Reject an advertisement carrying more entries than
pekko.remote.artery.advanced.compression.<table>.max — the setting that bounds the table on
the sending side, parsed here the same way ArterySettings parses it. When it is "off"
locally the setting is 0, there is no configured number to check against, and no bound is
applied.

The tradeoff worth a reviewer's attention: this bounds by the receiver's setting, but the
table is sized by the sender's. Configurations are normally uniform across a cluster, but
during a rolling change of actor-refs.max a node still on the smaller value would reject a
larger advertisement. I checked what that costs before choosing it: a fromBinary failure on
the inbound stream is caught at Codecs.scala:692-704, which logs an error and drops the
message — no quarantine — and InboundCompressions resends advertisements periodically
(InboundCompressions.scala:553, "The ActorRefCompressionAdvertisement message is resent
because it can be lost"). So the worst case for a skewed config is that compression is not
established for that association until both sides are updated; messages still flow uncompressed.
If reviewers would rather have headroom than exactness, the alternative is a separate setting
with a default well above the table maxes.

What I did not change

The review this came from also flagged ActorRefResolveCache as hash-floodable:
LruBoundedCache never grows and Unsafe.fastHash is seeded with two fixed public constants,
so colliding actor paths can be precomputed offline. I measured it rather than assume:
generating 4096 keys that all land in one slot took 558 ms, and cache operations then went from
2347 ns to 13631 ns, about 5.8x.

That is a real effect but far short of the degradation I expected, the cache is bounded at 1024
entries so nothing grows without limit, and closing it means changing the seed of a public
Unsafe.fastHash. 5.8x on one cache lookup does not seem to justify that, so I left it alone
and am recording the numbers here instead.

Result

An oversized advertisement is reported as a serialization failure. A legitimately sized one is
unaffected.

Tests

  • sbt "remote/testOnly org.apache.pekko.remote.serialization.*" — 195 passed, 1 pending

One new test in ArteryMessageSerializerSpec, checked to discriminate by reverting the
production file and re-running, where it fails with no exception was thrown:

  • reject a compression table advertisement with more entries than the configured maximum
    asserts a table of exactly max entries still deserializes, and that max + 1 is rejected

  • sbt "remote/mimaReportBinaryIssues" — no issues

  • sbt "remote/scalafmtCheckAll" headerCreateAll — clean

References

Touches the same method as #3507, so whichever merges second needs a trivial rebase; the two
guards are independent.

Motivation:
deserializeCompressionAdvertisement resolves every key in the advertised
table, and for actor refs that means parsing a path and populating the
resolve cache. The key list was unbounded, so the only limit was the
transport frame size. Measured, 10000 entries is 369 KB of wire and about
150 ms of CPU on the inbound control stream, against 9 KB for the 256
entry table a peer legitimately advertises.

Modification:
Reject an advertisement carrying more entries than
pekko.remote.artery.advanced.compression.<table>.max, the setting that
bounds the table on the sending side and is normally the same across a
cluster. When it is "off" locally there is no number to check against and
no bound is applied.

Result:
An oversized advertisement is reported as a serialization failure, which
the inbound stream logs and drops. Advertisements are resent periodically,
so a dropped one costs at most a delay in establishing compression.
@pjfanning

Copy link
Copy Markdown
Member Author

This is a fix that I would like to get into 1.7.1.

@pjfanning
pjfanning merged commit 6751ab9 into apache:main Sep 3, 2026
10 checks passed
@pjfanning
pjfanning deleted the bound-compression-advert branch September 3, 2026 08:16
@pjfanning pjfanning added this to the 2.0.0-M5 milestone Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants