fix: bounds check the lookup table indexes in gossip - #3508
Open
pjfanning wants to merge 1 commit into
Open
Conversation
Motivation: Gossip interns addresses, roles, hashes and app versions into tables and refers to them by index. Every index in gossipFromProto came straight off the wire into Vector.apply with no range check, so a negative or out of range one raised IndexOutOfBoundsException instead of a serialization failure. For a GossipEnvelope this matters more than usual: gossipEnvelopeFromProto defers the parse into a thunk, so the throw happens inside ClusterCoreDaemon when the gossip is read rather than on a deserialization thread. Modification: Look the indexes up through a helper that range checks first and reports a NotSerializableException naming the index and the table size. Every index gossipToProto writes is in range, so nothing a peer legitimately sends is affected. Result: Gossip that refers to a table entry the sender did not include is reported as a serialization failure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Gossip interns addresses, roles, hashes and app versions into tables and refers to them by
index. Every one of those indexes went straight from the wire into
Vector.applywith no rangecheck:
ClusterMessageSerializer:551,554:570:578:572:588:596:605GossipandGossipStatusA negative or out of range index raises
IndexOutOfBoundsExceptionrather than a serializationfailure. For a
GossipEnvelopethis lands somewhere unexpected:gossipEnvelopeFromProto(
:608-615) defers the whole parse into a() =>thunk, so the throw happens insideClusterCoreDaemonwhen the gossip is read, not on a deserialization thread where a failurewould be contained.
Modification
Look the indexes up through a helper that range checks first and reports a
NotSerializableExceptionnaming the index and the table size. Every indexgossipToProtowrites is in range by construction, so nothing a peer legitimately sends is affected.
Correction to what I reported earlier. I had also flagged
memberStatusFromIntandreachabilityStatusFromIntas reachableNoSuchElementExceptions on an unknown enum number.That is wrong:
ClusterMessages.protoisproto2, where an unrecognised enum value is moved tothe unknown fields and the accessor returns the default, so
getStatus.getNumberis always oneof the declared values and both maps are total over it. I had a check written for those two and
removed it rather than ship defensive code that cannot fire and cannot be tested.
Result
Gossip that refers to a table entry the sender did not include is reported as a serialization
failure naming the index and the size.
Tests
sbt "cluster/testOnly org.apache.pekko.cluster.protobuf.*"— 12 passedThree new tests, each checked to discriminate by reverting the production file and re-running;
all three then fail with
IndexOutOfBoundsException was thrown:reject gossip that refers to a lookup table entry it did not send— takes a realWelcome,decompresses it, and tampers one index at a time: an address index past the end, a negative
address index, a role index, a
seenentry, and a vector clock hash indexreject a gossip status that refers to a hash it did not send—GossipStatusreachesvectorClockFromProtowithout going throughGossip, and is not compressedreject a gossip envelope with a bad index when the gossip is read— asserts the failuresurfaces from
envelope.gossip, not fromfromBinary, which is where the deferred parse puts itsbt "cluster/mimaReportBinaryIssues"— no issuessbt "cluster/scalafmtCheckAll" headerCreateAll— cleanReferences
None.