Speed up initial in-memory Soroban state population by drebelsky · Pull Request #5252 · stellar/stellar-core

drebelsky · 2026-05-05T18:17:27Z

Related to #4902. Note that since that time, state churn has continued, so population now takes ~70s on a dev watcher. This PR changes the live state calculation from going through the buckets one-by-one using a hash map to a k-way merge among all the buckets. The merge is done using a loser tree, which gives us about half as many comparisons as using a heap. Running on a dev watcher speeds up from ~70s to ~30s.

Time for 3 runs on upstream vs patch

26.0.2-3192.9b5bee752.noble~do~not~use~in~prd~perftests 300

2026-05-05T18:09:57.683 GB4ZO [Perf INFO] Populated in-memory Soroban state in 31.077 sec
2026-05-05T18:09:57.683 GB4ZO [Perf INFO] Startup state load took 32.777 sec (full=true)

2026-05-05T18:10:33.150 GA6IH [Perf INFO] Populated in-memory Soroban state in 30.273 sec
2026-05-05T18:10:33.151 GA6IH [Perf INFO] Startup state load took 31.619 sec (full=true)

2026-05-05T18:11:08.694 GDECF [Perf INFO] Populated in-memory Soroban state in 30.091 sec
2026-05-05T18:11:08.698 GDECF [Perf INFO] Startup state load took 31.317 sec (full=true)

26.0.2-3190.8a71e20af.noble~perftests 300

2026-05-05T18:05:38.067 GDDEI [Perf INFO] Populated in-memory Soroban state in 68.910 sec
2026-05-05T18:05:38.067 GDDEI [Perf INFO] Startup state load took 70.809 sec (full=true)

2026-05-05T18:07:14.023 GDYRH [Perf INFO] Populated in-memory Soroban state in 70.991 sec
2026-05-05T18:07:14.023 GDYRH [Perf INFO] Startup state load took 73.228 sec (full=true)

2026-05-05T18:08:35.801 GA4XC [Perf INFO] Populated in-memory Soroban state in 74.710 sec
2026-05-05T18:08:35.801 GA4XC [Perf INFO] Startup state load took 76.958 sec (full=true)

Doing the k-way merge also has nicer memory scaling characteristics than the current approach: the amount of memory we use scales with the live state + number of buckets, instead of the current approach that scales with churn.

Additionally, the PR disables bucket merges until after the in-memory state is populated.

Copilot

Pull request overview

This PR speeds up startup-time reconstruction of the in-memory Soroban state by changing live-state discovery from per-bucket deduping to a merged scan across all buckets, and by deferring bucket-merge restart until after full state population. It fits into the ledger/bucket startup path that rebuilds Soroban state from the BucketList on node startup.

Changes:

Replace initializeStateFromSnapshot’s per-type bucket scans with a new “current live entries” scan that returns only the latest live version of each key.
Add bucket-snapshot support for k-way merged live-entry scanning, including a new ledger-key comparator used by the loser-tree merge.
Split bucket merge restart out of assumeState and invoke it later in full startup mode.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`src/ledger/LedgerManagerImpl.cpp`	Defers restarting bucket merges until after full Soroban state setup.
`src/ledger/InMemorySorobanState.cpp`	Switches snapshot initialization to current-live scans for Soroban entry types.
`src/ledger/ImmutableLedgerView.h`	Exposes a new current-live scan API on immutable/apply ledger views.
`src/ledger/ImmutableLedgerView.cpp`	Wires the new ledger-view scan API to the live bucket snapshot.
`src/bucket/LedgerCmp.h`	Declares a 3-way comparator for `LedgerKey` ordering.
`src/bucket/LedgerCmp.cpp`	Implements `LedgerKey` comparison logic used by merged scanning.
`src/bucket/BucketManager.h`	Adds an explicit `restartMerges` API.
`src/bucket/BucketManager.cpp`	Refactors merge restart out of `assumeState` into a separate method.
`src/bucket/BucketListSnapshot.h`	Adds snapshot API for scanning only current live entries of a type.
`src/bucket/BucketListSnapshot.cpp`	Implements the loser-tree/k-way merge scan over bucket entry streams.

bboston7

I'm by no means an expert on the bucket subsystem, but the algorithm implementation looks correct to me. I just had a few questions along the way.

bboston7 · 2026-06-01T17:33:37Z

+// Like LedgerEntryIdCmp, but only compares LedgerKeys, and does a 3-way
+// comparison instead of a less-than.
+std::partial_ordering compareLedgerKeys(LedgerKey const& a, LedgerKey const& b);


Why a partial ordering? Does a total ordering not exist for ledger keys?

The compare delegates to the operator <=> from xdrpp (which is important since we do need to match how LedgerEntryIdCmp does the ordering (and since, e.g., the value type in ScVal for CONTRACT_DATA is nested pretty deeply). I'll open a PR in xdrpp to fix it.

bboston7 · 2026-06-01T17:38:31Z

+SearchableLiveBucketListSnapshot::scanForLiveEntriesOfType(
+    LedgerEntryType type,
+    std::function<void(LedgerEntry const&, LedgerKey const&)> callback) const
+{


This should probably have a ZoneScoped

bboston7 · 2026-06-01T17:43:27Z

+    bool first = true;
+    LedgerKey last;
+    while (tree[0] != exhausted)
+    {
+        int index = tree[0];
+        auto& iter = iterators[index];
+        if (auto& key = iter.getKey(); first || key != last)
+        {
+            last = key;
+            auto& entry = iter.getEntry();
+            if (entry.type() == LIVEENTRY || entry.type() == INITENTRY)
+            {
+                callback(entry.liveEntry(), key);
+            }
+        }
+        first = false;
+
+        if (!iter.advance())
+        {
+            tree[index + numIterators] = exhausted;
+        }
+        int winner = tree[index + numIterators];
+
+        int i = (index + numIterators) / 2;
+        while (i > 0)
+        {
+            if (leftWins(tree[i], winner))
+            {
+                std::swap(tree[i], winner);
+            }
+            i /= 2;
+        }
+
+        tree[0] = winner;


Can you please add some comments to this? It's a little hard to figure out what's going on here. This is the actual k-way merge, right?

Agreed, it's still a little opaque.

- Add ZoneScoped - Add comments

SirTyson

Thanks for working on this, it looks like a nice change! It does look like we've added a fair bit of complexity to the scanning function, so I'd like to see some unit tests on the new algo and some randomized testing to make sure we've covered all our bases.

SirTyson · 2026-06-10T21:12:41Z

+} // namespace
+
+void
+SearchableLiveBucketListSnapshot::scanForLiveEntriesOfType(


I think this looks generally correct, but it's adding a good amount of complexity, I'd like to see a unit test specifically for this scanning function. Before it was pretty straight forward and indirectly tested, but given the k-way merge I think a more explicit test is warranted. Maybe we can test some of the loser tree edge cases, like a degenerate merge with just 1 bucket, 2 buckets, and some non powers of two. It might also be a good idea to hook this into the randomized bucket testing infra LedgerStateSnapshotTests,cpp or BucketIndexTests.cpp, where we just make sure we hit all the entries properly.

SirTyson · 2026-06-10T21:14:59Z

+
+        // Update tournament up the tree to the root
+        int i = (index + numIterators) / 2;
+        while (i > 0)


Nit: This while could be a for loop, which to me reads a little cleaner.

SirTyson · 2026-06-10T21:15:42Z

+    bool first = true;
+    LedgerKey last;
+    while (tree[0] != exhausted)
+    {
+        int index = tree[0];
+        auto& iter = iterators[index];
+        if (auto& key = iter.getKey(); first || key != last)
+        {
+            last = key;
+            auto& entry = iter.getEntry();
+            if (entry.type() == LIVEENTRY || entry.type() == INITENTRY)
+            {
+                callback(entry.liveEntry(), key);
+            }
+        }
+        first = false;
+
+        if (!iter.advance())
+        {
+            tree[index + numIterators] = exhausted;
+        }
+        int winner = tree[index + numIterators];
+
+        int i = (index + numIterators) / 2;
+        while (i > 0)
+        {
+            if (leftWins(tree[i], winner))
+            {
+                std::swap(tree[i], winner);
+            }
+            i /= 2;
+        }
+
+        tree[0] = winner;


Agreed, it's still a little opaque.

SirTyson · 2026-06-10T21:18:20Z

+        }
+    }
+
+    auto leftWins = [&iterators](int leftIndex, int rightIndex) -> bool {


More comments here would be helpful, this indicates that the smaller, newer version index wins, right?

SirTyson · 2026-06-10T21:21:47Z

-            return deletedKeys.find(lk) == deletedKeys.end();
+        auto contractDataHandler = [this](LedgerEntry const& le,
+                                          LedgerKey const&) {
+            createContractDataEntry(le);


Something I noticed in createContractDataEntry, we call xdr_size on le, which is a full recursive traversal of the SCVal for data types. Can we return this from readOne and just pipe it through? I remember you mentioned that XDR decode was a pretty significant bottleneck, that might be an easy win.

SirTyson · 2026-06-10T21:23:34Z

-            auto lk = LedgerEntryKey(be.liveEntry());
-            releaseAssertOrThrow(lk.type() == expectedType);
-            return deletedKeys.find(lk) == deletedKeys.end();
+        auto contractDataHandler = [this](LedgerEntry const& le,


Is it worth reserving mContractDataEntries and mCotnratCodeEntries here based on getRangeForType for the buckets?

SirTyson · 2026-06-10T21:26:29Z

+    {
+        return mKey;
+    }
+    bool advance();


Nit: Stylistically this is a little weird. Can we either inline the advance or just move the declaration to .h?

drebelsky added 2 commits May 5, 2026 10:21

Implement loser-tree merging for populating in-memory soroban state

e6d4a41

Delay restarting merges in startup path

9b5bee7

Copilot AI review requested due to automatic review settings May 5, 2026 18:17

Copilot started reviewing on behalf of drebelsky May 5, 2026 18:18 View session

Copilot AI reviewed May 5, 2026

View reviewed changes

Comment thread src/bucket/BucketListSnapshot.cpp Outdated

Ensure that BucketEntryIterator only yields entries of the proper type

c442d07

drebelsky requested a review from bboston7 May 11, 2026 20:58

bboston7 reviewed Jun 1, 2026

View reviewed changes

Address review feedback

6309ae3

- Add ZoneScoped - Add comments

drebelsky requested review from SirTyson and bboston7 June 5, 2026 17:56

SirTyson requested changes Jun 10, 2026

View reviewed changes

Conversation

drebelsky commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

bboston7 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SirTyson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

drebelsky commented May 5, 2026 •

edited

Loading