Skip to content

[#1071] Wrap the sources of the large values PDBStorageTest writes, so that four copies of a 63 MB value are not live at once - #1072

Merged
vharseko merged 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:tests/pdb-large-values-peak
Sep 23, 2026
Merged

vharseko merged 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:tests/pdb-large-values-peak

Conversation

@vharseko

Copy link
Copy Markdown
Member

Fixes #1071.

PDBStorageTest.testCanAddLargeValues writes values of 4, 32 and 63 MB, and build-maven (ubuntu-latest, 17) of #1069 (run 35304975160) ran the 512 MB of the test JVM out on the 32 MB put, in Value.ensureFit() - a PR which touches nothing near the test, in a fork of its own (reuseForks=false), so the margin spent was the test's.

Why

Each value was copied four times on its way into Persistit: the source array, ByteString.valueOfBytes() (a copy), toByteArray() in PDBStorage.bytesToValue() (another), and the value buffer of the exchange, which doubles up to 64 MB. With the 63 MB value that is about 250 MB of humongous arrays live at once, on top of the buffer pool of the test storage (76 MB, 20% of the quota) and the test server.

The change

Test only: the three sources are ByteString.wrap()ped rather than copied, which takes one copy per value out. The three puts stay in one transaction on purpose - the value buffer the 32 MB one grows to fits the 63 MB one without growing again - and the javadoc says so.

Measured

JDK 17.0.20, -Xmx512m -Xlog:gc* - the settings of the leg - worst pause and humongous regions, before -> after the collection:

variant worst pause humongous regions
master 436M -> 311M 300 -> 177
the three puts in three writes 452M -> 324M 302 -> 112
this PR 471M -> 165M 328 -> 31

The split over three writes was measured first and rejected: a fresh exchange grows its value buffer again for each put, and the live set is no smaller. PDBStorageTest is 10/10 on this head, under JDK 17 and JDK 26.

Left out

The two copies in main - valueOfBytes() callers elsewhere and toByteArray() in bytesToValue() - are as they were; a ByteSequence gives no zero-copy access to its backing array, and this is the test's margin to fix. Raising the -Xmx512m of the failsafe fork in the root pom.xml would widen the margin for every class, and is a change of its own.

…torageTest writes, so that four copies of a 63 MB value are not live at once

testCanAddLargeValues built each source with ByteString.valueOfBytes(), which copies, and a
value on its way into Persistit is copied twice more - toByteArray() in bytesToValue() and
the value buffer of the exchange, which doubles up to 64 MB. With the 63 MB value that is
four copies live at once on top of the buffer pool and the server: about 310 MB after a
collection in the 512 MB of the test JVM, and build-maven (ubuntu-latest, 17) of OpenIdentityPlatform#1069 ran
out of heap on the 32 MB put. Wrapping the sources leaves about 165 MB.

Measured under JDK 17 with -Xlog:gc*: the worst pause goes from 436M->311M with 300->177
humongous regions to 471M->165M with 328->31. Splitting the three puts over three writes
does not help - the value buffer grows again for each - so they stay in one transaction.
@vharseko vharseko added bug tests Test suites: fixing, enabling, un-disabling java Changes to Java sources labels Sep 18, 2026

@maximthomas maximthomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praise: The fix is the smallest change that takes the pressure off the heap, and it leaves the case's content unchanged.

  • ByteString.valueOfBytes is wrap(Arrays.copyOf(bytes, bytes.length)) (ByteString.java:201), so wrap(new byte[..]) at PDBStorageTest.java:159-162 saves one transient 4/32/63 MB humongous allocation per put. Nothing touches those arrays again, so wrapping them is safe.
  • The three puts stay in one storage.write(...), so the 63 MB put reuses the 64 MB exchange buffer that the 32 MB put grew (Value.ensureFit) instead of allocating another one.
  • The -Xlog:gc* figures in the commit message (177 -> 31 humongous regions) make the effect checkable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug java Changes to Java sources tests Test suites: fixing, enabling, un-disabling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PDBStorageTest.testCanAddLargeValues runs the 512 MB test JVM out of heap: four copies of a 63 MB value live at once

2 participants