[#1063] Give back what the open reserved rather than what the configuration says by then, and ask for a restart when the cache size changes

[vharseko]

Summary

PDBStorage and JEStorage reserve their cache size from the server's MemoryQuota when they open and release it
when they close - both times by reading the configuration they hold at that moment. A change of db-cache-size or
db-cache-percent on a running backend swaps that configuration (applyConfigurationChange ends in config = cfg)
without touching the quota or the cache, and neither property is marked as needing a restart. So the storage reserves
the old size and releases the new one at the next close - the disable an online import-ldif makes included - and
the quota drifts by the difference for the life of the JVM: a cache grown from 64 MB to 128 MB leaves the quota
believing 64 MB free that the server does not have; a shrink leaves 64 MB reserved by nobody. The running cache is
the old size throughout, and dsconfig reports the change applied.

Fixes #1063.

What changes

Both storages keep two numbers of their own instead of reading config twice:

• configuredCacheSize - the cache size of the configuration the storage opened with (for PDB, what the buffer pool
  was built to);
• reservedCacheSize - of it, what the quota granted. acquireMemory is a tryAcquire: refused, it reserves
  nothing, and the return value used to be ignored in both buildConfigurations, so a close released a size that
  was never taken. That is reachable without any change - the server does not call isConfigurationAcceptable for
  the backends it opens at startup.

close() gives back reservedCacheSize. isConfigurationChangeAcceptable admits a growth for the difference to
what is held rather than to config: once a change has been admitted but not applied, computeSize(config) is
already the new size while the reservation is the old one, and a second change was admitted for a difference nobody
would reserve. A size which does not grow past the one configured asks the quota for nothing, as before
(newSize <= max(reservedCacheSize, computeSize(config))): every change listener of the backend entry is asked about
every change, so after an open the quota refused - nothing held - a change of any other property, a disable, and the
disable TaskUtils.disableBackend makes for an online import, rebuild or restore would otherwise be refused.
applyConfigurationChange on an open storage whose cache size the change moves sets adminActionRequired and adds
NOTE_CONFIG_DB_CACHE_REQUIRES_RESTART (630), naming the size the backend runs with and the one now configured.

db-cache-size and db-cache-percent are marked component-restart in PDBBackendConfiguration.xml and
JEBackendConfiguration.xml, as db-directory is in the same files, so dsconfig says so too.

Why a restart rather than a resize

PersistIt sizes its buffer pool when the database opens and has no way to resize it (Persistit.setConfiguration
refuses a second configuration). JE could - je.maxMemory and je.maxMemoryPercent are mutable through
Environment.setMutableConfig - but JEStorage has never resized its environment, and it is not alone: none of the
JE properties the pluggable backend left without requires-admin-action is applied live since OPENDJ-1719 dropped
the setMutableConfig road of the old backend. Restoring that road for all of them is #1068; here the two
storages take the same shape, and the quota follows the cache that actually runs.

On master

The fix builds on the close() of #999 - the quota given back once, and since its last round after the database has
closed, in a finally - and on JEStorageTest, which #999 introduces. #999 is merged; the branch sits on master,
and the give-back of what the open reserved is the one inside that finally. Rebased onto master 0e039c6473
for round 1 of the review: the one conflict was JEStorageTest, where the replay cases of #1065 added their
imports and constants at the same places - a union, the test bodies merged on their own. #1070 (#1067) now asks
the quota in validateDbCacheSize instead of taking from it, leaving the reservation to the storage, as here.
Two commits: the fix, and round 1 on top.

Tests

PDBStorageTest and JEStorageTest, eleven cases each, over a mocked ServerContext with a fresh MemoryQuota:

• aCacheGrownWhileOpenIsGivenBackAsItWasTaken, aCacheShrunkWhileOpenIsGivenBackAsItWasTaken - the quota is
  back where it started after open, change, close;
• aCacheSizeChangedWhileOpenAsksForARestart - adminActionRequired, the message id and both sizes; a change
  back to the size opened with asks for nothing;
• aChangeWhichLeavesTheCacheSizeAloneAsksForNothing - a change of another property asks for nothing, as before;
• aCacheSizedByPercentAsksForARestartOnlyWhenThePercentChanges - db-cache-size 0 at percent 10: another
  property asks for nothing, percent 20 names memPercentToBytes(10) and memPercentToBytes(20);
• aStorageWhichIsNotOpenAsksForNoRestart - a storage constructed but never opened;
• aCacheSizeChangeIsAdmittedAgainstWhatTheStorageHolds - with 64 MB held and a change to 128 MB pending, 256 MB is
  refused and 192 MB admitted at 129 MB free;
• aChangeWhichLeavesTheCacheSizeAloneIsAdmittedAfterARefusedReservation - nothing held, 32 MB free, a
  db-txn-no-sync change is admitted;
• aGrowthAfterARefusedReservationIsMeasuredAgainstNothingHeld - same state, 80 MB is refused;
• aShrinkIsAdmittedWithTheQuotaExhausted - 64 MB held, 0 free, 32 MB is admitted;
• aReservationTheQuotaRefusedIsNotGivenBackOnClose - 32 MB free, a 64 MB cache opens, and the close leaves 32 MB.

Five of the six were red on the head of #999 before the fix (the sixth pins existing behaviour): +64 MB, −64 MB,
admission of 256 MB, no admin action, 96 MB after the close of a refused reservation.

Mutants, each run against both classes: close() releasing configuredCacheSize instead of the reserved size
(the refused-reservation case red, nothing else), admission against computeSize(config) (the admission case),
setAdminActionRequired dropped (the restart case), and the old release by config (grow, shrink and the refused
reservation). Round 1, on master 0e039c6473, one JVM per run: the admission of the first commit
(newSize <= reservedCacheSize), reservedCacheSize → configuredCacheSize in the admission, the short circuit
dropped, newCacheSize = cfg.getDBCacheSize(), the opened-with baseline moved after the note, and the open guard
dropped - each red on exactly its own case(s) in both classes. On the round head: PDBStorageTest 25/25,
JEStorageTest 22/22 (with the replay cases of #1065), FailedBackendOpenTest 8/8, BackendConfigManagerTestCase
11/11.

Regression set (one JVM per class, the fixed storages first on the classpath): FailedBackendOpenTest,
PDBTestCase, EncryptedPDBTestCase, JETestCase, EncryptedJETestCase, ReplayedConfigChangeTest,
OnDiskMergeImporterTest, PersistentCompressedSchemaTest, DN2IDTest, StateTest, ID2EntryTest,
ID2ChildrenCountTest, BulkCursorTest, DefaultIndexTest, ImportLDIFTestCase, RebuildIndexTestCase,
VerifyIndexTestCase, BackendConfigManagerTestCase - 240 tests, 0 failures (FailedBackendOpenTest and
PDBTestCase re-run after a collision on the admin port 65534 with another JVM on the machine).

Not in this PR

• A JE backend with an explicit db-cache-size takes that size from the server's memory quota twice per open and gives it back once #1067 - ConfigurableEnvironment.validateDbCacheSize took db-cache-size from the server's quota as a
  probe and never gave it back; fixed on master by [#1067] Ask the memory quota whether an explicit db-cache-size fits instead of taking it #1070.
• JE and PDB backend properties without requires-admin-action are neither applied to a running backend nor reported as needing a restart #1068 - the other JE properties (and PDB's db-checkpointer-wakeup-interval) that are neither applied live
  nor marked as needing a restart.

[vharseko]

Rebased onto master now that #999 is merged (1af0a1247d): its last round moved the quota give-back of close() into a finally after env.close() / db.close(), and the give-back of what the open reserved sits inside that block now (JEStorage.java, PDBStorage.java) - git range-diff against the previous head shows that one hunk and nothing else. Head is 725ff9af7d, one commit on master; the "Stacked on #999" section of the description is rewritten accordingly.

Re-run green on the rebased head: PDBStorageTest (20), JEStorageTest (9), FailedBackendOpenTest (8), PDBTestCase / JETestCase (35 each), BackendConfigManagerTestCase (11).

[vharseko]

Rebased onto master once more, now that #994 is merged (739ea68b7c): 725ff9af7d → 64f3cffd00. The one conflict was the tail of backend.properties — 624-627 of #994 now sit before 630; the range-diff is clean apart from that context, and the commit is otherwise the same. JEStorageTest and PDBStorageTest on the new head: 29/29 green. Description updated ("On master").

[vharseko]

Rebased onto master (7cebc65e3c) now that #998, #1003, #1004, #1015 and #1016 are merged: 64f3cffd00 → bd9d9d133f. The one conflict was the tail of backend.properties again - 628-629 of #998 (WARN_INDEX_ADD_DISCARDED_LEFTOVER_TREES, ERR_CONFIG_INDEX_ATTRIBUTE_ALREADY_INDEXED) now precede 630, which keeps its ordinal; git range-diff against the previous head shows that context and nothing else. #998 also made JEStorage's constructor public, which merged with the changes here without a conflict and is in place on the new head.

The restack carries no change of its own, so the suites are not re-run for it; test-compile of the reactor on the new head is green, and CI runs on the head. Description updated ("On master").

[maximthomas]

praise: The quota now follows the cache that actually runs, instead of the configuration as it stands at close.

• close() gives back reservedCacheSize in both PDBStorage and JEStorage, and buildConfiguration finally honours the result of acquireMemory (PDBStorage.java:1101, JEStorage.java:778).
• computeSize() reads serverContext.getMemoryQuota() rather than the memQuota field, which is null until the first open.
• The restart is declared in two places: component-restart in both XMLs, and adminActionRequired plus NOTE 630 in the ConfigChangeResult.

---

issue (blocking): After the quota refuses the open's reservation, isConfigurationChangeAcceptable refuses every change to the backend entry, including a change that leaves the cache size alone.

opendj-server-legacy/src/main/java/org/opends/server/backends/pdb/PDBStorage.java:1556, opendj-server-legacy/src/main/java/org/opends/server/backends/jeb/JEStorage.java:1286

A refused reservation leaves reservedCacheSize == 0 while the storage stays open. From then on, newSize <= reservedCacheSize is false for every change. The check falls through to isMemoryAvailable(newSize), which asks for the same amount the quota just refused, and it adds no reason. ConfigurationHandler.replaceEntry asks every change listener on the entry, and nothing filters by property (ConfigurationHandler.java:620-627, ConfigChangeListenerAdaptor.java:339-345). So the following all fail with UNWILLING_TO_PERFORM and an empty reason:

• a db-txn-no-sync change;
• dsconfig set-backend-prop --set enabled:false;
• the internal ds-cfg-enabled: false modify that TaskUtils.disableBackend makes for an online import-ldif, rebuild-index or restore (TaskUtils.java:203-210).

BASE accepted all of these through newSize <= computeSize(config). Startup opens are not checked against the quota, so this state needs no config change. It comes up with a third PDB/JE backend at the default db-cache-percent 50. It also comes up with one JE backend whose db-cache-size is more than half the reservable pool, because validateDbCacheSize has already taken that size and kept it (#1067). A server restart gets back to the same state.

    final long newSize = computeSize(newCfg);
    final MemoryQuota quota = serverContext.getMemoryQuota();
    // What does not grow past the size already configured asks the quota for nothing (BASE's rule);
    // a growth is measured against what this storage holds, which is what the next open adds to.
    return (newSize <= Math.max(reservedCacheSize, computeSize(config))
            || quota.isMemoryAvailable(newSize - reservedCacheSize))
        && checkConfigurationDirectories(newCfg, unacceptableReasons);

aCacheSizeChangeIsAdmittedAgainstWhatTheStorageHolds keeps its outcome under this fix: with 64 MB held and 128 MB pending, 256 MB is refused and 192 MB admitted at 129 MB free.

Pin: repeat the open of aReservationTheQuotaRefusedIsNotGivenBackOnClose (32 MB free, a 64 MB cache), then assert that a db-txn-no-sync-only change is acceptable. Do this in both classes. The assertion is red at this head.

    final PDBBackendCfg unchangedCache = createBackendCfg(SMALL_CACHE);
    when(unchangedCache.isDBTxnNoSync()).thenReturn(true);
    assertThat(storage.isConfigurationChangeAcceptable(unchangedCache, new ArrayList<LocalizableMessage>())).isTrue();

---

suggestion (non-blocking): Admission is only asserted where the reservation equals the configured size. A refused open, an unchanged size and a shrink are never checked.

opendj-server-legacy/src/test/java/org/opends/server/backends/pdb/PDBStorageTest.java:638, :663, opendj-server-legacy/src/test/java/org/opends/server/backends/jeb/JEStorageTest.java:311, :334

The only calls to isConfigurationChangeAcceptable come after a granted 64 MB open, where reservedCacheSize == configuredCacheSize. Two mutants survive all six cases in both classes (found by reading; not run):

• reservedCacheSize replaced by configuredCacheSize in the admission line;
• the newSize <= reservedCacheSize || short circuit dropped. In production that sends every shrink to Semaphore.tryAcquire(negative), which throws IllegalArgumentException.

The blocking issue above ships green for the same reason.

Pin: after the refused open of aReservationTheQuotaRefusedIsNotGivenBackOnClose, assert that createBackendCfg(SMALL_CACHE + SMALL_CACHE / 4) is not acceptable. It is 16 MB above what is configured but 80 MB above what is held, which kills the reserved → configured swap. Then, after a granted open with the quota drained to 0 free, assert that createBackendCfg(SMALL_CACHE / 2) is acceptable. That kills the dropped short circuit.

---

suggestion (non-blocking): The percent arm of the restart note is not pinned. Every case sizes the cache with db-cache-size.

opendj-server-legacy/src/test/java/org/opends/server/backends/pdb/PDBStorageTest.java:616

Each applyConfigurationChange call in the class (:570, :586, :604, :624, :644) uses createBackendCfg(n * SMALL_CACHE). None goes through db-cache-percent, which is the shipped default (size 0, percent 50). The mutant newCacheSize = cfg.getDBCacheSize() at PDBStorage.java:1649 survives. On a percent-sized backend, that mutant makes every unrelated change set adminActionRequired and emit a note saying "X bytes … 0 bytes".

Pin: open with createBackendCfg(0) and getDBCachePercent() stubbed to 10. Assert that a db-txn-no-sync-only change at percent 10 leaves adminActionRequired() false, and that a change to percent 20 sets it and names memPercentToBytes(10) and memPercentToBytes(20).

---

suggestion (non-blocking): No test shows that the "opened with" baseline stays fixed across two changes.

opendj-server-legacy/src/test/java/org/opends/server/backends/pdb/PDBStorageTest.java:598

Every restart-note case applies a single change to a storage that was just opened. So the mutant configuredCacheSize = newCacheSize; after the addMessage survives. In production, that mutant turns open 64 → change 128 → change back to 64 into a restart note for a pool that still runs at 64.

    final ConfigChangeResult back = storage.applyConfigurationChange(createBackendCfg(SMALL_CACHE));
    assertThat(back.adminActionRequired()).isFalse();
    assertThat(back.getMessages()).isEmpty();

Pin: append this to aCacheSizeChangedWhileOpenAsksForARestart in both classes.

---

suggestion (non-blocking): The open guard of the restart note (db != null / env != null) is not pinned.

opendj-server-legacy/src/test/java/org/opends/server/backends/jeb/JEStorageTest.java:271, opendj-server-legacy/src/main/java/org/opends/server/backends/pdb/PDBStorage.java:1650, opendj-server-legacy/src/main/java/org/opends/server/backends/jeb/JEStorage.java:1380

Every case that calls applyConfigurationChange opens the storage first. The mutant that drops the guard survives both classes. On a storage that has been constructed but never opened, the listener is already registered, and the mutant sets adminActionRequired and names "0 bytes opened with". The effect is small: the apply then fails in registerMonitoredDirectory anyway, at BASE as at this head.

Pin: construct a storage without opening it, apply a config with a different cache size, and assert that adminActionRequired() is false and that NOTE 630 is not among the messages.

---

issue (non-blocking): NOTE 630 calls the opened-with size "reserved", including when the quota refused the reservation.

opendj-server-legacy/src/messages/org/opends/messages/backend.properties:1173, opendj-server-legacy/src/main/java/org/opends/server/backends/pdb/PDBStorage.java:1657, opendj-server-legacy/src/main/java/org/opends/server/backends/jeb/JEStorage.java:1387

Both callers pass configuredCacheSize into "the memory reserved for it … stay at the %d bytes". After a refused open the storage holds 0 bytes of the quota. A later cache change that the quota now admits therefore reports memory as reserved when none is. On the granted road the text is accurate.

NOTE_CONFIG_DB_CACHE_REQUIRES_RESTART_630=The change to the database cache of backend %s will not take effect \
 until the backend is restarted: the cache the backend runs with stays at the %d bytes it was opened with until \
 then, and the %d bytes now configured are reserved by the next open

[vharseko]

Thanks, all six taken. Round head: cba9d511b5, one commit on top of the PR commit, which I first rebased onto master 0e039c6473. The one conflict was JEStorageTest against #1065, which added its replay cases at the same imports and constants; I resolved it as a union, the test bodies merged on their own, and the commit message is unchanged. #1070 now asks the quota in validateDbCacheSize rather than taking from it, which leaves the reservation to the storage, as this PR assumes.

Blocking: a refused reservation refuses every change of the backend entry. Confirmed by reading. ConfigurationHandler.replaceEntry asks every change listener, and TaskUtils.disableBackend goes through it, so an online import-ldif, rebuild-index or restore on such a backend fails with UNWILLING_TO_PERFORM and an empty reason. I used your rule as is, in both storages (PDBStorage.isConfigurationChangeAcceptable, JEStorage.isConfigurationChangeAcceptable):

return (newSize <= Math.max(reservedCacheSize, computeSize(config))
        || quota.isMemoryAvailable(newSize - reservedCacheSize))
    && checkConfigurationDirectories(newCfg, unacceptableReasons);

The comment above it now explains why a change that does not grow the size asks the quota for nothing. aCacheSizeChangeIsAdmittedAgainstWhatTheStorageHolds keeps its outcome. The pin is aChangeWhichLeavesTheCacheSizeAloneIsAdmittedAfterARefusedReservation, in both classes.

Admission pins. aGrowthAfterARefusedReservationIsMeasuredAgainstNothingHeld (80 MB, nothing held, 32 MB free → refused) and aShrinkIsAdmittedWithTheQuotaExhausted (granted 64 MB open, quota drained to 0, 32 MB → admitted). The refused open is shared through openWithTheReservationRefused().

Percent arm. aCacheSizedByPercentAsksForARestartOnlyWhenThePercentChanges: open at db-cache-size 0 and percent 10. A db-txn-no-sync-only change asks for nothing. Percent 20 asks for a restart and names memPercentToBytes(10) and memPercentToBytes(20). createBackendCfg gained a (size, percent) overload for it.

Fixed baseline across two changes. Your snippet is appended to aCacheSizeChangedWhileOpenAsksForARestart: 64 → 128 → 64, and the second apply asks for nothing.

Open guard. aStorageWhichIsNotOpenAsksForNoRestart: a storage that was constructed but never opened. adminActionRequired() is false and no message carries the ordinal of NOTE 630. The apply still records the registerMonitoredDirectory failure you mentioned. The test does not assert on it and closes the storage in a finally.

NOTE 630. Reworded to your text. The number and the arguments are unchanged:

NOTE_CONFIG_DB_CACHE_REQUIRES_RESTART_630=The change to the database cache of backend %s will not take effect \
 until the backend is restarted: the cache the backend runs with stays at the %d bytes it was opened with until \
 then, and the %d bytes now configured are reserved by the next open

Runs. PDBStorageTest 25/25 and JEStorageTest 22/22 on the round head (reactor verify; the JE count includes the eight replay cases of #1065), plus FailedBackendOpenTest 8/8 and BackendConfigManagerTestCase 11/11. Mutants, one JVM per run, each against both classes:

mutant	PDBStorageTest	JEStorageTest
admission back to newSize <= reservedCacheSize (the previous head)	aChangeWhichLeavesTheCacheSizeAloneIsAdmittedAfterARefusedReservation	same
reservedCacheSize → configuredCacheSize in the admission	aGrowthAfterARefusedReservationIsMeasuredAgainstNothingHeld	same
short circuit dropped	aChangeWhichLeavesTheCacheSizeAloneIsAdmittedAfterARefusedReservation, aShrinkIsAdmittedWithTheQuotaExhausted (IAE from tryAcquire)	same
newCacheSize = cfg.getDBCacheSize()	aCacheSizedByPercentAsksForARestartOnlyWhenThePercentChanges	same
configuredCacheSize = newCacheSize after the note	aCacheSizeChangedWhileOpenAsksForARestart	same
db != null / env != null dropped	aStorageWhichIsNotOpenAsksForNoRestart	same

Each mutant is red on exactly the case(s) listed and green elsewhere.