Disable LMDB value-cache eviction in new repo configs - #210
Open
tkuhn wants to merge 1 commit into
Open
Conversation
With eviction enabled (rdf4j's 60s default), unused value IDs are freed and then REUSED; a stale reference afterwards dereferences to a different value. That is the value-ID-remap corruption behind eight events on the fleet between 2026-08-20 and 2026-08-31 (eclipse-rdf4j/rdf4j#5970), where unrelated records shared identifiers, records were absorbed into others, and literals appeared in the wrong slots. Setting valueEvictionInterval to 0 disables the pass and is the workaround recommended upstream on 2026-08-31. It costs disk — dead values are never reclaimed — but not memory: gcIds() returns early when eviction is off, so nothing accumulates on the heap. Our stores are append-only apart from last30d pruning, so the disk cost is small. The eviction pass "is also run after opening the database" (ValueStore.java), which finally explains why every one of those eight events followed a restart of the affected store — something we had recorded repeatedly without a mechanism. This covers newly created shards only. Repositories that already exist keep the config.ttl they were created with and need a one-off sweep with rdf4j stopped. Also plumb NANOPUB_QUERY_FETCHING_SOCKET_TIMEOUT through docker-compose.yml. Utils.getHttpRequestConfig has always read it, but the query service declares an explicit environment list and no env_file, so setting it in .env did nothing. The 10s default is aggressive for a store that is slow rather than broken: the read raises QueryInterruptedException, the loader's retry loops treat that as transient, and the retries add load. The default is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Set
lmdb:valueEvictionInterval 0in both repository-config templates inTripleStore.java, and plumbNANOPUB_QUERY_FETCHING_SOCKET_TIMEOUTthroughdocker-compose.yml.Why
With eviction enabled (rdf4j's 60s default), unused value IDs are freed and then reused; a stale reference afterwards dereferences to a different value. That is the value-ID-remap corruption behind eight events on the fleet between 2026-08-20 and 2026-08-31 — unrelated nanopubs sharing identifiers, nanopubs absorbed into others, literals surfacing in the wrong slots.
Setting the interval to
0disables the pass. This is the workaround recommended upstream on 2026-08-31 while the underlying race is investigated.The eviction pass carries the comment "this is also run after opening the database" in
ValueStore.java. That explains something we had recorded repeatedly across those eight events without ever having a mechanism for it: every one of them followed a restart of the affected store. The trigger was the startup pass freeing and reusing IDs, not an unclean shutdown.Cost is disk, not memory. Dead values are never reclaimed or reused, but
gcIds()returns early when eviction is disabled, so nothing accumulates on the heap. Our stores are append-only apart fromlast30dpruning, so the disk cost is small.Scope — this covers new shards only
Repositories that already exist keep the
config.ttlthey were created with. Each host needs a one-off sweep of its ~2,800 configs with rdf4j stopped (it re-serialisesconfig.ttlon shutdown and would overwrite the edits). The sweep is independent of this release and does not depend on it. Script:kpxl-admin/scripts/disable-lmdb-value-eviction.sh.Because a restart with eviction still enabled is itself the exposure, the sweep should run before this release is deployed, not after.
Second change:
NANOPUB_QUERY_FETCHING_SOCKET_TIMEOUTUtils.getHttpRequestConfighas always read this variable, but thequeryservice declares an explicitenvironment:list and noenv_file, so setting it in.envsilently did nothing.The 10s default is aggressive for a store that is slow rather than broken: the read raises
QueryInterruptedException, the loader'swhile(!success)retry loops treat that as transient, and the retries add load, which makes the next read slower still. That loop produced ~17% HTTP 503 on federated/apion the OVH node on 2026-08-31 while status wasREADY(root cause there was a too-small heap; raising it cleared the 503s, but the amplification path remains).The default is unchanged at 10000 — this only makes it reachable.
Testing
517 tests pass. The config-template change is covered by existing repo-creation tests; the compose change validates with
docker compose config.Follow-up
Revisit once the upstream race is fixed — eviction exists for a reason and we are trading disk for safety.
🤖 Generated with Claude Code