[#1087] Copy the secret volume on every start of the Docker image, and run the server as PID 1 after the bootstrap - #1100
Conversation
f52171e to
9cc4413
Compare
|
@maximthomas, a round before your review: the first CI run of this PR failed in What failed. The existing "Docker test" step runs What changed.
Verified. All three docker steps of the workflow, run locally the way Actions runs them ( The PR description is updated to match. |
481572a to
631d048
Compare
|
@maximthomas, rebased onto the current The merge of #1091 made the second commit ( What #1091 changes for this PR: its |
maximthomas
left a comment
maximthomas
left a comment
There was a problem hiding this comment.
praise: the start sequence is right where #1087 and #1085 broke it.
./bin/stop-dsruns before.bootstrap-completeis written (run.sh:134-140), so the health check never reports the bootstrap's server healthy just before it goes down.copy_secretswrites each file next to its target withmktemp+chmod 600+mv -f(run.sh:56): the server never reads a half-copied keystore, and the key is readable by its user only.- Both roads end in one
start_server(run.sh:73-83), so the first start and every later one copy the volume the same way.
issue (blocking): the secret-volume step greps CN=secret-v1, but the runner's openssl x509 -subject prints CN = secret-v1.
.github/workflows/build.yml:588, :594, :605, :803, :809, :820
Both docker jobs run on ubuntu-latest (ubuntu-24.04), whose /usr/bin/openssl is 3.0.13. There get_nameopt() defaults to XN_FLAG_ONELINE (apps/lib/apps.c:198-200 at openssl-3.0.13), which includes XN_FLAG_SPC_EQ, so served() returns subject=CN = secret-v1. grep -q "CN=secret-v1" exits 1 under bash -eo pipefail, and build-docker and build-docker-alpine go red at the first certificate check, whatever LDAPS serves. The no-space form is the 3.2+ default (openssl-3.2.0 apps.c:189-192), which is what a Homebrew openssl prints locally. The step has not run in CI yet: run 36013203998 stopped at "Docker test".
served() { echo | openssl s_client -connect 127.0.0.1:1637 2>/dev/null | openssl x509 -noout -subject -nameopt RFC2253; }Or: grep -Eq 'CN ?= ?secret-v1' at each check. Same change in both jobs.
issue (blocking): the cert-manager recipe gives a keystore whose key has alias 1, not server-cert, so the LDAPS handler disables itself.
opendj-packages/opendj-docker/README.md:43, :66-83
Setup's self-signed options set the LDAPS handler's ssl-cert-nickname to server-cert (SecurityOptions.java:37, :104). cert-manager builds keystore.p12 with go-pkcs12 Encode, which puts only a localKeyID on the key bag and no friendlyName, and the JDK loads such a store (as PKCS12 or as JKS) with aliases=[1]. Once run.sh copies it over config/keystore, LDAPConnectionHandler.createSSLContext (:1330-1344) logs ERR_KEYSTORE_DOES_NOT_CONTAIN_ALIAS, drops the alias and disables the handler: a pod that follows the recipe serves no LDAPS, and the image's HEALTHCHECK, which probes LDAPS, never passes. CI misses this because its keystore() uses -alias server-cert. The same gap is in the prose at :43: nothing says which alias the key entry needs.
# the Certificate
spec:
secretName: opendj-tls
keystores:
jks:
create: true
alias: server-cert
passwordSecretRef: { name: opendj-keystore-password, key: password }
---
# the pod template of the StatefulSet
volumes:
- name: secrets
projected:
sources:
- secret:
name: opendj-tls
items: [{ key: keystore.jks, path: keystore }]
- secret:
name: opendj-keystore-password
items: [{ key: password, path: keystore.pin }]Also state at :43 that the key must be under alias server-cert. keystores.jks.alias needs cert-manager 1.15+. The other option is to set the LDAPS handler's ssl-cert-nickname to the alias the keystore actually has.
issue (non-blocking): nothing tests the copy on the restart road: the watcher has already copied v2 before docker stop.
.github/workflows/build.yml:598-605, :813-820; opendj-packages/opendj-docker/run.sh:92-100
The step waits at :598 until the watcher (SECRET_VOLUME_REFRESH=5) has copied keystore v2 into data/config, then stops and starts the container. At docker start, copy_secrets finds the files equal and copies nothing. A mutant whose [ -d ./data/config ] branch runs BASE's bare exec ./bin/start-ds --nodetach still serves CN=secret-v2 and stays green in both jobs. That is the restart-road half of #1087 ("later starts exec'ed the server before the copy"), and this step is its only test. Pin: recreate the container on the same data volume with the watcher off and a keystore renewed while it was down (after the -nameopt fix above):
docker rm -f test_secret
keystore secret-v3
docker run -d --memory="512m" -e SECRET_VOLUME_REFRESH=0 -p 127.0.0.1:1637:1636 --name=test_secret \
-v "$SECRETS":/var/secrets/opendj:ro -v test_secret_data:/opt/opendj/data "$IMAGE"
healthy
grep -q "CN=secret-v3" <<< "$(served)"Pin: the restart branch without copy_secrets serves v2, and grep -q "CN=secret-v3" goes red.
suggestion (non-blocking): no test runs with SECRET_VOLUME_REFRESH=0, so nothing pins the -gt 0 guard.
opendj-packages/opendj-docker/run.sh:77; .github/workflows/build.yml:591, :806
SECRET_VOLUME_REFRESH appears in CI only as =5. If the guard is dropped, or turned into -ge, 0 becomes while sleep 0; do copy_secrets; done: a busy loop, where the README says 0 means "copies them on start only". Continue in the REFRESH=0 container from the previous comment. The negation goes inside sh -c because bash -e ignores a failing ! cmd:
keystore secret-v4
sleep 15
docker exec test_secret sh -c '! cmp -s /var/secrets/opendj/keystore /opt/opendj/data/config/keystore'Pin: with the guard dropped or -ge 0, the busy loop copies v4 within the 15 s and the docker exec goes red.
suggestion (non-blocking): a SECRET_VOLUME_REFRESH that is not a whole number turns the watcher off and logs nothing.
opendj-packages/opendj-docker/run.sh:75-79
[ "$SECRET_VOLUME_REFRESH" -gt 0 ] 2>/dev/null returns 2 for 1m, 60s and 30.5, all of which sleep would accept, and the error goes to /dev/null. The log line at :75 is printed either way. An operator who writes 1m gets no refresh and no hint of it.
if [ "$SECRET_VOLUME_REFRESH" -gt 0 ] 2>/dev/null; then
watch_secrets &
elif [ "$SECRET_VOLUME_REFRESH" != 0 ]; then
echo "SECRET_VOLUME_REFRESH=$SECRET_VOLUME_REFRESH is not a whole number of seconds, the secret volume is copied on start only"
fisuggestion (non-blocking): if the keystore password changes while the server runs, the server keeps the old PIN in memory, and it no longer opens the copied keystore.
opendj-packages/opendj-docker/run.sh:65-68
FileBasedKeyManagerProvider sets keyStorePIN only at init or on a config change (:90), but getKeystore() reads the keystore file again on every call (:129-137). After the watcher copies a keystore with a new password, a dsconfig change to the LDAPS, StartTLS or HTTPS handler rebuilds the SSL context from keystore v2 with PIN v1. The change is rejected (ERR_FILE_KEYMANAGER_CANNOT_LOAD) until the next restart. Serving is not affected. This cannot happen with the recipe's fixed passwordSecretRef. Two options: the watcher skips a pass in which keystore.pin differs from the one in data/config (the copy at the next start applies it, and that is when it takes effect anyway), or the README says that a new keystore password needs a restart.
suggestion (non-blocking): on the first start the JVM is now PID 1, and it does not reap orphans left by a health probe that timed out.
opendj-packages/opendj-docker/run.sh:142, :82; opendj-packages/opendj-docker/Dockerfile:83
The HEALTHCHECK is in shell form and ends in || exit 1, so sh forks ldapsearch, and _client-script.sh forks java. When a probe runs past --timeout=30s, Docker sends SIGKILL to the exec'd sh only. The rest of the chain is reparented to PID 1, and a JVM never calls waitpid, so each orphan becomes a zombie. At BASE, bash on tail -f /dev/null reaped them on the first lifetime. The restart road already had java as PID 1. Each zombie costs one pid slot, and Kubernetes does not use this HEALTHCHECK. One README line would do: recommend docker run --init, which puts a PID 1 there that reaps and forwards SIGTERM.
suggestion (non-blocking): keystore and keystore.pin are compared and renamed one at a time, so one pass can mix two versions of the Secret.
opendj-packages/opendj-docker/run.sh:52-56
If the projected volume's ..data swap lands between the two cmp calls during a password rotation, data/config holds keystore v1 with pin v2 until the next pass. Today this can bite only a start that straddles the swap. It matters once #1101 makes an in-run copy live: compare the whole key*/trust* set first, and copy only when a second compare agrees.
|
@maximthomas, thanks for the review. Round 4 is in c099b70. Seven of the eight points are taken; the cert-manager alias one did not reproduce, details below.
The cert-manager keystore with alias Measured: I built a PKCS12 with What I did take from it is the gap in the prose. The restart road was not tested (non-blocking): taken. After the restart with v2, the step now stops and removes the container. It then renews the keystore to v3 while no container runs, creates a new container on the same data volume with
Both pins were checked against mutants, running the step locally the way Actions does (
A new keystore password while the server runs (non-blocking): taken as a README sentence. "The same holds for a new keystore password: the server keeps the one it started with." With #1101, a handshake after the change reads the PIN again ( PID 1 and orphaned probes (non-blocking): taken. The Health check section of the README now recommends
The PR description is updated to match. Edited: the first version said that #1101 reloads the store with the same |
c099b70 to
4abcc52
Compare
|
@maximthomas, rebased onto the current The merge of #1094 made two commits conflict:
The third and the fourth commit applied cleanly and are unchanged. |
maximthomas
left a comment
maximthomas
left a comment
There was a problem hiding this comment.
praise: Round 1's blocking points are settled, and the new REFRESH=0 run pins the copy on an existing instance.
served()now pipes throughopenssl x509 -noout -subject -nameopt RFC2253, soCN=secret-vNmatches on the runner's OpenSSL 3.0 too.- The REFRESH=0 container, created again on
test_secret_dataafterkeystore secret-v3, pins the start-time copy over an existing instance (your mutant table: red atCN=secret-v3). - The README's alias paragraph is right, and my round-1 issue on the cert-manager alias is withdrawn:
Installer.addCertificateArgumentspasses no nickname forSELF_SIGNED_CERTIFICATE, andConfigureDSthen removesds-cfg-ssl-cert-nicknamefrom the LDAP/LDAPS/HTTP/JMX handlers, so a key under alias1is served.
issue (non-blocking): The ERR trap of "Docker test secret volume" does not run for a failure inside healthy() or keystore().
.github/workflows/build.yml:627, :630-638, :901, :904-912
Actions runs the step as bash -eo pipefail without -E. A function does not inherit the ERR trap, and errexit ends the shell from inside the function. So a container that never turns healthy (timeout exits 124), or a failed keytool/mv, fails the step without the docker logs test_secret the trap is there to print. I checked this locally: bash -eo pipefail -c 'trap "echo ERR-TRAP-RAN" ERR; h() { bash -c "exit 124"; }; h' exits 124 and prints nothing, and with set -E the trap prints. Your other docker steps call timeout at top level, so their traps do run.
set -E
trap 'code=$?; echo "::group::container logs (test_secret)"; docker logs test_secret 2>&1 || true; echo "::endgroup::"; exit $code' ERRissue (non-blocking): If the keystore password changes while the server runs, the next dsconfig change to the LDAPS handler shuts LDAPS down. The new README sentence does not warn about this.
opendj-packages/opendj-docker/run.sh:79-83, opendj-packages/opendj-docker/README.md:64-65
watch_secrets is on by default and copies keystore and keystore.pin into data/config while the server runs. FileBasedKeyManagerProvider reads the PIN only at init (:90), but it reloads the keystore file on every getKeystore() (:129-138). A dsconfig modify of the LDAPS handler goes through LDAPConnectionHandler2.isConfigurationAcceptable:584 → createSSLContext. There containsAtLeastOneKey swallows the wrong-password failure, and disableAndWarnIfUseSSL sets enabled = false on the live handler before the change is refused. run() then stops the listener, and LDAPS stays closed until a restart. The flag flip during the dry run is an older server bug, but before this PR nothing in the image rewrote data/config under a running server. You can fix this in either of two ways. One: the watcher leaves a changed password to the next start, which is when it takes effect anyway. Two: the README says to restart before any change to a TLS connection handler.
pins_unchanged() {
local src
for src in "$SECRET_VOLUME"/*.pin; do
[ -f "$src" ] || continue
cmp -s "$src" "./data/config/$(basename -- "$src")" || return 1
done
}
watch_secrets() {
while sleep "$SECRET_VOLUME_REFRESH"; do
pins_unchanged && sync_secrets
done
}suggestion (non-blocking): The secret-volume step never changes keystore.pin and never mounts a truststore. So the sync_secrets repeat, the trust* glob and the skip of unchanged files are not pinned.
.github/workflows/build.yml:630-636, :904-910; opendj-packages/opendj-docker/run.sh:56-76
Three run.sh mutants pass both images' steps:
sync_secrets() { copy_secrets; }. One pass copies both changed files.- Dropping
"$SECRET_VOLUME"/trust*from:56. - Deleting
cmp -s "$src" "$dst" && continueat:59. Every start and every 5 s tick then copies identical bytes five times, and no check looks at that.
A truststore in the volume pins the glob. Counting the copy log line pins the skip. The repeat needs a Secret that changes in the middle of a pass, which CI cannot produce on demand. Not run here (no Docker daemon).
keystore() {
# ... as now, then:
docker run --rm --entrypoint keytool -v "$SECRETS":/secrets "$IMAGE" -exportcert -rfc -alias server-cert \
-keystore /secrets/keystore -storepass changeit -file /secrets/cert.pem
rm -f "$SECRETS/truststore.new"
docker run --rm --entrypoint keytool -v "$SECRETS":/secrets "$IMAGE" -importcert -noprompt -alias ca \
-file /secrets/cert.pem -keystore /secrets/truststore.new -storetype JKS -storepass changeit
mv -f "$SECRETS/truststore.new" "$SECRETS/truststore"
printf changeit > "$SECRETS/truststore.pin"
rm -f "$SECRETS/cert.pem"
}
# after the first `healthy`:
docker exec test_secret cmp -s /var/secrets/opendj/truststore /opt/opendj/data/config/truststore
sleep 12
test "$(docker logs test_secret 2>&1 | grep -c '^Copied keystore from the secret volume$')" -eq 1Pin: the trust* mutant goes red at cmp, and the mutant without the skip goes red at the count (5 or more instead of 1).
suggestion (non-blocking): Every CI keystore puts its key under alias server-cert, so nothing in CI pins the README's "the key entry may have any alias".
.github/workflows/build.yml:631, :905; opendj-packages/opendj-docker/README.md:50-52
server-cert is also the name setup gives its own key. Suppose a regression binds the handlers to that name, for example a -a server-cert in the SELF_SIGNED_CERTIFICATE case or a --certNickname server-cert in the default OPENDJ_SSL_OPTIONS. CI would still serve CN=secret-v1..v3. A cert-manager keystore.p12 (alias 1) would lose LDAPS. The "alias 1 is served" result in the PR body came from a local run.
docker run --rm --entrypoint keytool -v "$SECRETS":/secrets "$IMAGE" -genkeypair -alias "$1" \Pin: with a per-version alias, the handler-binding mutant fails at CN=secret-v1. If you also take the truststore pin above, use -alias "$1" in its -exportcert.
suggestion (non-blocking): The SECRET_VOLUME_REFRESH=0 check waits 15 s, and the default interval is 60 s. A mutant that treats 0 as "use the default" passes.
.github/workflows/build.yml:663-664, :937-938
Take [ "$SECRET_VOLUME_REFRESH" -gt 0 ] 2>/dev/null || SECRET_VOLUME_REFRESH=60 added after run.sh:47. Its watcher ticks at t0+60 s, and the ! cmp runs about 35-45 s after t0. So only the busy-loop mutant from your table (-ge 0) is caught. Asking the container whether a watcher exists answers this at once: with 0 no sleep process runs, and with any watcher one nearly always does. Keep the ! cmp for the busy loop. Not run here.
docker exec test_secret sh -c '! grep -sqx sleep /proc/[0-9]*/comm'suggestion (non-blocking): Only a health probe landing in a window of about 1 s pins the rule that the bootstrap's server stops before .bootstrap-complete is written.
opendj-packages/opendj-docker/run.sh:150-156; .github/workflows/build.yml:527-528
Swap the two and the HEALTHCHECK (test -f marker && ldapsearch, every 5 s during the start period) reports healthy only if it starts after the touch and finishes before stop-ds sends SIGTERM. Even then, "Docker test" goes red only if its 10 s poll reaches dsconfig create-backend while the foreground server is still starting. So a regression would come back as the same flake as in #1098.
Pin: check the order on the instance rather than through a race. For example, assert that .bootstrap-complete is not older than the first "The Directory Server is now stopped" line in logs/errors. Or log a line after the touch and assert that it follows stop-ds's output in docker logs.
suggestion (non-blocking): The exit status of ./bin/stop-ds is ignored. If the bootstrap's server takes more than 200 s to stop, exec start-ds fails with "already running" and ends the container.
opendj-packages/opendj-docker/run.sh:150
stop-ds waits for logs/server.pid in WaitForFileDelete, and that wait gives up after DirectoryServer.DEFAULT_TIMEOUT (200 s). The file is deleted only when the JVM exits. After a timeout, start_server execs start-ds, whose start check sees the lock still held and exits, and PID 1 goes with it. Not run: nothing I read bounds the stop time of a freshly bootstrapped server, and this machine has no Docker daemon. Failing with the reason says what happened:
./bin/stop-ds || { echo "The server the bootstrap started did not stop (stop-ds exited $?)"; exit 1; }nitpick (non-blocking): A SECRET_VOLUME_REFRESH with surrounding whitespace passes the -gt 0 guard, but the first sleep then fails. The values 00 and +0 get the "not a whole number" message.
opendj-packages/opendj-docker/run.sh:80, :91-94
[ "60 " -gt 0 ] and [ " 60" -gt 0 ] are true, and sleep "1 " exits 1 with "invalid time interval". So the loop ends at once, and the only trace is sleep's error line. 00 and +0 do switch the watcher off, but the log calls them invalid.
if [[ $SECRET_VOLUME_REFRESH =~ ^[0-9]+$ ]] && [ "$SECRET_VOLUME_REFRESH" -gt 0 ]; then
watch_secrets &
elif ! [[ $SECRET_VOLUME_REFRESH =~ ^0+$ ]]; then
echo "SECRET_VOLUME_REFRESH=$SECRET_VOLUME_REFRESH is not a whole number of seconds above 0, the secret volume is copied on start only"
fi
…the Docker image, and run the server as PID 1 after the bootstrap Keystores and truststores from SECRET_VOLUME were copied only on the first start, and only after setup had already started the server, so the mounted certificate was served from the first restart on and a renewed one never reached an instance on a persistent volume. They are now copied before every start, each file renamed over its target with mode 0600, and checked again every SECRET_VOLUME_REFRESH seconds while the server runs. On the first start the server setup started is stopped and started again with exec, as every later start does, so the server is PID 1 and stops cleanly on SIGTERM. The README documents the volume and a cert-manager setup that mounts its PKCS12 keystore under the names the image expects. Fixes OpenIdentityPlatform#1087 Fixes OpenIdentityPlatform#1085
…e in CI A step in both docker jobs mounts a keystore at SECRET_VOLUME and checks that LDAPS serves it from the first start, that the server is PID 1 and stops on SIGTERM, that a renewed keystore reaches the data volume while the server runs and is served after a restart, and that the copied PIN file is readable by the server's user only.
…alth check may probe it, and import online in the Docker test The server is PID 1 of the container from the first start on, so stop-ds in "Docker test" stopped the container and the offline import that followed had nothing to run in. The step now imports online and checks that docker stop stops the server cleanly, as OpenIdentityPlatform#1098 does. The server setup started is stopped before the bootstrap marker is written, so that the health check never reports it healthy just before it goes down.
…h the watcher off, and read the served subject in one format The runner's OpenSSL 3.0.13 prints the subject as `CN = secret-v1`, so the secret-volume step now asks for `-nameopt RFC2253`. The step also renews the keystore while no container runs and checks that a new container on the same data volume, with SECRET_VOLUME_REFRESH=0, serves it, and that with 0 a keystore renewed while the server runs is not copied. run.sh repeats the copy until a pass finds nothing left to copy, so that a Secret updated in the middle of a pass cannot leave a keystore of one version next to the password of the other, and it says so when SECRET_VOLUME_REFRESH is not a whole number of seconds above 0. The README says that the key may have any alias with the default options, that a new keystore password takes effect on the next restart, and recommends `docker run --init`.
… start, and pin the truststore, the alias and the order of the bootstrap's stop in CI While the server runs, the watcher no longer copies the .pin files, and it copies nothing while a key*.pin or trust*.pin on the volume differs from the one the server started with. The server keeps the password it started with, and a keystore it can no longer open with it makes the next dsconfig change to the LDAPS handler disable the handler. The next start copies the stores along with their new password. run.sh ends the container with a message naming the exit code of stop-ds when the server the bootstrap started does not stop, logs a line once the instance is marked bootstrapped, and takes SECRET_VOLUME_REFRESH only as digits: "60 " no longer passes the guard and fails sleep, and "00" no longer gets the "not a whole number" message. The README says so, and no longer says that a --certNickname in OPENDJ_SSL_OPTIONS binds the handlers to an alias: with --generateSelfSignedCertificate setup writes no nickname at all, only a keystore of its own given with --certNickname does. The secret-volume step runs with set -E, so its ERR trap prints the container logs for a failure inside its functions too. Each keystore holds its key under an alias of its own and comes with a truststore, and the step checks that the truststore is copied, that the keystore is copied only when it changed, that stop-ds ran before the instance was marked bootstrapped, that a keystore with a new password is left to the next start, and that SECRET_VOLUME_REFRESH=0 runs no watcher.
4abcc52 to
83b6119
Compare
|
@maximthomas, thanks for the review. Round 5 is in 83b6119, rebased onto the current The ERR trap inside functions (issue): taken. The step now starts with A new keystore password while the server runs (issue): taken, the watcher leaves it to the next start. You are right that this is worse than the rejected change I described in round 4. Checked in the code:
The watcher logs "A password on the secret volume changed, the secret volume is copied again on the next start" once per change. The README sentence says the same. Once #1101 lands, the server reads the PIN again on every load, and this hold can be relaxed to allow a hot rotation. That belongs in a follow-up, not in this PR. The The alias (suggestion): taken, and it turned up a wrong sentence in the README. Each keystore now holds its key under the alias
The order of stop-ds and the marker (suggestion): taken, through the log. The exit status of stop-ds (suggestion): taken as you wrote it: Whitespace, The step, extracted from the workflow and run the way Actions runs it (
The PR description is updated to match. |
maximthomas
left a comment
maximthomas
left a comment
There was a problem hiding this comment.
praise: Everything round 2 raised is fixed or pinned, and the watcher no longer hands a running server a keystore under a new password on the normal road.
watch_secretscopies only the stores and holds while akey*.pin/trust*.pinon the volume differs fromdata/config, logging that once (opendj-packages/opendj-docker/run.sh:95-106).set -E(.github/workflows/build.yml:627,:927),./bin/stop-ds || { …; exit 1; }(run.sh:173) and the^[0-9]+$guard (run.sh:114-117) fix round 2's trap issue, the ignored stop-ds status and the padded refresh value.- CI now pins the truststore copy (
build.yml:660), a per-version alias (:636) and the absence of a watcher underSECRET_VOLUME_REFRESH=0(! grep -sqx sleep /proc/[0-9]*/comm,:687).
2 participants
Problem
SECRET_VOLUMEwere copied only on the first start, and only aftersetup.shhad already started the server. The LDAPS handler had built its SSL context from the self-signed keystore by then, so the mounted certificate was served only from the first restart on. A renewed certificate never reached an instance on a persistent volume, because later startsexeced the server before the copy.run.shended intail -f /dev/nullwith bash as PID 1, which gets noSIGTERM, so the first container lifetime always ended with a kill.Change
opendj-packages/opendj-docker/run.sh:copy_secretscopies thekey*/trust*files ofSECRET_VOLUMEthat differ from those in./data/config. Each file is written next to its target, set to mode 0600 and renamed over it, so the server never reads a half copied file. It succeeds only when it copied something, andsync_secretsrepeats it until a pass finds nothing left to copy (at most 5 passes): a Secret updated between the copy ofkeystoreand that ofkeystore.pinwould otherwise leave a keystore of one version next to the password of the other.start_servercopies the volume, startswatch_secretsin the background whenSECRET_VOLUME_REFRESH(default 60 s,0disables) is set, logs that the volume is copied on start only when the value is not digits above 0 (1m,30.5,"60 "), andexecsstart-ds --nodetach. Both the first start and every later start end there.watch_secretscopies the stores but never a.pinfile, and it copies nothing while akey*.pin/trust*.pinon the volume differs from the one indata/config, logging that once. The server keeps the password it started with, but it opens the keystore file again when a connection handler checks a change to its configuration: a keystore it can no longer open with that password makes the nextdsconfigchange to the LDAPS handler disable the handler (LDAPConnectionHandler2.isConfigurationAcceptable→createSSLContext→disableAndWarnIfUseSSL, filed as A rejected change to the LDAPS connection handler stops the handler listening until a later change is accepted or the server restarts #1109). The next start copies the stores along with their new password.bin/stop-ds) and started again throughstart_server. The server is PID 1 in both cases (Docker image: a freshly bootstrapped container ignores SIGTERM and is killed without stopping the server #1085). The stop comes before the.bootstrap-completemarker is written: stopped after it, the health check could report the bootstrap's server healthy just before it went down, and the nextdsconfigthen failed with "Server Connection Closed" (found in [#1085] Stop the server cleanly on SIGTERM in a container that bootstrapped its instance #1098).stop-dsexits 0 when the server is not running, so a customBOOTSTRAPthat leaves it stopped still works; when it fails, the container ends with a message naming its exit code instead ofstart-dsrefusing to start next to a server still up. The line "The instance is bootstrapped, the health check may probe it" follows the marker.README.mdgets a "Certificates" section: what the volume holds, that with the defaultOPENDJ_SSL_OPTIONSthe key entry may have any alias (with--generateSelfSignedCertificatesetup binds no handler to one, even with a--certNickname; only a keystore of its own,--useJavaKeystoreor--usePkcs12keyStore, with a--certNicknamebinds them), when it is copied, that a new keystore or truststore password is left to the next start, and a cert-manager setup (spec.keystores.pkcs12plus a projected volume mappingkeystore.p12→keystoreand the password →keystore.pin). The "Health check" section recommendsdocker run --init: the server is PID 1, and a JVM does not reap the processes a timed-out probe leaves behind. A PEMkubernetes.io/tlsSecret cannot be used as is: OpenDJ reads keystores, and the Alpine image has noopensslfor a conversion.SECRET_VOLUME_REFRESHis added to the variables table and to both Dockerfiles..github/workflows/build.ymlgets a "Docker test secret volume" step in both docker jobs. It runs withset -E, so that its ERR trap prints the container logs for a failure inside its functions too. It mounts a keystore, whose key is under an alias named after its version, and a truststore atSECRET_VOLUME, and checks that:java;keystore.pinhas mode 0600, and the truststore is copied;stop-dsran before the instance was marked bootstrapped (the order of the two lines in the log);docker stoptakes under 30 s and exits with 143;SECRET_VOLUME_REFRESH=0serves it, so it can only have been copied by the start over the existing instance;SECRET_VOLUME_REFRESH=0, no watcher runs (nosleepprocess in the container), and a keystore renewed while the server runs is not copied.The existing "Docker test" step in both docker jobs ran
stop-ds, an offlineimport-ldif/rebuild-indexandstart-dsinside the container. That only worked while PID 1 wastail: with the server as PID 1,stop-dsstops the container, and the step failed withcontainer ... is not running. It now imports online, checks thatdocker stop -t 60stops the container in under 50 s and not with exit code 137, restarts it, rebuilds the indexes online and checks the 10000 entries. The step is word for word the one in #1098, which makes the same change for #1085, so the workflow merges cleanly whichever of the two lands first. The offline import path is no longer covered there: it cannot run inside a container whose PID 1 is the server.A certificate copied while the server runs is served from the next restart. The server side, loading a changed key store file without a restart, is #1095 (PR #1101). Once that lands, the background copy is picked up without a restart.
Verification
Images built from
openidentityplatform/opendj:latestand:alpinewith thisrun.sh(and the master health check), a JKSkeystore+keystore.pinmounted read-only at/var/secrets/opendj, andSECRET_VOLUME_REFRESH=5:latest)latestandalpine)CN=secret-v1bashjavadocker stopof the first startCN=secret-v1CN=secret-v2keystore, aliascertificatekeystore/keystore.pincprecreates the temp file, hence the explicitchmod)openssl x509 -subjectis called with-nameopt RFC2253: the runner's OpenSSL 3.0.13 printsCN = secret-v1by default, 3.2+ printsCN=secret-v1.The CI step, extracted from the workflow and run locally the way Actions runs it (
bash -eo pipefail), passes on both images with thisrun.sh(187 s onlatest, 190 s onalpine), and goes red against each of theserun.shmutants (run onlatest):execsstart-dswithout copyingCN=secret-v3-ge 0(a busy loop for0)! cmp -s0taken for the default interval! grep -sqx sleep! cmp -safter the new passwordtrust*dropped from the globcmpof the truststoreserver-certduring the bootstraphealthy(LDAPS disabled, "The key with alias '[server-cert]' was not found")stop-ds--certNickname server-certadded to the defaultOPENDJ_SSL_OPTIONSis not a regression: for a self-signed certificate setup writes no nickname to the handlers, and the step stays green.Against the published
openidentityplatform/opendj:latestwith the oldrun.shthe step fails at the first check (the self-signed certificate is served). A keystore without a friendlyName (alias1, as cert-manager's go-pkcs12 writes it) is served as well: setup binds no connection handler to an alias.All three docker steps of the workflow ("Docker test", "Docker test custom password", "Docker test secret volume"), extracted and run the same way against both images with the
run.shof the previous round, pass (the secret volume step has had two more checks since, with the times above; the other two steps do not mount a secret volume):latestdocker stop9 s, exit 143)alpinedocker stop9 s, exit 143)The old "Docker test" step fails against this
run.shatimport-ldif --offline(the container is gone afterstop-ds), as the first CI run of this PR showed.Found while evaluating the Helm chart proposal (discussion #1079).
Fixes #1087
Fixes #1085