Keep PKCS5S2 usable on a FIPS-restricted JCE, and name the key wrapping property when the runtime has no RSA-OAEP

[vharseko]

Problem

#1054 changes the config template to get a server installed on a FIPS 140-3 Java runtime: it disables the PKCS5S2 password storage scheme and sets key-wrapping-transformation to RSA/ECB/PKCS1Padding. The review there explains why neither belongs in the template — PKCS5S2 is PBKDF2-HMAC-SHA1 like the PBKDF2 scheme which stays enabled, and PKCS#1 v1.5 is the transformation #776 removed as a CodeQL java/rsa-without-oaep finding. What actually fails on such a runtime:

• PKCS5S2PasswordStorageScheme asked for SecureRandom "SHA1PRNG" by name, at initialization and in the offline encoder. A FIPS-restricted JCE registers no such algorithm, the scheme threw a message-less InitializationException, and the server refused to start.
• The crypto manager validates its key wrapping transformation when it is created. A runtime whose only RSA cipher is PKCS#1 v1.5 — a SunPKCS11 provider on its own, as on a Linux system in FIPS mode; JDK-6190389 is still open — cannot start the server with the RSA-OAEP default. Setup noticed (ConfigureDS.updateCryptoCipher), had no secure fallback left after Fix java/rsa-without-oaep CodeQL alert by dropping the RSA PKCS#1 v1.5 fallback #776, and silently kept the default; the failed start which followed named the cipher it could not get and nothing else.

Change

• PKCS5S2PasswordStorageScheme takes the provider's default SecureRandom, as the PBKDF2 family does, and its InitializationException — like the PBKDF2 schemes' — names the algorithm which is missing. The SHA1PRNG constant goes. The template keeps the scheme enabled.
• setup warns (WARN_CONFIGDS_KEY_WRAPPING_TRANSFORMATION_UNSUPPORTED) with the transformation, the reason and the property to set before the first start. The installer gives the warning itself once the configure step is over, since what ConfigureDS writes while it runs under setup reaches the setup log only; a standalone configure-ds prints it. At start, a transformation the runtime does not provide is refused with ERR_CRYPTOMGR_KEY_WRAPPING_TRANSFORMATION_UNSUPPORTED, which names the property. The rest of that check (MD5 key ID, wrap with the dummy certificate) keeps ERR_CRYPTOMGR_CANNOT_GET_PREFERRED_KEY_WRAPPING_CIPHER as it was, since changing the property would not help there. The choice itself stays with the administrator: the install guide gains "To Install OpenDJ Directory Server on a FIPS 140 Java Runtime". With the bundled BC-FIPS provider in its default mode, which offers RSA-OAEP, the default configuration works as it is. A SunPKCS11-only runtime gets the setup --doNotStart → ldifmodify → start-ds procedure, with the caveat that PKCS#1 v1.5 is disallowed for key transport by NIST SP 800-131A Rev. 2 and that every replica must use the same transformation. The ldifmodify examples of the install and admin guides used options the tool does not have and are corrected.
• A FIPS-approved key transport for such runtimes (SP 800-56B KTS-KEM-KWS over RSA/ECB/NoPadding) is Crypto manager: a FIPS-approved key transport for JVMs whose only RSA cipher is PKCS#1 v1.5 (SunPKCS11) #1056. BC-FIPS in approved-only mode refuses the 1024-bit key of the dummy certificate the start-time check wraps with; that is The start-time key wrapping check wraps with a 1024-bit RSA key, which BC-FIPS refuses in approved-only mode #1081.

Testing

The new tests withdraw the JCE providers which register a service, run the code without them, and put the providers back where they were (TestCaseUtils.withoutJceService):

• PKCS5S2PasswordStorageSchemeTestCase: the scheme initializes and encodes, online and offline, without SHA1PRNG (SUN withdrawn, BC-FIPS standing in for the digests); its initialization failure names PBKDF2WithHmacSHA1 (SunJCE withdrawn). The same failure test for PBKDF2PasswordStorageSchemeTestCase.
• ConfigureDSTestCase (new): the default transformation stays where the runtime has it. Without Cipher.RSA, or with a provider offering RSA/ECB/PKCS1Padding only, none is chosen (Fix java/rsa-without-oaep CodeQL alert by dropping the RSA PKCS#1 v1.5 fallback #776). The warning, both as the installer gets it and as configure-ds writes it, names the transformation and the property, and the configuration is not touched.
• CryptoManagerTestCase: an unsupported transformation is refused with the message that names key-wrapping-transformation. A supported one refused for another cause (MD5 withdrawn) is not.

Each test was run red before its fix, or against a mutant of it, and green after: PKCS5S2PasswordStorageSchemeTestCase 56/56, PBKDF2PasswordStorageSchemeTestCase 40/40, PBKDF2HmacSHA256/512PasswordStorageSchemeTestCase 39/39 each, CryptoManagerTestCase 32/32, ConfigureDSTestCase 5/5. The installer's call is not covered by a test: the setup road does not run in the unit tests.

Alternative to #1054. Related: #776, #1056.

[maximthomas]

praise: The fix stays secure and goes where the failures actually happen, without touching the template the way #1054 does.

• PKCS5S2PasswordStorageScheme now takes new SecureRandom() (:90, :252), as the PBKDF2 family does, and its InitializationException finally has a message (:97).
• supportedKeyWrappingTransformation still has only an OAEP alternative to fall back to, so #776 holds on a runtime without RSA-OAEP.
• TestCaseUtils.withoutJceService puts the withdrawn providers back at the positions they had (:374), so a later case in the same JVM sees the same provider order.

---

issue (blocking): Under setup, WARN_CONFIGDS_KEY_WRAPPING_TRANSFORMATION_UNSUPPORTED never reaches the user.

opendj-server-legacy/src/main/java/org/opends/server/tools/ConfigureDS.java:1237, opendj-server-legacy/src/main/java/org/opends/quicksetup/installer/Installer.java:1308

Setup runs ConfigureDS.configMain(args, System.out, System.err) in-process (InstallerHelper:117). There, System.err is the Installer's ApplicationPrintStream (Installer:253), and the whole configure step runs inside setNotifyListeners(false) (Installer:1308). ApplicationPrintStream.println reaches the CLI or GUI listeners only if (notifyListeners) (Application:248). Otherwise it only calls logger.info into the setup temp log, and a successful run deletes that log (Installer:311). configMain returns success after the warning. So setup --doNotStart on a SunPKCS11-only runtime still ends silently, as it did before this PR. Only a standalone configure-ds shows the warning. The description ("Setup prints a warning") and the code comment ("has to be told so here rather than by the failed start") both describe the setup road.

Raise the warning from the Installer, after the silenced window:

// Installer.configureServer, after the InvokeThread has finished (notifyListeners is true again)
try
{
  ConfigureDS.supportedKeyWrappingTransformation(defaultKeyWrappingTransformation);
}
catch (GeneralSecurityException e)
{
  notifyListeners(getFormattedWarning(
      WARN_CONFIGDS_KEY_WRAPPING_TRANSFORMATION_UNSUPPORTED.get(defaultKeyWrappingTransformation, e.getMessage())));
}

Or, if setup is not meant to warn: drop the setup claim from the description and reword the comment at ConfigureDS.java:1233-1236.

---

issue (blocking): The FIPS install procedure calls ldifmodify with options it does not have.

opendj-doc-generated-ref/src/main/asciidoc/install-guide/chap-install.adoc:826

The only ldifmodify in the distribution is opendj-ldap-toolkit's LDIFModify. It takes trailing arguments source_file [changes_files...] plus -o/--outputLDIF (LDIFModify.java:108, :119). --sourceLDIF, --changesLDIF and --targetLDIF exist nowhere in the tree. An administrator following the one documented way to set key-wrapping-transformation before the first start gets a usage error, and start-ds still refuses to start. (The same stale spelling is in admin-guide/chap-import-export.adoc:230, which predates this PR.)

$ ldifmodify \
 -o /path/to/opendj/config/config.ldif.new \
 /path/to/opendj/config/config.ldif \
 changes.ldif

---

question (non-blocking): Should the sentence added to ERR 664 apply only when the transformation itself is refused?

opendj-server-legacy/src/messages/org/opends/messages/core.properties:1197, opendj-server-legacy/src/main/java/org/opends/server/crypto/CryptoManagerImpl.java:424

The try that reports 664 does more than check the cipher. It also computes the instance key ID with MessageDigest "MD5" (:676) and wraps against a hard-coded 1024-bit RSA certificate. A runtime that has RSA-OAEP but lacks MD5, or refuses a 1024-bit wrap, passes ConfigureDS's Cipher.getInstance check without a warning. start-ds then fails with an instruction to change a property that is not the cause. The %s still carries the real exception. I did not run this on an NSS-FIPS or BC-FIPS approved-only JVM. As is, this is a Minor. It becomes a Major if the guide's "with the bundled BC-FIPS provider the default configuration works as it is" is meant to hold in approved-only mode. Splitting the try would give the digest and dummy-certificate failures a message of their own.

---

suggestion (non-blocking): No test runs the warning road of updateCryptoCipher.

opendj-server-legacy/src/test/java/org/opends/server/tools/ConfigureDSTestCase.java:50

Both cases call only the static supportedKeyWrappingTransformation. Three mutants keep every test green: deleting the printWrappedText(err, WARN_…) line, swapping its two arguments, and dropping the return so that the default gets rewritten. That is how the silencing above got through.

Pin: together with the fix above, run the setup road (or configMain on a test instance) under withoutJceService("Cipher", "RSA", …). Assert that the WARN text reaches the user-visible stream, and that ds-cfg-key-wrapping-transformation still holds the default in config.ldif.

---

suggestion (non-blocking): The #776 pin cannot see a weaker fallback. Withdrawing Cipher.RSA withdraws RSA/ECB/PKCS1Padding too.

opendj-server-legacy/src/test/java/org/opends/server/tools/ConfigureDSTestCase.java:46

Cipher.getInstance("RSA/ECB/PKCS1Padding") falls back to the Cipher.RSA service, and the fixture has removed it. So a mutant that adds "RSA/ECB/PKCS1Padding" to getAlternativeCipher (ConfigureDS.java:1365) gets null and rethrows, and the test passes against the regression its javadoc names.

withoutJceService("Cipher", "RSA", () ->
{
  final Provider weakOnly = new Provider("WeakRsaOnly", "1.0", "PKCS#1 v1.5 only") {};
  weakOnly.put("Cipher.RSA/ECB/PKCS1Padding", "com.sun.crypto.provider.RSACipher");
  Security.insertProviderAt(weakOnly, 1);
  try
  {
    ConfigureDS.supportedKeyWrappingTransformation(DEFAULT_KEY_WRAPPING_TRANSFORMATION);
    fail("a transformation was chosen on a runtime whose only RSA cipher is PKCS#1 v1.5");
  }
  catch (GeneralSecurityException expected)
  {
    // the #776 regression returns "RSA/ECB/PKCS1Padding" instead
  }
  finally
  {
    Security.removeProvider(weakOnly.getName());
  }
  return null;
});

Or drop "(#776)" from the javadoc.

---

suggestion (non-blocking): The asserts that the message names the missing algorithm are satisfied by the exception's own text.

opendj-server-legacy/src/test/java/org/opends/server/extensions/PKCS5S2PasswordStorageSchemeTestCase.java:186, opendj-server-legacy/src/test/java/org/opends/server/extensions/PBKDF2PasswordStorageSchemeTestCase.java:98

The second %s of ERR_PWSCHEME_CANNOT_INITIALIZE_MESSAGE_DIGEST is the NoSuchAlgorithmException, whose text is "PBKDF2WithHmacSHA1 SecretKeyFactory not available". So .get(getStorageSchemeName(), e) or .get("", e) in place of the algorithm keeps both cases green.

assertTrue(e.getMessage().contains("for the PBKDF2WithHmacSHA1 algorithm"), e.getMessage());

---

nitpick (non-blocking): testEncodesOfflineWithoutSha1Prng's javadoc names two roads that never reach PKCS5S2PasswordStorageScheme.encodeOffline.

opendj-server-legacy/src/test/java/org/opends/server/extensions/PKCS5S2PasswordStorageSchemeTestCase.java:153

No main code calls it. The initial root password goes through SaltedSHA512PasswordStorageScheme.encodeOffline (ConfigureDS.java:1138), and encode-password goes through the initialized scheme's encodePasswordWithScheme (EncodePassword.java:502). "An offline encoder a tool may call" is accurate. Or drop the clause.

[vharseko]

All seven points are taken, in adb4a39 (the branch is rebased onto the current master).

1. The setup warning (blocking). Confirmed: during the configure step the listeners are off and ApplicationPrintStream writes only to the setup log. ConfigureDS now builds the warning in a public unsupportedKeyWrappingTransformationWarning(). Installer.configureServer calls it after invokeLongOperation, once the listeners are back, and gives it with getFormattedWarning. That covers both the CLI and the GUI. Your snippet needed two changes: supportedKeyWrappingTransformation is package-private, and the installer has no default transformation of its own. So the method takes the default from CryptoManagerCfgDefn, as updateCryptoCipher does. A standalone configure-ds still prints the warning, and the code comment now says which road does what.

2. ldifmodify options (blocking). Fixed. The procedure now uses ldifmodify --outputLDIF <new> <source> <changes>. The same stale spelling was also in admin-guide/chap-import-export.adoc:230 and install-guide/chap-install.adoc:990 (the PDB-to-JE script), and both are fixed too. (ldif-diff in the same admin-guide section also takes trailing source target, not --sourceLDIF/--targetLDIF. I left it alone: it is outside this change.)

3. ERR 664. Split. isKeyWrappingTransformationSupported now checks Cipher.getInstance(transformation) first. Only that refusal names the property, as the new ERR_CRYPTOMGR_KEY_WRAPPING_TRANSFORMATION_UNSUPPORTED_766. ERR 664 is back to its original text for the rest of the check (MD5 key ID, MAC key, wrap with the dummy certificate).

On approved-only mode, I checked it: with bc-fips 2.1.3 and -Dorg.bouncycastle.fips.approved_only=true, the OAEP wrap with the dummy certificate's 1024-bit key fails with FipsUnapprovedOperationError: Attempt to use RSA key size outside of accepted range - requested keySize 1024 bits. That is an Error, so the catch (Exception) there does not report it as 664 either. OpenDJ does not turn approved-only mode on, so the guide now says the default configuration works with BC-FIPS "in its default mode" and that approved-only mode is not covered. The dummy key size is #1081.

4. Warning road untested. ConfigureDSTestCase now runs updateCryptoCipher itself, with and without Cipher.RSA, and asserts the text on its error stream. It also asserts the text unsupportedKeyWrappingTransformationWarning() returns to the installer. The tool gets no configuration handler, so a step that tried to rewrite the transformation would fail the test; that stands in for the config.ldif assertion. Mutants: deleting the printWrappedText line and swapping the two arguments both turn testWarningNamesTheTransformationAndThePropertyWhereTheRuntimeHasNoRsaOaep red. Dropping the return no longer compiles, since cipher is then not definitely assigned. The installer call itself is not covered by a test: the setup road does not run in the unit tests.

5. #776 pin. Added testNoKeyWrappingTransformationIsChosenWhereTheRuntimeHasOnlyPkcs1, with your provider. It first checks that Cipher.getInstance("RSA/ECB/PKCS1Padding") does resolve to the stand-in, so the case cannot pass on a fixture that offers nothing. Adding "RSA/ECB/PKCS1Padding" to getAlternativeCipher turns it red. The "(#776)" moved from the old case's javadoc to this one.

6. Message asserts. Both cases now assert contains("for the PBKDF2WithHmacSHA1 algorithm"). .get("", e) turns each of them red.

7. Offline encoder javadoc. Now "which a tool may call before any scheme is initialized".

Runs (JDK 26, locally): ConfigureDSTestCase 5/5, CryptoManagerTestCase 32/32 (with the new testKeyWrappingRefusalForAnotherCauseDoesNotNameTheProperty, and the old refusal case now pinned to ordinal 766; skipping the cipher step turns it red), PKCS5S2PasswordStorageSchemeTestCase 56/56, PBKDF2PasswordStorageSchemeTestCase 40/40.