openssl fails to load when BC FIPS jars are on the JRuby classloader #361

jruby-openssl 0.16.0 (which upgraded BC from 1.83 to 1.84 via #360) fails to load its native extension when BouncyCastle FIPS jars are present on the JRuby classloader. This is a regression from 0.15.x where the same scenario worked.

Error:

NameError: cannot load (ext) (org.jruby.ext.openssl.OpenSSL)
         load_ext at org/jruby/ext/jruby/JRubyUtilLibrary.java:234
           <main> at .../jruby-openssl-0.16.0-java/lib/jopenssl/load.rb:26

Environment:

• JRuby 10.0.5.0 (Ruby 3.4.5)
• Java 21.0.10 (OpenJDK, aarch64)
• jruby-openssl 0.16.0 (BC 1.84, commit 08595c3 upgraded BC)
• BC FIPS jars: bc-fips 2.0.0, bcpkix-fips 2.0.7, bctls-fips 2.0.19, bcutil-fips 2.0.3

Minimal reproduction:

Download BC FIPS jars from Maven Central:

mvn dependency:copy -Dartifact=org.bouncycastle:bc-fips:2.0.0 -DoutputDirectory=./lib
mvn dependency:copy -Dartifact=org.bouncycastle:bcpkix-fips:2.0.7 -DoutputDirectory=./lib
mvn dependency:copy -Dartifact=org.bouncycastle:bctls-fips:2.0.19 -DoutputDirectory=./lib
mvn dependency:copy -Dartifact=org.bouncycastle:bcutil-fips:2.0.3 -DoutputDirectory=./lib

Install 0.16.0:

gem install jruby-openssl -v 0.16.0

Run:

# test_bc_fips.rb
# Load BC FIPS jars onto JRuby classloader
Dir.glob("lib/bc*-fips-*.jar") { |jar| require jar }

# Activate 0.16.0 and load openssl
gem "jruby-openssl", "0.16.0"
require "openssl" 
puts OpenSSL::OPENSSL_VERSION

$ jruby test_bc_fips.rb
NameError: cannot load (ext) (org.jruby.ext.openssl.OpenSSL)
         load_ext at org/jruby/ext/jruby/JRubyUtilLibrary.java:234
           <main> at .../jruby-openssl-0.16.0-java/lib/jopenssl/load.rb:26

Same script with 0.15.7 works:

$ gem install jruby-openssl -v 0.15.7
$ jruby test_bc_fips.rb # (with 0.15.7 activated)
JRuby-OpenSSL 0.15.7

Claude analysis

This is what an LLM thought, I'm adding it to the end and qualifying that as it may be suspect to AI nonsense)

Analysis:

The problem is specifically in jopenssl.jar's class initialization path. When load_ext
is called, SecurityHelper's static initializer runs and calls
setBouncyCastleProvider() → newBouncyCastleProvider(BC_PROVIDER_CLASS) which does
Class.forName("org.bouncycastle.jce.provider.BouncyCastleProvider").newInstance().

When BC FIPS jars are on the classloader, some of the transitive classes that
BouncyCastleProvider references at construction time resolve from the FIPS jars rather
than the non-FIPS jars. With BC 1.84, this mixed-resolution causes the provider construction
(or subsequent SPI_ACCESSIBLE check via getCertificateFactory("X.509", provider)) to
fail, which cascades into a hard failure in load_ext.

With BC 1.83, the class layouts between FIPS and non-FIPS were compatible enough that
the hybrid classloader didn't cause a crash.

Setting jruby.openssl.load.jars=false does not help — the failure still occurs because
jopenssl.jar itself was compiled against BC 1.84, and its internal class references
resolve against whatever BC classes the JRuby classloader finds first (the FIPS ones).

The specific change: PR #360 upgraded
BC from 1.83 → 1.84. One visible Java code change was in PKey.java:
key.getParameters() → key.getParametersObject().toASN1Primitive() (the old method was
removed in BC 1.84). But the fundamental issue is that jopenssl.jar (all classes, not just
PKey) is now compiled and linked against BC 1.84 APIs which may not exist in the FIPS
variants of the same packages.

Suggested fix directions:

1. Catch and handle the failure in SecurityHelper.setBouncyCastleProvider() when BC FIPS
   classes are present — fall back gracefully instead of crashing load_ext
2. Detect BC FIPS on the classloader and skip BC non-FIPS provider registration entirely
3. Document the incompatibility with BC FIPS jars on the classloader