fail to start after mariadb upgrade from 10 to 11

[rstanonik]

We're running v3.3.1. We recently upgraded our mariadb from 10.3.39, to 11.3.2 and now Archivesspace fails to start.
I can connect to the db from the archivesspace server using the mysql cli.
Any thoughts?
Thanks
Ron

Specifics about our set up and the first part of the error in the logs And the diagnostic trace is included.

archivesspace-v3.3.1
openjdk version "1.8.0_402"
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
aspace_diagnostic_1710863787.txt

ArchivesSpaceThreadDump: Touch the file '/usr/local/archivesspace-v3.3.1/thread_dump_backend.txt' to trigger a thread dump
I, [2024-03-19T08:56:27.402961 #1465273] INFO -- : Thread-2002: Connecting to database: jdbc:mysql://lib-mariadb-dev.ucsd.edu:3306/archivesspace?user=[REDACTED]&password=[REDACTED]&useUnicode=true&characterEncoding=UTF-8. Max connections: 28
E, [2024-03-19T08:56:27.643013 #1465273] ERROR -- : Thread-2002: DB connection failed: Java::JavaSql::SQLException: java.lang.NullPointerException
E, [2024-03-19T08:56:27.643769 #1465273] ERROR -- : Thread-2002: Additional DB info: java.sql.SQLException: java.lang.NullPointerException: java.lang.NullPointerException
E, [2024-03-19T08:56:27.644634 #1465273] ERROR -- : Thread-2002: Additional DB info: java.lang.NullPointerException:
========================================================================
A trace file has been written to the following location: /usr/local/archivesspace-v3.3.1/data/tmp/aspace_diagnostic_1710863787.txt

[trevorthornton]

We ran into a problem when we upgraded to MariaDB 11 which might be the same one you're having. I think that the JDBC will attempt to connect with SSL when it's available, and MariaDB 11 has SSL enabled by default while 10 does not, so if you don't have a valid SSL certificate on the MariaDB server it won't connect. The fix for us was to add "&useSSL=false" to the connection string, i.e. AppConfig[:db_url] = "jdbc:mysql://localhost:3306/archivesspace?user=as&password=as123&useUnicode=true&characterEncoding=UTF-8&useSSL=false"

…

On Tue, Mar 19, 2024 at 2:07 PM rstanonik ***@***.***> wrote: We're running v3.3.1. We recently upgraded our mariadb from 10.3.39, to 11.3.2 and now Archivesspace fails to start. I can connect to the db from the archivesspace server using the mysql cli. Any thoughts? Thanks Ron Specifics about our set up and the first part of the error in the logs And the diagnostic trace is included. archivesspace-v3.3.1 openjdk version "1.8.0_402" NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)" aspace_diagnostic_1710863787.txt <https://github.com/archivesspace/tech-docs/files/14654510/aspace_diagnostic_1710863787.txt> ArchivesSpaceThreadDump: Touch the file '/usr/local/archivesspace-v3.3.1/thread_dump_backend.txt' to trigger a thread dump I, [2024-03-19T08:56:27.402961 #1465273] INFO -- : Thread-2002: Connecting to database: jdbc:mysql:// lib-mariadb-dev.ucsd.edu:3306/archivesspace?user=[REDACTED]&password=[REDACTED]&useUnicode=true&characterEncoding=UTF-8. Max connections: 28 E, [2024-03-19T08:56:27.643013 #1465273] ERROR -- : Thread-2002: DB connection failed: Java::JavaSql::SQLException: java.lang.NullPointerException E, [2024-03-19T08:56:27.643769 #1465273] ERROR -- : Thread-2002: Additional DB info: java.sql.SQLException: java.lang.NullPointerException: java.lang.NullPointerException E, [2024-03-19T08:56:27.644634 #1465273] ERROR -- : Thread-2002: Additional DB info: java.lang.NullPointerException: ======================================================================== A trace file has been written to the following location: /usr/local/archivesspace-v3.3.1/data/tmp/aspace_diagnostic_1710863787.txt — Reply to this email directly, view it on GitHub <#173>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMMLDUSDJLF7AXOBNGZ3TDYZB5HBAVCNFSM6AAAAABE6CJREOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGE4TKNRUGQ3TOOA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Trevor Thornton Applications Developer, Digital Library Initiatives North Carolina State University Libraries

[rstanonik]

Thanks! We'll try that.

[rstanonik]

Alas, that didn't fix it. Now I'm trying to turn on more logging/debugging.

[rstanonik]

We also tried disabling SSL on the mariadb server.

[trevorthornton]

Forgot to mention that you'll need to restart AS after making that change, but maybe you knew that already. When you restart, if there are still connection problems you should see the errors in logs/archivesspace.out

…

On Tue, Mar 19, 2024 at 3:31 PM rstanonik ***@***.***> wrote: We also tried disabling SSL on the mariadb server. — Reply to this email directly, view it on GitHub <#173 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMMLDSQ4OBDAWYFYOA3PFLYZCHC3AVCNFSM6AAAAABE6CJREOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBXHE3DMOJVGM> . You are receiving this because you commented.Message ID: ***@***.***>

-- Trevor Thornton Applications Developer, Digital Library Initiatives North Carolina State University Libraries

[rstanonik]

Thanks. I did restart. I used tcpdump to watch network traffic to/from port 3306 (mysql) and it was in clear text. When I started AS, I could see it connect to mysql, get some kind of collation info, then download some info about our items. It did that a second time. Although it was running at that point, it wasn't listening on 8080, so no user interface. The last thing in the AS log is

You're seeing this error because you use JRuby::Rack::ErrorApp::ShowStatus.

We're rolling the DB back to 10 to see if that still works.

[rstanonik]

Rolling mariadb back to 10 works.

Tried a fresh install of 3.3.1 (our current version) and 3.5.0 with mariadb 11. Both fail during setup-database.sh.

Loading ArchivesSpace configuration file from path: /usr/local/archivesspace/./c
onfig/config.rb
Sequel::DatabaseConnectionError: Java::JavaSql::SQLException: java.lang.NullPoin
terException
make_new at /usr/local/archivesspace/gems/gems/sequel-5.9.0/lib/seque
l/connection_pool.rb:143

So, something about mariadb 11 doesn't play well with Archivesspace. We're guessing something to do with utf8 handling. We tried some variants in AppConfig[:db_url] and in the DB creation, but no joy.

[rstanonik]

I updated the mysql connector (5.1.26 to 5.1.47) and seems to be working now