feat: boot-db-jdbc-postgresql and boot-db-jdbc-mysql driver modules

AGENTS.md

@@ -11,3 +11,9 @@
 
 Use asciidoc.
 Use one-sentence-per-line convention.
+
+## Database tests
+
+The `boot-db-jdbc-postgresql` and `boot-db-jdbc-mysql` tests run real servers with Testcontainers.
+They are skipped where Docker is unavailable, but not on CI, where `CI` is set.
+With rootless Podman, run them with `DOCKER_HOST=unix:///run/user/$(id -u)/podman/podman.sock TESTCONTAINERS_RYUK_DISABLED=true`.

README.adoc

@@ -65,6 +65,8 @@ An application makes two choices, each its own dependency:
 
 * A *driver*, which supplies the `JdbcConfig`: a driver module's typed config, or the generic `JdbcGenericConfig` (any database by JDBC URL, with the driver added separately).
 ** `boot-db-jdbc-sqlite`: `SqliteConfig` takes a file `path`, and defaults to WAL journaling, enforced foreign keys, a busy timeout, and a single-connection pool.
+** `boot-db-jdbc-postgresql`: `PostgresqlConfig` takes a `host`, `port`, `database`, and credentials, plus typed driver settings such as `sslMode`, `currentSchema`, and `applicationName`.
+** `boot-db-jdbc-mysql`: `MysqlConfig` takes a `host`, `port`, `database`, and credentials, plus typed driver settings such as `sslMode`, and defaults to Connector/J's prepared statement cache.
 * A *pool*, which supplies the `ConnectionPoolFactory`, configured with its own typed settings.
 ** `boot-db-pool-hikari`: HikariCP, configured by `HikariPoolConfig`. For HikariCP settings it doesn't cover, add a `HikariConfigCustomizer` to its set binding (Kodein `inBindSet`, Metro `@IntoSet`).
 
@@ -73,7 +75,7 @@ On top of the pool, two optional modules use its `DataSource`:
 * `boot-db-flyway`: `FlywayMigrationService` runs Flyway's `migrate` when the app starts, just after the pool, so a failed migration fails the boot and nothing uses an unmigrated schema.
 Migrations come from `classpath:db/migration` unless `FlywayConfig.locations` says otherwise.
 For Flyway settings `FlywayConfig` doesn't cover, add a `FlywayConfigurationCustomizer` to its set binding.
-flyway-core supports SQLite itself; other databases need their Flyway database module, which their driver modules will bring.
+flyway-core supports SQLite itself; the PostgreSQL and MySQL driver modules bring their Flyway database modules.
 * `boot-db-exposed`: `ExposedDatabaseService` owns an Exposed `Database` over the pool, configured by `ExposedConfig` (default isolation level, read-only, retries, query timeout, and so on), which applies to that `Database` only. On shutdown, before the pool closes, it unregisters the `Database` from Exposed's process-wide registry.
 `db.withTransaction { … }` runs a suspending transaction on the `Database`'s dispatcher (`Dispatchers.IO` by default); called inside another, it joins it, so several calls make one unit of work.
 `SchemaVerificationService` checks, after the migrations, that the schema matches the Exposed tables registered in its `Set<Table>` binding, and fails the boot listing the SQL the schema needs (or only warns, with `schemaVerification = WARN`).
@@ -102,7 +104,7 @@ db {
 ----
 
 With vanilla wiring, construct `JdbcDataSourceService(binder.config<SqliteConfig>("db.jdbc"), HikariConnectionPoolFactory(binder.config("db.pool")))`, pass it to `boot` as an app service, and hand its `dataSource` to whatever needs connections: e.g. `FlywayMigrationService(dataSource, binder.config("db.flyway"))`, and an `ExposedDatabaseService(dataSource, exposedConfig)`, as a service too, whose `database` goes to a `SchemaVerificationService` for its tables and to your repositories.
-With Kodein, import `jdbcDataSourceModule`, a driver config module (`sqliteConfigModule` or `jdbcGenericConfigModule`), and a pool module (`hikariPoolModule`), plus `flywayMigrationModule` and `exposedDatabaseModule` as needed; register tables with `inBindSet<Table> { add { singleton { MyTable } } }`.
+With Kodein, import `jdbcDataSourceModule`, a driver config module (`sqliteConfigModule`, `postgresqlConfigModule`, `mysqlConfigModule`, or `jdbcGenericConfigModule`), and a pool module (`hikariPoolModule`), plus `flywayMigrationModule` and `exposedDatabaseModule` as needed; register tables with `inBindSet<Table> { add { singleton { MyTable } } }`.
 With Metro, the `-metro` driver, pool, Flyway, and Exposed modules contribute everything; for the generic config, add `JdbcGenericConfigBindings` to your graph's `bindingContainers`; register tables with `@Provides @IntoSet fun myTable(): Table = MyTable`.
 A graph without a driver or a pool fails to compile with a missing binding.
 
@@ -176,7 +178,7 @@ See these three wiring options:
 * [ ] Create a boot-server-http-ktor-cohort module using https://github.com/sksamuel/cohort[Cohort]
 * [ ] Update the ktor example to integrate https://github.com/sksamuel/hoplite[Kompendium] — this is outside the scope of Bootable, but it would be a good example of how to integrate Bootable with other libraries
 * [ ] Make multiplatform — some modules are backend only e.g. Hoplite, Log4j2, but others could easily be adapted for multiplatform usage
-* [ ] Integrate more useful modules e.g. postgresql, and so on
+* [ ] Integrate more useful modules
 
 == Author
 

boot-db-jdbc-mysql/src/kodein/kotlin/com/github/rocketraman/bootable/db/jdbc/MysqlModule.kt

@@ -0,0 +1,16 @@
+package com.github.rocketraman.bootable.db.jdbc
+
+import com.github.rocketraman.bootable.config.ConfigBinder
+import com.github.rocketraman.bootable.config.config
+import org.kodein.di.DI
+import org.kodein.di.bindSingleton
+import org.kodein.di.instance
+
+/**
+ * A Kodein module that decodes a [MysqlConfig] from the `db.jdbc` config section as the [JdbcConfig].
+ * Import it alongside `jdbcDataSourceModule`, which builds the pool from it, and a pool module. Requires a `ConfigBinder` binding
+ * (e.g. from boot-config-hoplite's `hopliteConfigModule`).
+ */
+val mysqlConfigModule by DI.Module {
+  bindSingleton<JdbcConfig> { instance<ConfigBinder>().config<MysqlConfig>("db.jdbc") }
+}

boot-db-jdbc-mysql/src/main/kotlin/com/github/rocketraman/bootable/db/jdbc/MysqlConfig.kt

@@ -0,0 +1,98 @@
+package com.github.rocketraman.bootable.db.jdbc
+
+import kotlin.time.Duration
+
+/**
+ * A strongly typed [JdbcConfig] for a MySQL database, connected to with
+ * [MySQL Connector/J](https://dev.mysql.com/doc/connector-j/en/), which this module brings in. It brings Flyway's
+ * MySQL database module too, for `boot-db-flyway`.
+ *
+ * Its defaults turn on Connector/J's client-side prepared statement cache, sized as
+ * [HikariCP recommends](https://github.com/brettwooldridge/HikariCP/wiki/MySQL-Configuration): the driver
+ * otherwise parses every statement again each time it is prepared.
+ *
+ * Decoded from the `db.jdbc` config section:
+ *
+ * ```
+ * db {
+ *   jdbc {
+ *     host = db.example.com
+ *     database = app
+ *     username = app
+ *     password = ${?DB_PASSWORD}
+ *     sslMode = VERIFY_IDENTITY
+ *   }
+ * }
+ * ```
+ *
+ * The settings other than the address and credentials are passed to the driver as connection properties. Those
+ * left `null` keep the driver's default. For a server address this can't express, such as several hosts for
+ * failover, use [JdbcGenericConfig] with a JDBC URL instead.
+ *
+ * @property host The server's host name or IP address.
+ * @property port The server's port.
+ * @property database The database to connect to.
+ * @property sslMode Whether to use SSL, and how much of the server's certificate to verify.
+ * @property allowPublicKeyRetrieval Whether the driver may ask the server for its RSA public key, which the
+ *   `caching_sha2_password` authentication needs on a connection without SSL. Off by default in the driver, since a
+ *   server impersonating the real one could then read the password.
+ * @property connectTimeout How long to wait to open a connection to the server; the driver waits forever by
+ *   default.
+ * @property socketTimeout How long to wait for a read from the server, which bounds every query; the driver waits
+ *   forever by default.
+ * @property rewriteBatchedStatements Whether to rewrite a batch of statements into fewer, multi-row statements,
+ *   which is much faster.
+ * @property cachePrepStmts Whether to cache prepared statements' parsed form on each connection.
+ * @property prepStmtCacheSize How many prepared statements to cache on each connection.
+ * @property prepStmtCacheSqlLimit The longest SQL statement, in characters, that is cached.
+ */
+data class MysqlConfig(
+  val host: String = "localhost",
+  val port: Int = 3306,
+  val database: String,
+  override val username: String? = null,
+  override val password: String? = null,
+  val sslMode: MysqlSslMode? = null,
+  val allowPublicKeyRetrieval: Boolean? = null,
+  val connectTimeout: Duration? = null,
+  val socketTimeout: Duration? = null,
+  val rewriteBatchedStatements: Boolean? = null,
+  val cachePrepStmts: Boolean = true,
+  val prepStmtCacheSize: Int = 250,
+  val prepStmtCacheSqlLimit: Int = 2048,
+) : JdbcConfig {
+  override val jdbcUrl get() = "jdbc:mysql://${urlHost(host)}:$port/$database"
+  override val driverClassName get() = "com.mysql.cj.jdbc.Driver"
+
+  override val dataSourceProperties: Map<String, String>
+    get() = buildMap {
+      sslMode?.let { put("sslMode", it.name) }
+      allowPublicKeyRetrieval?.let { put("allowPublicKeyRetrieval", it.toString()) }
+      connectTimeout?.let { put("connectTimeout", it.inWholeMilliseconds.toString()) }
+      socketTimeout?.let { put("socketTimeout", it.inWholeMilliseconds.toString()) }
+      rewriteBatchedStatements?.let { put("rewriteBatchedStatements", it.toString()) }
+      put("cachePrepStmts", cachePrepStmts.toString())
+      if (cachePrepStmts) {
+        put("prepStmtCacheSize", prepStmtCacheSize.toString())
+        put("prepStmtCacheSqlLimit", prepStmtCacheSqlLimit.toString())
+      }
+    }
+
+  // keep the password out of logs and exception messages
+  override fun toString() =
+    "MysqlConfig(host=$host, port=$port, database=$database, username=$username, " +
+      "password=${password?.let { "****" }}, sslMode=$sslMode, allowPublicKeyRetrieval=$allowPublicKeyRetrieval, " +
+      "connectTimeout=$connectTimeout, socketTimeout=$socketTimeout, " +
+      "rewriteBatchedStatements=$rewriteBatchedStatements, cachePrepStmts=$cachePrepStmts, " +
+      "prepStmtCacheSize=$prepStmtCacheSize, prepStmtCacheSqlLimit=$prepStmtCacheSqlLimit)"
+}
+
+/**
+ * Connector/J's [SSL modes](https://dev.mysql.com/doc/connector-j/en/connector-j-connp-props-security.html#cj-conn-prop_sslMode).
+ * Only [VERIFY_CA] and [VERIFY_IDENTITY] protect against a server impersonating the real one; [VERIFY_IDENTITY]
+ * also checks the host name.
+ */
+enum class MysqlSslMode { DISABLED, PREFERRED, REQUIRED, VERIFY_CA, VERIFY_IDENTITY }
+
+/** A host for a JDBC URL: an IPv6 address must be bracketed, so its colons aren't read as the port. */
+private fun urlHost(host: String) = if (':' in host && !host.startsWith("[")) "[$host]" else host

boot-db-jdbc-mysql/src/metro/kotlin/com/github/rocketraman/bootable/db/jdbc/MysqlConfigBindings.kt

@@ -0,0 +1,20 @@
+package com.github.rocketraman.bootable.db.jdbc
+
+import com.github.rocketraman.bootable.boot.BootScope
+import com.github.rocketraman.bootable.config.ConfigBinder
+import com.github.rocketraman.bootable.config.config
+import dev.zacsweers.metro.BindingContainer
+import dev.zacsweers.metro.ContributesTo
+import dev.zacsweers.metro.Provides
+import dev.zacsweers.metro.SingleIn
+
+/**
+ * Provides a [MysqlConfig], decoded from the `db.jdbc` config section, as the [JdbcConfig] that
+ * `JdbcDataSourceBindings` (from boot-db-jdbc-metro) builds the pool from.
+ */
+@ContributesTo(BootScope::class)
+@BindingContainer
+object MysqlConfigBindings {
+  @Provides @SingleIn(BootScope::class)
+  fun mysqlConfig(binder: ConfigBinder): JdbcConfig = binder.config<MysqlConfig>("db.jdbc")
+}

boot-db-jdbc-mysql/src/test/kotlin/com/github/rocketraman/bootable/db/jdbc/MysqlConfigTest.kt

@@ -0,0 +1,55 @@
+package com.github.rocketraman.bootable.db.jdbc
+
+import de.infix.testBalloon.framework.core.testSuite
+import kotlin.time.Duration.Companion.seconds
+
+private val prepStmtCache = mapOf(
+  "cachePrepStmts" to "true",
+  "prepStmtCacheSize" to "250",
+  "prepStmtCacheSqlLimit" to "2048",
+)
+
+val MysqlConfigTest by testSuite {
+  test("the URL addresses the database") {
+    assert(MysqlConfig(database = "app").jdbcUrl == "jdbc:mysql://localhost:3306/app")
+    assert(MysqlConfig(host = "db", port = 3307, database = "app").jdbcUrl == "jdbc:mysql://db:3307/app")
+  }
+
+  test("an IPv6 host is bracketed") {
+    assert(MysqlConfig(host = "::1", database = "app").jdbcUrl == "jdbc:mysql://[::1]:3306/app")
+    assert(MysqlConfig(host = "[::1]", database = "app").jdbcUrl == "jdbc:mysql://[::1]:3306/app")
+  }
+
+  test("by default, only the prepared statement cache is set") {
+    assert(MysqlConfig(database = "app").dataSourceProperties == prepStmtCache)
+  }
+
+  test("turning off the prepared statement cache drops its sizes") {
+    assert(MysqlConfig(database = "app", cachePrepStmts = false).dataSourceProperties == mapOf("cachePrepStmts" to "false"))
+  }
+
+  test("settings become the driver's connection properties") {
+    val config = MysqlConfig(
+      database = "app",
+      sslMode = MysqlSslMode.VERIFY_IDENTITY,
+      allowPublicKeyRetrieval = true,
+      connectTimeout = 3.seconds,
+      socketTimeout = 60.seconds,
+      rewriteBatchedStatements = true,
+    )
+    assert(
+      config.dataSourceProperties == prepStmtCache + mapOf(
+        "sslMode" to "VERIFY_IDENTITY",
+        "allowPublicKeyRetrieval" to "true",
+        "connectTimeout" to "3000",
+        "socketTimeout" to "60000",
+        "rewriteBatchedStatements" to "true",
+      )
+    )
+  }
+
+  test("toString masks the password") {
+    val string = MysqlConfig(database = "app", username = "app", password = "secret").toString()
+    assert("secret" !in string && "password=****" in string && "username=app" in string)
+  }
+}

boot-db-jdbc-mysql/src/test/kotlin/com/github/rocketraman/bootable/db/jdbc/MysqlDatabaseTest.kt

@@ -0,0 +1,64 @@
+package com.github.rocketraman.bootable.db.jdbc
+
+import com.github.rocketraman.bootable.db.flyway.FlywayMigrationService
+import de.infix.testBalloon.framework.core.TestConfig
+import de.infix.testBalloon.framework.core.disable
+import de.infix.testBalloon.framework.core.testSuite
+import org.testcontainers.DockerClientFactory
+import org.testcontainers.mysql.MySQLContainer
+import javax.sql.DataSource
+
+/**
+ * Against a real MySQL server, in a container. Skipped where Docker is unavailable, except on CI, where it must
+ * run. With Podman, point `DOCKER_HOST` at its socket, e.g. `unix:///run/user/1000/podman/podman.sock`.
+ */
+val MysqlDatabaseTest by testSuite(
+  testConfig = if (System.getenv("CI") != null || DockerClientFactory.instance().isDockerAvailable) TestConfig
+  else TestConfig.disable()
+) {
+  val server = testFixture {
+    MySQLContainer("mysql:8.4").apply { start() }
+  } closeWith { stop() }
+
+  suspend fun config() = server().let {
+    MysqlConfig(
+      host = it.host,
+      port = it.firstMappedPort,
+      database = it.databaseName,
+      username = it.username,
+      password = it.password,
+    )
+  }
+
+  test("connections carry the configured settings") {
+    JdbcDataSourceService(config().copy(sslMode = MysqlSslMode.REQUIRED), HikariConnectionPoolFactory()).use {
+      // the second column of SHOW STATUS is the value: a cipher only on an SSL connection
+      assert(it.queryString("SHOW SESSION STATUS LIKE 'Ssl_cipher'", column = 2).isNotEmpty())
+    }
+  }
+
+  test("flyway migrates the database") {
+    JdbcDataSourceService(config(), HikariConnectionPoolFactory()).use {
+      FlywayMigrationService(it).start()
+      assert(it.queryString("SELECT message FROM greeting WHERE id = 1") == "hello")
+    }
+  }
+}
+
+private fun JdbcDataSourceService.use(block: (DataSource) -> Unit) {
+  start()
+  try {
+    block(dataSource)
+  } finally {
+    shutdown()
+  }
+}
+
+private fun DataSource.queryString(sql: String, column: Int = 1): String = connection.use { connection ->
+  connection.createStatement().use { statement ->
+    statement.executeQuery(sql).use { rs ->
+      rs.next()
+      rs.getString(column)
+    }
+  }
+}

boot-db-jdbc-mysql/src/test/kotlin/com/github/rocketraman/bootable/db/jdbc/MysqlWiringTest.kt

@@ -0,0 +1,88 @@
+package com.github.rocketraman.bootable.db.jdbc
+
+import com.github.rocketraman.bootable.boot.AppService
+import com.github.rocketraman.bootable.boot.BootScope
+import com.github.rocketraman.bootable.boot.BootableGraph
+import com.github.rocketraman.bootable.boot.bootableModule
+import com.github.rocketraman.bootable.config.config
+import com.github.rocketraman.bootable.config.hopliteConfigBinder
+import com.github.rocketraman.bootable.config.hopliteConfigModule
+import com.zaxxer.hikari.HikariDataSource
+import de.infix.testBalloon.framework.core.testSuite
+import dev.zacsweers.metro.DependencyGraph
+import dev.zacsweers.metro.createGraph
+import org.kodein.di.DI
+import org.kodein.di.direct
+import org.kodein.di.instance
+import javax.sql.DataSource
+import kotlin.time.Duration.Companion.seconds
+
+sealed interface TestScope
+
+@DependencyGraph(TestScope::class, additionalScopes = [BootScope::class])
+interface TestGraph : BootableGraph {
+  val dataSource: DataSource
+  val jdbcConfig: JdbcConfig
+}
+
+private val expectedConfig = MysqlConfig(
+  host = "db.test",
+  port = 3307,
+  database = "app",
+  username = "app",
+  password = "secret",
+  sslMode = MysqlSslMode.VERIFY_IDENTITY,
+  connectTimeout = 3.seconds,
+)
+
+/**
+ * The pool built from the decoded config. The server in the config doesn't exist, so nothing is started: the pool
+ * opens no connection until then.
+ */
+private fun HikariDataSource.assertBuiltFromConfig() {
+  assert(jdbcUrl == "jdbc:mysql://db.test:3307/app")
+  assert(username == "app" && password == "secret")
+  assert(dataSourceProperties.getProperty("sslMode") == "VERIFY_IDENTITY")
+  assert(poolName == "wiring-test")
+}
+
+/** Config comes from src/test/resources/application.conf, as it would in an application. */
+val MysqlWiringTest by testSuite {
+  testSuite("typed config decoded by") {
+    test("hoplite") {
+      assert(hopliteConfigBinder(report = false).config<MysqlConfig>("db.jdbc") == expectedConfig)
+    }
+
+    test("hoplite in strict mode, alongside the pool config") {
+      val binder = hopliteConfigBinder(report = false, strict = true)
+      binder.config<MysqlConfig>("db.jdbc")
+      binder.config<HikariPoolConfig>("db.pool")
+    }
+
+    test("kodein") {
+      val di = DI {
+        import(bootableModule)
+        import(hopliteConfigModule)
+        import(jdbcDataSourceModule)
+        import(mysqlConfigModule)
+        import(hikariPoolModule)
+      }.direct
+
+      assert(di.instance<JdbcConfig>() == expectedConfig)
+      val hikari = di.instance<DataSource>() as HikariDataSource
+      hikari.assertBuiltFromConfig()
+      assert(di.instance<Set<AppService>>().filterIsInstance<JdbcDataSourceService>().single().dataSource === hikari)
+      hikari.close()
+    }
+
+    test("metro") {
+      val graph = createGraph<TestGraph>()
+
+      assert(graph.jdbcConfig == expectedConfig)
+      val hikari = graph.dataSource as HikariDataSource
+      hikari.assertBuiltFromConfig()
+      assert(graph.appServices.filterIsInstance<JdbcDataSourceService>().single().dataSource === hikari)
+      hikari.close()
+    }
+  }
+}