From c8cc75c5d834f04bf585b7968d557fe4c504fda9 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 16 Sep 2026 02:51:24 +0800 Subject: [PATCH 1/4] [refactor](fe) Move the JDBC driver-jar policy into the connector SPI and add getPrimaryKeys/executeQuery Issue Number: close #xxx Related PR: #xxx Problem Summary: Part 1 of removing every JDBC data-source implementation from fe-core: the connector SPI grows what the streaming/CDC framework needs from a JDBC source, and the driver-jar policy stops being an engine service. - `ConnectorTableMetadataOps.getPrimaryKeys` (a JDBC table answers `isKey` for every column, so the real constraint needs its own method) and `ConnectorPassthroughSqlOps.executeQuery(session, sql, params)` with the `ConnectorQueryResult` value type; parameters bind positionally so a name a caller took from user input never becomes SQL text. - `ConnectorValidationContext` loses `validateAndResolveDriverPath` and `computeDriverChecksum`. The whole policy (url grammar, url white list, structural secure-path matching, bare-name resolution across the current and pre-2.1 drivers directories, checksum) moves to `DriverUrlPolicy` in fe-connector-spi, applied by the jdbc, iceberg and paimon connectors themselves; fe.conf's `jdbc_driver_secure_path` / `jdbc_driver_url_white_list` reach it through the engine environment. The only engine service left is `ConnectorContext.fetchPluginFile`, the cloud deployment's object-store copy of a missing driver jar. - The jdbc connector applies the policy at load time as well as at CREATE (the pre-SPI `JdbcClient` did too), so a catalog whose driver sits outside a since-tightened allow-list fails to connect instead of loading the jar. - Three fixes that had only landed in the fe-core client are ported to the connector clients: composite primary keys ordered by KEY_SEQ (#64740), the PostgreSQL neighbour-table column leak through LIKE wildcards (#63402), and PostgreSQL array element types (#61433; `bpchar[]` elements are STRING). - Plugin API version 8.0 -> 9.0, both surface baselines regenerated. A JDBC catalog now applies `jdbc_driver_secure_path` / `jdbc_driver_url_white_list` when it loads its driver, not only when it is created. PostgreSQL `char(n)[]` columns of a JDBC catalog map to `ARRAY` again instead of `ARRAY`. - Test: Unit Test - fe-connector-spi suite (incl. new DriverUrlPolicyTest), fe-connector-jdbc suite (new PostgreSQL/OceanBase/base-client tests, pre-create validation and load-time policy tests), fe-connector-paimon and fe-connector-iceberg suites. - Behavior changed: Yes (see release note) - Does this need documentation: No Co-Authored-By: Claude Opus 5 --- .../adbc/AdbcDriverPathResolver.java | 2 +- .../connector/iceberg/IcebergConnector.java | 10 +- .../connector/jdbc/JdbcConnectorMetadata.java | 15 + .../connector/jdbc/JdbcDorisConnector.java | 70 ++- .../jdbc/client/JdbcConnectorClient.java | 73 ++- .../jdbc/client/JdbcMySQLConnectorClient.java | 8 +- .../client/JdbcPostgreSQLConnectorClient.java | 11 +- .../jdbc/JdbcConnectorMetadataTest.java | 42 ++ .../jdbc/JdbcDorisConnectorTest.java | 108 ++++- .../doris/connector/jdbc/client/FakeJdbc.java | 230 ++++++++++ .../jdbc/client/JdbcConnectorClientTest.java | 101 +++++ .../JdbcOceanBaseConnectorClientTest.java | 82 ++++ .../JdbcPostgreSQLConnectorClientTest.java | 113 +++++ .../connector/paimon/PaimonConnector.java | 17 +- ...aimonConnectorPreCreateValidationTest.java | 156 ++++--- .../paimon/RecordingConnectorContext.java | 8 + .../doris/connector/spi/ConnectorContext.java | 22 + .../spi/ConnectorPassthroughSqlOps.java | 28 +- .../connector/spi/ConnectorQueryResult.java | 71 +++ .../spi/ConnectorTableMetadataOps.java | 17 +- .../spi/ConnectorValidationContext.java | 34 +- .../doris/connector/spi/DriverUrlPolicy.java | 421 ++++++++++++++++++ .../spi/ForwardingConnectorContext.java | 6 + .../spi/ConnectorPluginSurfaceTest.java | 7 +- .../connector/spi/DriverUrlPolicyTest.java | 296 ++++++++++++ .../resources/connector-metadata-methods.txt | 1 + .../resources/connector-plugin-surface.txt | 1 + fe/fe-connector/pom.xml | 2 +- .../connector/DefaultConnectorContext.java | 27 ++ .../DefaultConnectorValidationContext.java | 18 +- ...efaultConnectorContextEnvironmentTest.java | 35 ++ 31 files changed, 1841 insertions(+), 191 deletions(-) create mode 100644 fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/FakeJdbc.java create mode 100644 fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/JdbcConnectorClientTest.java create mode 100644 fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/JdbcOceanBaseConnectorClientTest.java create mode 100644 fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/JdbcPostgreSQLConnectorClientTest.java create mode 100644 fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorQueryResult.java create mode 100644 fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/DriverUrlPolicy.java create mode 100644 fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/DriverUrlPolicyTest.java diff --git a/fe/fe-connector/fe-connector-adbc/src/main/java/org/apache/doris/connector/adbc/AdbcDriverPathResolver.java b/fe/fe-connector/fe-connector-adbc/src/main/java/org/apache/doris/connector/adbc/AdbcDriverPathResolver.java index 778a39e8bf4705..4c7c08bbc25647 100644 --- a/fe/fe-connector/fe-connector-adbc/src/main/java/org/apache/doris/connector/adbc/AdbcDriverPathResolver.java +++ b/fe/fe-connector/fe-connector-adbc/src/main/java/org/apache/doris/connector/adbc/AdbcDriverPathResolver.java @@ -40,7 +40,7 @@ * file, and a URL each node downloads for itself cannot promise that. Rejecting remote schemes outright * keeps that failure -- which surfaces as an unreadable partition, far from its cause -- unreachable. * - *

This does NOT reuse {@code ConnectorValidationContext#validateAndResolveDriverPath}: that one resolves + *

This does NOT reuse the shared {@code DriverUrlPolicy} of fe-connector-spi: that one resolves * against {@code jdbc_drivers_dir} and enforces a {@code .jar} grammar. */ public final class AdbcDriverPathResolver { diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java index 0d263a6b52d1cb..ad8cdfcdd3c98c 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java @@ -38,6 +38,7 @@ import org.apache.doris.connector.spi.ConnectorTestResult; import org.apache.doris.connector.spi.ConnectorValidationContext; import org.apache.doris.connector.spi.DorisConnectorException; +import org.apache.doris.connector.spi.DriverUrlPolicy; import org.apache.doris.connector.spi.handle.ConnectorTableHandle; import org.apache.doris.connector.spi.mvcc.ConnectorMvccPartitionView; import org.apache.doris.connector.spi.procedure.ConnectorProcedureOps; @@ -1549,9 +1550,9 @@ static boolean pinPoolThreadsToClassLoader(ExecutorService pool, int poolSize, C /** * Enforces JDBC driver-url security at CREATE CATALOG (mirrors {@code PaimonConnector.preCreateValidation}): - * for the jdbc flavor a configured {@code iceberg.jdbc.driver_url} is routed through the engine's - * {@link ConnectorValidationContext#validateAndResolveDriverPath} hook (the FE format / - * {@code jdbc_driver_url_white_list} / {@code jdbc_driver_secure_path} gates), so a rejected url fails + * for the jdbc flavor a configured {@code iceberg.jdbc.driver_url} is checked against the FE's shared + * driver-jar policy ({@link DriverUrlPolicy}: format / {@code jdbc_driver_url_white_list} / + * {@code jdbc_driver_secure_path}, fed from the engine environment), so a rejected url fails * CREATE CATALOG before the jar is ever loaded by {@link #maybeRegisterJdbcDriver}. Non-jdbc flavors are * a no-op. */ @@ -1562,7 +1563,8 @@ public void preCreateValidation(ConnectorValidationContext validationContext) th } String driverUrl = IcebergJdbcMetaStoreProperties.of(properties).getDriverUrl(); if (StringUtils.isNotBlank(driverUrl)) { - validationContext.validateAndResolveDriverPath(driverUrl); + DriverUrlPolicy.resolve(driverUrl, + DriverUrlPolicy.Settings.fromContext(context, configuredDriversDir())); } } diff --git a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/JdbcConnectorMetadata.java b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/JdbcConnectorMetadata.java index 4ee67840d5deff..4ee58952c913aa 100644 --- a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/JdbcConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/JdbcConnectorMetadata.java @@ -22,6 +22,7 @@ import org.apache.doris.connector.spi.ConnectorColumn; import org.apache.doris.connector.spi.ConnectorMetadata; import org.apache.doris.connector.spi.ConnectorPassthroughSqlOps; +import org.apache.doris.connector.spi.ConnectorQueryResult; import org.apache.doris.connector.spi.ConnectorSession; import org.apache.doris.connector.spi.ConnectorStatementScopes; import org.apache.doris.connector.spi.ConnectorTableSchema; @@ -145,6 +146,15 @@ public ConnectorTableSchema getTableSchema( return new ConnectorTableSchema(tableName, columns, "JDBC", props.getRaw()); } + @Override + public List getPrimaryKeys(ConnectorSession session, ConnectorTableHandle handle) { + if (handle instanceof PassthroughQueryTableHandle) { + return Collections.emptyList(); + } + JdbcTableHandle jdbcHandle = (JdbcTableHandle) handle; + return client.getPrimaryKeys(jdbcHandle.getRemoteDbName(), jdbcHandle.getRemoteTableName()); + } + @Override public Optional getTableStatistics( ConnectorSession session, ConnectorTableHandle handle) { @@ -250,6 +260,11 @@ public void executeStmt(ConnectorSession session, String stmt) { client.executeStmt(stmt); } + @Override + public ConnectorQueryResult executeQuery(ConnectorSession session, String sql, List params) { + return client.executeQuery(sql, params); + } + @Override public ConnectorTableSchema getColumnsFromQuery(ConnectorSession session, String query) { List fields = client.getColumnsFromQuery(query); diff --git a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/JdbcDorisConnector.java b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/JdbcDorisConnector.java index 52f79f8e27809f..212f59516bed46 100644 --- a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/JdbcDorisConnector.java +++ b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/JdbcDorisConnector.java @@ -26,6 +26,7 @@ import org.apache.doris.connector.spi.ConnectorTestResult; import org.apache.doris.connector.spi.ConnectorValidationContext; import org.apache.doris.connector.spi.DorisConnectorException; +import org.apache.doris.connector.spi.DriverUrlPolicy; import org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider; import org.apache.doris.connector.spi.write.ConnectorWritePlanProvider; import org.apache.doris.thrift.TJdbcTable; @@ -37,7 +38,6 @@ import org.apache.logging.log4j.Logger; import org.apache.thrift.TSerializer; -import java.io.File; import java.io.IOException; import java.net.URI; import java.net.URISyntaxException; @@ -184,10 +184,10 @@ public void preCreateValidation(ConnectorValidationContext context) throws Excep if (driverUrl != null && !driverUrl.isEmpty()) { // Mandatory, non-configurable security rule, enforced on catalog creation only. checkDriverUrlSecurityRule(driverUrl); - context.validateAndResolveDriverPath(driverUrl); + String fullDriverUrl = DriverUrlPolicy.resolve(driverUrl, driverUrlSettings()); // 2. Compute and verify checksum. - String computedChecksum = context.computeDriverChecksum(driverUrl); + String computedChecksum = computeDriverChecksum(driverUrl, fullDriverUrl); String providedChecksum = context.getProperty(JdbcCatalogProperties.DRIVER_CHECKSUM); if (providedChecksum != null && !providedChecksum.isEmpty()) { if (!providedChecksum.equals(computedChecksum)) { @@ -252,7 +252,7 @@ private JdbcConnectorClient createClient() { JdbcDbType dbType = JdbcDbType.parseFromUrl(jdbcUrl); String user = props.getUser(); String password = props.getPassword(); - String driverUrl = resolveDriverUrl(props.getDriverUrl()); + String driverUrl = resolveDriverUrlForLoad(props.getDriverUrl()); String driverClass = props.getDriverClass(); int poolMinSize = props.getConnectionPoolMinSize(); int poolMaxSize = props.getConnectionPoolMaxSize(); @@ -286,46 +286,40 @@ public void close() throws IOException { } /** - * Resolves driver URL against the configured drivers directory. - * If the URL is a plain filename (e.g., "mysql-connector-j-8.4.0.jar"), - * resolves it under {@code drivers_dir} from this plugin's jdbc.conf, or fe.conf's - * {@code jdbc_drivers_dir}. + * The FE's driver-jar policy, fed from this plugin's own {@code jdbc.conf} (the drivers directory) and + * the engine environment (install root, allow-lists, the external plugin store). */ - private String resolveDriverUrl(String driverUrl) { + private DriverUrlPolicy.Settings driverUrlSettings() { + return DriverUrlPolicy.Settings.fromContext(context, JdbcConf.driversDir(context)); + } + + /** + * Resolves the driver_url the client loads its driver from, applying the FE's driver-jar policy + * ({@link DriverUrlPolicy}) at load time as well as at CREATE: the {@code jdbc_driver_secure_path} / + * {@code jdbc_driver_url_white_list} allow-lists, the bare-name lookup under the drivers directory + * (current and pre-2.1 default locations), and the external plugin store of a cloud deployment. + * A catalog whose driver sits outside a since-tightened allow-list therefore fails to connect rather + * than loading a jar the deployment no longer permits. + */ + private String resolveDriverUrlForLoad(String driverUrl) { if (driverUrl == null || driverUrl.isEmpty()) { return driverUrl; } - if (driverUrl.startsWith("file://") || driverUrl.startsWith("http://") - || driverUrl.startsWith("https://") || driverUrl.startsWith("/")) { - return driverUrl; + String resolved = DriverUrlPolicy.resolve(driverUrl, driverUrlSettings()); + if (!resolved.equals(driverUrl)) { + LOG.info("Resolved driver_url '{}' to '{}'", driverUrl, resolved); } - // Plain filename — resolve under the configured drivers directory. doris_home is engine-wide - // rather than this connector's setting, so it keeps coming from the engine environment. - String driversDir = JdbcConf.driversDir(context); - String dorisHome = JdbcConf.dorisHome(context); - if (driversDir != null && !driversDir.isEmpty()) { - String newPath = driversDir + "/" + driverUrl; - if (new File(newPath).exists()) { - return "file://" + newPath; - } - // Backward compatibility: check the old default directory - // (DORIS_HOME/jdbc_drivers) when the user hasn't customized jdbc_drivers_dir - if (dorisHome != null) { - String defaultNewDir = dorisHome + "/plugins/jdbc_drivers"; - if (driversDir.equals(defaultNewDir)) { - String oldPath = dorisHome + "/jdbc_drivers/" + driverUrl; - if (new File(oldPath).exists()) { - LOG.info("Resolved driver_url '{}' from old default directory: {}", - driverUrl, oldPath); - return "file://" + oldPath; - } - } - } - String resolved = "file://" + newPath; - LOG.info("Resolved driver_url '{}' to '{}' using jdbc_drivers_dir", driverUrl, resolved); - return resolved; + return resolved; + } + + /** The MD5 of the driver jar, remote fetches going through the engine's outbound-request hook. */ + private String computeDriverChecksum(String driverUrl, String fullDriverUrl) { + try { + return DriverUrlPolicy.checksum(fullDriverUrl, context.getHttpSecurityHook()); + } catch (IOException e) { + throw new DorisConnectorException("compute driver checksum from url: " + driverUrl + + " meet an IOException: " + e.getMessage(), e); } - return "file://" + driverUrl; } private TTableDescriptor buildTestTableDescriptor(ConnectorValidationContext context) { diff --git a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcConnectorClient.java b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcConnectorClient.java index e1ad6c33b523b3..5972babf50e966 100644 --- a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcConnectorClient.java +++ b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcConnectorClient.java @@ -18,6 +18,7 @@ package org.apache.doris.connector.jdbc.client; import org.apache.doris.connector.jdbc.JdbcDbType; +import org.apache.doris.connector.spi.ConnectorQueryResult; import org.apache.doris.connector.spi.ConnectorType; import org.apache.doris.connector.spi.DorisConnectorException; @@ -42,6 +43,7 @@ import java.util.List; import java.util.Map; import java.util.Set; +import java.util.TreeMap; import java.util.concurrent.ConcurrentHashMap; import java.util.function.Consumer; import java.util.function.UnaryOperator; @@ -438,19 +440,16 @@ public List getJdbcColumnsInfo(String remoteDbName, String remote } /** - * Get primary keys of one table. + * Get primary keys of one table, in key order. */ public List getPrimaryKeys(String remoteDbName, String remoteTableName) { Connection conn = getConnection(); ResultSet rs = null; - List primaryKeys = new ArrayList<>(); try { DatabaseMetaData databaseMetaData = conn.getMetaData(); String cat = getCatalogName(conn); rs = databaseMetaData.getPrimaryKeys(cat, remoteDbName, remoteTableName); - while (rs.next()) { - primaryKeys.add(rs.getString("COLUMN_NAME")); - } + return readPrimaryKeysInKeyOrder(rs); } catch (SQLException e) { throw new DorisConnectorException( "Failed to get primary keys for " + remoteDbName + "." + remoteTableName @@ -458,7 +457,29 @@ public List getPrimaryKeys(String remoteDbName, String remoteTableName) } finally { closeResources(rs, conn); } - return primaryKeys; + } + + /** + * Reads a {@link DatabaseMetaData#getPrimaryKeys} result into column names ordered by {@code KEY_SEQ}. + * The JDBC contract orders the rows by COLUMN_NAME, not by position in the key, so a composite key + * read in row order comes back alphabetized — and a UNIQUE KEY table built from it would key on the + * wrong column order. Drivers that report no KEY_SEQ (0 for every row) keep row order. + */ + protected static List readPrimaryKeysInKeyOrder(ResultSet rs) throws SQLException { + TreeMap byKeySeq = new TreeMap<>(); + List inRowOrder = new ArrayList<>(); + boolean allSeqsKnown = true; + while (rs.next()) { + String column = rs.getString("COLUMN_NAME"); + int keySeq = rs.getShort("KEY_SEQ"); + inRowOrder.add(column); + if (keySeq <= 0 || byKeySeq.containsKey(keySeq)) { + allSeqsKnown = false; + } else { + byKeySeq.put(keySeq, column); + } + } + return allSeqsKnown ? new ArrayList<>(byKeySeq.values()) : inRowOrder; } /** @@ -497,6 +518,46 @@ public void executeStmt(String origStmt) { } } + /** + * Runs a read-only query with positional parameters and materializes every row, values as the driver's + * {@code getObject} answers them. For the engine's small probe queries only. + */ + public ConnectorQueryResult executeQuery(String sql, List params) { + Connection conn = null; + PreparedStatement pstmt = null; + ResultSet rs = null; + try { + conn = getConnection(); + pstmt = conn.prepareStatement(sql); + if (params != null) { + for (int i = 0; i < params.size(); i++) { + pstmt.setObject(i + 1, params.get(i)); + } + } + rs = pstmt.executeQuery(); + ResultSetMetaData metaData = rs.getMetaData(); + int columnCount = metaData.getColumnCount(); + List columnNames = new ArrayList<>(columnCount); + for (int i = 1; i <= columnCount; i++) { + columnNames.add(metaData.getColumnLabel(i)); + } + List> rows = new ArrayList<>(); + while (rs.next()) { + List row = new ArrayList<>(columnCount); + for (int i = 1; i <= columnCount; i++) { + row.add(rs.getObject(i)); + } + rows.add(row); + } + return new ConnectorQueryResult(columnNames, rows); + } catch (SQLException e) { + throw new DorisConnectorException("Failed to execute query: " + sql + ": " + + getAllExceptionMessages(e), e); + } finally { + closeResources(rs, pstmt, conn); + } + } + /** * Get column metadata from a query by preparing it and reading ResultSetMetaData. */ diff --git a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcMySQLConnectorClient.java b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcMySQLConnectorClient.java index 9874d1b2460646..73a7d480437dee 100644 --- a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcMySQLConnectorClient.java +++ b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcMySQLConnectorClient.java @@ -215,20 +215,18 @@ public long getRowCount(String dbName, String tableName) { public List getPrimaryKeys(String remoteDbName, String remoteTableName) { Connection conn = getConnection(); ResultSet rs = null; - List primaryKeys = new ArrayList<>(); try { DatabaseMetaData databaseMetaData = conn.getMetaData(); + // MySQL exposes databases as JDBC catalogs, hence (db, null, table) rather than the base + // class's (catalog, schema, table). rs = databaseMetaData.getPrimaryKeys(remoteDbName, null, remoteTableName); - while (rs.next()) { - primaryKeys.add(rs.getString("COLUMN_NAME")); - } + return readPrimaryKeysInKeyOrder(rs); } catch (SQLException e) { throw new DorisConnectorException( "Failed to get primary keys for " + remoteDbName + "." + remoteTableName, e); } finally { closeResources(rs, conn); } - return primaryKeys; } @Override diff --git a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcPostgreSQLConnectorClient.java b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcPostgreSQLConnectorClient.java index 54b2eebd67c30d..8345644d867364 100644 --- a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcPostgreSQLConnectorClient.java +++ b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcPostgreSQLConnectorClient.java @@ -81,6 +81,13 @@ public List getJdbcColumnsInfo(String remoteDbName, String remote String cat = getCatalogName(conn); rs = getRemoteColumns(meta, cat, remoteDbName, remoteTableName); while (rs.next()) { + // getColumns treats schema/table as LIKE patterns and the pattern is deliberately left + // unescaped (see escapeSearchPattern), so a `_` in the name also matches a neighbour + // table (t_1 vs tx1); drop the rows that belong to one. + if (!remoteDbName.equals(rs.getString("TABLE_SCHEM")) + || !remoteTableName.equals(rs.getString("TABLE_NAME"))) { + continue; + } int sqlType = rs.getInt("DATA_TYPE"); if (sqlType == Types.ARRAY) { int arrayDim = getArrayDimensions(conn, remoteDbName, rs.getString("COLUMN_NAME"), @@ -247,7 +254,9 @@ private ConnectorType mapPgInnerType(String innerType, JdbcFieldInfo fieldInfo) case "bool": return ConnectorType.of("BOOLEAN"); case "bpchar": - return ConnectorType.of("CHAR", fieldInfo.requiredColumnSize(), -1); + // A Doris CHAR(n) measures bytes; a PostgreSQL char(n) measures characters. Inside an + // array there is no place to widen it, so the element stays a STRING (as the pre-SPI + // resolver did). case "varchar": case "text": case "json": diff --git a/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/JdbcConnectorMetadataTest.java b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/JdbcConnectorMetadataTest.java index f0c91cdf3a409f..bb0364d5cc3cb2 100644 --- a/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/JdbcConnectorMetadataTest.java +++ b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/JdbcConnectorMetadataTest.java @@ -21,11 +21,13 @@ import org.apache.doris.connector.jdbc.client.JdbcFieldInfo; import org.apache.doris.connector.spi.ConnectorColumn; import org.apache.doris.connector.spi.ConnectorPushdownOps; +import org.apache.doris.connector.spi.ConnectorQueryResult; import org.apache.doris.connector.spi.ConnectorSession; import org.apache.doris.connector.spi.ConnectorStatementScope; import org.apache.doris.connector.spi.ConnectorTableSchema; import org.apache.doris.connector.spi.ConnectorType; import org.apache.doris.connector.spi.handle.ConnectorColumnHandle; +import org.apache.doris.connector.spi.handle.PassthroughQueryTableHandle; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -283,6 +285,46 @@ void getTableSchemaStableOverSharedMutatedMemoUnderMutatingTypeConversion() { "the mutating type conversion's idempotent allowNull=true is reflected in the schema"); } + @Test + void primaryKeysAndProbeQueriesReachTheClient() { + List pkRequests = new ArrayList<>(); + List queries = new ArrayList<>(); + JdbcConnectorClient client = new JdbcConnectorClient("test_catalog", JdbcDbType.MYSQL, + "jdbc:mysql://h:3306/test_db", false, null, null, false, false) { + @Override + public List getPrimaryKeys(String remoteDbName, String remoteTableName) { + pkRequests.add(remoteDbName + "." + remoteTableName); + return Arrays.asList("k2", "k1"); + } + + @Override + public ConnectorQueryResult executeQuery(String sql, List params) { + queries.add(sql + params); + return new ConnectorQueryResult(Collections.singletonList("v"), + Collections.singletonList(Collections.singletonList("MYSQL"))); + } + + @Override + public ConnectorType jdbcTypeToConnectorType(JdbcFieldInfo fieldInfo) { + return ConnectorType.of("INT"); + } + }; + JdbcConnectorMetadata md = new JdbcConnectorMetadata(client, minimalCatalogProps()); + ConnectorSession session = sessionWithProps(Collections.emptyMap()); + + // The key order the client reports is the key order the engine gets - it builds a UNIQUE KEY on it. + Assertions.assertEquals(Arrays.asList("k2", "k1"), + md.getPrimaryKeys(session, new JdbcTableHandle("db", "t"))); + Assertions.assertEquals(Collections.singletonList("db.t"), pkRequests); + // A passthrough query has no table, hence no primary key. + Assertions.assertTrue(md.getPrimaryKeys(session, new PassthroughQueryTableHandle("select 1")).isEmpty()); + + ConnectorQueryResult result = md.executeQuery(session, "SHOW VARIABLES LIKE ?", + Collections.singletonList("ob_compatibility_mode")); + Assertions.assertEquals("MYSQL", result.getRows().get(0).get(0)); + Assertions.assertEquals(Collections.singletonList("SHOW VARIABLES LIKE ?[ob_compatibility_mode]"), queries); + } + @Test void allNamespacesArePrefixedWithConnectorType() throws Exception { // NORM (self-extending): reflect over every "*_NAMESPACE" constant this connector declares and assert each diff --git a/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/JdbcDorisConnectorTest.java b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/JdbcDorisConnectorTest.java index 8940d142c3f249..084d8cf4f91f53 100644 --- a/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/JdbcDorisConnectorTest.java +++ b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/JdbcDorisConnectorTest.java @@ -22,7 +22,9 @@ import org.apache.doris.connector.spi.ConnectorContractValidator; import org.apache.doris.connector.spi.ConnectorPassthroughSqlOps; import org.apache.doris.connector.spi.ConnectorSession; +import org.apache.doris.connector.spi.ConnectorValidationContext; import org.apache.doris.connector.spi.DorisConnectorException; +import org.apache.doris.connector.spi.DriverUrlPolicy; import org.apache.doris.connector.spi.handle.ConnectorTransaction; import org.apache.doris.connector.spi.handle.NoOpConnectorTransaction; import org.apache.doris.connector.spi.handle.WriteOperation; @@ -30,8 +32,12 @@ import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; import java.util.Collections; import java.util.EnumSet; import java.util.HashMap; @@ -40,6 +46,10 @@ class JdbcDorisConnectorTest { private static ConnectorContext testContext() { + return testContext(Collections.emptyMap()); + } + + private static ConnectorContext testContext(Map environment) { return new ConnectorContext() { @Override public String getCatalogName() { @@ -53,11 +63,43 @@ public long getCatalogId() { @Override public Map getEnvironment() { - return Collections.emptyMap(); + return environment; + } + }; + } + + /** A validation context over a plain property map, the way the engine's one wraps the catalog's. */ + private static ConnectorValidationContext validationContext(Map props) { + return new ConnectorValidationContext() { + @Override + public long getCatalogId() { + return 1L; + } + + @Override + public String getProperty(String key) { + return props.get(key); + } + + @Override + public void storeProperty(String key, String value) { + props.put(key, value); + } + + @Override + public void requestBeConnectivityTest(byte[] serializedDescriptor, int connectionTypeValue, + String testQuery) { + props.put("be_test_requested", "true"); } }; } + private static Path writeJar(Path dir, String name) throws IOException { + Path jar = dir.resolve(name); + Files.write(jar, "not really a jar".getBytes(StandardCharsets.UTF_8)); + return jar; + } + private static Map minimalProps() { Map props = new HashMap<>(); props.put("jdbc_url", "jdbc:mysql://localhost:3306/test"); @@ -119,6 +161,70 @@ void testDeclaresPassthroughSqlByImplementingTheOptionalInterface() { "jdbc must implement ConnectorPassthroughSqlOps or query()/EXECUTE_STMT stop admitting it"); } + @Test + void preCreateValidationResolvesTheDriverAndStoresItsChecksum(@TempDir Path dir) throws Exception { + // The driver-jar policy now runs inside the connector: a bare name resolves under the drivers + // directory the plugin was told about, and the checksum of that file is what the catalog stores. + writeJar(dir, "mysql.jar"); + Map env = new HashMap<>(); + env.put(DriverUrlPolicy.ENV_DRIVERS_DIR, dir.toString()); + env.put(DriverUrlPolicy.ENV_DORIS_HOME, dir.toString()); + Map props = new HashMap<>(); + props.put(JdbcCatalogProperties.JDBC_URL, "jdbc:mysql://localhost:3306/test"); + props.put(JdbcCatalogProperties.DRIVER_URL, "mysql.jar"); + props.put(JdbcCatalogProperties.DRIVER_CLASS, "com.mysql.cj.jdbc.Driver"); + props.put(JdbcCatalogProperties.TEST_CONNECTION, "false"); + JdbcDorisConnector connector = new JdbcDorisConnector(props, testContext(env)); + + connector.preCreateValidation(validationContext(props)); + + // The checksum is that of the file the bare name resolved to. + Assertions.assertEquals( + DriverUrlPolicy.checksum(dir.resolve("mysql.jar").toUri().toString(), null), + props.get(JdbcCatalogProperties.DRIVER_CHECKSUM)); + Assertions.assertNull(props.get("be_test_requested"), "test_connection=false requests no BE test"); + + // A checksum the user supplies is verified against the file, not overwritten. + props.put(JdbcCatalogProperties.DRIVER_CHECKSUM, "0000"); + DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, + () -> new JdbcDorisConnector(props, testContext(env)).preCreateValidation(validationContext(props))); + Assertions.assertTrue(e.getMessage().contains("does not match the computed checksum"), e.getMessage()); + } + + @Test + void preCreateValidationRejectsADriverOutsideTheAllowedPaths(@TempDir Path dir) throws Exception { + writeJar(dir, "evil.jar"); + Map env = new HashMap<>(); + env.put(DriverUrlPolicy.ENV_DRIVER_SECURE_PATH, "file:///opt/doris/jdbc_drivers"); + Map props = new HashMap<>(); + props.put(JdbcCatalogProperties.JDBC_URL, "jdbc:mysql://localhost:3306/test"); + props.put(JdbcCatalogProperties.DRIVER_URL, dir.resolve("evil.jar").toUri().toString()); + props.put(JdbcCatalogProperties.DRIVER_CLASS, "com.mysql.cj.jdbc.Driver"); + JdbcDorisConnector connector = new JdbcDorisConnector(props, testContext(env)); + + Assertions.assertThrows(IllegalArgumentException.class, + () -> connector.preCreateValidation(validationContext(props))); + Assertions.assertNull(props.get(JdbcCatalogProperties.DRIVER_CHECKSUM), "a rejected driver stores nothing"); + } + + @Test + void loadingTheDriverAppliesTheAllowedPathsToo(@TempDir Path dir) throws Exception { + // Not only CREATE: the same policy runs when the client is built, so a catalog whose driver sits + // outside a since-tightened jdbc_driver_secure_path fails to connect instead of loading the jar. + writeJar(dir, "evil.jar"); + Map env = new HashMap<>(); + env.put(DriverUrlPolicy.ENV_DRIVER_SECURE_PATH, "file:///opt/doris/jdbc_drivers"); + Map props = new HashMap<>(); + props.put(JdbcCatalogProperties.JDBC_URL, "jdbc:postgresql://localhost:5432/test"); + props.put(JdbcCatalogProperties.DRIVER_URL, dir.resolve("evil.jar").toUri().toString()); + props.put(JdbcCatalogProperties.DRIVER_CLASS, "java.lang.Object"); + JdbcDorisConnector connector = new JdbcDorisConnector(props, testContext(env)); + + IllegalArgumentException e = Assertions.assertThrows(IllegalArgumentException.class, + connector::getWritePlanProvider); + Assertions.assertTrue(e.getMessage().contains("does not match any allowed paths"), e.getMessage()); + } + @Test void testDoubleCloseNoException() throws IOException { JdbcDorisConnector connector = new JdbcDorisConnector(minimalProps(), testContext()); diff --git a/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/FakeJdbc.java b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/FakeJdbc.java new file mode 100644 index 00000000000000..434b8f92e6aa87 --- /dev/null +++ b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/FakeJdbc.java @@ -0,0 +1,230 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + + +package org.apache.doris.connector.jdbc.client; + +import java.lang.reflect.InvocationHandler; +import java.lang.reflect.Method; +import java.lang.reflect.Proxy; +import java.sql.Connection; +import java.sql.DatabaseMetaData; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.ResultSetMetaData; +import java.sql.SQLException; +import java.sql.Statement; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.function.Function; + +/** + * Reflection-proxy stand-ins for the handful of {@code java.sql} interfaces the metadata clients touch, + * so a client's row handling can be exercised without a database or a mocking framework (the connector + * poms carry none, deliberately). + * + *

A fake {@link ResultSet} is a list of rows, each a column-name-to-value map; {@code getXxx(label)} + * reads the current row, {@code getXxx(index)} reads by 1-based position in the row's insertion order. + * Anything not modelled throws, so a client reaching for more than the fake offers fails loud instead of + * silently reading {@code null}.

+ */ +final class FakeJdbc { + + private FakeJdbc() { + } + + /** A row of a fake result set, in insertion order. */ + static Map row(Object... labelValuePairs) { + Map row = new LinkedHashMap<>(); + for (int i = 0; i < labelValuePairs.length; i += 2) { + row.put((String) labelValuePairs[i], labelValuePairs[i + 1]); + } + return row; + } + + static ResultSet resultSet(List> rows) { + return (ResultSet) Proxy.newProxyInstance(FakeJdbc.class.getClassLoader(), + new Class[] {ResultSet.class}, new ResultSetHandler(rows)); + } + + /** + * A connection whose metadata answers {@code getColumns} and {@code getPrimaryKeys} with the given + * result sets, and whose statements run {@code queries} (sql -> result set); a prepared statement + * records every bound parameter into {@code boundParams} in bind order. + */ + static Connection connection(ResultSet columns, ResultSet primaryKeys, + Function queries, List boundParams) { + DatabaseMetaData meta = (DatabaseMetaData) Proxy.newProxyInstance(FakeJdbc.class.getClassLoader(), + new Class[] {DatabaseMetaData.class}, (proxy, method, args) -> { + switch (method.getName()) { + case "getColumns": + return columns; + case "getPrimaryKeys": + return primaryKeys; + case "getSearchStringEscape": + return "\\"; + default: + throw new UnsupportedOperationException("DatabaseMetaData." + method.getName()); + } + }); + return (Connection) Proxy.newProxyInstance(FakeJdbc.class.getClassLoader(), + new Class[] {Connection.class}, (proxy, method, args) -> { + switch (method.getName()) { + case "getMetaData": + return meta; + case "getCatalog": + case "getSchema": + return null; + case "prepareStatement": + return preparedStatement(queries.apply((String) args[0]), boundParams); + case "createStatement": + return statement(queries); + case "close": + return null; + default: + throw new UnsupportedOperationException("Connection." + method.getName()); + } + }); + } + + private static Statement statement(Function queries) { + return (Statement) Proxy.newProxyInstance(FakeJdbc.class.getClassLoader(), + new Class[] {Statement.class}, (proxy, method, args) -> { + switch (method.getName()) { + case "executeQuery": { + ResultSet result = queries.apply((String) args[0]); + if (result == null) { + throw new SQLException("no result set configured for: " + args[0]); + } + return result; + } + case "close": + return null; + default: + throw new UnsupportedOperationException("Statement." + method.getName()); + } + }); + } + + private static PreparedStatement preparedStatement(ResultSet result, List boundParams) { + return (PreparedStatement) Proxy.newProxyInstance(FakeJdbc.class.getClassLoader(), + new Class[] {PreparedStatement.class}, (proxy, method, args) -> { + switch (method.getName()) { + case "setObject": + case "setString": + boundParams.add(args[1]); + return null; + case "executeQuery": + if (result == null) { + throw new SQLException("no result set configured for this statement"); + } + return result; + case "close": + return null; + default: + throw new UnsupportedOperationException("PreparedStatement." + method.getName()); + } + }); + } + + private static final class ResultSetHandler implements InvocationHandler { + private final List> rows; + private int cursor = -1; + private boolean lastWasNull; + + private ResultSetHandler(List> rows) { + this.rows = rows; + } + + private Object value(Object[] args) throws SQLException { + if (cursor < 0 || cursor >= rows.size()) { + throw new SQLException("cursor is not on a row"); + } + Map row = rows.get(cursor); + Object v; + if (args[0] instanceof Integer) { + int index = (Integer) args[0]; + v = new ArrayList<>(row.values()).get(index - 1); + } else { + String label = (String) args[0]; + if (!row.containsKey(label)) { + throw new SQLException("no such column: " + label); + } + v = row.get(label); + } + lastWasNull = v == null; + return v; + } + + @Override + public Object invoke(Object proxy, Method method, Object[] args) throws Throwable { + switch (method.getName()) { + case "next": + cursor++; + return cursor < rows.size(); + case "getString": { + Object v = value(args); + return v == null ? null : v.toString(); + } + case "getObject": + return value(args); + case "getInt": { + Object v = value(args); + return v == null ? 0 : ((Number) v).intValue(); + } + case "getShort": { + Object v = value(args); + return v == null ? (short) 0 : ((Number) v).shortValue(); + } + case "getLong": { + Object v = value(args); + return v == null ? 0L : ((Number) v).longValue(); + } + case "getBoolean": { + Object v = value(args); + return v != null && (Boolean) v; + } + case "wasNull": + return lastWasNull; + case "getMetaData": + return metaData(); + case "close": + return null; + default: + throw new UnsupportedOperationException("ResultSet." + method.getName()); + } + } + + private ResultSetMetaData metaData() { + List labels = rows.isEmpty() ? new ArrayList<>() : new ArrayList<>(rows.get(0).keySet()); + return (ResultSetMetaData) Proxy.newProxyInstance(FakeJdbc.class.getClassLoader(), + new Class[] {ResultSetMetaData.class}, (proxy, method, args) -> { + switch (method.getName()) { + case "getColumnCount": + return labels.size(); + case "getColumnLabel": + case "getColumnName": + return labels.get((Integer) args[0] - 1); + default: + throw new UnsupportedOperationException("ResultSetMetaData." + method.getName()); + } + }); + } + } +} diff --git a/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/JdbcConnectorClientTest.java b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/JdbcConnectorClientTest.java new file mode 100644 index 00000000000000..d1b0a34be845cb --- /dev/null +++ b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/JdbcConnectorClientTest.java @@ -0,0 +1,101 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + + +package org.apache.doris.connector.jdbc.client; + +import org.apache.doris.connector.jdbc.JdbcDbType; +import org.apache.doris.connector.spi.ConnectorQueryResult; +import org.apache.doris.connector.spi.DorisConnectorException; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.sql.Connection; +import java.sql.ResultSet; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Map; + +public class JdbcConnectorClientTest { + + private static Map pk(String column, int keySeq) { + return FakeJdbc.row("COLUMN_NAME", column, "KEY_SEQ", keySeq); + } + + @Test + public void primaryKeysAreOrderedByKeySeqNotByRowOrder() throws Exception { + // DatabaseMetaData.getPrimaryKeys orders rows by COLUMN_NAME, so a composite key (b, a) arrives as + // (a, b); KEY_SEQ is the real position (#64740). + ResultSet rs = FakeJdbc.resultSet(Arrays.asList(pk("a", 2), pk("b", 1), pk("c", 3))); + Assertions.assertEquals(Arrays.asList("b", "a", "c"), JdbcConnectorClient.readPrimaryKeysInKeyOrder(rs)); + } + + @Test + public void primaryKeysKeepRowOrderWhenKeySeqIsNotReported() throws Exception { + ResultSet rs = FakeJdbc.resultSet(Arrays.asList(pk("a", 0), pk("b", 0))); + Assertions.assertEquals(Arrays.asList("a", "b"), JdbcConnectorClient.readPrimaryKeysInKeyOrder(rs)); + ResultSet dup = FakeJdbc.resultSet(Arrays.asList(pk("a", 1), pk("b", 1))); + Assertions.assertEquals(Arrays.asList("a", "b"), JdbcConnectorClient.readPrimaryKeysInKeyOrder(dup)); + Assertions.assertTrue(JdbcConnectorClient.readPrimaryKeysInKeyOrder( + FakeJdbc.resultSet(Collections.emptyList())).isEmpty()); + } + + private static JdbcConnectorClient clientOver(Connection connection) { + return new JdbcPostgreSQLConnectorClient("test_catalog", JdbcDbType.POSTGRESQL, + "jdbc:postgresql://localhost:5432/test", false, + Collections.emptyMap(), Collections.emptyMap(), false, false) { + @Override + public Connection getConnection() { + return connection; + } + }; + } + + @Test + public void executeQueryBindsParametersAndMaterializesRows() { + List bound = new ArrayList<>(); + ResultSet slots = FakeJdbc.resultSet(Arrays.asList( + FakeJdbc.row("slot_name", "doris_cdc_1", "active", Boolean.TRUE), + FakeJdbc.row("slot_name", "other", "active", null))); + Connection conn = FakeJdbc.connection(null, null, + sql -> sql.startsWith("SELECT slot_name") ? slots : null, bound); + + ConnectorQueryResult result = clientOver(conn).executeQuery( + "SELECT slot_name, active FROM pg_replication_slots WHERE slot_name = ?", + Collections.singletonList("doris_cdc_1")); + + // The caller's value reaches the source as a bound parameter, never as SQL text. + Assertions.assertEquals(Collections.singletonList("doris_cdc_1"), bound); + Assertions.assertEquals(Arrays.asList("slot_name", "active"), result.getColumnNames()); + Assertions.assertEquals(2, result.getRows().size()); + // Values are the driver's objects: a PostgreSQL boolean is a Boolean, SQL NULL is null. + Assertions.assertEquals(Boolean.TRUE, result.getRows().get(0).get(1)); + Assertions.assertNull(result.getRows().get(1).get(1)); + Assertions.assertEquals("other", result.getRows().get(1).get(0)); + } + + @Test + public void executeQueryFailuresCarryTheStatement() { + Connection conn = FakeJdbc.connection(null, null, sql -> null, new ArrayList<>()); + DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, + () -> clientOver(conn).executeQuery("SELECT 1", Collections.emptyList())); + Assertions.assertTrue(e.getMessage().contains("SELECT 1"), e.getMessage()); + } +} diff --git a/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/JdbcOceanBaseConnectorClientTest.java b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/JdbcOceanBaseConnectorClientTest.java new file mode 100644 index 00000000000000..45966da1b8d733 --- /dev/null +++ b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/JdbcOceanBaseConnectorClientTest.java @@ -0,0 +1,82 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + + +package org.apache.doris.connector.jdbc.client; + +import org.apache.doris.connector.jdbc.JdbcDbType; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.sql.Connection; +import java.sql.ResultSet; +import java.sql.Types; +import java.util.ArrayList; +import java.util.Collections; +import java.util.Optional; + +public class JdbcOceanBaseConnectorClientTest { + + private static JdbcOceanBaseConnectorClient clientAnswering(String compatibilityMode) { + ResultSet mode = FakeJdbc.resultSet(compatibilityMode == null + ? Collections.emptyList() + : Collections.singletonList(FakeJdbc.row("ob_compatibility_mode()", compatibilityMode))); + Connection conn = FakeJdbc.connection(null, null, + sql -> sql.contains("ob_compatibility_mode") ? mode : null, new ArrayList<>()); + return new JdbcOceanBaseConnectorClient("test_catalog", JdbcDbType.OCEANBASE, + "jdbc:oceanbase://localhost:2881/test", false, + Collections.emptyMap(), Collections.emptyMap(), false, false) { + @Override + public Connection getConnection() { + return conn; + } + }; + } + + private static JdbcFieldInfo number() { + // NUMBER(10,0) is an Oracle-mode type; MySQL mode has no such type name. + return new JdbcFieldInfo("col", Optional.of("NUMBER"), Types.NUMERIC, Optional.of(10), Optional.of(0), + Optional.empty()); + } + + @Test + public void oracleModeDelegatesToTheOracleMapping() { + JdbcOceanBaseConnectorClient client = clientAnswering("ORACLE"); + // The compatibility mode is probed lazily, on the first delegated call. + Assertions.assertEquals(JdbcDbType.OCEANBASE, client.getDbType()); + // NUMBER(10,0) is BIGINT under the Oracle mapping (10 integer digits do not fit an INT). + Assertions.assertEquals("BIGINT", client.jdbcTypeToConnectorType(number()).getTypeName()); + Assertions.assertEquals(JdbcDbType.OCEANBASE_ORACLE, client.getDbType()); + } + + @Test + public void mysqlModeDelegatesToTheMysqlMapping() { + JdbcOceanBaseConnectorClient client = clientAnswering("MYSQL"); + Assertions.assertEquals("UNSUPPORTED", client.jdbcTypeToConnectorType(number()).getTypeName()); + Assertions.assertEquals(JdbcDbType.OCEANBASE, client.getDbType()); + } + + @Test + public void unknownOrFailedProbeDefaultsToMysqlMode() { + for (JdbcOceanBaseConnectorClient client : new JdbcOceanBaseConnectorClient[] { + clientAnswering(null), clientAnswering("SOMETHING")}) { + Assertions.assertEquals("UNSUPPORTED", client.jdbcTypeToConnectorType(number()).getTypeName()); + Assertions.assertEquals(JdbcDbType.OCEANBASE, client.getDbType()); + } + } +} diff --git a/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/JdbcPostgreSQLConnectorClientTest.java b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/JdbcPostgreSQLConnectorClientTest.java new file mode 100644 index 00000000000000..5b5c4a1356476a --- /dev/null +++ b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/JdbcPostgreSQLConnectorClientTest.java @@ -0,0 +1,113 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + + +package org.apache.doris.connector.jdbc.client; + +import org.apache.doris.connector.jdbc.JdbcDbType; +import org.apache.doris.connector.spi.ConnectorType; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.sql.Connection; +import java.sql.Types; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +public class JdbcPostgreSQLConnectorClientTest { + + private static JdbcPostgreSQLConnectorClient client(Connection connection) { + return new JdbcPostgreSQLConnectorClient("test_catalog", JdbcDbType.POSTGRESQL, + "jdbc:postgresql://localhost:5432/test", false, + Collections.emptyMap(), Collections.emptyMap(), false, false) { + @Override + public Connection getConnection() { + return connection; + } + }; + } + + private static Map columnRow(String schema, String table, String column, String typeName, + int dataType) { + return FakeJdbc.row("TABLE_SCHEM", schema, "TABLE_NAME", table, "COLUMN_NAME", column, + "DATA_TYPE", dataType, "TYPE_NAME", typeName, "COLUMN_SIZE", 10, "DECIMAL_DIGITS", 0, + "NUM_PREC_RADIX", 10, "NULLABLE", 1, "REMARKS", "", "CHAR_OCTET_LENGTH", 0); + } + + @Test + public void columnsOfNeighbourTablesMatchedByLikeWildcardsAreDropped() { + // getColumns() treats the names as LIKE patterns and the client does not escape '_' (some drivers + // cannot take an escape there), so a lookup of t_1 also returns the columns of tx1 and of the same + // table in schema sX. Only the rows of the requested schema.table may come back (#63402). + List> rows = Arrays.asList( + columnRow("s_1", "t_1", "id", "int4", Types.INTEGER), + columnRow("s_1", "tx1", "leaked_col", "text", Types.VARCHAR), + columnRow("sx1", "t_1", "leaked_col2", "text", Types.VARCHAR), + columnRow("s_1", "t_1", "name", "varchar", Types.VARCHAR)); + Connection conn = FakeJdbc.connection(FakeJdbc.resultSet(rows), null, sql -> null, new ArrayList<>()); + + List fields = client(conn).getJdbcColumnsInfo("s_1", "t_1"); + + List names = new ArrayList<>(); + for (JdbcFieldInfo f : fields) { + names.add(f.getColumnName()); + } + Assertions.assertEquals(Arrays.asList("id", "name"), names); + } + + private static JdbcFieldInfo field(String typeName, int dataType, int dims) { + return new JdbcFieldInfo("col", Optional.of(typeName), dataType, Optional.of(10), Optional.of(0), + Optional.of(dims)); + } + + @Test + public void arrayElementTypes() { + JdbcPostgreSQLConnectorClient client = client(null); + + ConnectorType intArray = client.jdbcTypeToConnectorType(field("_int4", Types.ARRAY, 1)); + Assertions.assertEquals("ARRAY", intArray.getTypeName()); + Assertions.assertEquals("INT", intArray.getChildren().get(0).getTypeName()); + + ConnectorType nested = client.jdbcTypeToConnectorType(field("_int8", Types.ARRAY, 2)); + Assertions.assertEquals("ARRAY", nested.getTypeName()); + Assertions.assertEquals("ARRAY", nested.getChildren().get(0).getTypeName()); + Assertions.assertEquals("BIGINT", nested.getChildren().get(0).getChildren().get(0).getTypeName()); + + // A Doris CHAR(n) counts bytes, a PostgreSQL char(n) counts characters; inside an array there is + // nowhere to widen it, so the element is a STRING (as before the SPI migration), not CHAR(n). + ConnectorType charArray = client.jdbcTypeToConnectorType(field("_bpchar", Types.ARRAY, 1)); + Assertions.assertEquals("STRING", charArray.getChildren().get(0).getTypeName()); + + // Element types outside the mapped set degrade to STRING, matching the CDC client's mapping. + ConnectorType macaddrArray = client.jdbcTypeToConnectorType(field("_macaddr", Types.ARRAY, 1)); + Assertions.assertEquals("STRING", macaddrArray.getChildren().get(0).getTypeName()); + } + + @Test + public void scalarTypesAddedForStreaming() { + JdbcPostgreSQLConnectorClient client = client(null); + for (String pgType : new String[] {"macaddr8", "xml", "hstore"}) { + Assertions.assertEquals("STRING", + client.jdbcTypeToConnectorType(field(pgType, Types.OTHER, 0)).getTypeName(), pgType); + } + } +} diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java index b00efbcb57712c..6dcbeb9892be32 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java @@ -33,6 +33,7 @@ import org.apache.doris.connector.spi.ConnectorStorageContext; import org.apache.doris.connector.spi.ConnectorTestResult; import org.apache.doris.connector.spi.ConnectorValidationContext; +import org.apache.doris.connector.spi.DriverUrlPolicy; import org.apache.doris.connector.spi.handle.ConnectorTableHandle; import org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider; import org.apache.doris.filesystem.Location; @@ -620,11 +621,10 @@ private static boolean sameCacheKey(String required, String existing) { /** * Enforces JDBC driver-url security at CREATE CATALOG (rereview2 B-8b). For the JDBC flavor a * configured {@code driver_url} — read from either the {@code jdbc.driver_url} or the - * {@code paimon.jdbc.driver_url} alias — is routed through the engine's - * {@link ConnectorValidationContext#validateAndResolveDriverPath} hook, which applies the FE - * format / {@code jdbc_driver_url_white_list} / {@code jdbc_driver_secure_path} gates (legacy - * {@code JdbcResource.getFullDriverUrl}). A rejected url throws here, so CREATE CATALOG fails - * before the jar is ever loaded into the FE JVM by {@link #maybeRegisterJdbcDriver}. Mirrors + * {@code paimon.jdbc.driver_url} alias — is checked against the FE's shared driver-jar policy + * ({@link DriverUrlPolicy}: format / {@code jdbc_driver_url_white_list} / {@code jdbc_driver_secure_path}, + * fed from the engine environment). A rejected url throws here, so CREATE CATALOG fails before the jar + * is ever loaded into the FE JVM by {@link #maybeRegisterJdbcDriver}. Mirrors * {@code JdbcDorisConnector.preCreateValidation}; non-JDBC flavors are a no-op. */ @Override @@ -634,7 +634,8 @@ public void preCreateValidation(ConnectorValidationContext validationContext) th } String driverUrl = PaimonJdbcMetaStoreProperties.of(catalogProps.getRaw()).getDriverUrl(); if (StringUtils.isNotBlank(driverUrl)) { - validationContext.validateAndResolveDriverPath(driverUrl); + DriverUrlPolicy.resolve(driverUrl, + DriverUrlPolicy.Settings.fromContext(context, PaimonConf.driversDir(context))); } } @@ -664,8 +665,8 @@ private void maybeRegisterJdbcDriver() { * *

FE security validation (format / {@code jdbc_driver_url_white_list} / * {@code jdbc_driver_secure_path}) is enforced at CREATE CATALOG by {@link #preCreateValidation} - * via the engine's {@code ConnectorValidationContext.validateAndResolveDriverPath} hook — a - * rejected url fails catalog creation before this path is ever reached. Like the JDBC reference + * through the shared {@link DriverUrlPolicy} — a rejected url fails catalog creation before this + * path is ever reached. Like the JDBC reference * connector ({@code JdbcDorisConnector}), validation is CREATE-time only; catalogs reloaded after * an FE restart or reconfigured via ALTER CATALOG are not re-validated against a since-tightened * allow-list (a pre-existing fe-core gap shared by all plugin connectors — see deviations-log). diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorPreCreateValidationTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorPreCreateValidationTest.java index fb8b324da87d1f..f42673f2d7e6c5 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorPreCreateValidationTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorPreCreateValidationTest.java @@ -18,136 +18,134 @@ package org.apache.doris.connector.paimon; import org.apache.doris.connector.spi.ConnectorValidationContext; +import org.apache.doris.connector.spi.DriverUrlPolicy; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; -import java.util.ArrayList; -import java.util.Collections; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; import java.util.HashMap; -import java.util.List; import java.util.Map; /** * Tests for {@link PaimonConnector#preCreateValidation} (rereview2 B-8b): a JDBC-flavor catalog - * with a {@code driver_url} must route it through the engine's - * {@link ConnectorValidationContext#validateAndResolveDriverPath} security gate at CREATE CATALOG, - * before the jar is ever loaded into the FE JVM. Mirrors {@code JdbcDorisConnector.preCreateValidation}. + * with a {@code driver_url} must pass the FE's shared driver-jar policy ({@link DriverUrlPolicy}) at + * CREATE CATALOG, before the jar is ever loaded into the FE JVM. Mirrors + * {@code JdbcDorisConnector.preCreateValidation}. * - *

Offline: a hand-written {@link RecordingValidationContext} fake records each validated url and - * can simulate a rejected url. {@link RecordingConnectorContext} supplies the (unused-by-this-path) - * {@code ConnectorContext}. + *

Offline: the policy reads its settings off {@link RecordingConnectorContext#environment}, so a + * temporary drivers directory stands in for {@code jdbc_drivers_dir} and a {@code jdbc_driver_secure_path} + * that excludes the url simulates a rejected one. */ public class PaimonConnectorPreCreateValidationTest { - private static PaimonConnector connector(Map props) { - return new PaimonConnector(props, new RecordingConnectorContext()); + /** + * A context whose FE install root is {@code home}, with the default drivers directory + * ({@code home/plugins/jdbc_drivers}) — the layout in which the policy checks that a bare jar name + * actually exists. + */ + private static RecordingConnectorContext contextWithDriversDir(Path home) { + RecordingConnectorContext context = new RecordingConnectorContext(); + Map env = new HashMap<>(); + env.put(DriverUrlPolicy.ENV_DORIS_HOME, home.toString()); + env.put(DriverUrlPolicy.ENV_DRIVERS_DIR, home.resolve("plugins/jdbc_drivers").toString()); + context.environment = env; + return context; } + private static Path driversDirWith(Path home, String jar) throws IOException { + Path driversDir = Files.createDirectories(home.resolve("plugins/jdbc_drivers")); + Files.write(driversDir.resolve(jar), new byte[] {1}); + return home; + } + + /** Hand-written {@link ConnectorValidationContext} test double (no Mockito); nothing on it is consulted. */ + private static final ConnectorValidationContext VALIDATION_CONTEXT = new ConnectorValidationContext() { + @Override + public long getCatalogId() { + return 0; + } + + @Override + public String getProperty(String key) { + return null; + } + + @Override + public void storeProperty(String key, String value) { + } + + @Override + public void requestBeConnectivityTest(byte[] serializedDescriptor, int connectionTypeValue, + String testQuery) { + } + }; + @Test - public void validatesJdbcDriverUrl() throws Exception { + public void validatesJdbcDriverUrl(@TempDir Path dir) throws Exception { Map props = new HashMap<>(); props.put("paimon.catalog.type", "jdbc"); props.put("jdbc.driver_url", "mysql.jar"); - RecordingValidationContext ctx = new RecordingValidationContext(); - - connector(props).preCreateValidation(ctx); // WHY (BLOCKER B-8b): a jdbc driver_url is loaded into the FE JVM (URLClassLoader); CREATE - // CATALOG must route it through the engine's format / white-list / secure-path gate. MUTATION: - // dropping the preCreateValidation override -> validateAndResolveDriverPath never called -> red. - Assertions.assertEquals(Collections.singletonList("mysql.jar"), ctx.validatedDriverUrls); + // CATALOG must put it through the format / white-list / secure-path policy. A bare name that + // exists nowhere the policy looks is rejected; the same name present in the drivers directory + // passes. MUTATION: dropping the preCreateValidation override -> nothing rejected -> red. + Assertions.assertThrows(RuntimeException.class, () -> new PaimonConnector(props, + contextWithDriversDir(dir)).preCreateValidation(VALIDATION_CONTEXT)); + new PaimonConnector(props, contextWithDriversDir(driversDirWith(dir, "mysql.jar"))) + .preCreateValidation(VALIDATION_CONTEXT); } @Test - public void validatesPaimonJdbcDriverUrlAlias() throws Exception { + public void validatesPaimonJdbcDriverUrlAlias(@TempDir Path dir) throws Exception { Map props = new HashMap<>(); props.put("paimon.catalog.type", "jdbc"); props.put("paimon.jdbc.driver_url", "mysql.jar"); - RecordingValidationContext ctx = new RecordingValidationContext(); - - connector(props).preCreateValidation(ctx); - Assertions.assertEquals(Collections.singletonList("mysql.jar"), ctx.validatedDriverUrls, + Assertions.assertThrows(RuntimeException.class, () -> new PaimonConnector(props, + contextWithDriversDir(dir)).preCreateValidation(VALIDATION_CONTEXT), "the paimon.jdbc.driver_url alias must also be validated"); + new PaimonConnector(props, contextWithDriversDir(driversDirWith(dir, "mysql.jar"))) + .preCreateValidation(VALIDATION_CONTEXT); } @Test - public void skipsValidationForNonJdbcFlavor() throws Exception { + public void skipsValidationForNonJdbcFlavor(@TempDir Path dir) throws Exception { Map props = new HashMap<>(); props.put("paimon.catalog.type", "filesystem"); props.put("jdbc.driver_url", "mysql.jar"); - RecordingValidationContext ctx = new RecordingValidationContext(); - connector(props).preCreateValidation(ctx); - - Assertions.assertTrue(ctx.validatedDriverUrls.isEmpty(), - "non-JDBC flavors must not trigger driver-url validation"); + // The name exists nowhere, which would be rejected for the jdbc flavor. + new PaimonConnector(props, contextWithDriversDir(dir)).preCreateValidation(VALIDATION_CONTEXT); } @Test - public void skipsValidationWhenNoDriverUrl() throws Exception { + public void skipsValidationWhenNoDriverUrl(@TempDir Path dir) throws Exception { Map props = new HashMap<>(); props.put("paimon.catalog.type", "jdbc"); - RecordingValidationContext ctx = new RecordingValidationContext(); - - connector(props).preCreateValidation(ctx); - Assertions.assertTrue(ctx.validatedDriverUrls.isEmpty(), - "a jdbc catalog without a driver_url uses the platform driver -> nothing to validate"); + // a jdbc catalog without a driver_url uses the platform driver -> nothing to validate + new PaimonConnector(props, contextWithDriversDir(dir)).preCreateValidation(VALIDATION_CONTEXT); } @Test - public void propagatesRejectedDriverUrl() { + public void propagatesRejectedDriverUrl(@TempDir Path dir) throws Exception { Map props = new HashMap<>(); props.put("paimon.catalog.type", "jdbc"); props.put("jdbc.driver_url", "http://evil.test/x.jar"); - RecordingValidationContext ctx = new RecordingValidationContext(); - ctx.reject = true; + RecordingConnectorContext context = contextWithDriversDir(dir); + Map env = new HashMap<>(context.environment); + env.put(DriverUrlPolicy.ENV_DRIVER_SECURE_PATH, "http://good.test/drivers"); + context.environment = env; - // WHY (BLOCKER B-8b): a disallowed url must FAIL CREATE CATALOG — the hook throws and the + // WHY (BLOCKER B-8b): a disallowed url must FAIL CREATE CATALOG — the policy throws and the // connector must let it propagate, not swallow it. MUTATION: catching the exception -> no // throw -> red. Assertions.assertThrows(IllegalArgumentException.class, - () -> connector(props).preCreateValidation(ctx)); - } - - /** Hand-written {@link ConnectorValidationContext} test double (no Mockito). */ - private static final class RecordingValidationContext implements ConnectorValidationContext { - final List validatedDriverUrls = new ArrayList<>(); - boolean reject; - - @Override - public long getCatalogId() { - return 0; - } - - @Override - public String getProperty(String key) { - return null; - } - - @Override - public void storeProperty(String key, String value) { - } - - @Override - public String validateAndResolveDriverPath(String driverUrl) throws Exception { - validatedDriverUrls.add(driverUrl); - if (reject) { - throw new IllegalArgumentException("disallowed driver url: " + driverUrl); - } - return "file:///resolved/" + driverUrl; - } - - @Override - public String computeDriverChecksum(String driverUrl) { - return "deadbeef"; - } - - @Override - public void requestBeConnectivityTest(byte[] serializedDescriptor, int connectionTypeValue, - String testQuery) { - } + () -> new PaimonConnector(props, context).preCreateValidation(VALIDATION_CONTEXT)); } } diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/RecordingConnectorContext.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/RecordingConnectorContext.java index e858c0debab111..cf4ed3e06963aa 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/RecordingConnectorContext.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/RecordingConnectorContext.java @@ -127,6 +127,14 @@ public long getCatalogId() { return 0; } + /** The engine environment the fake hands the connector (default: none). */ + Map environment = Collections.emptyMap(); + + @Override + public Map getEnvironment() { + return environment; + } + @Override public T executeAuthenticated(Callable task) throws Exception { authCount++; diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorContext.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorContext.java index da568c66b471e9..91ecbdf8436770 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorContext.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorContext.java @@ -19,6 +19,7 @@ import java.util.Collections; import java.util.Map; +import java.util.Optional; import java.util.concurrent.Callable; /** @@ -47,6 +48,8 @@ public interface ConnectorContext { *

  • {@code doris_home} — the DORIS_HOME path
    • *
  • {@code hadoop_config_dir} — the configured Hadoop resource directory
    • *
  • {@code jdbc_drivers_dir} — the configured JDBC drivers directory
    • + *
  • {@code jdbc_driver_secure_path}, {@code jdbc_driver_url_white_list} — the FE's driver-jar + * allow-lists, consumed through {@link DriverUrlPolicy}
    • * */ default Map getEnvironment() { @@ -177,4 +180,23 @@ default Connector createSiblingConnector(String catalogType, Map default ConnectorStorageContext getStorageContext() { return ConnectorStorageContext.NOOP; } + + /** + * Fetches a plugin file that is missing from the FE's local plugin directory from the deployment's + * external plugin store, when the deployment has one (a cloud deployment keeps its JDBC driver jars + * and Java UDF jars in its object store and copies them down on first use). + * + *

    {@code category} names the store's file category ({@code "jdbc_drivers"}), {@code fileName} is the + * bare file name and {@code targetPath} is where the engine should place the copy. Returns the local + * path of the fetched file, or {@link Optional#empty()} when this deployment has no such store — the + * caller then reports the file as missing. Throws when the store exists but the fetch fails; the message + * names the file.

    + * + *

    The only engine service left on the driver-jar path: the policy that decides whether a jar may be + * loaded at all is {@link DriverUrlPolicy}, applied by the connector. Fetching needs the engine's + * metaservice client, which no plugin has.

    + */ + default Optional fetchPluginFile(String category, String fileName, String targetPath) { + return Optional.empty(); + } } diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorPassthroughSqlOps.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorPassthroughSqlOps.java index 02202f729b07ad..016b6627836d44 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorPassthroughSqlOps.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorPassthroughSqlOps.java @@ -17,6 +17,8 @@ package org.apache.doris.connector.spi; +import java.util.List; + /** * Passing a SQL string through to the remote source untouched, for a connector that fronts a system which * speaks SQL itself. @@ -27,14 +29,14 @@ * {@code SUPPORTS_PASSTHROUGH_QUERY} flag existed and was removed: it was a second overridable answer to the * question "can this connector run my SQL", and a connector could declare it while implementing nothing).

    * - *

    Both methods take a SQL string the user wrote. A connector that implements them owns the consequences: - * the engine does not parse, rewrite or authorize the statement beyond the catalog-level privilege check on - * the entry points, so an implementation must send it under the catalog's own credentials and must not widen - * what those credentials can reach.

    + *

    All three methods take a SQL string the caller wrote. A connector that implements them owns the + * consequences: the engine does not parse, rewrite or authorize the statement beyond the catalog-level + * privilege check on the entry points, so an implementation must send it under the catalog's own credentials + * and must not widen what those credentials can reach.

    * - *

    Minimum implementation set: whichever of the two the connector actually supports. Each defaults to + *

    Minimum implementation set: whichever of the three the connector actually supports. Each defaults to * refusing, so implementing the interface for the {@code query()} TVF alone does not silently claim - * {@code CALL EXECUTE_STMT} as well.

    + * {@code CALL EXECUTE_STMT} or {@link #executeQuery} as well.

    */ public interface ConnectorPassthroughSqlOps { @@ -53,4 +55,18 @@ default void executeStmt(ConnectorSession session, String stmt) { default ConnectorTableSchema getColumnsFromQuery(ConnectorSession session, String query) { throw new DorisConnectorException("getColumnsFromQuery not supported"); } + + /** + * Runs a read-only query on the remote source and returns its rows, for an engine-side probe that must + * look at the source before it commits to something (a streaming job checking that a replication slot + * exists or which compatibility mode a server runs in, say). + * + *

    {@code params} bind positionally to the {@code ?} placeholders of {@code sql}, so a name the caller + * took from user input reaches the source as a bound value and never as SQL text. The result is fully + * materialized; this is for the handful of rows a probe reads, not for data. Values are whatever the + * driver's {@code getObject} answers — see {@link ConnectorQueryResult}.

    + */ + default ConnectorQueryResult executeQuery(ConnectorSession session, String sql, List params) { + throw new DorisConnectorException("executeQuery not supported"); + } } diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorQueryResult.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorQueryResult.java new file mode 100644 index 00000000000000..08c10c69660aed --- /dev/null +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorQueryResult.java @@ -0,0 +1,71 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + + +package org.apache.doris.connector.spi; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Objects; + +/** + * The fully materialized rows of a probe query run on the remote source through + * {@link ConnectorPassthroughSqlOps#executeQuery}. + * + *

    Column names are the driver's result-set labels in result order; every row has one value per column, + * in the same order, and a value is whatever the driver's {@code getObject} answered (a {@link Boolean} for + * a PostgreSQL {@code boolean}, a {@link String} for a MySQL {@code SHOW VARIABLES} value, {@code null} for + * SQL NULL). Callers must not assume every value is a {@code String}.

    + * + *

    Immutable. The whole result lives in memory, so it is meant for the small answers an engine-side probe + * reads (a variable, a catalog row), not for data.

    + */ +public final class ConnectorQueryResult { + + private final List columnNames; + private final List> rows; + + public ConnectorQueryResult(List columnNames, List> rows) { + this.columnNames = Collections.unmodifiableList(new ArrayList<>( + Objects.requireNonNull(columnNames, "columnNames"))); + List> copied = new ArrayList<>(Objects.requireNonNull(rows, "rows").size()); + for (List row : rows) { + copied.add(Collections.unmodifiableList(new ArrayList<>(row))); + } + this.rows = Collections.unmodifiableList(copied); + } + + /** The result's column labels, in result order. */ + public List getColumnNames() { + return columnNames; + } + + /** The rows, each holding one value per column in {@link #getColumnNames()} order. */ + public List> getRows() { + return rows; + } + + public boolean isEmpty() { + return rows.isEmpty(); + } + + @Override + public String toString() { + return "ConnectorQueryResult{columns=" + columnNames + ", rows=" + rows.size() + "}"; + } +} diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorTableMetadataOps.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorTableMetadataOps.java index 4d29727e1b84d9..ccfd3113a755cd 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorTableMetadataOps.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorTableMetadataOps.java @@ -53,7 +53,7 @@ *
  • System tables: {@link #listSupportedSysTables} plus {@link #getSysTableHandle}; * {@link #isPartitionValuesSysTable} only for a system table served by the engine's generic * partition-values function rather than by a native scan.
    • - *
  • Optional: {@link #getTableComment}, {@link #renderShowCreateTableDdl}.
    • + *
  • Optional: {@link #getTableComment}, {@link #renderShowCreateTableDdl}, {@link #getPrimaryKeys}.
    • * * *

    Note that {@link #getTableComment} addresses a table by NAME, not by handle. A heterogeneous gateway @@ -218,6 +218,21 @@ default String getTableComment(ConnectorSession session, return ""; } + /** + * The names of the table's primary-key columns in key order, or an empty list when the table declares + * none or the source has no notion of one. + * + *

    This is a different question from {@link ConnectorColumn#isKey()}: that flag drives DESCRIBE's + * "Key" column and a source may answer it for every column (a JDBC table does, by the legacy convention), + * whereas a caller that builds a Doris UNIQUE KEY table from the remote schema — the streaming/CDC + * framework — needs the real constraint. The default answers "no primary key"; a connector whose source + * exposes the constraint (every JDBC dialect does, via {@code DatabaseMetaData.getPrimaryKeys}) + * overrides it.

    + */ + default List getPrimaryKeys(ConnectorSession session, ConnectorTableHandle handle) { + return Collections.emptyList(); + } + /** * Builds the Thrift {@code TTableDescriptor} that BE needs for query execution. * diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorValidationContext.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorValidationContext.java index 4a0d24cf726cfb..d6b880a314bfb1 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorValidationContext.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorValidationContext.java @@ -15,18 +15,21 @@ // specific language governing permissions and limitations // under the License. + package org.apache.doris.connector.spi; /** * Context provided to connectors during pre-creation validation (CREATE CATALOG). * - *

    The engine implements this interface to expose infrastructure services - * (driver validation, checksum computation, BE connectivity testing) that - * connectors may need during validation. Each connector type calls only the - * services relevant to its own validation logic.

    + *

    The engine implements this interface to expose the infrastructure services a connector may need + * while it validates a catalog it is about to create: reading and storing catalog properties, and + * deferring a BE-side connectivity test to the engine. Each connector type calls only the services + * relevant to its own validation logic.

    * - *

    This keeps connector-specific validation inside the connector while - * the engine provides the underlying capabilities.

    + *

    Validating and resolving a driver jar is NOT an engine service any more: the policy lives with the + * connectors, in {@link DriverUrlPolicy}, fed from {@link ConnectorContext#getEnvironment()}. Keeping it + * here meant the engine carried one connector family's file-format rules; every connector that loads a + * driver jar now applies the one shared policy itself.

    */ public interface ConnectorValidationContext { @@ -39,25 +42,6 @@ public interface ConnectorValidationContext { /** Stores a computed property back into the catalog configuration. */ void storeProperty(String key, String value); - /** - * Validates a driver URL: format, whitelist, secure_path, file existence. - * Returns the resolved full driver URL. - * - * @param driverUrl the raw driver URL from catalog properties - * @return the resolved, validated full URL - * @throws Exception if the driver URL is invalid or inaccessible - */ - String validateAndResolveDriverPath(String driverUrl) throws Exception; - - /** - * Computes the MD5 checksum for a driver file at the given URL. - * - * @param driverUrl the driver URL to checksum - * @return the hex-encoded MD5 checksum - * @throws Exception if checksum computation fails - */ - String computeDriverChecksum(String driverUrl) throws Exception; - /** * Registers a BE→external connectivity test request. The engine will * execute this test after {@code preCreateValidation()} returns by diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/DriverUrlPolicy.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/DriverUrlPolicy.java new file mode 100644 index 00000000000000..6e4e08165e1b7b --- /dev/null +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/DriverUrlPolicy.java @@ -0,0 +1,421 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + + +package org.apache.doris.connector.spi; + +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLConnection; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Objects; +import java.util.Optional; + +/** + * The FE's policy for a driver jar a connector is about to load into the FE JVM: which {@code driver_url} + * forms are accepted, where a bare jar name resolves, and which locations the deployment allows. + * + *

    One implementation, applied by every connector that loads a driver jar (the jdbc connector for its + * own driver, the iceberg and paimon connectors for their JDBC-backed metastores), so the three cannot + * drift apart and the engine carries none of it. The inputs are the FE settings the engine forwards through + * {@link ConnectorContext#getEnvironment()} — {@code jdbc_drivers_dir}, {@code doris_home}, + * {@code jdbc_driver_secure_path}, {@code jdbc_driver_url_white_list} — bundled into a {@link Settings}.

    + * + *

    {@link #resolve} is the whole rule, unchanged from the engine-side resolver it replaces:

    + *
      + *
    1. the url must be {@code file://…}, {@code http://…}, {@code https://…}, or a bare {@code name.jar}; + * anything else — including a url {@link URI} cannot parse — is rejected (fail closed, so a malformed + * url can never slip past the location checks below);
      2. + *
    2. when {@code jdbc_driver_url_white_list} is set, the url must be listed verbatim;
      3. + *
    3. a bare jar name resolves under the drivers directory: the file there, else (when that directory is + * the default one) the pre-2.1 default {@code DORIS_HOME/jdbc_drivers}, else the deployment's external + * plugin store ({@link Settings#getMissingFileFetcher()}), else "does not exist";
      4. + *
    4. a scheme-bearing url must sit under one of the {@code jdbc_driver_secure_path} prefixes, matched + * structurally (component-based) so that neither prefix confusion ({@code /opt/drivers} vs + * {@code /opt/drivers-evil}) nor path traversal can escape the allowed location; {@code "*"} or blank + * allows all.
      5. + *
    + */ +public final class DriverUrlPolicy { + + /** Timeout for both connecting and reading a driver jar for its checksum. 10 seconds is long enough. */ + private static final int HTTP_TIMEOUT_MS = 10000; + + /** Category name of driver jars in the deployment's external plugin store. */ + public static final String PLUGIN_FILE_CATEGORY_JDBC_DRIVERS = "jdbc_drivers"; + + /** Environment key: the directory a bare {@code driver_url} resolves under (fe.conf {@code jdbc_drivers_dir}). */ + public static final String ENV_DRIVERS_DIR = "jdbc_drivers_dir"; + /** Environment key: the FE install root. */ + public static final String ENV_DORIS_HOME = "doris_home"; + /** Environment key: fe.conf {@code jdbc_driver_secure_path}, semicolon-separated prefixes; "*" or blank = all. */ + public static final String ENV_DRIVER_SECURE_PATH = "jdbc_driver_secure_path"; + /** Environment key: fe.conf {@code jdbc_driver_url_white_list}, comma-separated exact urls; blank = unset. */ + public static final String ENV_DRIVER_URL_WHITE_LIST = "jdbc_driver_url_white_list"; + + private DriverUrlPolicy() { + } + + /** + * Fetches a bare-named jar that is absent from the local drivers directory from the deployment's external + * plugin store. Answers the local path of the copy, or empty when the deployment has no such store. + * Adapts {@link ConnectorContext#fetchPluginFile} for {@link #PLUGIN_FILE_CATEGORY_JDBC_DRIVERS}. + */ + @FunctionalInterface + public interface MissingFileFetcher { + Optional fetch(String fileName, String targetPath); + } + + /** The FE-level inputs of the policy. Build one from the connector's context with {@link #fromContext}. */ + public static final class Settings { + private final String driversDir; + private final String dorisHome; + private final String securePath; + private final List urlWhiteList; + private final MissingFileFetcher missingFileFetcher; + + /** + * @param driversDir directory a bare jar name resolves under; blank falls back to + * {@code /plugins/jdbc_drivers} + * @param dorisHome the FE install root; blank means "." + * @param securePath semicolon-separated allowed prefixes; {@code "*"}, blank or null allows all + * @param urlWhiteList exact urls allowed; empty means no white list + * @param missingFileFetcher the external plugin store, or null when the deployment has none + */ + public Settings(String driversDir, String dorisHome, String securePath, List urlWhiteList, + MissingFileFetcher missingFileFetcher) { + this.driversDir = driversDir; + this.dorisHome = dorisHome; + this.securePath = securePath; + List whiteList = new ArrayList<>(); + if (urlWhiteList != null) { + for (String entry : urlWhiteList) { + if (entry != null && !entry.isEmpty()) { + whiteList.add(entry); + } + } + } + this.urlWhiteList = Collections.unmodifiableList(whiteList); + this.missingFileFetcher = missingFileFetcher; + } + + /** + * Reads the FE settings off the connector's context. {@code driversDir} is passed in rather than read + * here because which settings file it comes from is the connector's business (its own {@code .conf} + * first, then fe.conf's {@code jdbc_drivers_dir}); the other three are engine-wide and stay in the + * environment. The external plugin store is {@link ConnectorContext#fetchPluginFile}. + */ + public static Settings fromContext(ConnectorContext context, String driversDir) { + Map env = context.getEnvironment() == null + ? Collections.emptyMap() : context.getEnvironment(); + String whiteList = env.get(ENV_DRIVER_URL_WHITE_LIST); + List urls = whiteList == null || whiteList.trim().isEmpty() + ? Collections.emptyList() : Arrays.asList(whiteList.split(",")); + return new Settings(driversDir, env.get(ENV_DORIS_HOME), env.get(ENV_DRIVER_SECURE_PATH), urls, + (fileName, targetPath) -> + context.fetchPluginFile(PLUGIN_FILE_CATEGORY_JDBC_DRIVERS, fileName, targetPath)); + } + + public String getDriversDir() { + return driversDir; + } + + public String getDorisHome() { + return dorisHome; + } + + public String getSecurePath() { + return securePath; + } + + public List getUrlWhiteList() { + return urlWhiteList; + } + + public MissingFileFetcher getMissingFileFetcher() { + return missingFileFetcher; + } + + /** The directory a bare jar name resolves under, with the default applied. */ + String effectiveDriversDir() { + if (driversDir != null && !driversDir.trim().isEmpty()) { + return driversDir; + } + return defaultDriversDir(); + } + + /** {@code /plugins/jdbc_drivers}: the drivers directory of a deployment that configured none. */ + String defaultDriversDir() { + String home = dorisHome == null || dorisHome.trim().isEmpty() ? "." : dorisHome; + return home + "/plugins/jdbc_drivers"; + } + + /** {@code /jdbc_drivers}, the drivers directory of releases before 2.1, still consulted. */ + String legacyDriversDir() { + String home = dorisHome == null || dorisHome.trim().isEmpty() ? "." : dorisHome; + return home + "/jdbc_drivers"; + } + } + + /** + * Validates {@code driverUrl} against the policy and resolves it to the full, scheme-bearing url the + * driver is loaded from. + * + * @throws IllegalArgumentException when the url is malformed or outside the allowed locations + * @throws RuntimeException when a bare jar name exists nowhere the policy looks (message names the file), + * or the external plugin store fails to deliver it + */ + public static String resolve(String driverUrl, Settings settings) { + Objects.requireNonNull(driverUrl, "driverUrl"); + Objects.requireNonNull(settings, "settings"); + if (!(driverUrl.startsWith("file://") || driverUrl.startsWith("http://") + || driverUrl.startsWith("https://") || driverUrl.matches("^[^:/]+\\.jar$"))) { + throw new IllegalArgumentException("Invalid driver URL format. Supported formats are: " + + "file://xxx.jar, http://xxx.jar, https://xxx.jar, or xxx.jar (without prefix)."); + } + + URI uri; + try { + uri = new URI(driverUrl); + } catch (URISyntaxException e) { + // Fail closed: an unparsable URL must never be silently accepted, otherwise the + // allowed-path check below could be bypassed by a malformed URL. + throw new IllegalArgumentException("Invalid driver URL: " + driverUrl, e); + } + + String schema = uri.getScheme(); + checkWhiteList(driverUrl, settings); + if (schema == null && !driverUrl.startsWith("/")) { + // A scheme-less driver_url is a plain jar file name resolved under the drivers directory. This + // resolver is also on the lazy load path of pre-existing catalogs (with no create/alter or replay + // context), so it deliberately applies no new restriction here: an unmodified historical catalog + // must keep resolving exactly as before. The mandatory bare-name grammar is enforced only when a + // catalog is created or altered, by the jdbc connector's checkDriverUrlSecurityRule. + return resolveBareName(driverUrl, settings); + } + + // "*" or an empty/blank value means allow all (the documented, backward-compatible contract). + String securePath = settings.getSecurePath(); + if (securePath == null || securePath.trim().isEmpty() || "*".equals(securePath.trim())) { + return driverUrl; + } + + if (!isDriverUrlAllowed(driverUrl, uri, securePath)) { + throw new IllegalArgumentException("Driver URL does not match any allowed paths: " + driverUrl); + } + return driverUrl; + } + + /** + * The MD5 of the jar at {@code fullDriverUrl} (as returned by {@link #resolve}), hex-encoded. A remote url + * goes through {@code hook} first, which is how the engine's outbound-request policy (SSRF checks) applies + * to a jar the connector fetches itself. + */ + public static String checksum(String fullDriverUrl, ConnectorHttpSecurityHook hook) throws IOException { + Objects.requireNonNull(fullDriverUrl, "fullDriverUrl"); + boolean remote = !(fullDriverUrl.startsWith("/") || fullDriverUrl.startsWith("file://")); + ConnectorHttpSecurityHook effectiveHook = hook == null ? ConnectorHttpSecurityHook.NOOP : hook; + try { + if (remote) { + effectiveHook.beforeRequest(fullDriverUrl); + } + URLConnection conn = new URL(fullDriverUrl).openConnection(); + conn.setConnectTimeout(HTTP_TIMEOUT_MS); + conn.setReadTimeout(HTTP_TIMEOUT_MS); + try (InputStream inputStream = conn.getInputStream()) { + MessageDigest digest = MessageDigest.getInstance("MD5"); + byte[] buf = new byte[4096]; + int bytesRead; + while ((bytesRead = inputStream.read(buf)) >= 0) { + digest.update(buf, 0, bytesRead); + } + return toHex(digest.digest()); + } + } catch (NoSuchAlgorithmException e) { + throw new IOException("could not find algorithm: " + e.getMessage(), e); + } catch (IOException e) { + throw e; + } catch (Exception e) { + throw new IOException(e.getMessage(), e); + } finally { + if (remote) { + effectiveHook.afterRequest(); + } + } + } + + private static void checkWhiteList(String driverUrl, Settings settings) { + // For compatibility with cloud mode, both `jdbc_driver_url_white_list` and + // `jdbc_driver_secure_path` gate a driver url. + List whiteList = settings.getUrlWhiteList(); + if (!whiteList.isEmpty() && !whiteList.contains(driverUrl)) { + throw new IllegalArgumentException("Driver URL does not match any allowed paths" + driverUrl); + } + } + + private static String resolveBareName(String driverUrl, Settings settings) { + String driversDir = settings.effectiveDriversDir(); + if (!driversDir.equals(settings.defaultDriversDir())) { + // The deployment configured its own drivers directory: resolve there, nothing else to consult. + return "file://" + driversDir + "/" + driverUrl; + } + // The default directory is in use. Its location moved from DORIS_HOME/jdbc_drivers to + // DORIS_HOME/plugins/jdbc_drivers, so the old location is still consulted for jars that never moved. + String targetPath = driversDir + "/" + driverUrl; + if (new File(targetPath).exists()) { + return "file://" + targetPath; + } + String oldTargetPath = settings.legacyDriversDir() + "/" + driverUrl; + if (new File(oldTargetPath).exists()) { + return "file://" + oldTargetPath; + } + MissingFileFetcher fetcher = settings.getMissingFileFetcher(); + if (fetcher != null) { + Optional fetched; + try { + fetched = fetcher.fetch(driverUrl, targetPath); + } catch (Exception e) { + throw new RuntimeException("Cannot download JDBC driver from cloud: " + driverUrl + + ". Please retry later or check your driver has been uploaded to cloud. Error: " + + rootCauseMessage(e), e); + } + if (fetched != null && fetched.isPresent()) { + return "file://" + fetched.get(); + } + } + throw new RuntimeException("JDBC driver file does not exist: " + driverUrl); + } + + /** + * Whether {@code driverUrl} falls under one of the semicolon-separated prefixes in {@code securePath}. + * Matching is structural (component-based) rather than a raw string prefix, so that neither prefix + * confusion ({@code /opt/drivers} vs {@code /opt/drivers-evil}) nor path traversal + * ({@code /opt/drivers/../etc}) can slip a driver outside the allowed location. + */ + private static boolean isDriverUrlAllowed(String driverUrl, URI uri, String securePath) { + String scheme = uri.getScheme(); + List allowedPaths = new ArrayList<>(); + for (String p : securePath.split(";")) { + String trimmed = p.trim(); + if (!trimmed.isEmpty()) { + allowedPaths.add(trimmed); + } + } + if ("http".equalsIgnoreCase(scheme) || "https".equalsIgnoreCase(scheme)) { + URI candidate = uri.normalize(); + return allowedPaths.stream().anyMatch(allowed -> remoteUrlMatches(candidate, allowed)); + } + // Only file:// reaches here; bare absolute paths and bare "*.jar" are handled earlier. + // A local file URL must carry no authority, query or fragment. Otherwise validation (which + // looks only at URI.getPath()) and the consumers (URLClassLoader / checksum, which act on the + // whole original URL) would address different objects — e.g. "file://attacker/dir/x.jar" is + // fetched from a remote authority, and "file:///dir/x.jar?evil" maps to a sibling file. + String authority = uri.getRawAuthority(); + if ((authority != null && !authority.isEmpty()) + || uri.getRawQuery() != null || uri.getRawFragment() != null) { + return false; + } + Path candidate = toLocalPath(driverUrl).normalize(); + return allowedPaths.stream() + .map(allowed -> toLocalPath(allowed).normalize()) + .anyMatch(candidate::startsWith); + } + + /** + * Turns a {@code file://} URL or a plain filesystem path into a {@link Path} for structural comparison. + * A {@code file://} URL is decoded exactly once via {@link URI#getPath()} so that percent-encoded + * segments (e.g. {@code %2e%2e}) are resolved into the same representation the driver-loading + * consumers ({@code URL.openStream} / {@code URLClassLoader}) will use; otherwise an encoded parent + * segment would survive normalization and escape the allowed directory. + */ + private static Path toLocalPath(String pathOrUrl) { + if (pathOrUrl.startsWith("file:")) { + try { + String decoded = new URI(pathOrUrl).getPath(); + if (decoded != null && !decoded.isEmpty()) { + return Paths.get(decoded); + } + } catch (URISyntaxException ignored) { + // fall through to literal stripping below + } + int sep = pathOrUrl.indexOf("//"); + return Paths.get(sep >= 0 ? pathOrUrl.substring(sep + 2) : pathOrUrl.substring("file:".length())); + } + return Paths.get(pathOrUrl); + } + + /** + * Structural match for remote (http/https) driver URLs: scheme, host and port must be equal, and the + * candidate path must sit under the allowed path (component-based). A bare path prefix (no scheme) can + * never authorize a remote URL. + */ + private static boolean remoteUrlMatches(URI candidate, String allowedPath) { + URI base; + try { + base = new URI(allowedPath).normalize(); + } catch (URISyntaxException e) { + return false; + } + if (base.getScheme() == null) { + return false; + } + // Scheme/host/port and the path prefix must match, and the resource-selecting components + // (user-info and query) that the checksum/classloader consumers act on must match exactly too, + // otherwise e.g. ".../download?id=approved" would authorize ".../download?id=evil". + return base.getScheme().equalsIgnoreCase(candidate.getScheme()) + && base.getHost() != null && base.getHost().equalsIgnoreCase(candidate.getHost()) + && base.getPort() == candidate.getPort() + && Objects.equals(base.getUserInfo(), candidate.getUserInfo()) + && Objects.equals(base.getRawQuery(), candidate.getRawQuery()) + && pathIsUnder(candidate.getPath(), base.getPath()); + } + + private static boolean pathIsUnder(String candidatePath, String basePath) { + Path candidate = Paths.get(candidatePath == null || candidatePath.isEmpty() ? "/" : candidatePath).normalize(); + Path base = Paths.get(basePath == null || basePath.isEmpty() ? "/" : basePath).normalize(); + return candidate.startsWith(base); + } + + private static String rootCauseMessage(Throwable t) { + Throwable p = t; + while (p.getCause() != null) { + p = p.getCause(); + } + return p.getMessage() == null ? p.getClass().getName() : p.getMessage(); + } + + private static String toHex(byte[] bytes) { + StringBuilder sb = new StringBuilder(bytes.length * 2); + for (byte b : bytes) { + sb.append(Character.forDigit((b >> 4) & 0xF, 16)).append(Character.forDigit(b & 0xF, 16)); + } + return sb.toString().toLowerCase(Locale.ROOT); + } +} diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ForwardingConnectorContext.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ForwardingConnectorContext.java index b77b8fc9f82a8e..00289bc539f6b7 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ForwardingConnectorContext.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ForwardingConnectorContext.java @@ -19,6 +19,7 @@ import java.util.Map; import java.util.Objects; +import java.util.Optional; import java.util.concurrent.Callable; /** @@ -109,4 +110,9 @@ public Connector createSiblingConnector(String catalogType, Map public ConnectorStorageContext getStorageContext() { return delegate.getStorageContext(); } + + @Override + public Optional fetchPluginFile(String category, String fileName, String targetPath) { + return delegate.fetchPluginFile(category, fileName, targetPath); + } } diff --git a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java index 22ba216e0a20c9..886b9c1c9abf4a 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java +++ b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java @@ -82,7 +82,12 @@ public void connectorApiMajorTracksTheRecordedSurfaceChange() throws IOException version.load(in); } // Latest-schema publication is explicit in major 8; older engines cannot honor the opt-in contract. - Assertions.assertEquals("8.0", version.getProperty("api.version")); + // Major 9 moved the driver-jar policy out of the engine: ConnectorValidationContext lost its two + // driver methods (validateAndResolveDriverPath / computeDriverChecksum), ConnectorContext gained + // fetchPluginFile, ConnectorMetadata gained getPrimaryKeys and ConnectorPassthroughSqlOps gained + // executeQuery. A plugin built against major 8 calls methods that no longer exist and must be + // refused rather than run against a contract it did not compile against. + Assertions.assertEquals("9.0", version.getProperty("api.version")); } /** Root entry points plus provider/handle types returned to connector plugins. */ diff --git a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/DriverUrlPolicyTest.java b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/DriverUrlPolicyTest.java new file mode 100644 index 00000000000000..ec4a43c3fee972 --- /dev/null +++ b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/DriverUrlPolicyTest.java @@ -0,0 +1,296 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + + +package org.apache.doris.connector.spi; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * The driver-url policy, case for case as the engine-side resolver it replaced ({@code JdbcResource + * .getFullDriverUrl}) was tested: url grammar, the url white list, structural secure-path matching for local + * and remote urls, bare-name resolution across the current and pre-2.1 default directories, and the external + * plugin store fallback. + */ +public class DriverUrlPolicyTest { + + private static DriverUrlPolicy.Settings settings(String securePath) { + return new DriverUrlPolicy.Settings("/opt/doris/plugins/jdbc_drivers", "/opt/doris", securePath, + Collections.emptyList(), null); + } + + private static DriverUrlPolicy.Settings allowAll() { + return settings("*"); + } + + @Test + public void validUrlsPassThroughUnchanged() { + for (String url : new String[] {"file://path/to/driver.jar", "http://example.com/driver.jar", + "https://example.com/driver.jar"}) { + Assertions.assertEquals(url, DriverUrlPolicy.resolve(url, allowAll())); + } + } + + @Test + public void bareNameMissingEverywhereIsReported() { + RuntimeException e = Assertions.assertThrows(RuntimeException.class, + () -> DriverUrlPolicy.resolve("driver.jar", allowAll())); + Assertions.assertTrue(e.getMessage().contains("JDBC driver file does not exist: driver.jar"), + e.getMessage()); + } + + @Test + public void malformedUrlsAreRejected() { + for (String url : new String[] {"/mnt/path/to/driver.jar", "ftp://example.com/driver.jar", "", + "example.com/driver"}) { + Assertions.assertThrows(IllegalArgumentException.class, + () -> DriverUrlPolicy.resolve(url, allowAll()), url); + } + } + + @Test + public void unparsableUrlFailsClosed() { + Assertions.assertThrows(IllegalArgumentException.class, + () -> DriverUrlPolicy.resolve("http://exa mple.com/driver.jar", allowAll())); + } + + @Test + public void whiteListMustListTheUrlVerbatim() { + DriverUrlPolicy.Settings listed = new DriverUrlPolicy.Settings(null, null, "*", + List.of("http://good.com/a.jar", ""), null); + Assertions.assertEquals("http://good.com/a.jar", DriverUrlPolicy.resolve("http://good.com/a.jar", listed)); + Assertions.assertThrows(IllegalArgumentException.class, + () -> DriverUrlPolicy.resolve("http://good.com/b.jar", listed)); + // An all-blank white list is no white list. + DriverUrlPolicy.Settings blank = new DriverUrlPolicy.Settings(null, null, "*", List.of("", ""), null); + Assertions.assertEquals("http://good.com/b.jar", DriverUrlPolicy.resolve("http://good.com/b.jar", blank)); + } + + @Test + public void securePathRejectsPrefixConfusion() { + // A directory that merely shares a string prefix must NOT be allowed. + Assertions.assertThrows(IllegalArgumentException.class, () -> DriverUrlPolicy.resolve( + "file:///opt/doris/jdbc_drivers-evil/x.jar", settings("file:///opt/doris/jdbc_drivers"))); + } + + @Test + public void securePathRejectsPathTraversal() { + Assertions.assertThrows(IllegalArgumentException.class, () -> DriverUrlPolicy.resolve( + "file:///opt/doris/jdbc_drivers/../../etc/x.jar", settings("file:///opt/doris/jdbc_drivers"))); + } + + @Test + public void securePathAllowsPathUnderAllowedDir() { + String url = "file:///opt/doris/jdbc_drivers/sub/x.jar"; + Assertions.assertEquals(url, DriverUrlPolicy.resolve(url, settings("file:///opt/doris/jdbc_drivers"))); + } + + @Test + public void securePathAcceptsSemicolonSeparatedPrefixes() { + String url = "file:///var/lib/drivers/x.jar"; + Assertions.assertEquals(url, DriverUrlPolicy.resolve(url, + settings("file:///opt/doris/jdbc_drivers; file:///var/lib/drivers ;"))); + } + + @Test + public void securePathRejectsHostConfusion() { + Assertions.assertThrows(IllegalArgumentException.class, () -> DriverUrlPolicy.resolve( + "http://good.com.evil.com/x.jar", settings("http://good.com/"))); + } + + @Test + public void securePathAllowsRemoteUnderAllowedHost() { + String url = "http://good.com/drivers/x.jar"; + Assertions.assertEquals(url, DriverUrlPolicy.resolve(url, settings("http://good.com/drivers"))); + } + + @Test + public void securePathWildcardAndBlankAllowAll() { + String url = "file:///any/where/x.jar"; + Assertions.assertEquals(url, DriverUrlPolicy.resolve(url, settings("*"))); + Assertions.assertEquals(url, DriverUrlPolicy.resolve(url, settings(""))); + Assertions.assertEquals(url, DriverUrlPolicy.resolve(url, settings(null))); + } + + @Test + public void securePathRejectsEncodedTraversal() { + // %2e%2e decodes to "..", which must be resolved the same way the classloader resolves it. + Assertions.assertThrows(IllegalArgumentException.class, () -> DriverUrlPolicy.resolve( + "file:///opt/doris/jdbc_drivers/%2e%2e/%2e%2e/etc/x.jar", settings("file:///opt/doris/jdbc_drivers"))); + } + + @Test + public void securePathRejectsRemoteQueryMismatch() { + // A query-bearing URL must not be authorized by a query-less allowed prefix. + Assertions.assertThrows(IllegalArgumentException.class, () -> DriverUrlPolicy.resolve( + "http://good.com/drivers/x.jar?id=evil", settings("http://good.com/drivers"))); + } + + @Test + public void securePathRejectsRemoteUserInfoMismatch() { + Assertions.assertThrows(IllegalArgumentException.class, () -> DriverUrlPolicy.resolve( + "http://user@good.com/drivers/x.jar", settings("http://good.com/drivers"))); + } + + @Test + public void securePathRejectsFileAuthority() { + // A non-local authority makes consumers fetch a remote object though the path matches. + Assertions.assertThrows(IllegalArgumentException.class, () -> DriverUrlPolicy.resolve( + "file://attacker.example/opt/doris/jdbc_drivers/evil.jar", settings("file:///opt/doris/jdbc_drivers"))); + } + + @Test + public void securePathRejectsFileQuery() { + Assertions.assertThrows(IllegalArgumentException.class, () -> DriverUrlPolicy.resolve( + "file:///opt/doris/jdbc_drivers/x.jar?evil", settings("file:///opt/doris/jdbc_drivers"))); + } + + @Test + public void bareNameResolvesUnderCustomDriversDirWithoutLookingAtTheFile() { + // A deployment-configured directory is authoritative: the name is resolved there, historically + // valid characters (e.g. '+') included, and no new restriction applies on this lazy load path. + DriverUrlPolicy.Settings custom = new DriverUrlPolicy.Settings("/opt/doris/jdbc_drivers", "/opt/doris", + "*", Collections.emptyList(), null); + Assertions.assertEquals("file:///opt/doris/jdbc_drivers/legacy+patched.jar", + DriverUrlPolicy.resolve("legacy+patched.jar", custom)); + } + + @Test + public void bareNameResolvesInTheDefaultDirThenThePre21Dir(@TempDir Path home) throws IOException { + Path newDir = Files.createDirectories(home.resolve("plugins/jdbc_drivers")); + Path oldDir = Files.createDirectories(home.resolve("jdbc_drivers")); + Files.write(newDir.resolve("new.jar"), new byte[] {1}); + Files.write(oldDir.resolve("old.jar"), new byte[] {2}); + DriverUrlPolicy.Settings defaults = new DriverUrlPolicy.Settings(newDir.toString(), home.toString(), "*", + Collections.emptyList(), null); + + Assertions.assertEquals("file://" + newDir.resolve("new.jar"), DriverUrlPolicy.resolve("new.jar", defaults)); + Assertions.assertEquals("file://" + oldDir.resolve("old.jar"), DriverUrlPolicy.resolve("old.jar", defaults)); + Assertions.assertThrows(RuntimeException.class, () -> DriverUrlPolicy.resolve("none.jar", defaults)); + } + + @Test + public void bareNameFallsBackToTheExternalPluginStore(@TempDir Path home) throws IOException { + Path newDir = Files.createDirectories(home.resolve("plugins/jdbc_drivers")); + List requests = new ArrayList<>(); + DriverUrlPolicy.Settings cloud = new DriverUrlPolicy.Settings(newDir.toString(), home.toString(), "*", + Collections.emptyList(), (name, target) -> { + requests.add(name + "->" + target); + return Optional.of(target); + }); + Assertions.assertEquals("file://" + newDir.resolve("cloud.jar"), DriverUrlPolicy.resolve("cloud.jar", cloud)); + Assertions.assertEquals(List.of("cloud.jar->" + newDir.resolve("cloud.jar")), requests); + + DriverUrlPolicy.Settings failing = new DriverUrlPolicy.Settings(newDir.toString(), home.toString(), "*", + Collections.emptyList(), (name, target) -> { + throw new IllegalStateException("bucket unreachable"); + }); + RuntimeException e = Assertions.assertThrows(RuntimeException.class, + () -> DriverUrlPolicy.resolve("cloud.jar", failing)); + Assertions.assertTrue(e.getMessage().contains("has been uploaded to cloud"), e.getMessage()); + Assertions.assertTrue(e.getMessage().contains("bucket unreachable"), e.getMessage()); + + DriverUrlPolicy.Settings noStore = new DriverUrlPolicy.Settings(newDir.toString(), home.toString(), "*", + Collections.emptyList(), (name, target) -> Optional.empty()); + Assertions.assertThrows(RuntimeException.class, () -> DriverUrlPolicy.resolve("cloud.jar", noStore)); + } + + @Test + public void settingsFromContextReadTheEngineEnvironment() { + Map env = new HashMap<>(); + env.put(DriverUrlPolicy.ENV_DORIS_HOME, "/opt/doris"); + env.put(DriverUrlPolicy.ENV_DRIVER_SECURE_PATH, "file:///opt/doris/jdbc_drivers"); + env.put(DriverUrlPolicy.ENV_DRIVER_URL_WHITE_LIST, "http://good.com/a.jar,http://good.com/b.jar"); + List fetches = new ArrayList<>(); + ConnectorContext context = new ConnectorContext() { + @Override + public String getCatalogName() { + return "c"; + } + + @Override + public long getCatalogId() { + return 1; + } + + @Override + public Map getEnvironment() { + return env; + } + + @Override + public Optional fetchPluginFile(String category, String fileName, String targetPath) { + fetches.add(category + ":" + fileName); + return Optional.empty(); + } + }; + DriverUrlPolicy.Settings settings = DriverUrlPolicy.Settings.fromContext(context, "/custom/drivers"); + Assertions.assertEquals("/custom/drivers", settings.getDriversDir()); + Assertions.assertEquals("/opt/doris", settings.getDorisHome()); + Assertions.assertEquals("file:///opt/doris/jdbc_drivers", settings.getSecurePath()); + Assertions.assertEquals(List.of("http://good.com/a.jar", "http://good.com/b.jar"), settings.getUrlWhiteList()); + settings.getMissingFileFetcher().fetch("x.jar", "/tmp/x.jar"); + Assertions.assertEquals(List.of(DriverUrlPolicy.PLUGIN_FILE_CATEGORY_JDBC_DRIVERS + ":x.jar"), fetches); + + DriverUrlPolicy.Settings unset = DriverUrlPolicy.Settings.fromContext(context, null); + Assertions.assertEquals("/opt/doris/plugins/jdbc_drivers", unset.effectiveDriversDir()); + } + + @Test + public void checksumIsTheHexMd5OfTheFile(@TempDir Path dir) throws IOException { + Path jar = dir.resolve("d.jar"); + Files.write(jar, "hello".getBytes(StandardCharsets.UTF_8)); + Assertions.assertEquals("5d41402abc4b2a76b9719d911017c592", + DriverUrlPolicy.checksum(jar.toUri().toString(), null)); + Assertions.assertThrows(IOException.class, + () -> DriverUrlPolicy.checksum(dir.resolve("missing.jar").toUri().toString(), null)); + } + + @Test + public void checksumRoutesRemoteUrlsThroughTheSecurityHook() { + List events = new ArrayList<>(); + ConnectorHttpSecurityHook hook = new ConnectorHttpSecurityHook() { + @Override + public void beforeRequest(String url) { + events.add("before:" + url); + throw new IllegalStateException("blocked"); + } + + @Override + public void afterRequest() { + events.add("after"); + } + }; + IOException e = Assertions.assertThrows(IOException.class, + () -> DriverUrlPolicy.checksum("http://127.0.0.1:1/x.jar", hook)); + Assertions.assertTrue(e.getMessage().contains("blocked"), e.getMessage()); + Assertions.assertEquals(List.of("before:http://127.0.0.1:1/x.jar", "after"), events); + } +} diff --git a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-metadata-methods.txt b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-metadata-methods.txt index b69475f19975ec..a2a57f9e8f4441 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-metadata-methods.txt +++ b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-metadata-methods.txt @@ -36,6 +36,7 @@ getDatabase(org.apache.doris.connector.spi.ConnectorSession,java.lang.String) getMvccPartitionView(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle) getPartitionFreshnessMillis(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.lang.String) getPartitionsFreshnessMillis(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.List) +getPrimaryKeys(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle) getSyntheticScanPredicates(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot) getSysTableHandle(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.lang.String) getTableComment(org.apache.doris.connector.spi.ConnectorSession,java.lang.String,java.lang.String) diff --git a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt index 663b6a8b7fdda8..da9c54f649d3b6 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt +++ b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt @@ -35,6 +35,7 @@ org.apache.doris.connector.spi.ConnectorCapability#enum:SUPPORTS_USER_SESSION org.apache.doris.connector.spi.ConnectorCapability#enum:SUPPORTS_VIEW org.apache.doris.connector.spi.ConnectorContext#createSiblingConnector(java.lang.String,java.util.Map):org.apache.doris.connector.spi.Connector org.apache.doris.connector.spi.ConnectorContext#executeAuthenticated(java.util.concurrent.Callable):java.lang.Object +org.apache.doris.connector.spi.ConnectorContext#fetchPluginFile(java.lang.String,java.lang.String,java.lang.String):java.util.Optional org.apache.doris.connector.spi.ConnectorContext#getCatalogId():long org.apache.doris.connector.spi.ConnectorContext#getCatalogName():java.lang.String org.apache.doris.connector.spi.ConnectorContext#getConnectorConfig():java.util.Map diff --git a/fe/fe-connector/pom.xml b/fe/fe-connector/pom.xml index f744982603411a..c00ef144e591f3 100644 --- a/fe/fe-connector/pom.xml +++ b/fe/fe-connector/pom.xml @@ -55,7 +55,7 @@ under the License. of the latter two means bumping this property as well (and fe-extension-spi means bumping all five families). --> - 8.0 + 9.0 diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java b/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java index 810846c19a1734..d4f69c68afb69b 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java @@ -24,6 +24,7 @@ import org.apache.doris.common.Config; import org.apache.doris.common.EnvUtils; import org.apache.doris.common.Version; +import org.apache.doris.common.plugin.CloudPluginDownloader; import org.apache.doris.common.util.LocationPath; import org.apache.doris.connector.spi.Connector; import org.apache.doris.connector.spi.ConnectorBrokerAddress; @@ -62,8 +63,10 @@ import java.util.Collections; import java.util.HashMap; import java.util.List; +import java.util.Locale; import java.util.Map; import java.util.Objects; +import java.util.Optional; import java.util.concurrent.Callable; import java.util.function.Function; import java.util.function.Supplier; @@ -212,6 +215,25 @@ public ConnectorStorageContext getStorageContext() { return this; } + /** + * A cloud deployment keeps its plugin files (JDBC driver jars, Java UDF jars) in its object store and + * copies one down on first use; every other deployment has no such store, so the plugin file simply + * does not exist. {@code category} is the store's file category, which is the plugin type's name. + */ + @Override + public Optional fetchPluginFile(String category, String fileName, String targetPath) { + if (!Config.isCloudMode()) { + return Optional.empty(); + } + CloudPluginDownloader.PluginType type; + try { + type = CloudPluginDownloader.PluginType.valueOf(category.toUpperCase(Locale.ROOT)); + } catch (IllegalArgumentException e) { + throw new IllegalArgumentException("Unknown plugin file category: " + category, e); + } + return Optional.of(CloudPluginDownloader.downloadFromCloud(type, fileName, targetPath)); + } + @Override public T executeAuthenticated(Callable task) throws Exception { return authSupplier.get().execute(task); @@ -606,6 +628,11 @@ private static Map buildEnvironment() { // HMS resources may be read before storage binding publishes this process-global FE setting. env.put("hadoop_config_dir", Config.hadoop_config_dir); env.put("jdbc_drivers_dir", Config.jdbc_drivers_dir); + // The driver-jar allow-lists are fe.conf-only security settings shared by every connector that loads + // a driver jar into the FE JVM (jdbc, iceberg, paimon); the policy that reads them is + // org.apache.doris.connector.spi.DriverUrlPolicy, keyed by these exact names. + env.put("jdbc_driver_secure_path", Config.jdbc_driver_secure_path); + env.put("jdbc_driver_url_white_list", String.join(",", Config.jdbc_driver_url_white_list)); env.put("force_sqlserver_jdbc_encrypt_false", String.valueOf(Config.force_sqlserver_jdbc_encrypt_false)); // HMS metastore client socket-timeout default (C4): the metastore-spi cannot read FE Config diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorValidationContext.java b/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorValidationContext.java index e794a6207b496e..8199e52e6fae7b 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorValidationContext.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorValidationContext.java @@ -18,7 +18,6 @@ package org.apache.doris.connector; import org.apache.doris.catalog.Env; -import org.apache.doris.catalog.JdbcResource; import org.apache.doris.common.DdlException; import org.apache.doris.common.FeConstants; import org.apache.doris.connector.spi.ConnectorValidationContext; @@ -39,9 +38,10 @@ /** * Engine-side implementation of {@link ConnectorValidationContext}. * - *

    Provides driver validation (via {@link JdbcResource}), checksum computation, - * and deferred BE→external connectivity testing (via BRPC) as infrastructure - * services that connectors can call during pre-creation validation.

    + *

    Provides property access and deferred BE→external connectivity testing (via BRPC) as + * infrastructure services that connectors can call during pre-creation validation. Driver-jar + * validation is not an engine service: connectors apply the shared + * {@code org.apache.doris.connector.spi.DriverUrlPolicy} themselves.

    * *

    Connectors register a BE connectivity test via {@link #requestBeConnectivityTest}; * the engine calls {@link #executePendingBeTests()} after validation to send @@ -77,16 +77,6 @@ public void storeProperty(String key, String value) { catalogProperty.addProperty(key, value); } - @Override - public String validateAndResolveDriverPath(String driverUrl) throws Exception { - return JdbcResource.getFullDriverUrl(driverUrl); - } - - @Override - public String computeDriverChecksum(String driverUrl) throws Exception { - return JdbcResource.computeObjectChecksum(driverUrl); - } - @Override public void requestBeConnectivityTest(byte[] serializedDescriptor, int connectionTypeValue, String testQuery) { diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextEnvironmentTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextEnvironmentTest.java index 4a3adb8904d8b7..027df8405cc20f 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextEnvironmentTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextEnvironmentTest.java @@ -18,10 +18,14 @@ package org.apache.doris.connector; import org.apache.doris.common.Config; +import org.apache.doris.connector.spi.DriverUrlPolicy; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import java.util.Arrays; +import java.util.Map; + public class DefaultConnectorContextEnvironmentTest { @Test @@ -36,4 +40,35 @@ public void forwardsConfiguredHadoopResourceDirectory() { Config.hadoop_config_dir = previous; } } + + @Test + public void forwardsTheDriverJarAllowListsUnderThePolicyKeys() { + // The connectors read these through DriverUrlPolicy.Settings.fromContext, keyed by the fe.conf names; + // the white list travels as one comma-joined value. + String previousPath = Config.jdbc_driver_secure_path; + String[] previousList = Config.jdbc_driver_url_white_list; + try { + Config.jdbc_driver_secure_path = "file:///opt/doris/jdbc_drivers"; + Config.jdbc_driver_url_white_list = new String[] {"http://a/x.jar", "http://b/y.jar"}; + Map env = new DefaultConnectorContext("test", 1L).getEnvironment(); + Assertions.assertEquals("file:///opt/doris/jdbc_drivers", env.get(DriverUrlPolicy.ENV_DRIVER_SECURE_PATH)); + Assertions.assertEquals("http://a/x.jar,http://b/y.jar", + env.get(DriverUrlPolicy.ENV_DRIVER_URL_WHITE_LIST)); + Assertions.assertEquals(Config.jdbc_drivers_dir, env.get(DriverUrlPolicy.ENV_DRIVERS_DIR)); + Assertions.assertEquals(Arrays.asList("http://a/x.jar", "http://b/y.jar"), + DriverUrlPolicy.Settings.fromContext(new DefaultConnectorContext("test", 1L), null) + .getUrlWhiteList()); + } finally { + Config.jdbc_driver_secure_path = previousPath; + Config.jdbc_driver_url_white_list = previousList; + } + } + + @Test + public void hasNoPluginFileStoreOutsideCloudMode() { + // A non-cloud deployment has no object store to fetch a missing driver jar from: the policy then + // reports the file as missing rather than the context inventing a path. + Assertions.assertFalse(new DefaultConnectorContext("test", 1L) + .fetchPluginFile("jdbc_drivers", "x.jar", "/tmp/x.jar").isPresent()); + } } From 00a4cd97628a11e0e2561e242fd5acff48cd3a00 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 16 Sep 2026 02:51:24 +0800 Subject: [PATCH 2/4] [refactor](fe) Serve streaming-job source metadata through the connector SPI and delete the fe-core JDBC clients ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: Part 2: the streaming/CDC framework was the last user of the fe-core copy of the JDBC dialect clients (`datasource/jdbc/client/*`, 13 classes and ~3000 lines that duplicated fe-connector-jdbc and had drifted from it). It now goes through the same connector plugin a JDBC catalog uses: - `StreamingSourceClient` (fe-core, `job/util`) opens a temporary connector via `ConnectorFactory` with the job's own source properties, acquires metadata through the `PluginDrivenMetadata` funnel, converts the connector schema with `ConnectorColumnConverter`, and runs the framework's probe queries through `executeQuery`. The four call sites (`generateCreateTableCmds`, the `cdc_stream` TVF, the OceanBase compatibility-mode check and the PostgreSQL slot/publication validator) keep their logic; only the client changes. - The fe-core clients, `JdbcFieldSchema` and their six tests are deleted; `JdbcClientException.getAllExceptionMessages` becomes `Util.getAllExceptionMessages`. - A streaming job now needs the jdbc connector plugin (bundled by default) and fails with a clear message without it. ### Release note None ### Check List (For Author) - Test: Unit Test - StreamingSourceClientTest (new), StreamingJobUtilsTest, DataSourceConfigValidatorTest, CdcStreamTableValuedFunctionTest, PostgresResourceValidatorTest (four new cases), UtilTest. - Behavior changed: Yes. The FE-side metadata connection of a streaming job is built by the connector, whose URL normalization forces `tinyInt1isBit=false` and `yearIsDateType=false`; the job's persisted URL and the BE CDC reader are unchanged. PostgreSQL array columns whose element type the old client rejected now map to `ARRAY`, as the CDC client already did. - Does this need documentation: No Co-Authored-By: Claude Opus 5 --- .../org/apache/doris/common/util/Util.java | 20 + .../jdbc/client/JdbcClickHouseClient.java | 247 -------- .../datasource/jdbc/client/JdbcClient.java | 588 ------------------ .../jdbc/client/JdbcClientConfig.java | 265 -------- .../jdbc/client/JdbcClientException.java | 66 -- .../datasource/jdbc/client/JdbcDB2Client.java | 113 ---- .../jdbc/client/JdbcGbaseClient.java | 154 ----- .../jdbc/client/JdbcMySQLClient.java | 480 -------------- .../jdbc/client/JdbcOceanBaseClient.java | 77 --- .../jdbc/client/JdbcOracleClient.java | 246 -------- .../jdbc/client/JdbcPostgreSQLClient.java | 228 ------- .../jdbc/client/JdbcSQLServerClient.java | 217 ------- .../jdbc/client/JdbcSapHanaClient.java | 102 --- .../jdbc/client/JdbcTrinoClient.java | 91 --- .../datasource/jdbc/util/JdbcFieldSchema.java | 129 ---- .../doris/job/common/DataSourceType.java | 12 +- .../streaming/DataSourceConfigValidator.java | 32 +- .../streaming/PostgresResourceValidator.java | 58 +- .../doris/job/util/StreamingJobUtils.java | 39 +- .../doris/job/util/StreamingSourceClient.java | 201 ++++++ .../CdcStreamTableValuedFunction.java | 15 +- .../apache/doris/common/util/UtilTest.java | 11 + .../jdbc/client/JdbcClickHouseClientTest.java | 92 --- .../jdbc/client/JdbcClientExceptionTest.java | 130 ---- .../jdbc/client/JdbcMySQLClientTest.java | 37 -- .../jdbc/client/JdbcOceanBaseClientTest.java | 105 ---- .../jdbc/client/JdbcSQLServerClientTest.java | 165 ----- .../jdbc/util/JdbcFieldSchemaTest.java | 57 -- .../DataSourceConfigValidatorTest.java | 75 ++- .../PostgresResourceValidatorTest.java | 127 ++++ .../doris/job/util/StreamingJobUtilsTest.java | 51 +- .../job/util/StreamingSourceClientTest.java | 235 +++++++ .../CdcStreamTableValuedFunctionTest.java | 12 +- 33 files changed, 736 insertions(+), 3741 deletions(-) delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClickHouseClient.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClient.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClientConfig.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClientException.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcDB2Client.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcGbaseClient.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcMySQLClient.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcOceanBaseClient.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcOracleClient.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcPostgreSQLClient.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcSQLServerClient.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcSapHanaClient.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcTrinoClient.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/util/JdbcFieldSchema.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/job/util/StreamingSourceClient.java delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcClickHouseClientTest.java delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcClientExceptionTest.java delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcMySQLClientTest.java delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcOceanBaseClientTest.java delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcSQLServerClientTest.java delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/util/JdbcFieldSchemaTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/job/util/StreamingSourceClientTest.java diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/util/Util.java b/fe/fe-core/src/main/java/org/apache/doris/common/util/Util.java index 2f706a4c660492..90959aeaf81b76 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/common/util/Util.java +++ b/fe/fe-core/src/main/java/org/apache/doris/common/util/Util.java @@ -539,6 +539,26 @@ public static String getRootCauseMessage(Throwable t) { return rootCause; } + /** + * The messages of {@code throwable} and of every cause below it, joined with {@code " | Caused by: "}, + * for an error report that should show the whole chain (a connection failure wrapped by a client + * wrapped by a validator) rather than only the root. + */ + public static String getAllExceptionMessages(Throwable throwable) { + StringBuilder sb = new StringBuilder(); + while (throwable != null) { + String message = throwable.getMessage(); + if (message != null && !message.isEmpty()) { + if (sb.length() > 0) { + sb.append(" | Caused by: "); + } + sb.append(message); + } + throwable = throwable.getCause(); + } + return sb.toString(); + } + public static String getRootCauseWithSuppressedMessage(Throwable t) { String rootCause; Throwable p = t; diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClickHouseClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClickHouseClient.java deleted file mode 100644 index fec623e88f2b91..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClickHouseClient.java +++ /dev/null @@ -1,247 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.jdbc.client; - -import org.apache.doris.catalog.ArrayType; -import org.apache.doris.catalog.ScalarType; -import org.apache.doris.catalog.Type; -import org.apache.doris.datasource.jdbc.util.JdbcFieldSchema; - -import com.google.common.collect.Lists; - -import java.sql.Connection; -import java.sql.DatabaseMetaData; -import java.sql.ResultSet; -import java.sql.SQLException; -import java.util.List; -import java.util.Optional; -import java.util.function.Consumer; - -public class JdbcClickHouseClient extends JdbcClient { - - private final Boolean databaseTermIsCatalog; - - protected JdbcClickHouseClient(JdbcClientConfig jdbcClientConfig) { - super(jdbcClientConfig); - try (Connection conn = getConnection()) { - DatabaseMetaData databaseMetaData = conn.getMetaData(); - this.databaseTermIsCatalog = isDatabaseTermCatalog( - databaseMetaData, databaseMetaData.getDriverVersion()); - } catch (SQLException e) { - throw new JdbcClientException("Failed to initialize JdbcClickHouseClient: %s", e.getMessage()); - } - } - - @Override - public List getDatabaseNameList() { - Connection conn = null; - ResultSet rs = null; - List remoteDatabaseNames = Lists.newArrayList(); - try { - conn = getConnection(); - if (isOnlySpecifiedDatabase && includeDatabaseMap.isEmpty() && excludeDatabaseMap.isEmpty()) { - if (databaseTermIsCatalog) { - remoteDatabaseNames.add(conn.getCatalog()); - } else { - remoteDatabaseNames.add(conn.getSchema()); - } - } else { - if (databaseTermIsCatalog) { - rs = conn.getMetaData().getCatalogs(); - } else { - rs = conn.getMetaData().getSchemas(conn.getCatalog(), null); - } - while (rs.next()) { - remoteDatabaseNames.add(rs.getString(1)); - } - } - } catch (SQLException e) { - throw new JdbcClientException("failed to get database name list from jdbc", e); - } finally { - close(rs, conn); - } - return filterDatabaseNames(remoteDatabaseNames); - } - - @Override - protected void processTable(String remoteDbName, String remoteTableName, String[] tableTypes, - Consumer resultSetConsumer) { - Connection conn = null; - ResultSet rs = null; - try { - conn = super.getConnection(); - DatabaseMetaData databaseMetaData = conn.getMetaData(); - if (databaseTermIsCatalog) { - rs = databaseMetaData.getTables(remoteDbName, null, remoteTableName, tableTypes); - } else { - rs = databaseMetaData.getTables(null, remoteDbName, remoteTableName, tableTypes); - } - resultSetConsumer.accept(rs); - } catch (SQLException e) { - throw new JdbcClientException("Failed to process table", e); - } finally { - close(rs, conn); - } - } - - @Override - protected ResultSet getRemoteColumns(DatabaseMetaData databaseMetaData, String catalogName, String remoteDbName, - String remoteTableName) throws SQLException { - if (databaseTermIsCatalog) { - return databaseMetaData.getColumns(remoteDbName, null, remoteTableName, null); - } else { - return databaseMetaData.getColumns(catalogName, remoteDbName, remoteTableName, null); - } - } - - @Override - protected String getCatalogName(Connection conn) throws SQLException { - if (databaseTermIsCatalog) { - return null; - } else { - return conn.getCatalog(); - } - } - - @Override - protected String[] getTableTypes() { - // ClickHouse JDBC V2 filters engines by these vendor-specific table type names. - return new String[] {"TABLE", "VIEW", "SYSTEM TABLE", "REMOTE TABLE", "MATERIALIZED VIEW"}; - } - - @Override - protected Type jdbcTypeToDoris(JdbcFieldSchema fieldSchema) { - - String ckType = fieldSchema.getDataTypeName().orElse("unknown"); - - if (ckType.startsWith("LowCardinality")) { - fieldSchema.setAllowNull(true); - ckType = ckType.substring(15, ckType.length() - 1); - if (ckType.startsWith("Nullable")) { - ckType = ckType.substring(9, ckType.length() - 1); - } - } else if (ckType.startsWith("Nullable")) { - fieldSchema.setAllowNull(true); - ckType = ckType.substring(9, ckType.length() - 1); - } - - if (ckType.startsWith("Decimal")) { - String[] accuracy = ckType.substring(8, ckType.length() - 1).split(", "); - int precision = Integer.parseInt(accuracy[0]); - int scale = Integer.parseInt(accuracy[1]); - return createDecimalOrStringType(precision, scale); - } - - if ("String".contains(ckType) - || ckType.startsWith("Enum") - || ckType.startsWith("IPv") - || "UUID".contains(ckType) - || ckType.startsWith("FixedString")) { - return ScalarType.createStringType(); - } - - if (ckType.startsWith("DateTime")) { - // DateTime with second precision - if (ckType.startsWith("DateTime(") || ckType.equals("DateTime")) { - return ScalarType.createDatetimeV2Type(0); - } else { - // DateTime64 with millisecond precision - // Datetime64(6) / DateTime64(6, 'Asia/Shanghai') - String[] accuracy = ckType.substring(11, ckType.length() - 1).split(", "); - int precision = Integer.parseInt(accuracy[0]); - if (precision > 6) { - precision = JDBC_DATETIME_SCALE; - } - return ScalarType.createDatetimeV2Type(precision); - } - } - - if (ckType.startsWith("Array")) { - String cktype = ckType.substring(6, ckType.length() - 1); - fieldSchema.setDataTypeName(Optional.of(cktype)); - Type type = jdbcTypeToDoris(fieldSchema); - return ArrayType.create(type, true); - } - - switch (ckType) { - case "Bool": - return Type.BOOLEAN; - case "Int8": - return Type.TINYINT; - case "Int16": - case "UInt8": - return Type.SMALLINT; - case "Int32": - case "UInt16": - return Type.INT; - case "Int64": - case "UInt32": - return Type.BIGINT; - case "Int128": - case "UInt64": - return Type.LARGEINT; - case "Int256": - case "UInt128": - case "UInt256": - return ScalarType.createStringType(); - case "Float32": - return Type.FLOAT; - case "Float64": - return Type.DOUBLE; - case "Date": - case "Date32": - return ScalarType.createDateV2Type(); - default: - return Type.UNSUPPORTED; - } - } - - /** - * Determine whether the driver version is greater than or equal to 0.5.0. - */ - private static boolean isNewClickHouseDriver(String driverVersion) { - if (driverVersion == null) { - throw new JdbcClientException("Driver version cannot be null"); - } - try { - String[] versionParts = driverVersion.split("\\."); - int majorVersion = Integer.parseInt(versionParts[0]); - int minorVersion = Integer.parseInt(versionParts[1]); - // Determine whether it is greater than or equal to 0.5.x - return (majorVersion > 0) || (majorVersion == 0 && minorVersion >= 5); - } catch (NumberFormatException | ArrayIndexOutOfBoundsException e) { - throw new JdbcClientException("Invalid clickhouse driver version format: " + driverVersion, e); - } - } - - static boolean isDatabaseTermCatalog(DatabaseMetaData databaseMetaData, String driverVersion) - throws SQLException { - return isNewClickHouseDriver(driverVersion) && databaseMetaData.supportsCatalogsInDataManipulation(); - } - - /** - * Get the driver version. - */ - public String getJdbcDriverVersion() { - try (Connection conn = getConnection()) { - return conn.getMetaData().getDriverVersion(); - } catch (SQLException e) { - throw new JdbcClientException("Failed to get jdbc driver version", e); - } - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClient.java deleted file mode 100644 index 8b496f82dd7d47..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClient.java +++ /dev/null @@ -1,588 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.jdbc.client; - -import org.apache.doris.catalog.Column; -import org.apache.doris.catalog.JdbcResource; -import org.apache.doris.catalog.ScalarType; -import org.apache.doris.catalog.Type; -import org.apache.doris.cloud.security.SecurityChecker; -import org.apache.doris.common.DdlException; -import org.apache.doris.common.util.Util; -import org.apache.doris.datasource.jdbc.util.JdbcFieldSchema; - -import com.google.common.collect.ImmutableSet; -import com.google.common.collect.Lists; -import com.zaxxer.hikari.HikariDataSource; -import lombok.Getter; -import org.apache.logging.log4j.LogManager; -import org.apache.logging.log4j.Logger; - -import java.net.MalformedURLException; -import java.net.URL; -import java.net.URLClassLoader; -import java.sql.Connection; -import java.sql.DatabaseMetaData; -import java.sql.PreparedStatement; -import java.sql.ResultSet; -import java.sql.ResultSetMetaData; -import java.sql.SQLException; -import java.sql.Statement; -import java.util.Collections; -import java.util.List; -import java.util.Map; -import java.util.Optional; -import java.util.Set; -import java.util.concurrent.ConcurrentHashMap; -import java.util.function.Consumer; - -@Getter -public abstract class JdbcClient { - private static final Logger LOG = LogManager.getLogger(JdbcClient.class); - private static final int HTTP_TIMEOUT_MS = 10000; - protected static final int JDBC_DATETIME_SCALE = 6; - - private static final Map classLoaderMap = new ConcurrentHashMap<>(); - - private String catalogName; - protected String dbType; - protected String jdbcUser; - protected ClassLoader classLoader = null; - protected HikariDataSource dataSource = null; - protected boolean isOnlySpecifiedDatabase; - protected Map includeDatabaseMap; - protected Map excludeDatabaseMap; - protected boolean enableMappingVarbinary; - protected boolean enableMappingTimestampTz; - - public static JdbcClient createJdbcClient(JdbcClientConfig jdbcClientConfig) { - String dbType = parseDbType(jdbcClientConfig.getJdbcUrl()); - switch (dbType) { - case JdbcResource.MYSQL: - return new JdbcMySQLClient(jdbcClientConfig); - case JdbcResource.OCEANBASE: - JdbcOceanBaseClient jdbcOceanBaseClient = new JdbcOceanBaseClient(jdbcClientConfig); - return jdbcOceanBaseClient.createClient(jdbcClientConfig); - case JdbcResource.POSTGRESQL: - return new JdbcPostgreSQLClient(jdbcClientConfig); - case JdbcResource.ORACLE: - return new JdbcOracleClient(jdbcClientConfig); - case JdbcResource.SQLSERVER: - return new JdbcSQLServerClient(jdbcClientConfig); - case JdbcResource.CLICKHOUSE: - return new JdbcClickHouseClient(jdbcClientConfig); - case JdbcResource.SAP_HANA: - return new JdbcSapHanaClient(jdbcClientConfig); - case JdbcResource.TRINO: - case JdbcResource.PRESTO: - return new JdbcTrinoClient(jdbcClientConfig); - case JdbcResource.DB2: - return new JdbcDB2Client(jdbcClientConfig); - case JdbcResource.GBASE: - return new JdbcGbaseClient(jdbcClientConfig); - default: - throw new IllegalArgumentException("Unsupported DB type: " + dbType); - } - } - - protected JdbcClient(JdbcClientConfig jdbcClientConfig) { - setJdbcDriverSystemProperties(); - this.catalogName = jdbcClientConfig.getCatalog(); - this.jdbcUser = jdbcClientConfig.getUser(); - this.isOnlySpecifiedDatabase = Boolean.parseBoolean(jdbcClientConfig.getOnlySpecifiedDatabase()); - this.includeDatabaseMap = - Optional.ofNullable(jdbcClientConfig.getIncludeDatabaseMap()).orElse(Collections.emptyMap()); - this.excludeDatabaseMap = - Optional.ofNullable(jdbcClientConfig.getExcludeDatabaseMap()).orElse(Collections.emptyMap()); - String jdbcUrl = jdbcClientConfig.getJdbcUrl(); - this.dbType = parseDbType(jdbcUrl); - initializeClassLoader(jdbcClientConfig); - initializeDataSource(jdbcClientConfig); - this.enableMappingVarbinary = jdbcClientConfig.isEnableMappingVarbinary(); - this.enableMappingTimestampTz = jdbcClientConfig.isEnableMappingTimestampTz(); - } - - protected void setJdbcDriverSystemProperties() { - System.setProperty("com.zaxxer.hikari.useWeakReferences", "true"); - } - - // Initialize DataSource - private void initializeDataSource(JdbcClientConfig config) { - ClassLoader oldClassLoader = Thread.currentThread().getContextClassLoader(); - try { - Thread.currentThread().setContextClassLoader(this.classLoader); - dataSource = new HikariDataSource(); - dataSource.setDriverClassName(config.getDriverClass()); - dataSource.setJdbcUrl(SecurityChecker.getInstance().getSafeJdbcUrl(config.getJdbcUrl())); - dataSource.setUsername(config.getUser()); - dataSource.setPassword(config.getPassword()); - dataSource.setMinimumIdle(config.getConnectionPoolMinSize()); // default 1 - dataSource.setMaximumPoolSize(config.getConnectionPoolMaxSize()); // default 10 - // set connection timeout to 5s. - // The default is 30s, which is too long. - // Because when querying information_schema db, BE will call thrift rpc(default timeout is 30s) - // to FE to get schema info, and may create connection here, if we set it too long and the url is invalid, - // it may cause the thrift rpc timeout. - dataSource.setConnectionTimeout(config.getConnectionPoolMaxWaitTime()); // default 5000 - dataSource.setMaxLifetime(config.getConnectionPoolMaxLifeTime()); // default 30 min - dataSource.setIdleTimeout(config.getConnectionPoolMaxLifeTime() / 2L); // default 15 min - dataSource.setConnectionTestQuery(getTestQuery()); - LOG.info("JdbcClient set" - + " ConnectionPoolMinSize = " + config.getConnectionPoolMinSize() - + ", ConnectionPoolMaxSize = " + config.getConnectionPoolMaxSize() - + ", ConnectionPoolMaxWaitTime = " + config.getConnectionPoolMaxWaitTime() - + ", ConnectionPoolMaxLifeTime = " + config.getConnectionPoolMaxLifeTime()); - } catch (Exception e) { - // If driver class loading failed (Hikari wraps it), clean cache and prompt retry - String msg = e.getMessage(); - if (msg != null && msg.contains("Failed to load driver class")) { - try { - URL url = new URL(JdbcResource.getFullDriverUrl(config.getDriverUrl())); - classLoaderMap.remove(url); - // Prompt user to verify driver validity and retry - throw new JdbcClientException( - String.format("Failed to load driver class `%s`. " - + "Please check that the driver JAR is valid and retry.", - config.getDriverClass()), e); - } catch (MalformedURLException ignore) { - // ignore invalid URL when cleaning cache - } - } - throw new JdbcClientException(e.getMessage(), e); - } finally { - Thread.currentThread().setContextClassLoader(oldClassLoader); - } - } - - private synchronized void initializeClassLoader(JdbcClientConfig config) { - try { - URL[] urls = {new URL(JdbcResource.getFullDriverUrl(config.getDriverUrl()))}; - if (classLoaderMap.containsKey(urls[0]) && classLoaderMap.get(urls[0]) != null) { - this.classLoader = classLoaderMap.get(urls[0]); - } else { - ClassLoader parent = getClass().getClassLoader(); - this.classLoader = URLClassLoader.newInstance(urls, parent); - classLoaderMap.put(urls[0], this.classLoader); - } - } catch (MalformedURLException e) { - throw new RuntimeException("Failed to load JDBC driver from path: " - + config.getDriverUrl(), e); - } - } - - public static String parseDbType(String jdbcUrl) { - try { - return JdbcResource.parseDbType(jdbcUrl); - } catch (DdlException e) { - throw new JdbcClientException("Failed to parse db type from jdbcUrl: " + jdbcUrl, e); - } - } - - public void closeClient() { - dataSource.close(); - dataSource = null; - } - - public Connection getConnection() throws JdbcClientException { - ClassLoader oldClassLoader = Thread.currentThread().getContextClassLoader(); - Connection conn; - try { - Thread.currentThread().setContextClassLoader(this.classLoader); - conn = dataSource.getConnection(); - } catch (Exception e) { - String errorMessage = String.format( - "Catalog `%s` can not connect to jdbc due to error: %s", - this.getCatalogName(), JdbcClientException.getAllExceptionMessages(e)); - throw new JdbcClientException(errorMessage, e); - } finally { - Thread.currentThread().setContextClassLoader(oldClassLoader); - } - return conn; - } - - public void close(Object... resources) { - for (Object resource : resources) { - if (resource != null) { - try { - if (resource instanceof ResultSet) { - ((ResultSet) resource).close(); - } else if (resource instanceof Statement) { - ((Statement) resource).close(); - } else if (resource instanceof Connection) { - ((Connection) resource).close(); - } - } catch (SQLException e) { - LOG.warn("Failed to close resource: {}", e.getMessage(), e); - } - } - } - } - - /** - * Execute stmt direct via jdbc - * - * @param origStmt, the raw stmt string - */ - public void executeStmt(String origStmt) { - Connection conn = null; - Statement stmt = null; - try { - conn = getConnection(); - stmt = conn.createStatement(); - int effectedRows = stmt.executeUpdate(origStmt); - if (LOG.isDebugEnabled()) { - LOG.debug("finished to execute dml stmt: {}, effected rows: {}", origStmt, effectedRows); - } - } catch (SQLException e) { - throw new JdbcClientException("Failed to execute stmt. error: " + e.getMessage(), e); - } finally { - close(stmt, conn); - } - } - - /** - * Execute query via jdbc - * - * @param query, the query string - * @return List - */ - public List getColumnsFromQuery(String query) { - Connection conn = null; - PreparedStatement pstmt = null; - List columns = Lists.newArrayList(); - try { - conn = getConnection(); - pstmt = conn.prepareStatement(query); - ResultSetMetaData metaData = pstmt.getMetaData(); - if (metaData == null) { - throw new JdbcClientException("Query not supported: Failed to get ResultSetMetaData from query: %s", - query); - } else { - List schemas = getSchemaFromResultSetMetaData(metaData); - for (JdbcFieldSchema schema : schemas) { - columns.add(new Column(schema.getColumnName(), jdbcTypeToDoris(schema), true, null, true, null, - true, -1)); - } - } - } catch (SQLException e) { - throw new JdbcClientException("Failed to get columns from query: %s", e, query); - } finally { - close(pstmt, conn); - } - return columns; - } - - /** - * Get schema from ResultSetMetaData - * - * @param metaData, the ResultSetMetaData - * @return List - */ - public List getSchemaFromResultSetMetaData(ResultSetMetaData metaData) throws SQLException { - List schemas = Lists.newArrayList(); - for (int i = 1; i <= metaData.getColumnCount(); i++) { - schemas.add(new JdbcFieldSchema(metaData, i)); - } - return schemas; - } - - // This part used to process meta-information of database, table and column. - - /** - * get all database name through JDBC - * - * @return list of database names - */ - public List getDatabaseNameList() { - Connection conn = null; - ResultSet rs = null; - List remoteDatabaseNames = Lists.newArrayList(); - try { - conn = getConnection(); - if (isOnlySpecifiedDatabase && includeDatabaseMap.isEmpty() && excludeDatabaseMap.isEmpty()) { - String currentDatabase = conn.getSchema(); - remoteDatabaseNames.add(currentDatabase); - } else { - rs = conn.getMetaData().getSchemas(conn.getCatalog(), getSchemaPatternForDatabaseNameList()); - while (rs.next()) { - remoteDatabaseNames.add(rs.getString("TABLE_SCHEM")); - } - } - } catch (SQLException e) { - throw new JdbcClientException("failed to get database name list from jdbc", e); - } finally { - close(rs, conn); - } - return filterDatabaseNames(remoteDatabaseNames); - } - - /** - * Schema pattern passed to {@link java.sql.DatabaseMetaData#getSchemas(String, String)} when listing - * remote database names. - * - *

    The default {@code null} follows JDBC semantics of "schema name should not be used to narrow - * the search", preserving the existing generic behavior. Subclasses should override this only when - * a driver treats {@code null} specially and does not return the schemas Doris expects. - */ - protected String getSchemaPatternForDatabaseNameList() { - return null; - } - - /** - * get all tables of one database - */ - public List getTablesNameList(String remoteDbName) { - List remoteTablesNames = Lists.newArrayList(); - String[] tableTypes = getTableTypes(); - processTable(remoteDbName, null, tableTypes, (rs) -> { - try { - while (rs.next()) { - remoteTablesNames.add(rs.getString("TABLE_NAME")); - } - } catch (SQLException e) { - throw new JdbcClientException("failed to get all tables for remote database: `%s`", e, remoteDbName); - } - }); - return remoteTablesNames; - } - - /** - * get table comment - */ - public String getTableComment(String remoteDbName, String remoteTableName) { - return ""; - } - - public boolean isTableExist(String remoteDbName, String remoteTableName) { - final boolean[] isExist = {false}; - String[] tableTypes = getTableTypes(); - processTable(remoteDbName, remoteTableName, tableTypes, (rs) -> { - try { - if (rs.next()) { - isExist[0] = true; - } - } catch (SQLException e) { - throw new JdbcClientException("failed to judge if table exist for table %s in db %s", - e, remoteTableName, remoteDbName); - } - }); - return isExist[0]; - } - - /** - * get all columns of one table - */ - public List getJdbcColumnsInfo(String remoteDbName, String remoteTableName) { - Connection conn = null; - ResultSet rs = null; - List tableSchema = Lists.newArrayList(); - try { - conn = getConnection(); - DatabaseMetaData databaseMetaData = conn.getMetaData(); - String catalogName = getCatalogName(conn); - rs = getRemoteColumns(databaseMetaData, catalogName, remoteDbName, remoteTableName); - while (rs.next()) { - tableSchema.add(new JdbcFieldSchema(rs)); - } - } catch (SQLException e) { - throw new JdbcClientException("failed to get jdbc columns info for remote table `%s.%s`: %s", - remoteDbName, remoteTableName, Util.getRootCauseMessage(e)); - } finally { - close(rs, conn); - } - return tableSchema; - } - - public List getColumnsFromJdbc(String remoteDbName, String remoteTableName) { - List jdbcTableSchema = getJdbcColumnsInfo(remoteDbName, remoteTableName); - List dorisTableSchema = Lists.newArrayListWithCapacity(jdbcTableSchema.size()); - for (JdbcFieldSchema field : jdbcTableSchema) { - dorisTableSchema.add(new Column(field.getColumnName(), - jdbcTypeToDoris(field), true, null, - field.isAllowNull(), field.getRemarks(), - true, -1)); - } - return dorisTableSchema; - } - - /** - * get primary keys of one table - */ - public List getPrimaryKeys(String remoteDbName, String remoteTableName) { - Connection conn = getConnection(); - ResultSet rs = null; - List primaryKeys = Lists.newArrayList(); - try { - DatabaseMetaData databaseMetaData = conn.getMetaData(); - String catalogName = getCatalogName(conn); - rs = databaseMetaData.getPrimaryKeys(catalogName, remoteDbName, remoteTableName); - while (rs.next()) { - String fieldName = rs.getString("COLUMN_NAME"); - primaryKeys.add(fieldName); - } - } catch (SQLException e) { - throw new JdbcClientException("failed to get jdbc primary key info for remote table `%s.%s`: %s", - remoteDbName, remoteTableName, Util.getRootCauseMessage(e)); - } finally { - close(rs, conn); - } - return primaryKeys; - } - - // protected methods, for subclass to override - protected String getCatalogName(Connection conn) throws SQLException { - return conn.getCatalog(); - } - - protected String[] getTableTypes() { - return new String[] {"TABLE", "VIEW"}; - } - - protected void processTable(String remoteDbName, String remoteTableName, String[] tableTypes, - Consumer resultSetConsumer) { - Connection conn = null; - ResultSet standardRs = null; - Statement stmt = null; - ResultSet customRs = null; - - try { - conn = getConnection(); - DatabaseMetaData databaseMetaData = conn.getMetaData(); - String catalogName = getCatalogName(conn); - - // 1. Process standard tables from getTables() method - standardRs = databaseMetaData.getTables(catalogName, remoteDbName, remoteTableName, tableTypes); - resultSetConsumer.accept(standardRs); - - // 2. Process additional tables from custom SQL query (if any) - String additionalQuery = getAdditionalTablesQuery(remoteDbName, remoteTableName, tableTypes); - if (additionalQuery != null && !additionalQuery.trim().isEmpty()) { - stmt = conn.createStatement(); - customRs = stmt.executeQuery(additionalQuery); - resultSetConsumer.accept(customRs); - } - } catch (SQLException e) { - throw new JdbcClientException("Failed to process table", e); - } finally { - close(customRs, stmt, standardRs, conn); - } - } - - protected String modifyTableNameIfNecessary(String remoteTableName) { - return remoteTableName; - } - - protected boolean isTableModified(String modifiedTableName, String actualTableName) { - return false; - } - - protected ResultSet getRemoteColumns(DatabaseMetaData databaseMetaData, String catalogName, String remoteDbName, - String remoteTableName) throws SQLException { - return databaseMetaData.getColumns(catalogName, remoteDbName, remoteTableName, null); - } - - protected List filterDatabaseNames(List remoteDbNames) { - Set filterInternalDatabases = getFilterInternalDatabases(); - List filteredDatabaseNames = Lists.newArrayList(); - for (String databaseName : remoteDbNames) { - if (isOnlySpecifiedDatabase) { - if (!excludeDatabaseMap.isEmpty() && excludeDatabaseMap.containsKey(databaseName)) { - continue; - } - if (!includeDatabaseMap.isEmpty() && !includeDatabaseMap.containsKey(databaseName)) { - continue; - } - } - if (filterInternalDatabases.contains(databaseName.toLowerCase())) { - continue; - } - filteredDatabaseNames.add(databaseName); - } - return filteredDatabaseNames; - } - - protected Set getFilterInternalDatabases() { - return ImmutableSet.builder() - .add("information_schema") - .add("performance_schema") - .add("mysql") - .build(); - } - - protected abstract Type jdbcTypeToDoris(JdbcFieldSchema fieldSchema); - - /** - * Get additional SQL query for tables that cannot be retrieved from standard getTables() method. - * For example, Oracle SYNONYM tables need custom SQL query. - *

    - * Default implementation returns null, meaning no additional query is needed. - * Subclasses can override this method to provide custom SQL queries. - * - * @param remoteDbName database name - * @param remoteTableName table name (can be null for all tables) - * @param tableTypes table types array - * @return SQL query string, or null if no additional query needed - */ - protected String getAdditionalTablesQuery(String remoteDbName, String remoteTableName, String[] tableTypes) { - // Default implementation: most databases don't need additional queries - return null; - } - - protected Type createDecimalOrStringType(int precision, int scale) { - if (precision <= ScalarType.MAX_DECIMAL128_PRECISION && precision > 0) { - return ScalarType.createDecimalV3Type(precision, scale); - } - return ScalarType.createStringType(); - } - - public void testConnection() { - String testQuery = getTestQuery(); - Connection conn = null; - Statement stmt = null; - ResultSet rs = null; - try { - conn = getConnection(); - stmt = conn.createStatement(); - rs = stmt.executeQuery(testQuery); - if (!rs.next()) { - throw new JdbcClientException( - "Failed to test connection in FE: query executed but returned no results."); - } - } catch (SQLException e) { - throw new JdbcClientException("Failed to test connection in FE: " + e.getMessage(), e); - } finally { - close(rs, stmt, conn); - } - } - - public String getTestQuery() { - return "select 1"; - } - - public String getJdbcDriverVersion() { - Connection conn = null; - try { - conn = getConnection(); - return conn.getMetaData().getDriverVersion(); - } catch (SQLException e) { - throw new JdbcClientException("Failed to get jdbc driver version", e); - } finally { - close(conn); - } - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClientConfig.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClientConfig.java deleted file mode 100644 index a35f908ed4b709..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClientConfig.java +++ /dev/null @@ -1,265 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - - -package org.apache.doris.datasource.jdbc.client; - -import org.apache.doris.catalog.JdbcResource; -import org.apache.doris.datasource.CatalogProperty; -import org.apache.doris.datasource.ExternalCatalog; - -import com.google.common.collect.Maps; - -import java.util.Map; - -public class JdbcClientConfig implements Cloneable { - private String catalog; - private String user; - private String password; - private String jdbcUrl; - private String driverUrl; - private String driverClass; - private String onlySpecifiedDatabase; - private String isLowerCaseMetaNames; - private String metaNamesMapping; - private int connectionPoolMinSize; - private int connectionPoolMaxSize; - private int connectionPoolMaxWaitTime; - private int connectionPoolMaxLifeTime; - private boolean connectionPoolKeepAlive; - // Whether to enable mapping BINARY to doris VARBINARY - // default: false, mapping to doris string type - private boolean enableMappingVarbinary; - // default: false, mapping to doris datetime type - private boolean enableMappingTimestampTz; - - private Map includeDatabaseMap; - private Map excludeDatabaseMap; - private Map customizedProperties; - - public JdbcClientConfig() { - this.onlySpecifiedDatabase = JdbcResource.getDefaultPropertyValue(JdbcResource.ONLY_SPECIFIED_DATABASE); - this.isLowerCaseMetaNames = JdbcResource.getDefaultPropertyValue(ExternalCatalog.LOWER_CASE_META_NAMES); - this.metaNamesMapping = JdbcResource.getDefaultPropertyValue(ExternalCatalog.META_NAMES_MAPPING); - this.connectionPoolMinSize = Integer.parseInt( - JdbcResource.getDefaultPropertyValue(JdbcResource.CONNECTION_POOL_MIN_SIZE)); - this.connectionPoolMaxSize = Integer.parseInt( - JdbcResource.getDefaultPropertyValue(JdbcResource.CONNECTION_POOL_MAX_SIZE)); - this.connectionPoolMaxWaitTime = Integer.parseInt( - JdbcResource.getDefaultPropertyValue(JdbcResource.CONNECTION_POOL_MAX_WAIT_TIME)); - this.connectionPoolMaxLifeTime = Integer.parseInt( - JdbcResource.getDefaultPropertyValue(JdbcResource.CONNECTION_POOL_MAX_LIFE_TIME)); - this.connectionPoolKeepAlive = Boolean.parseBoolean( - JdbcResource.getDefaultPropertyValue(JdbcResource.CONNECTION_POOL_KEEP_ALIVE)); - this.includeDatabaseMap = Maps.newHashMap(); - this.excludeDatabaseMap = Maps.newHashMap(); - this.customizedProperties = Maps.newHashMap(); - this.enableMappingVarbinary = Boolean.parseBoolean( - JdbcResource.getDefaultPropertyValue(CatalogProperty.ENABLE_MAPPING_VARBINARY)); - this.enableMappingTimestampTz = Boolean.parseBoolean( - JdbcResource.getDefaultPropertyValue(CatalogProperty.ENABLE_MAPPING_TIMESTAMP_TZ)); - } - - @Override - public JdbcClientConfig clone() { - try { - JdbcClientConfig cloned = (JdbcClientConfig) super.clone(); - - cloned.connectionPoolMinSize = connectionPoolMinSize; - cloned.connectionPoolMaxSize = connectionPoolMaxSize; - cloned.connectionPoolMaxLifeTime = connectionPoolMaxLifeTime; - cloned.connectionPoolMaxWaitTime = connectionPoolMaxWaitTime; - cloned.connectionPoolKeepAlive = connectionPoolKeepAlive; - cloned.includeDatabaseMap = Maps.newHashMap(includeDatabaseMap); - cloned.excludeDatabaseMap = Maps.newHashMap(excludeDatabaseMap); - cloned.customizedProperties = Maps.newHashMap(customizedProperties); - return cloned; - } catch (CloneNotSupportedException e) { - throw new RuntimeException(e); - } - } - - public String getCatalog() { - return catalog; - } - - public JdbcClientConfig setCatalog(String catalog) { - this.catalog = catalog; - return this; - } - - public String getUser() { - return user; - } - - public JdbcClientConfig setUser(String user) { - this.user = user; - return this; - } - - public String getPassword() { - return password; - } - - public JdbcClientConfig setPassword(String password) { - this.password = password; - return this; - } - - public String getJdbcUrl() { - return jdbcUrl; - } - - public JdbcClientConfig setJdbcUrl(String jdbcUrl) { - this.jdbcUrl = jdbcUrl; - return this; - } - - public String getDriverUrl() { - return driverUrl; - } - - public JdbcClientConfig setDriverUrl(String driverUrl) { - this.driverUrl = driverUrl; - return this; - } - - public String getDriverClass() { - return driverClass; - } - - public JdbcClientConfig setDriverClass(String driverClass) { - this.driverClass = driverClass; - return this; - } - - public String getOnlySpecifiedDatabase() { - return onlySpecifiedDatabase; - } - - public JdbcClientConfig setOnlySpecifiedDatabase(String onlySpecifiedDatabase) { - this.onlySpecifiedDatabase = onlySpecifiedDatabase; - return this; - } - - public String getIsLowerCaseMetaNames() { - return isLowerCaseMetaNames; - } - - public JdbcClientConfig setIsLowerCaseMetaNames(String isLowerCaseTableNames) { - this.isLowerCaseMetaNames = isLowerCaseTableNames; - return this; - } - - public String getMetaNamesMapping() { - return metaNamesMapping; - } - - public JdbcClientConfig setMetaNamesMapping(String metaNamesMapping) { - this.metaNamesMapping = metaNamesMapping; - return this; - } - - public int getConnectionPoolMinSize() { - return connectionPoolMinSize; - } - - public JdbcClientConfig setConnectionPoolMinSize(int connectionPoolMinSize) { - this.connectionPoolMinSize = connectionPoolMinSize; - return this; - } - - public int getConnectionPoolMaxSize() { - return connectionPoolMaxSize; - } - - public JdbcClientConfig setConnectionPoolMaxSize(int connectionPoolMaxSize) { - this.connectionPoolMaxSize = connectionPoolMaxSize; - return this; - } - - public int getConnectionPoolMaxLifeTime() { - return connectionPoolMaxLifeTime; - } - - public JdbcClientConfig setConnectionPoolMaxLifeTime(int connectionPoolMaxLifeTime) { - this.connectionPoolMaxLifeTime = connectionPoolMaxLifeTime; - return this; - } - - public int getConnectionPoolMaxWaitTime() { - return connectionPoolMaxWaitTime; - } - - public JdbcClientConfig setConnectionPoolMaxWaitTime(int connectionPoolMaxWaitTime) { - this.connectionPoolMaxWaitTime = connectionPoolMaxWaitTime; - return this; - } - - public boolean isConnectionPoolKeepAlive() { - return connectionPoolKeepAlive; - } - - public JdbcClientConfig setConnectionPoolKeepAlive(boolean connectionPoolKeepAlive) { - this.connectionPoolKeepAlive = connectionPoolKeepAlive; - return this; - } - - public Map getIncludeDatabaseMap() { - return includeDatabaseMap; - } - - public JdbcClientConfig setIncludeDatabaseMap(Map includeDatabaseMap) { - this.includeDatabaseMap = includeDatabaseMap; - return this; - } - - public Map getExcludeDatabaseMap() { - return excludeDatabaseMap; - } - - public JdbcClientConfig setExcludeDatabaseMap(Map excludeDatabaseMap) { - this.excludeDatabaseMap = excludeDatabaseMap; - return this; - } - - public JdbcClientConfig setEnableMappingVarbinary(boolean enableMappingVarbinary) { - this.enableMappingVarbinary = enableMappingVarbinary; - return this; - } - - public boolean isEnableMappingVarbinary() { - return enableMappingVarbinary; - } - - public JdbcClientConfig setEnableMappingTimestampTz(boolean enableMappingTimestampTz) { - this.enableMappingTimestampTz = enableMappingTimestampTz; - return this; - } - - public boolean isEnableMappingTimestampTz() { - return enableMappingTimestampTz; - } - - public void setCustomizedProperties(Map customizedProperties) { - this.customizedProperties = customizedProperties; - } - - public Map getCustomizedProperties() { - return customizedProperties; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClientException.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClientException.java deleted file mode 100644 index b07662459daa5b..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClientException.java +++ /dev/null @@ -1,66 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.jdbc.client; - -public class JdbcClientException extends RuntimeException { - public JdbcClientException(String format, Throwable cause, Object... msg) { - super(formatMessage(format, msg), cause); - } - - public JdbcClientException(String format, Object... msg) { - super(formatMessage(format, msg)); - } - - private static String formatMessage(String format, Object... msg) { - if (msg == null || msg.length == 0) { - return format; - } else { - return String.format(format, escapePercentInArgs(msg)); - } - } - - private static Object[] escapePercentInArgs(Object... args) { - if (args == null) { - return null; - } - Object[] escapedArgs = new Object[args.length]; - for (int i = 0; i < args.length; i++) { - if (args[i] instanceof String) { - escapedArgs[i] = ((String) args[i]).replace("%", "%%"); - } else { - escapedArgs[i] = args[i]; - } - } - return escapedArgs; - } - - public static String getAllExceptionMessages(Throwable throwable) { - StringBuilder sb = new StringBuilder(); - while (throwable != null) { - String message = throwable.getMessage(); - if (message != null && !message.isEmpty()) { - if (sb.length() > 0) { - sb.append(" | Caused by: "); - } - sb.append(message); - } - throwable = throwable.getCause(); - } - return sb.toString(); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcDB2Client.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcDB2Client.java deleted file mode 100644 index 95730fbb5ac2ff..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcDB2Client.java +++ /dev/null @@ -1,113 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.jdbc.client; - -import org.apache.doris.catalog.ScalarType; -import org.apache.doris.catalog.Type; -import org.apache.doris.datasource.jdbc.util.JdbcFieldSchema; - -import com.google.common.collect.Lists; - -import java.sql.Connection; -import java.sql.ResultSet; -import java.sql.SQLException; -import java.util.List; - -public class JdbcDB2Client extends JdbcClient { - - protected JdbcDB2Client(JdbcClientConfig jdbcClientConfig) { - super(jdbcClientConfig); - } - - public String getTestQuery() { - return "select 1 from sysibm.sysdummy1"; - } - - @Override - public List getDatabaseNameList() { - Connection conn = null; - ResultSet rs = null; - List remoteDatabaseNames = Lists.newArrayList(); - try { - conn = getConnection(); - if (isOnlySpecifiedDatabase && includeDatabaseMap.isEmpty() && excludeDatabaseMap.isEmpty()) { - String currentDatabase = conn.getSchema().trim(); - remoteDatabaseNames.add(currentDatabase); - } else { - rs = conn.getMetaData().getSchemas(conn.getCatalog(), null); - while (rs.next()) { - remoteDatabaseNames.add(rs.getString("TABLE_SCHEM").trim()); - } - } - } catch (SQLException e) { - throw new JdbcClientException("failed to get database name list from jdbc", e); - } finally { - close(rs, conn); - } - return filterDatabaseNames(remoteDatabaseNames); - } - - @Override - protected Type jdbcTypeToDoris(JdbcFieldSchema fieldSchema) { - String db2Type = fieldSchema.getDataTypeName().orElse("unknown"); - switch (db2Type) { - case "SMALLINT": - return Type.SMALLINT; - case "INTEGER": - return Type.INT; - case "BIGINT": - return Type.BIGINT; - case "DECFLOAT": - case "DECIMAL": { - int precision = fieldSchema.getColumnSize().orElse(0); - int scale = fieldSchema.getDecimalDigits().orElse(0); - return createDecimalOrStringType(precision, scale); - } - case "DOUBLE": - return Type.DOUBLE; - case "REAL": - return Type.FLOAT; - case "CHAR": - return ScalarType.createCharType(fieldSchema.requiredColumnSize()); - case "VARCHAR": - case "LONG VARCHAR": - return ScalarType.createVarcharType(fieldSchema.requiredColumnSize()); - case "DATE": - return ScalarType.createDateV2Type(); - case "TIMESTAMP": { - // postgres can support microsecond - int scale = fieldSchema.getDecimalDigits().orElse(0); - if (scale > 6) { - scale = 6; - } - return ScalarType.createDatetimeV2Type(scale); - } - case "TIME": - case "CLOB": - case "VARGRAPHIC": - case "LONG VARGRAPHIC": - case "XML": - return ScalarType.createStringType(); - case "BLOB": - return enableMappingVarbinary ? ScalarType.createVarbinaryType(fieldSchema.requiredColumnSize()) - : ScalarType.createStringType(); - default: - return Type.UNSUPPORTED; - } - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcGbaseClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcGbaseClient.java deleted file mode 100644 index 6121ef2dbfc2fb..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcGbaseClient.java +++ /dev/null @@ -1,154 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.jdbc.client; - -import org.apache.doris.catalog.ScalarType; -import org.apache.doris.catalog.Type; -import org.apache.doris.common.util.Util; -import org.apache.doris.datasource.jdbc.util.JdbcFieldSchema; - -import com.google.common.collect.Lists; - -import java.sql.Connection; -import java.sql.DatabaseMetaData; -import java.sql.ResultSet; -import java.sql.SQLException; -import java.sql.Types; -import java.util.List; -import java.util.function.Consumer; - -public class JdbcGbaseClient extends JdbcClient { - - protected JdbcGbaseClient(JdbcClientConfig jdbcClientConfig) { - super(jdbcClientConfig); - } - - @Override - public List getDatabaseNameList() { - Connection conn = null; - ResultSet rs = null; - List remoteDatabaseNames = Lists.newArrayList(); - try { - conn = getConnection(); - if (isOnlySpecifiedDatabase && includeDatabaseMap.isEmpty() && excludeDatabaseMap.isEmpty()) { - String currentDatabase = conn.getCatalog(); - remoteDatabaseNames.add(currentDatabase); - } else { - rs = conn.getMetaData().getCatalogs(); - while (rs.next()) { - remoteDatabaseNames.add(rs.getString("TABLE_CAT")); - } - } - } catch (SQLException e) { - throw new JdbcClientException("failed to get database name list from jdbc", e); - } finally { - close(rs, conn); - } - return filterDatabaseNames(remoteDatabaseNames); - } - - @Override - protected void processTable(String remoteDbName, String remoteTableName, String[] tableTypes, - Consumer resultSetConsumer) { - Connection conn = null; - ResultSet rs = null; - try { - conn = super.getConnection(); - DatabaseMetaData databaseMetaData = conn.getMetaData(); - rs = databaseMetaData.getTables(remoteDbName, null, remoteTableName, tableTypes); - resultSetConsumer.accept(rs); - } catch (SQLException e) { - throw new JdbcClientException("Failed to process table", e); - } finally { - close(rs, conn); - } - } - - @Override - protected ResultSet getRemoteColumns(DatabaseMetaData databaseMetaData, String catalogName, String remoteDbName, - String remoteTableName) throws SQLException { - return databaseMetaData.getColumns(remoteDbName, null, remoteTableName, null); - } - - @Override - public List getJdbcColumnsInfo(String remoteDbName, String remoteTableName) { - Connection conn = null; - ResultSet rs = null; - List tableSchema = Lists.newArrayList(); - try { - conn = getConnection(); - DatabaseMetaData databaseMetaData = conn.getMetaData(); - String catalogName = getCatalogName(conn); - rs = getRemoteColumns(databaseMetaData, catalogName, remoteDbName, remoteTableName); - while (rs.next()) { - JdbcFieldSchema field = new JdbcFieldSchema(rs); - tableSchema.add(field); - } - } catch (SQLException e) { - throw new JdbcClientException("failed to get jdbc columns info for remote table `%s.%s`: %s", - remoteDbName, remoteTableName, Util.getRootCauseMessage(e)); - } finally { - close(rs, conn); - } - return tableSchema; - } - - @Override - protected Type jdbcTypeToDoris(JdbcFieldSchema fieldSchema) { - switch (fieldSchema.getDataType()) { - case Types.TINYINT: - return Type.TINYINT; - case Types.SMALLINT: - return Type.SMALLINT; - case Types.INTEGER: - return Type.INT; - case Types.BIGINT: - return Type.BIGINT; - case Types.FLOAT: - case Types.REAL: - return Type.FLOAT; - case Types.DOUBLE: - return Type.DOUBLE; - case Types.NUMERIC: - case Types.DECIMAL: { - int precision = fieldSchema.getColumnSize() - .orElseThrow(() -> new IllegalArgumentException("Precision not present")); - int scale = fieldSchema.getDecimalDigits() - .orElseThrow(() -> new JdbcClientException("Scale not present")); - return createDecimalOrStringType(precision, scale); - } - case Types.DATE: - return Type.DATEV2; - case Types.TIMESTAMP: { - int scale = fieldSchema.getDecimalDigits().orElse(0); - if (scale > 6) { - scale = 6; - } - return ScalarType.createDatetimeV2Type(scale); - } - case Types.CHAR: - return ScalarType.createCharType(fieldSchema.requiredColumnSize()); - case Types.TIME: - case Types.VARCHAR: - case Types.LONGVARCHAR: - return ScalarType.createStringType(); - default: - return Type.UNSUPPORTED; - } - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcMySQLClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcMySQLClient.java deleted file mode 100644 index bcdc9ccc03bd69..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcMySQLClient.java +++ /dev/null @@ -1,480 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.jdbc.client; - -import org.apache.doris.catalog.ArrayType; -import org.apache.doris.catalog.ScalarType; -import org.apache.doris.catalog.Type; -import org.apache.doris.common.util.Util; -import org.apache.doris.datasource.jdbc.util.JdbcFieldSchema; - -import com.google.common.base.Preconditions; -import com.google.common.base.Strings; -import com.google.common.collect.ImmutableList; -import com.google.common.collect.ImmutableSet; -import com.google.common.collect.Lists; -import com.google.common.collect.Maps; - -import java.sql.Connection; -import java.sql.DatabaseMetaData; -import java.sql.ResultSet; -import java.sql.SQLException; -import java.sql.Statement; -import java.util.List; -import java.util.Locale; -import java.util.Map; -import java.util.Optional; -import java.util.Set; -import java.util.TreeMap; -import java.util.function.Consumer; - -public class JdbcMySQLClient extends JdbcClient { - - private boolean convertDateToNull = false; - private boolean isDoris = false; - - protected JdbcMySQLClient(JdbcClientConfig jdbcClientConfig) { - super(jdbcClientConfig); - convertDateToNull = isConvertDatetimeToNull(jdbcClientConfig); - Connection conn = null; - Statement stmt = null; - ResultSet rs = null; - try { - conn = super.getConnection(); - stmt = conn.createStatement(); - rs = stmt.executeQuery("SHOW VARIABLES LIKE 'version_comment'"); - if (rs.next()) { - String versionComment = rs.getString("Value"); - isDoris = isDorisCompatibleVersionComment(versionComment); - } - } catch (SQLException | JdbcClientException e) { - closeClient(); - throw new JdbcClientException("Failed to initialize JdbcMySQLClient: %s", e.getMessage()); - } finally { - close(rs, stmt, conn); - } - } - - protected JdbcMySQLClient(JdbcClientConfig jdbcClientConfig, String dbType) { - super(jdbcClientConfig); - convertDateToNull = isConvertDatetimeToNull(jdbcClientConfig); - this.dbType = dbType; - } - - static boolean isDorisCompatibleVersionComment(String versionComment) { - if (Strings.isNullOrEmpty(versionComment)) { - return false; - } - String lowerVersionComment = versionComment.toLowerCase(Locale.ROOT); - return lowerVersionComment.contains("doris") - || lowerVersionComment.contains("selectdb") - || lowerVersionComment.contains("velodb") - || (lowerVersionComment.contains("enterprise version") - && lowerVersionComment.contains("cloud mode")); - } - - @Override - public String getTableComment(String remoteDbName, String remoteTableName) { - ImmutableList.Builder tableCommentBuilder = ImmutableList.builder(); - String[] tableTypes = getTableTypes(); - processTable(remoteDbName, remoteTableName, tableTypes, (rs) -> { - try { - while (rs.next()) { - tableCommentBuilder.add(Strings.nullToEmpty(rs.getString("REMARKS"))); - } - } catch (SQLException e) { - throw new JdbcClientException( - "failed to get table's comment for remote database: `%s`, remote table: `%s`", - e, remoteDbName, remoteTableName); - } - }); - - ImmutableList tableComment = tableCommentBuilder.build(); - Preconditions.checkArgument(tableComment.size() == 1, "Multiple tables `%s` are matched", remoteTableName); - return tableComment.get(0); - } - - @Override - protected void setJdbcDriverSystemProperties() { - super.setJdbcDriverSystemProperties(); - System.setProperty("com.mysql.cj.disableAbandonedConnectionCleanup", "true"); - } - - @Override - public List getDatabaseNameList() { - Connection conn = null; - ResultSet rs = null; - List remoteDatabaseNames = Lists.newArrayList(); - try { - conn = getConnection(); - if (isOnlySpecifiedDatabase && includeDatabaseMap.isEmpty() && excludeDatabaseMap.isEmpty()) { - String currentDatabase = conn.getCatalog(); - remoteDatabaseNames.add(currentDatabase); - } else { - rs = conn.getMetaData().getCatalogs(); - while (rs.next()) { - remoteDatabaseNames.add(rs.getString("TABLE_CAT")); - } - } - } catch (SQLException e) { - throw new JdbcClientException("failed to get database name list from jdbc", e); - } finally { - close(rs, conn); - } - return filterDatabaseNames(remoteDatabaseNames); - } - - @Override - protected void processTable(String remoteDbName, String remoteTableName, String[] tableTypes, - Consumer resultSetConsumer) { - Connection conn = null; - ResultSet rs = null; - try { - conn = super.getConnection(); - DatabaseMetaData databaseMetaData = conn.getMetaData(); - rs = databaseMetaData.getTables(remoteDbName, null, remoteTableName, tableTypes); - resultSetConsumer.accept(rs); - } catch (SQLException e) { - throw new JdbcClientException("Failed to process table", e); - } finally { - close(rs, conn); - } - } - - @Override - protected String[] getTableTypes() { - return new String[] {"TABLE", "VIEW", "SYSTEM VIEW"}; - } - - @Override - protected ResultSet getRemoteColumns(DatabaseMetaData databaseMetaData, String catalogName, String remoteDbName, - String remoteTableName) throws SQLException { - return databaseMetaData.getColumns(remoteDbName, null, remoteTableName, null); - } - - /** - * get all columns of one table - */ - @Override - public List getJdbcColumnsInfo(String remoteDbName, String remoteTableName) { - Connection conn = null; - ResultSet rs = null; - List tableSchema = Lists.newArrayList(); - try { - conn = getConnection(); - DatabaseMetaData databaseMetaData = conn.getMetaData(); - String catalogName = getCatalogName(conn); - rs = getRemoteColumns(databaseMetaData, catalogName, remoteDbName, remoteTableName); - - Map mapFieldtoType = Maps.newHashMap(); - if (isDoris) { - mapFieldtoType = getColumnsDataTypeUseQuery(remoteDbName, remoteTableName); - } - - while (rs.next()) { - JdbcFieldSchema field = new JdbcFieldSchema(rs, mapFieldtoType); - tableSchema.add(field); - } - } catch (SQLException e) { - throw new JdbcClientException("failed to get jdbc columns info for remote table `%s.%s`: %s", - remoteDbName, remoteTableName, Util.getRootCauseMessage(e)); - } finally { - close(rs, conn); - } - return tableSchema; - } - - @Override - public List getPrimaryKeys(String remoteDbName, String remoteTableName) { - Connection conn = getConnection(); - ResultSet rs = null; - // getPrimaryKeys orders rows by COLUMN_NAME, not KEY_SEQ; reorder by the 1-based KEY_SEQ - // to keep the real composite-PK column order. - TreeMap primaryKeys = new TreeMap<>(); - try { - DatabaseMetaData databaseMetaData = conn.getMetaData(); - rs = databaseMetaData.getPrimaryKeys(remoteDbName, null, remoteTableName); - while (rs.next()) { - primaryKeys.put(rs.getShort("KEY_SEQ"), rs.getString("COLUMN_NAME")); - } - } catch (SQLException e) { - throw new JdbcClientException("failed to get jdbc primary key info for remote table `%s.%s`: %s", - remoteDbName, remoteTableName, Util.getRootCauseMessage(e)); - } finally { - close(rs, conn); - } - return Lists.newArrayList(primaryKeys.values()); - } - - protected String getCatalogName(Connection conn) throws SQLException { - return null; - } - - protected Set getFilterInternalDatabases() { - return ImmutableSet.builder() - .add("information_schema") - .add("performance_schema") - .add("mysql") - .add("sys") - .build(); - } - - @Override - protected Type jdbcTypeToDoris(JdbcFieldSchema fieldSchema) { - // For Doris type - if (isDoris) { - return dorisTypeToDoris(fieldSchema); - } - // For mysql type: "INT UNSIGNED": - // fieldSchema.getDataTypeName().orElse("unknown").split(" ")[0] == "INT" - // fieldSchema.getDataTypeName().orElse("unknown").split(" ")[1] == "UNSIGNED" - String[] typeFields = fieldSchema.getDataTypeName().orElse("unknown").split(" "); - String mysqlType = typeFields[0]; - // For unsigned int, should extend the type. - if (typeFields.length > 1 && "UNSIGNED".equals(typeFields[1])) { - switch (mysqlType) { - case "TINYINT": - return Type.SMALLINT; - case "SMALLINT": - case "MEDIUMINT": - return Type.INT; - case "INT": - return Type.BIGINT; - case "BIGINT": - return Type.LARGEINT; - case "DECIMAL": { - int precision = fieldSchema.requiredColumnSize() + 1; - int scale = fieldSchema.requiredDecimalDigits(); - return createDecimalOrStringType(precision, scale); - } - case "DOUBLE": - // As of MySQL 8.0.17, the UNSIGNED attribute is deprecated - // for columns of type FLOAT, DOUBLE, and DECIMAL (and any synonyms) - // https://dev.mysql.com/doc/refman/8.0/en/numeric-type-syntax.html - // The maximum value may cause errors due to insufficient accuracy - return Type.DOUBLE; - case "FLOAT": - return Type.FLOAT; - default: - throw new JdbcClientException("Unknown UNSIGNED type of mysql, type: [" + mysqlType + "]"); - } - } - switch (mysqlType) { - case "BOOLEAN": - return Type.BOOLEAN; - case "TINYINT": - return Type.TINYINT; - case "SMALLINT": - case "YEAR": - return Type.SMALLINT; - case "MEDIUMINT": - case "INT": - return Type.INT; - case "BIGINT": - return Type.BIGINT; - case "DATE": - if (convertDateToNull) { - fieldSchema.setAllowNull(true); - } - return ScalarType.createDateV2Type(); - case "TIMESTAMP": { - int columnSize = fieldSchema.requiredColumnSize(); - int scale = columnSize > 19 ? columnSize - 20 : 0; - if (scale > 6) { - scale = 6; - } - if (convertDateToNull) { - fieldSchema.setAllowNull(true); - } - return enableMappingTimestampTz ? ScalarType.createTimeStampTzType(scale) - : ScalarType.createDatetimeV2Type(scale); - } - case "DATETIME": { - // mysql can support microsecond - // use columnSize to calculate the precision of timestamp/datetime - int columnSize = fieldSchema.requiredColumnSize(); - int scale = columnSize > 19 ? columnSize - 20 : 0; - if (scale > 6) { - scale = 6; - } - if (convertDateToNull) { - fieldSchema.setAllowNull(true); - } - return ScalarType.createDatetimeV2Type(scale); - } - case "FLOAT": - return Type.FLOAT; - case "DOUBLE": - return Type.DOUBLE; - case "DECIMAL": { - int precision = fieldSchema.requiredColumnSize(); - int scale = fieldSchema.requiredDecimalDigits(); - return createDecimalOrStringType(precision, scale); - } - case "CHAR": - return ScalarType.createCharType(fieldSchema.requiredColumnSize()); - case "VARCHAR": - return ScalarType.createVarcharType(fieldSchema.requiredColumnSize()); - case "TINYBLOB": - case "BLOB": - case "MEDIUMBLOB": - case "LONGBLOB": - case "BINARY": - case "VARBINARY": - return enableMappingVarbinary ? ScalarType.createVarbinaryType(fieldSchema.requiredColumnSize()) - : ScalarType.createStringType(); - case "BIT": - if (fieldSchema.requiredColumnSize() == 1) { - return Type.BOOLEAN; - } else { - return ScalarType.createStringType(); - } - case "JSON": - case "TIME": - case "TINYTEXT": - case "TEXT": - case "MEDIUMTEXT": - case "LONGTEXT": - case "STRING": - case "SET": - case "ENUM": - return ScalarType.createStringType(); - default: - return Type.UNSUPPORTED; - } - } - - private boolean isConvertDatetimeToNull(JdbcClientConfig jdbcClientConfig) { - // Check if the JDBC URL contains "zeroDateTimeBehavior=convertToNull" or "zeroDateTimeBehavior=convert_to_null" - String jdbcUrl = jdbcClientConfig.getJdbcUrl().toLowerCase(); - return jdbcUrl.contains("zerodatetimebehavior=converttonull") - || jdbcUrl.contains("zerodatetimebehavior=convert_to_null"); - } - - /** - * get all columns like DatabaseMetaData.getColumns in mysql-jdbc-connector - */ - private Map getColumnsDataTypeUseQuery(String remoteDbName, String remoteTableName) { - Connection conn = null; - Statement stmt = null; - ResultSet resultSet = null; - Map fieldToType = Maps.newHashMap(); - - StringBuilder queryBuf = new StringBuilder("SHOW FULL COLUMNS FROM "); - queryBuf.append("`").append(remoteTableName).append("`"); - queryBuf.append(" FROM "); - queryBuf.append("`").append(remoteDbName).append("`"); - try { - conn = getConnection(); - stmt = conn.createStatement(); - resultSet = stmt.executeQuery(queryBuf.toString()); - while (resultSet.next()) { - // get column name - String fieldName = resultSet.getString("Field"); - // get original type name - String typeName = resultSet.getString("Type"); - fieldToType.put(fieldName, typeName); - } - } catch (SQLException e) { - throw new JdbcClientException("failed to get jdbc columns info for remote table `%s.%s`: %s", - remoteDbName, remoteTableName, Util.getRootCauseMessage(e)); - } finally { - close(resultSet, stmt, conn); - } - return fieldToType; - } - - private Type dorisTypeToDoris(JdbcFieldSchema fieldSchema) { - String type = fieldSchema.getDataTypeName().orElse("unknown").toUpperCase(); - if (type == null || type.isEmpty()) { - return Type.UNSUPPORTED; - } - - String upperType = type.toUpperCase(); - - // For ARRAY type - if (upperType.startsWith("ARRAY")) { - String innerType = upperType.substring(6, upperType.length() - 1).trim(); - JdbcFieldSchema innerFieldSchema = new JdbcFieldSchema(fieldSchema); - innerFieldSchema.setDataTypeName(Optional.of(innerType)); - Type arrayInnerType = dorisTypeToDoris(innerFieldSchema); - return ArrayType.create(arrayInnerType, true); - } - - int openParen = upperType.indexOf("("); - String baseType = (openParen == -1) ? upperType : upperType.substring(0, openParen); - - switch (baseType) { - case "BOOL": - case "BOOLEAN": - return Type.BOOLEAN; - case "TINYINT": - return Type.TINYINT; - case "INT": - return Type.INT; - case "SMALLINT": - return Type.SMALLINT; - case "BIGINT": - return Type.BIGINT; - case "LARGEINT": - return Type.LARGEINT; - case "FLOAT": - return Type.FLOAT; - case "DOUBLE": - return Type.DOUBLE; - case "DECIMAL": - case "DECIMALV3": { - int precision = fieldSchema.requiredColumnSize(); - int scale = fieldSchema.requiredDecimalDigits(); - return createDecimalOrStringType(precision, scale); - } - case "DATE": - case "DATEV2": - return ScalarType.createDateV2Type(); - case "DATETIME": - case "DATETIMEV2": { - int scale = (openParen == -1) ? 0 - : Integer.parseInt(upperType.substring(openParen + 1, upperType.length() - 1)); - if (scale > 6) { - scale = 6; - } - return ScalarType.createDatetimeV2Type(scale); - } - case "CHAR": - case "CHARACTER": - return ScalarType.createCharType(fieldSchema.requiredColumnSize()); - case "VARCHAR": - return ScalarType.createVarcharType(fieldSchema.requiredColumnSize()); - case "STRING": - case "TEXT": - case "JSON": - case "JSONB": - return ScalarType.createStringType(); - case "HLL": - return ScalarType.createHllType(); - case "BITMAP": - return Type.BITMAP; - case "QUANTILE_STATE": - return Type.QUANTILE_STATE; - case "VARBINARY": - return ScalarType.createVarbinaryType(fieldSchema.requiredColumnSize()); - default: - return Type.UNSUPPORTED; - } - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcOceanBaseClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcOceanBaseClient.java deleted file mode 100644 index 8dda77213350d6..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcOceanBaseClient.java +++ /dev/null @@ -1,77 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.jdbc.client; - -import org.apache.doris.catalog.JdbcResource; -import org.apache.doris.catalog.Type; -import org.apache.doris.datasource.jdbc.util.JdbcFieldSchema; - -import java.sql.Connection; -import java.sql.ResultSet; -import java.sql.SQLException; -import java.sql.Statement; - -public class JdbcOceanBaseClient extends JdbcClient { - - public JdbcOceanBaseClient(JdbcClientConfig jdbcClientConfig) { - super(jdbcClientConfig); - } - - public JdbcClient createClient(JdbcClientConfig jdbcClientConfig) throws JdbcClientException { - Connection conn = null; - Statement stmt = null; - ResultSet rs = null; - try { - conn = super.getConnection(); - stmt = conn.createStatement(); - rs = stmt.executeQuery("SHOW VARIABLES LIKE 'ob_compatibility_mode'"); - if (rs.next()) { - String compatibilityMode = rs.getString(2); - if ("MYSQL".equalsIgnoreCase(compatibilityMode)) { - return new JdbcMySQLClient(jdbcClientConfig, JdbcResource.OCEANBASE); - } else if ("ORACLE".equalsIgnoreCase(compatibilityMode)) { - setOracleMode(); - return new JdbcOracleClient(jdbcClientConfig, JdbcResource.OCEANBASE_ORACLE); - } else { - throw new JdbcClientException("Unsupported OceanBase compatibility mode: " + compatibilityMode); - } - } else { - throw new JdbcClientException("Failed to determine OceanBase compatibility mode"); - } - } catch (SQLException e) { - throw new JdbcClientException("Failed to initialize JdbcOceanBaseClient: %s", e.getMessage()); - } finally { - close(rs, stmt, conn); - closeClient(); - } - } - - @Override - protected Type jdbcTypeToDoris(JdbcFieldSchema fieldSchema) { - throw new UnsupportedOperationException("JdbcOceanBaseClient does not support jdbcTypeToDoris"); - } - - @Override - public String getTestQuery() { - return "SELECT 1 FROM DUAL"; - } - - void setOracleMode() { - this.dbType = JdbcResource.OCEANBASE_ORACLE; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcOracleClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcOracleClient.java deleted file mode 100644 index 996608545cdb2e..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcOracleClient.java +++ /dev/null @@ -1,246 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.jdbc.client; - -import org.apache.doris.catalog.ScalarType; -import org.apache.doris.catalog.Type; -import org.apache.doris.common.util.Util; -import org.apache.doris.datasource.jdbc.util.JdbcFieldSchema; - -import com.google.common.base.Strings; -import com.google.common.collect.ImmutableSet; -import com.google.common.collect.Lists; - -import java.sql.Connection; -import java.sql.DatabaseMetaData; -import java.sql.ResultSet; -import java.sql.SQLException; -import java.sql.Statement; -import java.util.List; -import java.util.Set; - -public class JdbcOracleClient extends JdbcClient { - - protected JdbcOracleClient(JdbcClientConfig jdbcClientConfig) { - super(jdbcClientConfig); - } - - protected JdbcOracleClient(JdbcClientConfig jdbcClientConfig, String dbType) { - super(jdbcClientConfig); - this.dbType = dbType; - } - - @Override - public String getTestQuery() { - return "SELECT 1 FROM dual"; - } - - @Override - public List getJdbcColumnsInfo(String remoteDbName, String remoteTableName) { - Connection conn = null; - ResultSet rs = null; - List tableSchema = Lists.newArrayList(); - Statement stmt = null; - ResultSet isSynonymRs = null; - ResultSet synonymInfoRs = null; - try { - conn = getConnection(); - DatabaseMetaData databaseMetaData = conn.getMetaData(); - String catalogName = getCatalogName(conn); - String modifiedTableName; - boolean isModify = false; - if (remoteTableName.contains("/")) { - modifiedTableName = modifyTableNameIfNecessary(remoteTableName); - isModify = !modifiedTableName.equals(remoteTableName); - if (isModify) { - rs = getRemoteColumns(databaseMetaData, catalogName, remoteDbName, modifiedTableName); - } else { - rs = getRemoteColumns(databaseMetaData, catalogName, remoteDbName, remoteTableName); - } - } else { - rs = getRemoteColumns(databaseMetaData, catalogName, remoteDbName, remoteTableName); - } - while (rs.next()) { - if (isModify && isTableModified(rs.getString("TABLE_NAME"), remoteTableName)) { - continue; - } - tableSchema.add(new JdbcFieldSchema(rs)); - } - if (tableSchema.isEmpty()) { - // maybe the table is a synonym - stmt = conn.createStatement(); - isSynonymRs = stmt.executeQuery( - "SELECT OBJECT_TYPE FROM ALL_OBJECTS WHERE OBJECT_NAME = '" + remoteTableName - + "' AND OWNER = '" + remoteDbName + "'"); - if (isSynonymRs.next() && "SYNONYM".equalsIgnoreCase(isSynonymRs.getString("OBJECT_TYPE"))) { - // if it is a synonym, get the actual table name and owner(database) - String additionalTablesQuery = getAdditionalTablesQuery(remoteDbName, remoteTableName, null); - synonymInfoRs = stmt.executeQuery(additionalTablesQuery); - while (synonymInfoRs.next()) { - String baseTableName = synonymInfoRs.getString("BASE_TABLE_NAME"); - String baseTableOwner = synonymInfoRs.getString("BASE_TABLE_OWNER"); - return getJdbcColumnsInfo(baseTableOwner, baseTableName); - } - } - } - } catch (SQLException e) { - throw new JdbcClientException("failed to get table name list from jdbc for table %s:%s", e, remoteTableName, - Util.getRootCauseMessage(e)); - } finally { - close(rs, conn, stmt, isSynonymRs, synonymInfoRs); - } - return tableSchema; - } - - @Override - protected String modifyTableNameIfNecessary(String remoteTableName) { - return remoteTableName.replace("/", "%"); - } - - @Override - protected boolean isTableModified(String modifiedTableName, String actualTableName) { - return !modifiedTableName.equals(actualTableName); - } - - @Override - protected Set getFilterInternalDatabases() { - return ImmutableSet.builder() - .add("ctxsys") - .add("flows_files") - .add("mdsys") - .add("outln") - .add("sys") - .add("system") - .add("xdb") - .add("xs$null") - .build(); - } - - @Override - protected String getAdditionalTablesQuery(String remoteDbName, String remoteTableName, String[] tableTypes) { - StringBuilder sb = new StringBuilder( - "SELECT SYNONYM_NAME as TABLE_NAME, TABLE_OWNER as BASE_TABLE_OWNER, TABLE_NAME as BASE_TABLE_NAME " - + "FROM ALL_SYNONYMS"); - List conditions = Lists.newArrayList(); - if (!Strings.isNullOrEmpty(remoteDbName)) { - conditions.add("OWNER = '" + remoteDbName + "'"); - } - if (!Strings.isNullOrEmpty(remoteTableName)) { - conditions.add("SYNONYM_NAME = '" + remoteTableName + "'"); - } - if (!conditions.isEmpty()) { - sb.append(" WHERE ").append(String.join(" AND ", conditions)); - } - return sb.toString(); - } - - @Override - protected Type jdbcTypeToDoris(JdbcFieldSchema fieldSchema) { - String oracleType = fieldSchema.getDataTypeName().orElse("unknown"); - if (oracleType.startsWith("INTERVAL")) { - oracleType = oracleType.substring(0, 8); - } else if (oracleType.startsWith("TIMESTAMP")) { - // oracle can support nanosecond, will lose precision - int scale = fieldSchema.getDecimalDigits().orElse(0); - if (scale > 6) { - scale = 6; - } - if (oracleType.contains("LOCAL TIME ZONE")) { - //TIMESTAMP(s) WITH LOCAL TIME ZONE - return enableMappingTimestampTz ? ScalarType.createTimeStampTzType(scale) - : ScalarType.createDatetimeV2Type(scale); - } else if (oracleType.contains("TIME ZONE")) { - //TIMESTAMP(s) WITH TIME ZONE - return Type.UNSUPPORTED; - } else { - //TIMESTAMP(s) - oracleType = "TIMESTAMP"; - return ScalarType.createDatetimeV2Type(scale); - } - } - switch (oracleType) { - /** - * The data type NUMBER(p,s) of oracle has some different of doris decimal type in semantics. - * For Oracle Number(p,s) type: - * 1. if s<0 , it means this is an Interger. - * This NUMBER(p,s) has (p+|s| ) significant digit, and rounding will be performed at s position. - * eg: if we insert 1234567 into NUMBER(5,-2) type, then the oracle will store 1234500. - * In this case, Doris will use INT type (TINYINT/SMALLINT/INT/.../LARGEINT). - * 2. if s>=0 && s

    =0 && s>p, it means this is a decimal(like 0.xxxxx). - * p represents how many digits can be left to the left after the decimal point, - * the figure after the decimal point s will be rounded. - * eg: we can not insert 0.0123456 into NUMBER(5,7) type, - * because there must be two zeros on the right side of the decimal point, - * we can insert 0.0012345 into NUMBER(5,7) type. - * In this case, Doris will use DECIMAL(s,s) - * 4. if we don't specify p and s for NUMBER(p,s), just NUMBER, the p and s of NUMBER are uncertain. - * In this case, doris can not determine p and s, so doris can not determine data type. - */ - case "NUMBER": - int precision = fieldSchema.getColumnSize().orElse(0); - int scale = fieldSchema.getDecimalDigits().orElse(0); - if (scale <= 0) { - precision -= scale; - if (precision < 3) { - return Type.TINYINT; - } else if (precision < 5) { - return Type.SMALLINT; - } else if (precision < 10) { - return Type.INT; - } else if (precision < 19) { - return Type.BIGINT; - } else if (precision < 39) { - // LARGEINT supports up to 38 numbers. - return Type.LARGEINT; - } else { - return ScalarType.createStringType(); - } - } - // scale > 0 - if (precision < scale) { - precision = scale; - } - return createDecimalOrStringType(precision, scale); - case "FLOAT": - return Type.DOUBLE; - case "DATE": - // can save date and time with second precision - return ScalarType.createDatetimeV2Type(0); - case "VARCHAR2": - case "NVARCHAR2": - case "CHAR": - case "NCHAR": - case "LONG": - case "RAW": - case "LONG RAW": - case "INTERVAL": - case "CLOB": - return ScalarType.createStringType(); - case "BLOB": - return enableMappingVarbinary ? ScalarType.createVarbinaryType(fieldSchema.requiredColumnSize()) - : ScalarType.createStringType(); - case "NCLOB": - case "BFILE": - case "BINARY_FLOAT": - case "BINARY_DOUBLE": - default: - return Type.UNSUPPORTED; - } - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcPostgreSQLClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcPostgreSQLClient.java deleted file mode 100644 index 46d94f90b4b678..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcPostgreSQLClient.java +++ /dev/null @@ -1,228 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.jdbc.client; - -import org.apache.doris.catalog.ArrayType; -import org.apache.doris.catalog.ScalarType; -import org.apache.doris.catalog.Type; -import org.apache.doris.common.util.Util; -import org.apache.doris.datasource.jdbc.util.JdbcFieldSchema; - -import com.google.common.collect.Lists; -import org.apache.logging.log4j.LogManager; -import org.apache.logging.log4j.Logger; - -import java.sql.Connection; -import java.sql.DatabaseMetaData; -import java.sql.PreparedStatement; -import java.sql.ResultSet; -import java.sql.SQLException; -import java.sql.Types; -import java.util.Arrays; -import java.util.List; -import java.util.Optional; - -public class JdbcPostgreSQLClient extends JdbcClient { - private static final Logger LOG = LogManager.getLogger(JdbcPostgreSQLClient.class); - - private static final String[] supportedInnerType = new String[] { - "int2", "int4", "int8", "smallserial", "serial", - "bigserial", "float4", "float8", "numeric", - "timestamp", "timestamptz", "date", "bool", - "bpchar", "varchar", "text", - "json", "jsonb", "uuid" - }; - - protected JdbcPostgreSQLClient(JdbcClientConfig jdbcClientConfig) { - super(jdbcClientConfig); - } - - @Override - public List getJdbcColumnsInfo(String remoteDbName, String remoteTableName) { - Connection conn = null; - ResultSet rs = null; - List tableSchema = Lists.newArrayList(); - try { - conn = getConnection(); - DatabaseMetaData databaseMetaData = conn.getMetaData(); - String catalogName = getCatalogName(conn); - rs = getRemoteColumns(databaseMetaData, catalogName, remoteDbName, remoteTableName); - while (rs.next()) { - // getColumns treats schema/table as LIKE patterns; drop rows pulled in via `_`/`%`. - if (!remoteDbName.equals(rs.getString("TABLE_SCHEM")) - || !remoteTableName.equals(rs.getString("TABLE_NAME"))) { - continue; - } - int dataType = rs.getInt("DATA_TYPE"); - int arrayDimensions = 0; - if (dataType == Types.ARRAY) { - String columnName = rs.getString("COLUMN_NAME"); - PreparedStatement pstmt = null; - ResultSet arrayRs = null; - try { - pstmt = conn.prepareStatement( - String.format("SELECT array_ndims(\"%s\") FROM \"%s\".\"%s\"" - + " WHERE \"%s\" IS NOT NULL LIMIT 1", - columnName, remoteDbName, remoteTableName, - columnName)); - arrayRs = pstmt.executeQuery(); - if (arrayRs.next()) { - arrayDimensions = arrayRs.getInt(1); - } - } catch (SQLException ex) { - LOG.warn("Failed to get array dimensions for column {}: {}", - columnName, Util.getRootCauseMessage(ex)); - } finally { - close(arrayRs, null); - if (pstmt != null) { - try { - pstmt.close(); - } catch (SQLException ex) { - LOG.warn("Failed to close prepared statement: {}", Util.getRootCauseMessage(ex)); - } - } - } - } - tableSchema.add(new JdbcFieldSchema(rs, arrayDimensions)); - } - } catch (SQLException e) { - throw new JdbcClientException("failed to get jdbc columns info for remote table `%s.%s`: %s", - remoteDbName, remoteTableName, Util.getRootCauseMessage(e)); - } finally { - close(rs, conn); - } - return tableSchema; - } - - @Override - protected String[] getTableTypes() { - return new String[] {"TABLE", "PARTITIONED TABLE", "VIEW", "MATERIALIZED VIEW", "FOREIGN TABLE"}; - } - - @Override - protected Type jdbcTypeToDoris(JdbcFieldSchema fieldSchema) { - String pgType = fieldSchema.getDataTypeName().orElse("unknown"); - switch (pgType) { - case "int2": - case "smallserial": - return Type.SMALLINT; - case "int4": - case "serial": - return Type.INT; - case "int8": - case "bigserial": - return Type.BIGINT; - case "numeric": { - int precision = fieldSchema.getColumnSize().orElse(0); - int scale = fieldSchema.getDecimalDigits().orElse(0); - return createDecimalOrStringType(precision, scale); - } - case "float4": - return Type.FLOAT; - case "float8": - return Type.DOUBLE; - case "bpchar": - return ScalarType.createCharType(fieldSchema.requiredColumnSize()); - case "timestamp": { - // postgres can support microsecond - int scale = fieldSchema.getDecimalDigits().orElse(0); - if (scale > 6) { - scale = 6; - } - return ScalarType.createDatetimeV2Type(scale); - } - case "timestamptz": { - int scale = fieldSchema.getDecimalDigits().orElse(0); - if (scale > 6) { - scale = 6; - } - return enableMappingTimestampTz ? ScalarType.createTimeStampTzType(scale) - : ScalarType.createDatetimeV2Type(scale); - } - case "date": - return ScalarType.createDateV2Type(); - case "bool": - return Type.BOOLEAN; - case "bit": - if (fieldSchema.getColumnSize().orElse(0) == 1) { - return Type.BOOLEAN; - } else { - return ScalarType.createStringType(); - } - case "point": - case "line": - case "lseg": - case "box": - case "path": - case "polygon": - case "circle": - case "varchar": - case "text": - case "time": - case "timetz": - case "interval": - case "cidr": - case "inet": - case "macaddr": - case "macaddr8": - case "varbit": - case "uuid": - case "xml": - case "hstore": - case "json": - case "jsonb": - return ScalarType.createStringType(); - case "bytea": // https://www.postgresql.org/docs/12/datatype-binary.html#DATATYPE-BINARY-TABLE - return enableMappingVarbinary ? ScalarType.createVarbinaryType(fieldSchema.requiredColumnSize()) - : ScalarType.createStringType(); - default: { - if (fieldSchema.getDataType() == Types.ARRAY && pgType.startsWith("_")) { - return convertArrayType(fieldSchema); - } else { - return Type.UNSUPPORTED; - } - } - } - } - - private Type convertArrayType(JdbcFieldSchema fieldSchema) { - int arrayDimensions = fieldSchema.getArrayDimensions().orElse(0); - if (arrayDimensions == 0) { - LOG.warn("postgres array type without dimensions"); - return Type.UNSUPPORTED; - } - - String innerType = fieldSchema.getDataTypeName().orElse("unknown").substring(1); - - boolean isSupported = Arrays.asList(supportedInnerType).contains(innerType); - if (!isSupported) { - return Type.UNSUPPORTED; - } - if (innerType.equals("bpchar")) { - innerType = "text"; - } - JdbcFieldSchema innerFieldSchema = new JdbcFieldSchema(fieldSchema); - innerFieldSchema.setDataTypeName(Optional.of(innerType)); - Type arrayInnerType = jdbcTypeToDoris(innerFieldSchema); - Type arrayType = ArrayType.create(arrayInnerType, true); - for (int i = 1; i < arrayDimensions; i++) { - arrayType = ArrayType.create(arrayType, true); - } - return arrayType; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcSQLServerClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcSQLServerClient.java deleted file mode 100644 index d1aaa42ef3cd37..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcSQLServerClient.java +++ /dev/null @@ -1,217 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.jdbc.client; - -import org.apache.doris.catalog.ScalarType; -import org.apache.doris.catalog.Type; -import org.apache.doris.datasource.jdbc.util.JdbcFieldSchema; - -import java.sql.Types; -import java.util.Locale; -import java.util.regex.Matcher; -import java.util.regex.Pattern; - -public class JdbcSQLServerClient extends JdbcClient { - - // TYPE_NAME of an IDENTITY column decorates the base type: "int identity", "decimal() identity", - // "numeric(18, 0) identity", "decimal(18,0) IDENTITY(1,1)". IDENTITY is only allowed on these base types. - private static final Pattern IDENTITY_TYPE_NAME = Pattern.compile( - "^(tinyint|smallint|int|bigint|decimal|numeric)\\s*(\\([^)]*\\))?\\s+identity(\\s*\\([^)]*\\))?$", - Pattern.CASE_INSENSITIVE); - - protected JdbcSQLServerClient(JdbcClientConfig jdbcClientConfig) { - super(jdbcClientConfig); - } - - @Override - protected String getSchemaPatternForDatabaseNameList() { - // "%" is a JDBC schemaPattern wildcard that matches all schemas. mssql-jdbc 13.4 filters - // built-in schemas when catalog is non-empty and schemaPattern is null. - return "%"; - } - - /** - * The base type of an IDENTITY column's TYPE_NAME, or the name unchanged. - *

    - * The decoration is trusted only when {@code DATA_TYPE} is the code of the named base type: an alias type - * may legally be named like that ({@code CREATE TYPE dbo.[int identity] FROM varchar(10)}, or - * {@code dbo.[int alias]}), and it then has to be resolved by its code, not by the words of its name. - */ - static String identityBaseType(String typeName, int dataType) { - Matcher matcher = IDENTITY_TYPE_NAME.matcher(typeName); - if (!matcher.matches()) { - return typeName; - } - String baseType = matcher.group(1).toLowerCase(Locale.ROOT); - boolean codeMatches; - switch (baseType) { - case "tinyint": - codeMatches = dataType == Types.TINYINT; - break; - case "smallint": - codeMatches = dataType == Types.SMALLINT; - break; - case "int": - codeMatches = dataType == Types.INTEGER; - break; - case "bigint": - codeMatches = dataType == Types.BIGINT; - break; - default: - codeMatches = dataType == Types.DECIMAL || dataType == Types.NUMERIC; - break; - } - return codeMatches ? baseType : typeName; - } - - @Override - protected Type jdbcTypeToDoris(JdbcFieldSchema fieldSchema) { - String originSqlserverType = fieldSchema.getDataTypeName().orElse("unknown"); - // An IDENTITY column is reported as "int identity" or "decimal(18,0) identity": only the base type - // is matched below. Any other name is matched as it is: system type names are single words, and a - // user-defined alias type may be named with spaces or parentheses ("int alias") and must not be - // mistaken for the system type its name starts with. - String sqlserverType = identityBaseType(originSqlserverType, fieldSchema.getDataType()); - - switch (sqlserverType) { - case "bit": - return Type.BOOLEAN; - case "tinyint": - case "smallint": - return Type.SMALLINT; - case "int": - return Type.INT; - case "bigint": - return Type.BIGINT; - case "real": - return Type.FLOAT; - case "float": - return Type.DOUBLE; - case "money": - return ScalarType.createDecimalV3Type(19, 4); - case "smallmoney": - return ScalarType.createDecimalV3Type(10, 4); - case "decimal": - case "numeric": { - int precision = fieldSchema.getColumnSize().orElse(0); - int scale = fieldSchema.getDecimalDigits().orElse(0); - return createDecimalOrStringType(precision, scale); - } - case "date": - return ScalarType.createDateV2Type(); - case "datetime": - case "datetime2": - case "smalldatetime": { - // postgres can support microsecond - int scale = fieldSchema.getDecimalDigits().orElse(0); - if (scale > 6) { - scale = 6; - } - return ScalarType.createDatetimeV2Type(scale); - } - case "char": - case "varchar": - case "nchar": - case "nvarchar": - case "text": - case "ntext": - case "time": - case "datetimeoffset": - case "uniqueidentifier": - case "timestamp": - return ScalarType.createStringType(); - case "image": - case "binary": - case "varbinary": - return enableMappingVarbinary ? ScalarType.createVarbinaryType(fieldSchema.requiredColumnSize()) - : ScalarType.createStringType(); - case "xml": - case "sql_variant": - case "geometry": - case "geography": - case "hierarchyid": - case "json": - case "vector": - // SQL Server system types that Doris does not support. They are listed explicitly - // so that they never reach the JDBC type code fallback below. - return Type.UNSUPPORTED; - default: - return jdbcTypeCodeToDoris(fieldSchema); - } - } - - /** - * Fallback for type names that are not SQL Server system types. - *

    - * User-defined alias types ({@code CREATE TYPE dbo.my_type FROM varchar(50)}) are reported by - * {@code DatabaseMetaData.getColumns()} with {@code TYPE_NAME} set to the alias name, so they can not be - * matched by name. {@code DATA_TYPE}, {@code COLUMN_SIZE} and {@code DECIMAL_DIGITS} still describe the - * base type, so the standard {@link Types} code is used to resolve the Doris type. The mapping mirrors - * the name based one above. - *

    - * Binary codes are deliberately not mapped: mssql-jdbc also reports CLR user-defined types - * (geometry, geography, hierarchyid, ...) as {@link Types#VARBINARY}, so they can not be told apart from - * an alias over a binary type by the type code alone. Vendor specific codes stay unsupported as well. - */ - private Type jdbcTypeCodeToDoris(JdbcFieldSchema fieldSchema) { - switch (fieldSchema.getDataType()) { - case Types.BIT: - case Types.BOOLEAN: - return Type.BOOLEAN; - // SQL Server tinyint is unsigned (0 to 255), so it needs SMALLINT - case Types.TINYINT: - case Types.SMALLINT: - return Type.SMALLINT; - case Types.INTEGER: - return Type.INT; - case Types.BIGINT: - return Type.BIGINT; - case Types.REAL: - return Type.FLOAT; - case Types.FLOAT: - case Types.DOUBLE: - return Type.DOUBLE; - case Types.DECIMAL: - case Types.NUMERIC: { - // money and smallmoney are reported as DECIMAL(19,4) and DECIMAL(10,4) - int precision = fieldSchema.getColumnSize().orElse(0); - int scale = fieldSchema.getDecimalDigits().orElse(0); - return createDecimalOrStringType(precision, scale); - } - case Types.DATE: - return ScalarType.createDateV2Type(); - case Types.TIMESTAMP: { - int scale = fieldSchema.getDecimalDigits().orElse(0); - if (scale > 6) { - scale = 6; - } - return ScalarType.createDatetimeV2Type(scale); - } - case Types.CHAR: - case Types.VARCHAR: - case Types.LONGVARCHAR: - case Types.NCHAR: - case Types.NVARCHAR: - case Types.LONGNVARCHAR: - case Types.TIME: - return ScalarType.createStringType(); - default: - return Type.UNSUPPORTED; - } - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcSapHanaClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcSapHanaClient.java deleted file mode 100644 index e89b6268b900f7..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcSapHanaClient.java +++ /dev/null @@ -1,102 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.jdbc.client; - -import org.apache.doris.catalog.ScalarType; -import org.apache.doris.catalog.Type; -import org.apache.doris.datasource.jdbc.util.JdbcFieldSchema; - -public class JdbcSapHanaClient extends JdbcClient { - protected JdbcSapHanaClient(JdbcClientConfig jdbcClientConfig) { - super(jdbcClientConfig); - } - - @Override - protected String[] getTableTypes() { - return new String[] {"TABLE", "VIEW", "OLAP VIEW", "JOIN VIEW", "HIERARCHY VIEW", "CALC VIEW"}; - } - - @Override - public String getTestQuery() { - return "SELECT 1 FROM DUMMY"; - } - - @Override - protected Type jdbcTypeToDoris(JdbcFieldSchema fieldSchema) { - String hanaType = fieldSchema.getDataTypeName().orElse("unknown"); - switch (hanaType) { - case "TINYINT": - return Type.TINYINT; - case "SMALLINT": - return Type.SMALLINT; - case "INTEGER": - return Type.INT; - case "BIGINT": - return Type.BIGINT; - case "SMALLDECIMAL": - case "DECIMAL": { - if (!fieldSchema.getDecimalDigits().isPresent()) { - return Type.DOUBLE; - } else { - int precision = fieldSchema.getColumnSize().orElse(0); - int scale = fieldSchema.getDecimalDigits().orElse(0); - return createDecimalOrStringType(precision, scale); - } - } - case "REAL": - return Type.FLOAT; - case "DOUBLE": - return Type.DOUBLE; - case "TIMESTAMP": { - // postgres can support microsecond - int scale = fieldSchema.getDecimalDigits().orElse(0); - if (scale > 6) { - scale = 6; - } - return ScalarType.createDatetimeV2Type(scale); - } - case "SECONDDATE": - // SECONDDATE with second precision - return ScalarType.createDatetimeV2Type(0); - case "DATE": - return ScalarType.createDateV2Type(); - case "BOOLEAN": - return Type.BOOLEAN; - case "CHAR": - case "NCHAR": - return ScalarType.createCharType(fieldSchema.requiredColumnSize()); - case "TIME": - case "VARCHAR": - case "NVARCHAR": - case "ALPHANUM": - case "SHORTTEXT": - case "CLOB": - case "NCLOB": - case "TEXT": - case "BINTEXT": - case "BINARY": - case "VARBINARY": - return ScalarType.createStringType(); - case "BLOB": - case "ST_GEOMETRY": - case "ST_POINT": - default: - return Type.UNSUPPORTED; - } - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcTrinoClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcTrinoClient.java deleted file mode 100644 index 6c818a41cb6aca..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcTrinoClient.java +++ /dev/null @@ -1,91 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.jdbc.client; - -import org.apache.doris.catalog.ArrayType; -import org.apache.doris.catalog.ScalarType; -import org.apache.doris.catalog.Type; -import org.apache.doris.datasource.jdbc.util.JdbcFieldSchema; - -import java.util.Optional; - -public class JdbcTrinoClient extends JdbcClient { - protected JdbcTrinoClient(JdbcClientConfig jdbcClientConfig) { - super(jdbcClientConfig); - } - - @Override - protected Type jdbcTypeToDoris(JdbcFieldSchema fieldSchema) { - String trinoType = fieldSchema.getDataTypeName().orElse("unknown"); - switch (trinoType) { - case "integer": - return Type.INT; - case "bigint": - return Type.BIGINT; - case "smallint": - return Type.SMALLINT; - case "tinyint": - return Type.TINYINT; - case "double": - return Type.DOUBLE; - case "real": - return Type.FLOAT; - case "boolean": - return Type.BOOLEAN; - case "date": - return ScalarType.createDateV2Type(); - case "json": - return ScalarType.createStringType(); - default: - break; - } - - if (trinoType.startsWith("decimal")) { - String[] split = trinoType.split("\\("); - String[] precisionAndScale = split[1].split(","); - int precision = Integer.parseInt(precisionAndScale[0]); - int scale = Integer.parseInt(precisionAndScale[1].substring(0, precisionAndScale[1].length() - 1)); - return createDecimalOrStringType(precision, scale); - } - - if (trinoType.startsWith("char")) { - return ScalarType.createCharType(fieldSchema.requiredColumnSize()); - } - - if (trinoType.startsWith("timestamp")) { - int scale = fieldSchema.getDecimalDigits().orElse(0); - if (scale > 6) { - scale = 6; - } - return ScalarType.createDatetimeV2Type(scale); - } - - if (trinoType.startsWith("array")) { - String trinoArrType = trinoType.substring(6, trinoType.length() - 1); - fieldSchema.setDataTypeName(Optional.of(trinoArrType)); - Type type = jdbcTypeToDoris(fieldSchema); - return ArrayType.create(type, true); - } - - if (trinoType.startsWith("varchar") || trinoType.startsWith("time")) { - return ScalarType.createStringType(); - } - - return Type.UNSUPPORTED; - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/util/JdbcFieldSchema.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/util/JdbcFieldSchema.java deleted file mode 100644 index 5e1f0a87bbacb9..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/util/JdbcFieldSchema.java +++ /dev/null @@ -1,129 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.jdbc.util; - -import lombok.Data; - -import java.sql.DatabaseMetaData; -import java.sql.ResultSet; -import java.sql.ResultSetMetaData; -import java.sql.SQLException; -import java.util.Map; -import java.util.Optional; - -@Data -public class JdbcFieldSchema { - protected String columnName; - // The SQL type of the corresponding java.sql.types (Type ID) - protected int dataType; - // The SQL type of the corresponding java.sql.types (Type Name) - protected Optional dataTypeName; - // For CHAR/DATA, columnSize means the maximum number of chars. - // For NUMERIC/DECIMAL, columnSize means precision. - protected Optional columnSize; - protected Optional decimalDigits; - protected Optional arrayDimensions; - // Base number (usually 10 or 2) - protected int numPrecRadix; - // column description - protected String remarks; - // This length is the maximum number of bytes for CHAR type - // for utf8 encoding, if columnSize=10, then charOctetLength=30 - // because for utf8 encoding, a Chinese character takes up 3 bytes - protected int charOctetLength; - protected boolean isAllowNull; - - public JdbcFieldSchema(JdbcFieldSchema other) { - this.columnName = other.columnName; - this.dataType = other.dataType; - this.dataTypeName = other.dataTypeName; - this.columnSize = other.columnSize; - this.decimalDigits = other.decimalDigits; - this.arrayDimensions = other.arrayDimensions; - this.numPrecRadix = other.numPrecRadix; - this.remarks = other.remarks; - this.charOctetLength = other.charOctetLength; - this.isAllowNull = other.isAllowNull; - } - - public JdbcFieldSchema(ResultSet rs) throws SQLException { - this.columnName = rs.getString("COLUMN_NAME"); - this.dataType = getInteger(rs, "DATA_TYPE").orElseThrow(() -> new IllegalStateException("DATA_TYPE is null")); - this.dataTypeName = Optional.ofNullable(rs.getString("TYPE_NAME")); - this.columnSize = getInteger(rs, "COLUMN_SIZE"); - this.decimalDigits = getInteger(rs, "DECIMAL_DIGITS"); - this.numPrecRadix = rs.getInt("NUM_PREC_RADIX"); - this.isAllowNull = rs.getInt("NULLABLE") != DatabaseMetaData.columnNoNulls; - this.remarks = rs.getString("REMARKS"); - this.charOctetLength = rs.getInt("CHAR_OCTET_LENGTH"); - } - - public JdbcFieldSchema(ResultSet rs, int arrayDimensions) throws SQLException { - this.columnName = rs.getString("COLUMN_NAME"); - this.dataType = getInteger(rs, "DATA_TYPE").orElseThrow(() -> new IllegalStateException("DATA_TYPE is null")); - this.dataTypeName = Optional.ofNullable(rs.getString("TYPE_NAME")); - this.columnSize = getInteger(rs, "COLUMN_SIZE"); - this.decimalDigits = getInteger(rs, "DECIMAL_DIGITS"); - this.numPrecRadix = rs.getInt("NUM_PREC_RADIX"); - this.isAllowNull = rs.getInt("NULLABLE") != DatabaseMetaData.columnNoNulls; - this.remarks = rs.getString("REMARKS"); - this.charOctetLength = rs.getInt("CHAR_OCTET_LENGTH"); - this.arrayDimensions = Optional.of(arrayDimensions); - } - - public JdbcFieldSchema(ResultSet rs, Map dataTypeOverrides) throws SQLException { - this.columnName = rs.getString("COLUMN_NAME"); - this.dataType = getInteger(rs, "DATA_TYPE").orElseThrow(() -> new IllegalStateException("DATA_TYPE is null")); - this.dataTypeName = Optional.ofNullable(dataTypeOverrides.getOrDefault(columnName, rs.getString("TYPE_NAME"))); - this.columnSize = getInteger(rs, "COLUMN_SIZE"); - this.decimalDigits = getInteger(rs, "DECIMAL_DIGITS"); - this.numPrecRadix = rs.getInt("NUM_PREC_RADIX"); - this.isAllowNull = rs.getInt("NULLABLE") != 0; - this.remarks = rs.getString("REMARKS"); - this.charOctetLength = rs.getInt("CHAR_OCTET_LENGTH"); - } - - public JdbcFieldSchema(ResultSetMetaData metaData, int columnIndex) throws SQLException { - String columnLabel = metaData.getColumnLabel(columnIndex); - this.columnName = columnLabel == null || columnLabel.isEmpty() - ? metaData.getColumnName(columnIndex) - : columnLabel; - this.dataType = metaData.getColumnType(columnIndex); - this.dataTypeName = Optional.ofNullable(metaData.getColumnTypeName(columnIndex)); - this.columnSize = Optional.of(metaData.getPrecision(columnIndex)); - this.decimalDigits = Optional.of(metaData.getScale(columnIndex)); - this.arrayDimensions = Optional.of(0); - } - - public int requiredColumnSize() { - return columnSize.orElseThrow(() -> new IllegalStateException("column size not present")); - } - - public int requiredDecimalDigits() { - return decimalDigits.orElseThrow(() -> new IllegalStateException("decimal digits not present")); - } - - protected static Optional getInteger(ResultSet resultSet, String columnLabel) - throws SQLException { - int value = resultSet.getInt(columnLabel); - if (resultSet.wasNull()) { - return Optional.empty(); - } - return Optional.of(value); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/job/common/DataSourceType.java b/fe/fe-core/src/main/java/org/apache/doris/job/common/DataSourceType.java index 3eec77e757b5da..d0e9782fd5b73a 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/job/common/DataSourceType.java +++ b/fe/fe-core/src/main/java/org/apache/doris/job/common/DataSourceType.java @@ -17,8 +17,18 @@ package org.apache.doris.job.common; +/** + * The source databases a streaming (CDC) job can read. Every one of them is reached, for metadata + * discovery on the FE, through the connector plugin named by {@link #connectorType()}; the BE-side CDC + * client is chosen separately by the streaming framework. + */ public enum DataSourceType { MYSQL, POSTGRES, - OCEANBASE + OCEANBASE; + + /** The connector plugin type ({@code ConnectorProvider.getType()}) that serves this source's metadata. */ + public String connectorType() { + return "jdbc"; + } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/DataSourceConfigValidator.java b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/DataSourceConfigValidator.java index f75bf03c56e043..cf0354204ed171 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/DataSourceConfigValidator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/DataSourceConfigValidator.java @@ -17,12 +17,13 @@ package org.apache.doris.job.extensions.insert.streaming; -import org.apache.doris.datasource.jdbc.client.JdbcClient; -import org.apache.doris.datasource.jdbc.client.JdbcClientException; +import org.apache.doris.common.util.Util; +import org.apache.doris.connector.spi.ConnectorQueryResult; import org.apache.doris.job.cdc.DataSourceConfigKeys; import org.apache.doris.job.common.DataSourceType; import org.apache.doris.job.exception.JobException; import org.apache.doris.job.util.StreamingJobUtils; +import org.apache.doris.job.util.StreamingSourceClient; import org.apache.doris.nereids.trees.plans.commands.LoadCommand; import com.fasterxml.jackson.databind.JsonNode; @@ -30,9 +31,8 @@ import com.google.common.collect.ImmutableSortedSet; import com.google.common.collect.Sets; -import java.sql.Connection; -import java.sql.ResultSet; -import java.sql.Statement; +import java.util.Collections; +import java.util.List; import java.util.Map; import java.util.Set; import java.util.regex.Pattern; @@ -155,17 +155,17 @@ public static void validateSourceBeforeTableCreation( private static void validateOceanBaseCompatibilityMode(Map sourceProperties) throws JobException { - // jdbc:mysql routes through JdbcMySQLClient so Connector/J is initialized consistently. - JdbcClient jdbcClient = StreamingJobUtils.getJdbcClient( - DataSourceType.OCEANBASE, sourceProperties); - try (Connection connection = jdbcClient.getConnection(); - Statement statement = connection.createStatement(); - ResultSet resultSet = statement.executeQuery( - "SHOW VARIABLES LIKE 'ob_compatibility_mode'")) { - if (!resultSet.next()) { + // The jdbc_url is jdbc:mysql://, so the connector talks to OceanBase through the MySQL dialect and + // the compatibility mode is read the way a MySQL client reads a variable. + try (StreamingSourceClient sourceClient = StreamingJobUtils.openSourceClient( + DataSourceType.OCEANBASE, sourceProperties)) { + ConnectorQueryResult result = sourceClient.executeQuery( + "SHOW VARIABLES LIKE 'ob_compatibility_mode'", Collections.emptyList()); + if (result.isEmpty()) { throw new JobException("Failed to determine OceanBase compatibility mode"); } - String compatibilityMode = resultSet.getString(2); + List row = result.getRows().get(0); + String compatibilityMode = row.size() > 1 && row.get(1) != null ? row.get(1).toString() : null; if ("MYSQL".equalsIgnoreCase(compatibilityMode)) { return; } @@ -180,9 +180,7 @@ private static void validateOceanBaseCompatibilityMode(Map sourc } catch (Exception e) { throw new JobException( "Failed to validate OceanBase compatibility mode: " - + JdbcClientException.getAllExceptionMessages(e), e); - } finally { - jdbcClient.closeClient(); + + Util.getAllExceptionMessages(e), e); } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/PostgresResourceValidator.java b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/PostgresResourceValidator.java index ae6277b3028b79..fd5e3468a60976 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/PostgresResourceValidator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/PostgresResourceValidator.java @@ -18,19 +18,18 @@ package org.apache.doris.job.extensions.insert.streaming; import org.apache.doris.common.Config; -import org.apache.doris.datasource.jdbc.client.JdbcClient; +import org.apache.doris.connector.spi.ConnectorQueryResult; import org.apache.doris.job.cdc.DataSourceConfigKeys; import org.apache.doris.job.common.DataSourceType; import org.apache.doris.job.exception.JobException; import org.apache.doris.job.util.StreamingJobUtils; +import org.apache.doris.job.util.StreamingSourceClient; import org.apache.commons.lang3.StringUtils; import java.nio.charset.StandardCharsets; -import java.sql.Connection; -import java.sql.PreparedStatement; -import java.sql.ResultSet; import java.util.ArrayList; +import java.util.Collections; import java.util.HashSet; import java.util.List; import java.util.Map; @@ -61,8 +60,8 @@ public static void validate(Map sourceProperties, String jobId, qualifiedTables.add(pgSchema + "." + name); } - JdbcClient jdbcClient = StreamingJobUtils.getJdbcClient(DataSourceType.POSTGRES, sourceProperties); - try (Connection conn = jdbcClient.getConnection()) { + try (StreamingSourceClient conn = StreamingJobUtils.openSourceClient(DataSourceType.POSTGRES, + sourceProperties)) { boolean pubExists = publicationExists(conn, publicationName); if (!pubExists && pubUserProvided) { throw new JobException( @@ -112,8 +111,6 @@ public static void validate(Map sourceProperties, String jobId, throw new JobException( "Failed to validate PG resources for publication " + publicationName + ": " + e.getMessage(), e); - } finally { - jdbcClient.closeClient(); } } @@ -140,26 +137,20 @@ private static void checkDatabaseNameLength(String database) throws JobException } } - private static boolean publicationExists(Connection conn, String publicationName) throws Exception { - try (PreparedStatement ps = conn.prepareStatement("SELECT 1 FROM pg_publication WHERE pubname = ?")) { - ps.setString(1, publicationName); - try (ResultSet rs = ps.executeQuery()) { - return rs.next(); - } - } + private static boolean publicationExists(StreamingSourceClient conn, String publicationName) throws Exception { + ConnectorQueryResult rs = conn.executeQuery("SELECT 1 FROM pg_publication WHERE pubname = ?", + Collections.singletonList(publicationName)); + return !rs.isEmpty(); } - private static List findMissingTables(Connection conn, String publicationName, List tables) - throws Exception { + private static List findMissingTables(StreamingSourceClient conn, String publicationName, + List tables) throws Exception { Set covered = new HashSet<>(); - try (PreparedStatement ps = conn.prepareStatement( - "SELECT schemaname, tablename FROM pg_publication_tables WHERE pubname = ?")) { - ps.setString(1, publicationName); - try (ResultSet rs = ps.executeQuery()) { - while (rs.next()) { - covered.add(rs.getString(1) + "." + rs.getString(2)); - } - } + ConnectorQueryResult rs = conn.executeQuery( + "SELECT schemaname, tablename FROM pg_publication_tables WHERE pubname = ?", + Collections.singletonList(publicationName)); + for (List row : rs.getRows()) { + covered.add(row.get(0) + "." + row.get(1)); } List missing = new ArrayList<>(); for (String table : tables) { @@ -171,16 +162,13 @@ private static List findMissingTables(Connection conn, String publicatio } /** Returns the slot's active flag, or null when the slot does not exist. */ - private static Boolean queryReplicationSlotActive(Connection conn, String slotName) throws Exception { - try (PreparedStatement ps = conn.prepareStatement( - "SELECT active FROM pg_replication_slots WHERE slot_name = ?")) { - ps.setString(1, slotName); - try (ResultSet rs = ps.executeQuery()) { - if (!rs.next()) { - return null; - } - return rs.getBoolean(1); - } + private static Boolean queryReplicationSlotActive(StreamingSourceClient conn, String slotName) throws Exception { + ConnectorQueryResult rs = conn.executeQuery("SELECT active FROM pg_replication_slots WHERE slot_name = ?", + Collections.singletonList(slotName)); + if (rs.isEmpty()) { + return null; } + // The driver hands a PostgreSQL boolean back as a Boolean; anything else reads as "not active". + return Boolean.TRUE.equals(rs.getRows().get(0).get(0)); } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/job/util/StreamingJobUtils.java b/fe/fe-core/src/main/java/org/apache/doris/job/util/StreamingJobUtils.java index bb5d88f48cf327..4503f1d32e426e 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/job/util/StreamingJobUtils.java +++ b/fe/fe-core/src/main/java/org/apache/doris/job/util/StreamingJobUtils.java @@ -33,8 +33,6 @@ import org.apache.doris.common.util.SmallFileMgr; import org.apache.doris.common.util.SmallFileMgr.SmallFile; import org.apache.doris.datasource.InternalCatalog; -import org.apache.doris.datasource.jdbc.client.JdbcClient; -import org.apache.doris.datasource.jdbc.client.JdbcClientConfig; import org.apache.doris.job.cdc.DataSourceConfigKeys; import org.apache.doris.job.cdc.split.SnapshotSplit; import org.apache.doris.job.common.DataSourceType; @@ -263,15 +261,14 @@ private static ConnectContext buildConnectContext() { return ctx; } - public static JdbcClient getJdbcClient(DataSourceType sourceType, Map properties) { - JdbcClientConfig config = new JdbcClientConfig(); - config.setCatalog(sourceType.name()); - config.setUser(properties.get(DataSourceConfigKeys.USER)); - config.setPassword(properties.get(DataSourceConfigKeys.PASSWORD)); - config.setDriverClass(properties.get(DataSourceConfigKeys.DRIVER_CLASS)); - config.setDriverUrl(properties.get(DataSourceConfigKeys.DRIVER_URL)); - config.setJdbcUrl(properties.get(DataSourceConfigKeys.JDBC_URL)); - return JdbcClient.createJdbcClient(config); + /** + * Opens the source database of a streaming job for metadata discovery, through its connector plugin. + * The job's source properties use the same names a JDBC catalog does ({@code jdbc_url}, {@code user}, + * {@code password}, {@code driver_url}, {@code driver_class}), so they are handed over as they are. + */ + public static StreamingSourceClient openSourceClient(DataSourceType sourceType, Map properties) + throws JobException { + return StreamingSourceClient.open(sourceType, properties); } public static Backend selectBackend(String cloudCluster) throws JobException { @@ -379,10 +376,10 @@ public static LinkedHashMap> generateCreate excludeTablesList = Arrays.asList(excludeTables.split(",")); } - JdbcClient jdbcClient = getJdbcClient(sourceType, properties); + StreamingSourceClient sourceClient = openSourceClient(sourceType, properties); try { String database = getRemoteDbName(sourceType, properties); - List tablesNameList = jdbcClient.getTablesNameList(database); + List tablesNameList = sourceClient.listTables(database); if (tablesNameList.isEmpty()) { throw new JobException("No tables found in database " + database); } @@ -406,7 +403,7 @@ public static LinkedHashMap> generateCreate continue; } - List primaryKeys = jdbcClient.getPrimaryKeys(database, table); + List primaryKeys = sourceClient.getPrimaryKeys(database, table); if (primaryKeys.isEmpty()) { noPrimaryKeyTables.add(table); } @@ -421,8 +418,8 @@ public static LinkedHashMap> generateCreate Set excludeColumns = parseExcludeColumns(properties, table); if (targetDatabase.isTableExist(targetTableName)) { if (!excludeColumns.isEmpty()) { - Set columnNames = jdbcClient.getJdbcColumnsInfo(database, table).stream() - .map(field -> field.getColumnName()) + Set columnNames = sourceClient.getColumns(database, table).stream() + .map(Column::getName) .collect(Collectors.toSet()); validateExcludeColumns(excludeColumns, table, columnNames, primaryKeys); } @@ -430,7 +427,7 @@ public static LinkedHashMap> generateCreate continue; } - List columns = getColumns(jdbcClient, database, table, primaryKeys); + List columns = getColumns(sourceClient, database, table, primaryKeys); if (!excludeColumns.isEmpty()) { Set columnNames = columns.stream().map(Column::getName).collect(Collectors.toSet()); validateExcludeColumns(excludeColumns, table, columnNames, primaryKeys); @@ -488,15 +485,15 @@ public static LinkedHashMap> generateCreate } return createtblCmds; } finally { - jdbcClient.closeClient(); + sourceClient.close(); } } - public static List getColumns(JdbcClient jdbcClient, + public static List getColumns(StreamingSourceClient sourceClient, String database, String table, - List primaryKeys) { - List columns = jdbcClient.getColumnsFromJdbc(database, table); + List primaryKeys) throws JobException { + List columns = sourceClient.getColumns(database, table); columns.forEach(col -> { Preconditions.checkArgument(!col.getType().isUnsupported(), "Unsupported column type, table:[%s], column:[%s]", table, col.getName()); diff --git a/fe/fe-core/src/main/java/org/apache/doris/job/util/StreamingSourceClient.java b/fe/fe-core/src/main/java/org/apache/doris/job/util/StreamingSourceClient.java new file mode 100644 index 00000000000000..678d7caf55f6bb --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/job/util/StreamingSourceClient.java @@ -0,0 +1,201 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + + +package org.apache.doris.job.util; + +import org.apache.doris.catalog.Column; +import org.apache.doris.connector.ConnectorFactory; +import org.apache.doris.connector.ConnectorSessionBuilder; +import org.apache.doris.connector.DefaultConnectorContext; +import org.apache.doris.connector.spi.Connector; +import org.apache.doris.connector.spi.ConnectorMetadata; +import org.apache.doris.connector.spi.ConnectorPassthroughSqlOps; +import org.apache.doris.connector.spi.ConnectorQueryResult; +import org.apache.doris.connector.spi.ConnectorSession; +import org.apache.doris.connector.spi.ConnectorStatementScope; +import org.apache.doris.connector.spi.handle.ConnectorTableHandle; +import org.apache.doris.datasource.connector.converter.ConnectorColumnConverter; +import org.apache.doris.datasource.plugin.PluginDrivenMetadata; +import org.apache.doris.job.common.DataSourceType; +import org.apache.doris.job.exception.JobException; + +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; + +import java.io.IOException; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.function.Supplier; + +/** + * The streaming (CDC) framework's view of a source database, served by the source's connector plugin + * through the connector SPI: the tables of a remote database, their columns as Doris columns, their + * primary keys, and the small probe queries the framework runs before it starts a job. + * + *

    One instance is one temporary connector — its own connection pool, its own driver classloader + * (shared per driver url with every catalog) — opened for one piece of work and closed with it, exactly + * as the framework used its own JDBC client before. Metadata is acquired through the engine's + * {@link PluginDrivenMetadata} funnel like every other caller; nothing here knows what a JDBC driver is. + * The source properties are the job's ({@code jdbc_url}, {@code user}, {@code password}, + * {@code driver_url}, {@code driver_class}, ...), which are exactly the property names a JDBC catalog + * takes; the connector ignores the streaming-only keys.

    + */ +public class StreamingSourceClient implements AutoCloseable { + + private static final Logger LOG = LogManager.getLogger(StreamingSourceClient.class); + + private final DataSourceType sourceType; + private final Connector connector; + private final ConnectorSession session; + private final Map handles = new HashMap<>(); + + StreamingSourceClient(DataSourceType sourceType, Connector connector, ConnectorSession session) { + this.sourceType = sourceType; + this.connector = connector; + this.session = session; + } + + /** + * Opens a client over {@code sourceProps} through the connector plugin that serves {@code sourceType}. + * + * @throws JobException when that plugin is not installed, or the connector rejects the properties + */ + public static StreamingSourceClient open(DataSourceType sourceType, Map sourceProps) + throws JobException { + String connectorType = sourceType.connectorType(); + Connector connector; + try { + connector = ConnectorFactory.createConnector(connectorType, sourceProps, + new DefaultConnectorContext(sourceType.name(), -1L)); + } catch (RuntimeException e) { + throw new JobException("Failed to open streaming source " + sourceType + ": " + e.getMessage(), e); + } + if (connector == null) { + throw new JobException("Streaming source " + sourceType + " requires the '" + connectorType + + "' connector plugin, which is not installed"); + } + ConnectorSession session = ConnectorSessionBuilder.create() + .withCatalogId(-1L) + .withCatalogName(sourceType.name()) + .withCatalogProperties(sourceProps) + .withStatementScope(ConnectorStatementScope.NONE) + .build(); + return new StreamingSourceClient(sourceType, connector, session); + } + + /** The tables of a remote database (a MySQL database, a PostgreSQL schema). */ + public List listTables(String remoteDb) { + return onPluginClassLoader(() -> metadata().listTableNames(session, remoteDb)); + } + + public boolean tableExists(String remoteDb, String table) { + return onPluginClassLoader(() -> handle(remoteDb, table).isPresent()); + } + + /** + * The table's columns as Doris columns, converted from the connector's schema exactly as a catalog + * table's are. + * + * @throws JobException when the table does not exist + */ + public List getColumns(String remoteDb, String table) throws JobException { + ConnectorTableHandle handle = requireHandle(remoteDb, table); + return onPluginClassLoader(() -> ConnectorColumnConverter.convertColumns( + metadata().getTableSchema(session, handle).getColumns())); + } + + /** + * The table's primary-key column names in key order; empty when it has none. + * + * @throws JobException when the table does not exist + */ + public List getPrimaryKeys(String remoteDb, String table) throws JobException { + ConnectorTableHandle handle = requireHandle(remoteDb, table); + return onPluginClassLoader(() -> metadata().getPrimaryKeys(session, handle)); + } + + /** + * Runs a read-only probe query on the source with positional parameters and returns its rows. + * + * @throws JobException when the source's connector cannot run queries + */ + public ConnectorQueryResult executeQuery(String sql, List params) throws JobException { + ConnectorMetadata metadata = onPluginClassLoader(this::metadata); + if (!(metadata instanceof ConnectorPassthroughSqlOps)) { + throw new JobException("Streaming source " + sourceType + " does not support probe queries"); + } + return onPluginClassLoader(() -> ((ConnectorPassthroughSqlOps) metadata).executeQuery(session, sql, params)); + } + + @Override + public void close() { + Thread thread = Thread.currentThread(); + ClassLoader previous = thread.getContextClassLoader(); + thread.setContextClassLoader(connector.getClass().getClassLoader()); + try { + connector.close(); + } catch (IOException e) { + LOG.warn("Failed to close streaming source {}", sourceType, e); + } finally { + thread.setContextClassLoader(previous); + } + } + + private ConnectorMetadata metadata() { + return PluginDrivenMetadata.get(session, connector); + } + + private ConnectorTableHandle requireHandle(String remoteDb, String table) throws JobException { + Optional handle = onPluginClassLoader(() -> handle(remoteDb, table)); + if (!handle.isPresent()) { + throw new JobException("Table " + remoteDb + "." + table + " does not exist in streaming source " + + sourceType); + } + return handle.get(); + } + + private Optional handle(String remoteDb, String table) { + String key = remoteDb + "." + table; + ConnectorTableHandle cached = handles.get(key); + if (cached != null) { + return Optional.of(cached); + } + Optional handle = metadata().getTableHandle(session, remoteDb, table); + handle.ifPresent(h -> handles.put(key, h)); + return handle; + } + + /** + * Runs {@code body} with the thread-context classloader pinned to the connector's plugin loader, the + * engine-side convention at every plugin boundary (a plugin's by-name reflection resolves against the + * context loader; unpinned it would find fe-core's copies of shared classes). + */ + private T onPluginClassLoader(Supplier body) { + Thread thread = Thread.currentThread(); + ClassLoader previous = thread.getContextClassLoader(); + thread.setContextClassLoader(connector.getClass().getClassLoader()); + try { + return body.get(); + } finally { + thread.setContextClassLoader(previous); + } + } + +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/CdcStreamTableValuedFunction.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/CdcStreamTableValuedFunction.java index e4323cef82ab15..b55d254fe65491 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/CdcStreamTableValuedFunction.java +++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/CdcStreamTableValuedFunction.java @@ -22,13 +22,14 @@ import org.apache.doris.catalog.Column; import org.apache.doris.catalog.PrimitiveType; import org.apache.doris.common.AnalysisException; -import org.apache.doris.datasource.jdbc.client.JdbcClient; import org.apache.doris.job.cdc.DataSourceConfigKeys; import org.apache.doris.job.cdc.request.FetchRecordRequest; import org.apache.doris.job.common.DataSourceType; +import org.apache.doris.job.exception.JobException; import org.apache.doris.job.extensions.insert.streaming.DataSourceConfigValidator; import org.apache.doris.job.extensions.insert.streaming.StreamingJdbcUrlNormalizer; import org.apache.doris.job.util.StreamingJobUtils; +import org.apache.doris.job.util.StreamingSourceClient; import org.apache.doris.thrift.TBrokerFileStatus; import org.apache.doris.thrift.TFileType; @@ -218,20 +219,20 @@ private void generateFileStatus() { public List getTableColumns() throws AnalysisException { DataSourceType dataSourceType = DataSourceType.valueOf(processedParams.get(DataSourceConfigKeys.TYPE).toUpperCase()); - JdbcClient jdbcClient = StreamingJobUtils.getJdbcClient(dataSourceType, processedParams); - try { + try (StreamingSourceClient sourceClient = StreamingJobUtils.openSourceClient(dataSourceType, + processedParams)) { String database = StreamingJobUtils.getRemoteDbName(dataSourceType, processedParams); String table = processedParams.get(DataSourceConfigKeys.TABLE); - if (!jdbcClient.isTableExist(database, table)) { + if (!sourceClient.tableExists(database, table)) { throw new AnalysisException("Table does not exist: " + table); } - List columns = new ArrayList<>(jdbcClient.getColumnsFromJdbc(database, table)); + List columns = new ArrayList<>(sourceClient.getColumns(database, table)); if (includeDeleteSign) { columns.add(new Column(Column.DELETE_SIGN, PrimitiveType.TINYINT, false)); } return columns; - } finally { - jdbcClient.closeClient(); + } catch (JobException e) { + throw new AnalysisException(e.getMessage(), e); } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/common/util/UtilTest.java b/fe/fe-core/src/test/java/org/apache/doris/common/util/UtilTest.java index db4f7f028f4d65..2a8241f9a702b3 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/common/util/UtilTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/common/util/UtilTest.java @@ -120,6 +120,17 @@ public void strictBooleanPropertyParsesAndDefaults() throws AnalysisException { Assertions.assertNull(Util.getOptionalBooleanProperty(properties, "missing_optional_flag")); } + @Test + public void allExceptionMessagesJoinTheCauseChain() { + Exception root = new IllegalStateException("root"); + Exception middle = new RuntimeException("middle", root); + Exception top = new Exception("top", middle); + Assertions.assertEquals("top | Caused by: middle | Caused by: root", Util.getAllExceptionMessages(top)); + // Causes without a message are skipped rather than rendered as "null". + Assertions.assertEquals("top", Util.getAllExceptionMessages(new Exception("top", new RuntimeException()))); + Assertions.assertEquals("", Util.getAllExceptionMessages(null)); + } + @Test public void strictBooleanPropertyRejectsInvalidValue() { Map properties = new HashMap<>(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcClickHouseClientTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcClickHouseClientTest.java deleted file mode 100644 index dafb7ade3c7d2a..00000000000000 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcClickHouseClientTest.java +++ /dev/null @@ -1,92 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.jdbc.client; - -import org.junit.jupiter.api.Assertions; -import org.junit.jupiter.api.Test; -import org.mockito.Answers; -import org.mockito.Mockito; - -import java.lang.reflect.Method; -import java.sql.DatabaseMetaData; - -public class JdbcClickHouseClientTest { - - @Test - public void testDatabaseTermFollowsDriverMetadata() throws Exception { - DatabaseMetaData databaseMetaData = Mockito.mock(DatabaseMetaData.class); - - Mockito.when(databaseMetaData.supportsCatalogsInDataManipulation()).thenReturn(false); - Assertions.assertFalse(JdbcClickHouseClient.isDatabaseTermCatalog(databaseMetaData, "0.9.8")); - - Mockito.when(databaseMetaData.supportsCatalogsInDataManipulation()).thenReturn(true); - Assertions.assertTrue(JdbcClickHouseClient.isDatabaseTermCatalog(databaseMetaData, "0.7.1")); - - Assertions.assertFalse(JdbcClickHouseClient.isDatabaseTermCatalog(databaseMetaData, "0.4.2")); - } - - @Test - public void testClickHouseSpecificTableTypesAreVisible() { - JdbcClickHouseClient client = Mockito.mock(JdbcClickHouseClient.class, Answers.CALLS_REAL_METHODS); - - Assertions.assertArrayEquals( - new String[] {"TABLE", "VIEW", "SYSTEM TABLE", "REMOTE TABLE", "MATERIALIZED VIEW"}, - client.getTableTypes()); - } - - @Test - public void testIsNewClickHouseDriver() { - try { - Method method = JdbcClickHouseClient.class.getDeclaredMethod("isNewClickHouseDriver", String.class); - method.setAccessible(true); - - // Valid test cases - Assertions.assertTrue((boolean) method.invoke(null, "0.5.0")); // Major version 0, Minor version 5 - Assertions.assertTrue((boolean) method.invoke(null, "1.0.0")); // Major version 1 - Assertions.assertTrue((boolean) method.invoke(null, "0.6.3 (revision: a6a8a22)")); // Major version 0, Minor version 6 - Assertions.assertFalse((boolean) method.invoke(null, "0.4.2 (revision: 1513b27)")); // Major version 0, Minor version 4 - - // Invalid version formats - try { - method.invoke(null, "invalid.version"); // Invalid version format - Assertions.fail("Expected JdbcClientException for invalid version 'invalid.version'"); - } catch (Exception e) { - Assertions.assertTrue(e.getCause() instanceof JdbcClientException); - Assertions.assertTrue(e.getCause().getMessage().contains("Invalid clickhouse driver version format")); - } - - try { - method.invoke(null, ""); // Empty version - Assertions.fail("Expected JdbcClientException for empty version"); - } catch (Exception e) { - Assertions.assertTrue(e.getCause() instanceof JdbcClientException); - Assertions.assertTrue(e.getCause().getMessage().contains("Invalid clickhouse driver version format")); - } - - try { - method.invoke(null, (Object) null); // Null version - Assertions.fail("Expected JdbcClientException for null version"); - } catch (Exception e) { - Assertions.assertTrue(e.getCause() instanceof JdbcClientException); - Assertions.assertTrue(e.getCause().getMessage().contains("Driver version cannot be null")); - } - } catch (Exception e) { - Assertions.fail("Exception occurred while testing isNewClickHouseDriver: " + e.getMessage()); - } - } -} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcClientExceptionTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcClientExceptionTest.java deleted file mode 100644 index 040d4e9109f8cc..00000000000000 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcClientExceptionTest.java +++ /dev/null @@ -1,130 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.jdbc.client; - -import org.junit.jupiter.api.Assertions; -import org.junit.jupiter.api.Test; - -public class JdbcClientExceptionTest { - - @Test - public void testExceptionWithoutArgs() { - String message = "An error occurred."; - JdbcClientException exception = new JdbcClientException(message); - - Assertions.assertEquals(message, exception.getMessage()); - Assertions.assertNull(exception.getCause()); - } - - @Test - public void testExceptionWithFormattingArgs() { - String format = "Error code: %d, message: %s"; - int errorCode = 404; - String errorMsg = "Not Found"; - JdbcClientException exception = new JdbcClientException(format, errorCode, errorMsg); - - String expectedMessage = String.format(format, errorCode, errorMsg); - Assertions.assertEquals(expectedMessage, exception.getMessage()); - Assertions.assertNull(exception.getCause()); - } - - @Test - public void testExceptionWithPercentInFormatString() { - String format = "Usage is at 80%%, threshold is %d%%"; - int threshold = 75; - JdbcClientException exception = new JdbcClientException(format, threshold); - - String expectedMessage = String.format(format, threshold); - Assertions.assertEquals(expectedMessage, exception.getMessage()); - Assertions.assertNull(exception.getCause()); - } - - @Test - public void testExceptionWithPercentInArgs() { - String format = "Invalid input: %s"; - String input = "50% discount"; - JdbcClientException exception = new JdbcClientException(format, input); - - String expectedMessage = String.format(format, input.replace("%", "%%")); - Assertions.assertEquals(expectedMessage, exception.getMessage()); - Assertions.assertNull(exception.getCause()); - } - - @Test - public void testExceptionWithCause() { - String message = "Database connection failed."; - Exception cause = new Exception("Timeout occurred"); - JdbcClientException exception = new JdbcClientException(message, cause); - - Assertions.assertEquals(message, exception.getMessage()); - Assertions.assertEquals(cause, exception.getCause()); - } - - @Test - public void testExceptionWithFormattingArgsAndCause() { - String format = "Failed to execute query: %s"; - String query = "SELECT * FROM users"; - Exception cause = new Exception("Syntax error"); - JdbcClientException exception = new JdbcClientException(format, cause, query); - - String expectedMessage = String.format(format, query); - Assertions.assertEquals(expectedMessage, exception.getMessage()); - Assertions.assertEquals(cause, exception.getCause()); - } - - @Test - public void testExceptionWithPercentInArgsAndCause() { - String format = "File path: %s"; - String filePath = "C:\\Program Files\\App%20Data"; - Exception cause = new Exception("File not found"); - JdbcClientException exception = new JdbcClientException(format, cause, filePath); - - String expectedMessage = String.format(format, filePath.replace("%", "%%")); - Assertions.assertEquals(expectedMessage, exception.getMessage()); - Assertions.assertEquals(cause, exception.getCause()); - } - - @Test - public void testExceptionWithNoFormattingNeeded() { - String message = "Simple error message."; - JdbcClientException exception = new JdbcClientException(message, (Object[]) null); - - Assertions.assertEquals(message, exception.getMessage()); - Assertions.assertNull(exception.getCause()); - } - - @Test - public void testExceptionWithNullArgs() { - String format = "Error occurred: %s"; - JdbcClientException exception = new JdbcClientException(format, (Object[]) null); - - // Since args are null, message should remain unformatted - Assertions.assertEquals(format, exception.getMessage()); - Assertions.assertNull(exception.getCause()); - } - - @Test - public void testExceptionWithEmptyArgs() { - String format = "Error occurred: %s"; - JdbcClientException exception = new JdbcClientException(format, new Object[]{}); - - // Since args are empty, message should remain unformatted - Assertions.assertEquals(format, exception.getMessage()); - Assertions.assertNull(exception.getCause()); - } -} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcMySQLClientTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcMySQLClientTest.java deleted file mode 100644 index 6379c0db52aae2..00000000000000 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcMySQLClientTest.java +++ /dev/null @@ -1,37 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.jdbc.client; - -import org.junit.jupiter.api.Assertions; -import org.junit.jupiter.api.Test; - -public class JdbcMySQLClientTest { - - @Test - public void testIsDorisCompatibleVersionComment() { - Assertions.assertTrue(JdbcMySQLClient.isDorisCompatibleVersionComment("Apache Doris version 3.1.0")); - Assertions.assertTrue(JdbcMySQLClient.isDorisCompatibleVersionComment("SelectDB Cloud version 4.0.5")); - Assertions.assertTrue(JdbcMySQLClient.isDorisCompatibleVersionComment("VeloDB version 2.1.0")); - Assertions.assertTrue(JdbcMySQLClient.isDorisCompatibleVersionComment( - "enterprise version enterprise-4.0.5-rc01-0724569463d (Cloud Mode)")); - - Assertions.assertFalse(JdbcMySQLClient.isDorisCompatibleVersionComment("MySQL Community Server - GPL")); - Assertions.assertFalse(JdbcMySQLClient.isDorisCompatibleVersionComment("")); - Assertions.assertFalse(JdbcMySQLClient.isDorisCompatibleVersionComment(null)); - } -} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcOceanBaseClientTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcOceanBaseClientTest.java deleted file mode 100644 index 4addc9caaa2b1d..00000000000000 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcOceanBaseClientTest.java +++ /dev/null @@ -1,105 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.jdbc.client; - -import com.zaxxer.hikari.HikariDataSource; -import org.junit.jupiter.api.Assertions; -import org.junit.jupiter.api.BeforeEach; -import org.junit.jupiter.api.Test; -import org.mockito.InOrder; -import org.mockito.MockedConstruction; -import org.mockito.Mockito; - -import java.sql.Connection; -import java.sql.ResultSet; -import java.sql.Statement; - -public class JdbcOceanBaseClientTest { - private Connection connection; - private Statement statement; - private ResultSet resultSet; - - @BeforeEach - public void setUp() throws Exception { - connection = Mockito.mock(Connection.class); - statement = Mockito.mock(Statement.class); - resultSet = Mockito.mock(ResultSet.class); - Mockito.when(connection.createStatement()).thenReturn(statement); - Mockito.when(statement.executeQuery("SHOW VARIABLES LIKE 'ob_compatibility_mode'")).thenReturn(resultSet); - } - - @Test - public void testCloseTemporaryDataSourceAfterCreatingClient() throws Exception { - Mockito.when(resultSet.next()).thenReturn(true); - Mockito.when(resultSet.getString(2)).thenReturn("MYSQL"); - - try (MockedConstruction mockedDataSources = mockDataSources()) { - JdbcOceanBaseClient oceanBaseClient = new JdbcOceanBaseClient(createConfig()); - JdbcClient client = oceanBaseClient.createClient(createConfig()); - - Assertions.assertTrue(client instanceof JdbcMySQLClient); - Assertions.assertEquals(2, mockedDataSources.constructed().size()); - HikariDataSource temporaryDataSource = mockedDataSources.constructed().get(0); - HikariDataSource clientDataSource = mockedDataSources.constructed().get(1); - assertTemporaryResourcesClosed(temporaryDataSource); - Mockito.verify(clientDataSource, Mockito.never()).close(); - - client.closeClient(); - Mockito.verify(clientDataSource).close(); - } - } - - @Test - public void testCloseTemporaryDataSourceWhenCompatibilityModeIsMissing() throws Exception { - Mockito.when(resultSet.next()).thenReturn(false); - - try (MockedConstruction mockedDataSources = mockDataSources()) { - JdbcOceanBaseClient oceanBaseClient = new JdbcOceanBaseClient(createConfig()); - - JdbcClientException exception = Assertions.assertThrows( - JdbcClientException.class, () -> oceanBaseClient.createClient(createConfig())); - - Assertions.assertEquals("Failed to determine OceanBase compatibility mode", exception.getMessage()); - Assertions.assertEquals(1, mockedDataSources.constructed().size()); - assertTemporaryResourcesClosed(mockedDataSources.constructed().get(0)); - } - } - - private MockedConstruction mockDataSources() { - return Mockito.mockConstruction(HikariDataSource.class, (mock, context) -> - Mockito.when(mock.getConnection()).thenReturn(connection)); - } - - private JdbcClientConfig createConfig() { - return new JdbcClientConfig() - .setCatalog("oceanbase_catalog") - .setUser("user") - .setPassword("password") - .setJdbcUrl("jdbc:oceanbase://localhost:2881/test") - .setDriverUrl("file:///tmp/oceanbase-jdbc.jar") - .setDriverClass("com.oceanbase.jdbc.Driver"); - } - - private void assertTemporaryResourcesClosed(HikariDataSource temporaryDataSource) throws Exception { - InOrder inOrder = Mockito.inOrder(resultSet, statement, connection, temporaryDataSource); - inOrder.verify(resultSet).close(); - inOrder.verify(statement).close(); - inOrder.verify(connection).close(); - inOrder.verify(temporaryDataSource).close(); - } -} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcSQLServerClientTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcSQLServerClientTest.java deleted file mode 100644 index d7b8ad645bfe8d..00000000000000 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcSQLServerClientTest.java +++ /dev/null @@ -1,165 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.jdbc.client; - -import org.apache.doris.catalog.ScalarType; -import org.apache.doris.catalog.Type; -import org.apache.doris.datasource.jdbc.util.JdbcFieldSchema; - -import org.junit.jupiter.api.Assertions; -import org.junit.jupiter.api.Test; -import org.mockito.Answers; -import org.mockito.Mockito; - -import java.sql.DatabaseMetaData; -import java.sql.ResultSet; -import java.sql.SQLException; -import java.sql.Types; - -public class JdbcSQLServerClientTest { - - // ODBC type codes that mssql-jdbc passes through DatabaseMetaData.getColumns() unchanged - private static final int SQL_VARIANT = -150; - private static final int SQL_SS_TIMESTAMPOFFSET = -155; - - private final JdbcSQLServerClient client = Mockito.mock(JdbcSQLServerClient.class, Answers.CALLS_REAL_METHODS); - - /** - * Builds the schema of one DatabaseMetaData.getColumns() row as reported by mssql-jdbc. - * For a user-defined alias type, TYPE_NAME is the alias name while DATA_TYPE, COLUMN_SIZE - * and DECIMAL_DIGITS describe the base type. - */ - private static JdbcFieldSchema column(String typeName, int dataType, int columnSize, int decimalDigits) - throws SQLException { - ResultSet rs = Mockito.mock(ResultSet.class); - Mockito.when(rs.getString("COLUMN_NAME")).thenReturn("col"); - Mockito.when(rs.getInt("DATA_TYPE")).thenReturn(dataType); - Mockito.when(rs.getString("TYPE_NAME")).thenReturn(typeName); - Mockito.when(rs.getInt("COLUMN_SIZE")).thenReturn(columnSize); - Mockito.when(rs.getInt("DECIMAL_DIGITS")).thenReturn(decimalDigits); - Mockito.when(rs.getInt("NULLABLE")).thenReturn(DatabaseMetaData.columnNullable); - return new JdbcFieldSchema(rs); - } - - @Test - public void testAliasTypeIsResolvedByJdbcTypeCode() throws SQLException { - // CREATE TYPE dbo.customtexttype FROM varchar(50), the case reported in #67793 - Assertions.assertEquals(Type.STRING, client.jdbcTypeToDoris(column("customtexttype", Types.VARCHAR, 50, 0))); - // sysname is a built-in alias over nvarchar(128) - Assertions.assertEquals(Type.STRING, client.jdbcTypeToDoris(column("sysname", Types.NVARCHAR, 128, 0))); - Assertions.assertEquals(Type.STRING, client.jdbcTypeToDoris(column("alias_nchar", Types.NCHAR, 10, 0))); - Assertions.assertEquals(Type.STRING, - client.jdbcTypeToDoris(column("alias_text", Types.LONGVARCHAR, Integer.MAX_VALUE, 0))); - Assertions.assertEquals(Type.STRING, - client.jdbcTypeToDoris(column("alias_ntext", Types.LONGNVARCHAR, Integer.MAX_VALUE / 2, 0))); - Assertions.assertEquals(Type.STRING, client.jdbcTypeToDoris(column("alias_time", Types.TIME, 16, 7))); - // uniqueidentifier is reported as CHAR(36) - Assertions.assertEquals(Type.STRING, client.jdbcTypeToDoris(column("alias_guid", Types.CHAR, 36, 0))); - - Assertions.assertEquals(Type.BOOLEAN, client.jdbcTypeToDoris(column("alias_bit", Types.BIT, 1, 0))); - // SQL Server tinyint is unsigned, so it keeps the SMALLINT mapping of the name based path - Assertions.assertEquals(Type.SMALLINT, client.jdbcTypeToDoris(column("alias_tinyint", Types.TINYINT, 3, 0))); - Assertions.assertEquals(Type.SMALLINT, client.jdbcTypeToDoris(column("alias_smallint", Types.SMALLINT, 5, 0))); - Assertions.assertEquals(Type.INT, client.jdbcTypeToDoris(column("alias_int", Types.INTEGER, 10, 0))); - Assertions.assertEquals(Type.BIGINT, client.jdbcTypeToDoris(column("alias_bigint", Types.BIGINT, 19, 0))); - Assertions.assertEquals(Type.FLOAT, client.jdbcTypeToDoris(column("alias_real", Types.REAL, 24, 0))); - Assertions.assertEquals(Type.DOUBLE, client.jdbcTypeToDoris(column("alias_float", Types.DOUBLE, 53, 0))); - - Assertions.assertEquals(ScalarType.createDecimalV3Type(10, 2), - client.jdbcTypeToDoris(column("alias_decimal", Types.DECIMAL, 10, 2))); - Assertions.assertEquals(ScalarType.createDecimalV3Type(38, 10), - client.jdbcTypeToDoris(column("alias_numeric", Types.NUMERIC, 38, 10))); - // money is reported as DECIMAL(19,4), the same as the name based mapping produces - Assertions.assertEquals(ScalarType.createDecimalV3Type(19, 4), - client.jdbcTypeToDoris(column("alias_money", Types.DECIMAL, 19, 4))); - - Assertions.assertEquals(Type.DATEV2, client.jdbcTypeToDoris(column("alias_date", Types.DATE, 10, 0))); - Assertions.assertEquals(ScalarType.createDatetimeV2Type(3), - client.jdbcTypeToDoris(column("alias_datetime", Types.TIMESTAMP, 23, 3))); - // datetime2 defaults to 7 fractional digits, Doris supports at most 6 - Assertions.assertEquals(ScalarType.createDatetimeV2Type(6), - client.jdbcTypeToDoris(column("alias_datetime2", Types.TIMESTAMP, 27, 7))); - Assertions.assertEquals(ScalarType.createDatetimeV2Type(0), - client.jdbcTypeToDoris(column("alias_smalldatetime", Types.TIMESTAMP, 16, 0))); - } - - @Test - public void testUnknownTypesStayUnsupported() throws SQLException { - // CLR user-defined types are reported as VARBINARY, exactly like an alias over varbinary, - // so binary codes must not be resolved by the fallback - Assertions.assertEquals(Type.UNSUPPORTED, - client.jdbcTypeToDoris(column("geometry", Types.VARBINARY, Integer.MAX_VALUE, 0))); - Assertions.assertEquals(Type.UNSUPPORTED, - client.jdbcTypeToDoris(column("my_clr_type", Types.VARBINARY, 8000, 0))); - Assertions.assertEquals(Type.UNSUPPORTED, client.jdbcTypeToDoris(column("alias_binary", Types.BINARY, 20, 0))); - Assertions.assertEquals(Type.UNSUPPORTED, - client.jdbcTypeToDoris(column("alias_image", Types.LONGVARBINARY, Integer.MAX_VALUE, 0))); - // vendor specific type codes - Assertions.assertEquals(Type.UNSUPPORTED, client.jdbcTypeToDoris(column("sql_variant", SQL_VARIANT, 8000, 0))); - Assertions.assertEquals(Type.UNSUPPORTED, - client.jdbcTypeToDoris(column("alias_datetimeoffset", SQL_SS_TIMESTAMPOFFSET, 34, 7))); - // explicitly unsupported system types keep that behavior whatever type code the driver reports - Assertions.assertEquals(Type.UNSUPPORTED, - client.jdbcTypeToDoris(column("xml", Types.LONGNVARCHAR, Integer.MAX_VALUE / 2, 0))); - Assertions.assertEquals(Type.UNSUPPORTED, - client.jdbcTypeToDoris(column("json", Types.LONGNVARCHAR, Integer.MAX_VALUE / 2, 0))); - Assertions.assertEquals(Type.UNSUPPORTED, - client.jdbcTypeToDoris(column("hierarchyid", Types.VARBINARY, 892, 0))); - } - - @Test - public void testAliasNamedLikeASystemTypeIsResolvedByJdbcTypeCode() throws SQLException { - // A delimited alias name may contain spaces and parentheses ([int alias], [decimal(18,0) identity]); - // it is reported as is, and the base type is still what DATA_TYPE says - Assertions.assertEquals(Type.STRING, client.jdbcTypeToDoris(column("int alias", Types.VARCHAR, 50, 0))); - Assertions.assertEquals(Type.STRING, - client.jdbcTypeToDoris(column("decimal(18,0) identity", Types.NVARCHAR, 20, 0))); - Assertions.assertEquals(Type.STRING, client.jdbcTypeToDoris(column("int identity", Types.VARCHAR, 10, 0))); - Assertions.assertEquals(Type.STRING, - client.jdbcTypeToDoris(column("bigint identity", Types.NVARCHAR, 20, 0))); - Assertions.assertEquals(ScalarType.createDatetimeV2Type(3), - client.jdbcTypeToDoris(column("varchar(50) alias", Types.TIMESTAMP, 23, 3))); - - // The IDENTITY decoration of a real system type, in the forms the driver versions report it in - Assertions.assertEquals(Type.INT, client.jdbcTypeToDoris(column("int identity", Types.INTEGER, 10, 0))); - Assertions.assertEquals(Type.BIGINT, client.jdbcTypeToDoris(column("bigint identity", Types.BIGINT, 19, 0))); - Assertions.assertEquals(Type.SMALLINT, - client.jdbcTypeToDoris(column("tinyint identity", Types.TINYINT, 3, 0))); - Assertions.assertEquals(ScalarType.createDecimalV3Type(18, 0), - client.jdbcTypeToDoris(column("decimal identity", Types.DECIMAL, 18, 0))); - Assertions.assertEquals(ScalarType.createDecimalV3Type(18, 0), - client.jdbcTypeToDoris(column("decimal() identity", Types.DECIMAL, 18, 0))); - Assertions.assertEquals(ScalarType.createDecimalV3Type(18, 0), - client.jdbcTypeToDoris(column("numeric(18, 0) identity", Types.NUMERIC, 18, 0))); - Assertions.assertEquals(ScalarType.createDecimalV3Type(18, 0), - client.jdbcTypeToDoris(column("decimal(18,0) IDENTITY(1,1)", Types.DECIMAL, 18, 0))); - } - - @Test - public void testSystemTypeNamesTakePrecedence() throws SQLException { - // the name based mapping is unchanged, the type code is only consulted for unknown names - Assertions.assertEquals(Type.SMALLINT, client.jdbcTypeToDoris(column("tinyint", Types.TINYINT, 3, 0))); - Assertions.assertEquals(Type.INT, client.jdbcTypeToDoris(column("int identity", Types.INTEGER, 10, 0))); - Assertions.assertEquals(ScalarType.createDecimalV3Type(19, 4), - client.jdbcTypeToDoris(column("money", Types.DECIMAL, 19, 4))); - Assertions.assertEquals(Type.STRING, client.jdbcTypeToDoris(column("varbinary", Types.VARBINARY, 20, 0))); - Assertions.assertEquals(Type.STRING, client.jdbcTypeToDoris(column("timestamp", Types.BINARY, 8, 0))); - Assertions.assertEquals(Type.STRING, - client.jdbcTypeToDoris(column("datetimeoffset", SQL_SS_TIMESTAMPOFFSET, 34, 7))); - } -} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/util/JdbcFieldSchemaTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/util/JdbcFieldSchemaTest.java deleted file mode 100644 index bed0faf9f3dd67..00000000000000 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/util/JdbcFieldSchemaTest.java +++ /dev/null @@ -1,57 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.jdbc.util; - -import org.junit.jupiter.api.Assertions; -import org.junit.jupiter.api.Test; -import org.mockito.Mockito; - -import java.sql.ResultSetMetaData; -import java.sql.Types; - -public class JdbcFieldSchemaTest { - - private ResultSetMetaData metaData = Mockito.mock(ResultSetMetaData.class); - - @Test - public void testUseColumnLabelForQueryAlias() throws Exception { - Mockito.when(metaData.getColumnLabel(1)).thenReturn("t1"); - Mockito.when(metaData.getColumnType(1)).thenReturn(Types.VARCHAR); - Mockito.when(metaData.getColumnTypeName(1)).thenReturn("VARCHAR"); - Mockito.when(metaData.getPrecision(1)).thenReturn(64); - Mockito.when(metaData.getScale(1)).thenReturn(0); - - JdbcFieldSchema schema = new JdbcFieldSchema(metaData, 1); - - Assertions.assertEquals("t1", schema.getColumnName()); - } - - @Test - public void testFallbackToColumnNameWhenLabelMissing() throws Exception { - Mockito.when(metaData.getColumnLabel(1)).thenReturn(""); - Mockito.when(metaData.getColumnName(1)).thenReturn("username"); - Mockito.when(metaData.getColumnType(1)).thenReturn(Types.VARCHAR); - Mockito.when(metaData.getColumnTypeName(1)).thenReturn("VARCHAR"); - Mockito.when(metaData.getPrecision(1)).thenReturn(64); - Mockito.when(metaData.getScale(1)).thenReturn(0); - - JdbcFieldSchema schema = new JdbcFieldSchema(metaData, 1); - - Assertions.assertEquals("username", schema.getColumnName()); - } -} diff --git a/fe/fe-core/src/test/java/org/apache/doris/job/extensions/insert/streaming/DataSourceConfigValidatorTest.java b/fe/fe-core/src/test/java/org/apache/doris/job/extensions/insert/streaming/DataSourceConfigValidatorTest.java index 231c7210e7d727..edd10de8b9538e 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/job/extensions/insert/streaming/DataSourceConfigValidatorTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/job/extensions/insert/streaming/DataSourceConfigValidatorTest.java @@ -17,21 +17,22 @@ package org.apache.doris.job.extensions.insert.streaming; -import org.apache.doris.datasource.jdbc.client.JdbcClient; +import org.apache.doris.connector.spi.ConnectorQueryResult; import org.apache.doris.job.cdc.DataSourceConfigKeys; import org.apache.doris.job.common.DataSourceType; import org.apache.doris.job.exception.JobException; import org.apache.doris.job.util.StreamingJobUtils; +import org.apache.doris.job.util.StreamingSourceClient; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; import org.mockito.MockedStatic; import org.mockito.Mockito; -import java.sql.Connection; -import java.sql.ResultSet; -import java.sql.Statement; +import java.util.Arrays; +import java.util.Collections; import java.util.HashMap; +import java.util.List; import java.util.Map; public class DataSourceConfigValidatorTest { @@ -476,28 +477,28 @@ public void testCompatibilityModePreflightIsNoopForExistingSources() throws Exce @Test public void testOceanBaseMysqlCompatibilityModePasses() throws Exception { - JdbcClient jdbcClient = mockOceanBaseCompatibilityMode(true, "MYSQL"); + StreamingSourceClient sourceClient = mockOceanBaseCompatibilityMode(true, "MYSQL"); try (MockedStatic utils = Mockito.mockStatic(StreamingJobUtils.class)) { - utils.when(() -> StreamingJobUtils.getJdbcClient( + utils.when(() -> StreamingJobUtils.openSourceClient( Mockito.eq(DataSourceType.OCEANBASE), Mockito.anyMap())) - .thenReturn(jdbcClient); + .thenReturn(sourceClient); DataSourceConfigValidator.validateSourceBeforeTableCreation( DataSourceType.OCEANBASE, new HashMap<>()); } - Mockito.verify(jdbcClient).closeClient(); + Mockito.verify(sourceClient).close(); } @Test public void testOceanBaseOracleCompatibilityModeIsRejected() throws Exception { - JdbcClient jdbcClient = mockOceanBaseCompatibilityMode(true, "ORACLE"); + StreamingSourceClient sourceClient = mockOceanBaseCompatibilityMode(true, "ORACLE"); try (MockedStatic utils = Mockito.mockStatic(StreamingJobUtils.class)) { - utils.when(() -> StreamingJobUtils.getJdbcClient( + utils.when(() -> StreamingJobUtils.openSourceClient( Mockito.eq(DataSourceType.OCEANBASE), Mockito.anyMap())) - .thenReturn(jdbcClient); + .thenReturn(sourceClient); JobException exception = Assertions.assertThrows(JobException.class, () -> DataSourceConfigValidator.validateSourceBeforeTableCreation( @@ -505,17 +506,17 @@ public void testOceanBaseOracleCompatibilityModeIsRejected() throws Exception { Assertions.assertTrue(exception.getMessage().contains("Oracle compatibility mode")); } - Mockito.verify(jdbcClient).closeClient(); + Mockito.verify(sourceClient).close(); } @Test public void testOceanBaseUnknownCompatibilityModeIsRejected() throws Exception { - JdbcClient jdbcClient = mockOceanBaseCompatibilityMode(true, "UNKNOWN"); + StreamingSourceClient sourceClient = mockOceanBaseCompatibilityMode(true, "UNKNOWN"); try (MockedStatic utils = Mockito.mockStatic(StreamingJobUtils.class)) { - utils.when(() -> StreamingJobUtils.getJdbcClient( + utils.when(() -> StreamingJobUtils.openSourceClient( Mockito.eq(DataSourceType.OCEANBASE), Mockito.anyMap())) - .thenReturn(jdbcClient); + .thenReturn(sourceClient); JobException exception = Assertions.assertThrows(JobException.class, () -> DataSourceConfigValidator.validateSourceBeforeTableCreation( @@ -523,17 +524,17 @@ public void testOceanBaseUnknownCompatibilityModeIsRejected() throws Exception { Assertions.assertTrue(exception.getMessage().contains("UNKNOWN")); } - Mockito.verify(jdbcClient).closeClient(); + Mockito.verify(sourceClient).close(); } @Test public void testOceanBaseEmptyCompatibilityModeResultIsRejected() throws Exception { - JdbcClient jdbcClient = mockOceanBaseCompatibilityMode(false, null); + StreamingSourceClient sourceClient = mockOceanBaseCompatibilityMode(false, null); try (MockedStatic utils = Mockito.mockStatic(StreamingJobUtils.class)) { - utils.when(() -> StreamingJobUtils.getJdbcClient( + utils.when(() -> StreamingJobUtils.openSourceClient( Mockito.eq(DataSourceType.OCEANBASE), Mockito.anyMap())) - .thenReturn(jdbcClient); + .thenReturn(sourceClient); JobException exception = Assertions.assertThrows(JobException.class, () -> DataSourceConfigValidator.validateSourceBeforeTableCreation( @@ -541,20 +542,19 @@ public void testOceanBaseEmptyCompatibilityModeResultIsRejected() throws Excepti Assertions.assertTrue(exception.getMessage().contains("Failed to determine")); } - Mockito.verify(jdbcClient).closeClient(); + Mockito.verify(sourceClient).close(); } @Test public void testOceanBaseCompatibilityModeQueryFailurePreservesCause() throws Exception { - JdbcClient jdbcClient = Mockito.mock(JdbcClient.class); - Connection connection = Mockito.mock(Connection.class); - Mockito.when(jdbcClient.getConnection()).thenReturn(connection); - Mockito.when(connection.createStatement()).thenThrow(new IllegalStateException("query failed")); + StreamingSourceClient sourceClient = Mockito.mock(StreamingSourceClient.class); + Mockito.when(sourceClient.executeQuery(Mockito.anyString(), Mockito.anyList())) + .thenThrow(new IllegalStateException("query failed")); try (MockedStatic utils = Mockito.mockStatic(StreamingJobUtils.class)) { - utils.when(() -> StreamingJobUtils.getJdbcClient( + utils.when(() -> StreamingJobUtils.openSourceClient( Mockito.eq(DataSourceType.OCEANBASE), Mockito.anyMap())) - .thenReturn(jdbcClient); + .thenReturn(sourceClient); JobException exception = Assertions.assertThrows(JobException.class, () -> DataSourceConfigValidator.validateSourceBeforeTableCreation( @@ -563,21 +563,18 @@ public void testOceanBaseCompatibilityModeQueryFailurePreservesCause() throws Ex Assertions.assertNotNull(exception.getCause()); } - Mockito.verify(jdbcClient).closeClient(); + Mockito.verify(sourceClient).close(); } - private JdbcClient mockOceanBaseCompatibilityMode(boolean hasResult, String mode) + private StreamingSourceClient mockOceanBaseCompatibilityMode(boolean hasResult, String mode) throws Exception { - JdbcClient jdbcClient = Mockito.mock(JdbcClient.class); - Connection connection = Mockito.mock(Connection.class); - Statement statement = Mockito.mock(Statement.class); - ResultSet resultSet = Mockito.mock(ResultSet.class); - Mockito.when(jdbcClient.getConnection()).thenReturn(connection); - Mockito.when(connection.createStatement()).thenReturn(statement); - Mockito.when(statement.executeQuery("SHOW VARIABLES LIKE 'ob_compatibility_mode'")) - .thenReturn(resultSet); - Mockito.when(resultSet.next()).thenReturn(hasResult); - Mockito.when(resultSet.getString(2)).thenReturn(mode); - return jdbcClient; + StreamingSourceClient sourceClient = Mockito.mock(StreamingSourceClient.class); + List> rows = hasResult + ? Collections.singletonList(Arrays.asList("ob_compatibility_mode", mode)) + : Collections.emptyList(); + Mockito.when(sourceClient.executeQuery(Mockito.eq("SHOW VARIABLES LIKE 'ob_compatibility_mode'"), + Mockito.anyList())) + .thenReturn(new ConnectorQueryResult(Arrays.asList("Variable_name", "Value"), rows)); + return sourceClient; } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/job/extensions/insert/streaming/PostgresResourceValidatorTest.java b/fe/fe-core/src/test/java/org/apache/doris/job/extensions/insert/streaming/PostgresResourceValidatorTest.java index 0dc550530f67f9..519c26bbec60fd 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/job/extensions/insert/streaming/PostgresResourceValidatorTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/job/extensions/insert/streaming/PostgresResourceValidatorTest.java @@ -17,15 +17,25 @@ package org.apache.doris.job.extensions.insert.streaming; +import org.apache.doris.connector.spi.ConnectorQueryResult; import org.apache.doris.job.cdc.DataSourceConfigKeys; +import org.apache.doris.job.common.DataSourceType; import org.apache.doris.job.exception.JobException; +import org.apache.doris.job.util.StreamingJobUtils; +import org.apache.doris.job.util.StreamingSourceClient; import org.apache.commons.lang3.StringUtils; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import org.mockito.ArgumentCaptor; +import org.mockito.MockedStatic; +import org.mockito.Mockito; +import java.util.ArrayList; +import java.util.Arrays; import java.util.Collections; import java.util.HashMap; +import java.util.List; import java.util.Map; public class PostgresResourceValidatorTest { @@ -42,4 +52,121 @@ public void testRejectMultibyteOverLongDatabaseName() { () -> PostgresResourceValidator.validate(props, "1", Collections.emptyList())); Assertions.assertTrue(e.getMessage().contains("bytes"), e.getMessage()); } + + private static ConnectorQueryResult rows(List> rows) { + return new ConnectorQueryResult(Collections.singletonList("c"), rows); + } + + /** + * A source whose publication {@code pubName} covers {@code coveredTables} and whose slot {@code slotName} + * reports {@code slotActive} (null = no such slot); every other name is unknown. + */ + private static StreamingSourceClient source(String pubName, List coveredTables, String slotName, + Boolean slotActive) throws Exception { + StreamingSourceClient client = Mockito.mock(StreamingSourceClient.class); + Mockito.when(client.executeQuery(Mockito.contains("FROM pg_publication WHERE"), Mockito.anyList())) + .thenAnswer(inv -> rows(pubName.equals(param(inv.getArgument(1))) + ? Collections.singletonList(Collections.singletonList(1)) : Collections.emptyList())); + Mockito.when(client.executeQuery(Mockito.contains("FROM pg_publication_tables"), Mockito.anyList())) + .thenAnswer(inv -> { + List> covered = new ArrayList<>(); + if (pubName.equals(param(inv.getArgument(1)))) { + for (String t : coveredTables) { + covered.add(Arrays.asList(t.split("\\.")[0], t.split("\\.")[1])); + } + } + return rows(covered); + }); + Mockito.when(client.executeQuery(Mockito.contains("FROM pg_replication_slots"), Mockito.anyList())) + .thenAnswer(inv -> rows(slotName.equals(param(inv.getArgument(1))) && slotActive != null + ? Collections.singletonList(Collections.singletonList(slotActive)) + : Collections.emptyList())); + return client; + } + + private static Object param(List params) { + return params.isEmpty() ? null : params.get(0); + } + + private static Map pgProps() { + Map props = new HashMap<>(); + props.put(DataSourceConfigKeys.DATABASE, "db"); + props.put(DataSourceConfigKeys.SCHEMA, "public"); + return props; + } + + @Test + public void defaultNamesPassWhenNothingConflicts() throws Exception { + StreamingSourceClient client = source("none", Collections.emptyList(), "none", null); + try (MockedStatic utils = Mockito.mockStatic(StreamingJobUtils.class)) { + serving(utils, client); + PostgresResourceValidator.validate(pgProps(), "7", Collections.singletonList("t")); + } + // The names reach the source as bound parameters, never spliced into the SQL text. + ArgumentCaptor sql = ArgumentCaptor.forClass(String.class); + Mockito.verify(client, Mockito.atLeastOnce()).executeQuery(sql.capture(), Mockito.anyList()); + for (String statement : sql.getAllValues()) { + Assertions.assertTrue(statement.contains("= ?"), statement); + Assertions.assertFalse(statement.contains("doris_"), statement); + } + Mockito.verify(client).close(); + } + + private static void serving(MockedStatic utils, StreamingSourceClient client) { + utils.when(() -> StreamingJobUtils.openSourceClient(Mockito.eq(DataSourceType.POSTGRES), + Mockito.anyMap())).thenReturn(client); + } + + @Test + public void userProvidedPublicationMustExistAndCoverTheTables() throws Exception { + Map props = pgProps(); + props.put(DataSourceConfigKeys.PUBLICATION_NAME, "my_pub"); + StreamingSourceClient withoutPub = source("other_pub", Collections.emptyList(), "none", null); + StreamingSourceClient partialPub = source("my_pub", Collections.singletonList("public.a"), "none", null); + try (MockedStatic utils = Mockito.mockStatic(StreamingJobUtils.class)) { + serving(utils, withoutPub); + JobException missing = Assertions.assertThrows(JobException.class, + () -> PostgresResourceValidator.validate(props, "7", Collections.singletonList("t"))); + Assertions.assertTrue(missing.getMessage().contains("publication does not exist: my_pub"), + missing.getMessage()); + + serving(utils, partialPub); + JobException uncovered = Assertions.assertThrows(JobException.class, + () -> PostgresResourceValidator.validate(props, "7", Arrays.asList("a", "t"))); + Assertions.assertTrue(uncovered.getMessage().contains("missing required tables: [public.t]"), + uncovered.getMessage()); + } + } + + @Test + public void activeDorisOwnedSlotIsAConflict() throws Exception { + String defaultSlot = DataSourceConfigKeys.defaultSlotName("7"); + // The driver reports the slot's active flag as a Boolean, not a String. + StreamingSourceClient activeSlot = source("none", Collections.emptyList(), defaultSlot, Boolean.TRUE); + StreamingSourceClient idleSlot = source("none", Collections.emptyList(), defaultSlot, Boolean.FALSE); + try (MockedStatic utils = Mockito.mockStatic(StreamingJobUtils.class)) { + serving(utils, activeSlot); + JobException e = Assertions.assertThrows(JobException.class, + () -> PostgresResourceValidator.validate(pgProps(), "7", Collections.singletonList("t"))); + Assertions.assertTrue(e.getMessage().contains("is active, held by another consumer"), e.getMessage()); + + // An inactive Doris-owned slot from an earlier run is simply reused. + serving(utils, idleSlot); + PostgresResourceValidator.validate(pgProps(), "7", Collections.singletonList("t")); + } + } + + @Test + public void userProvidedSlotMustExist() throws Exception { + Map props = pgProps(); + props.put(DataSourceConfigKeys.SLOT_NAME, "my_slot"); + StreamingSourceClient otherSlot = source("none", Collections.emptyList(), "other", null); + try (MockedStatic utils = Mockito.mockStatic(StreamingJobUtils.class)) { + serving(utils, otherSlot); + JobException e = Assertions.assertThrows(JobException.class, + () -> PostgresResourceValidator.validate(props, "7", Collections.singletonList("t"))); + Assertions.assertTrue(e.getMessage().contains("replication slot does not exist: my_slot"), + e.getMessage()); + } + } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/job/util/StreamingJobUtilsTest.java b/fe/fe-core/src/test/java/org/apache/doris/job/util/StreamingJobUtilsTest.java index dcc96e6686e916..7b87ec53dcc871 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/job/util/StreamingJobUtilsTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/job/util/StreamingJobUtilsTest.java @@ -25,7 +25,6 @@ import org.apache.doris.catalog.PrimitiveType; import org.apache.doris.catalog.ScalarType; import org.apache.doris.datasource.InternalCatalog; -import org.apache.doris.datasource.jdbc.client.JdbcClient; import org.apache.doris.job.cdc.DataSourceConfigKeys; import org.apache.doris.job.common.DataSourceType; import org.apache.doris.job.exception.JobException; @@ -49,7 +48,7 @@ public class StreamingJobUtilsTest { @Mock - private JdbcClient jdbcClient; + private StreamingSourceClient sourceClient; @BeforeEach public void setUp() { @@ -71,8 +70,8 @@ public void testGetColumnsWithPrimaryKeySorting() throws Exception { mockColumns.add(new Column("name", ScalarType.createVarcharType(50))); mockColumns.add(new Column("address", ScalarType.createVarcharType(200))); - Mockito.when(jdbcClient.getColumnsFromJdbc(ArgumentMatchers.anyString(), ArgumentMatchers.anyString())).thenReturn(mockColumns); - List result = StreamingJobUtils.getColumns(jdbcClient, database, table, primaryKeys); + Mockito.when(sourceClient.getColumns(ArgumentMatchers.anyString(), ArgumentMatchers.anyString())).thenReturn(mockColumns); + List result = StreamingJobUtils.getColumns(sourceClient, database, table, primaryKeys); // Verify primary keys are at the front in correct order Assertions.assertEquals(5, result.size()); @@ -103,8 +102,8 @@ public void testGetColumnsWithVarcharTypeConversion() throws Exception { mockColumns.add(new Column("short_name", ScalarType.createVarcharType(50))); mockColumns.add(new Column("long_name", ScalarType.createVarcharType(20000))); - Mockito.when(jdbcClient.getColumnsFromJdbc(ArgumentMatchers.anyString(), ArgumentMatchers.anyString())).thenReturn(mockColumns); - List result = StreamingJobUtils.getColumns(jdbcClient, database, table, primaryKeys); + Mockito.when(sourceClient.getColumns(ArgumentMatchers.anyString(), ArgumentMatchers.anyString())).thenReturn(mockColumns); + List result = StreamingJobUtils.getColumns(sourceClient, database, table, primaryKeys); // Verify varchar length multiplication by 3 Column shortName = result.stream() @@ -133,8 +132,8 @@ public void testGetColumnsWithStringTypeAsPrimaryKey() throws Exception { mockColumns.add(new Column("id", ScalarType.createStringType())); mockColumns.add(new Column("name", ScalarType.createVarcharType(50))); - Mockito.when(jdbcClient.getColumnsFromJdbc(ArgumentMatchers.anyString(), ArgumentMatchers.anyString())).thenReturn(mockColumns); - List result = StreamingJobUtils.getColumns(jdbcClient, database, table, primaryKeys); + Mockito.when(sourceClient.getColumns(ArgumentMatchers.anyString(), ArgumentMatchers.anyString())).thenReturn(mockColumns); + List result = StreamingJobUtils.getColumns(sourceClient, database, table, primaryKeys); // Verify string type primary key is converted to varchar Column idColumn = result.stream() @@ -157,8 +156,8 @@ public void testGetColumnsWithEmptyPrimaryKeys() throws Exception { mockColumns.add(new Column("col2", ScalarType.createVarcharType(100))); mockColumns.add(new Column("col3", ScalarType.createType(PrimitiveType.BIGINT))); - Mockito.when(jdbcClient.getColumnsFromJdbc(ArgumentMatchers.anyString(), ArgumentMatchers.anyString())).thenReturn(mockColumns); - List result = StreamingJobUtils.getColumns(jdbcClient, database, table, primaryKeys); + Mockito.when(sourceClient.getColumns(ArgumentMatchers.anyString(), ArgumentMatchers.anyString())).thenReturn(mockColumns); + List result = StreamingJobUtils.getColumns(sourceClient, database, table, primaryKeys); // Verify columns maintain original order when no primary keys Assertions.assertEquals(3, result.size()); @@ -181,8 +180,8 @@ public void testGetColumnsWithMultiplePrimaryKeys() throws Exception { mockColumns.add(new Column("pk3", ScalarType.createType(PrimitiveType.INT))); mockColumns.add(new Column("data3", ScalarType.createVarcharType(50))); - Mockito.when(jdbcClient.getColumnsFromJdbc(ArgumentMatchers.anyString(), ArgumentMatchers.anyString())).thenReturn(mockColumns); - List result = StreamingJobUtils.getColumns(jdbcClient, database, table, primaryKeys); + Mockito.when(sourceClient.getColumns(ArgumentMatchers.anyString(), ArgumentMatchers.anyString())).thenReturn(mockColumns); + List result = StreamingJobUtils.getColumns(sourceClient, database, table, primaryKeys); // Verify primary keys are sorted in the order defined in primaryKeys list Assertions.assertEquals(6, result.size()); @@ -205,10 +204,10 @@ public void testGetColumnsWithUnsupportedColumnType() throws Exception { mockColumns.add(new Column("id", ScalarType.createType(PrimitiveType.INT))); mockColumns.add(new Column("unsupported_col", new ScalarType(PrimitiveType.UNSUPPORTED))); - Mockito.when(jdbcClient.getColumnsFromJdbc(ArgumentMatchers.anyString(), ArgumentMatchers.anyString())).thenReturn(mockColumns); + Mockito.when(sourceClient.getColumns(ArgumentMatchers.anyString(), ArgumentMatchers.anyString())).thenReturn(mockColumns); // This should throw IllegalArgumentException due to unsupported column type try { - StreamingJobUtils.getColumns(jdbcClient, database, table, primaryKeys); + StreamingJobUtils.getColumns(sourceClient, database, table, primaryKeys); Assertions.fail("Expected IllegalArgumentException to be thrown"); } catch (IllegalArgumentException e) { // Verify the exception message contains expected information @@ -230,8 +229,8 @@ public void testGetColumnsWithVarcharPrimaryKeyLengthMultiplication() throws Exc mockColumns.add(new Column("pk_varchar", ScalarType.createVarcharType(100))); mockColumns.add(new Column("normal_varchar", ScalarType.createVarcharType(50))); - Mockito.when(jdbcClient.getColumnsFromJdbc(ArgumentMatchers.anyString(), ArgumentMatchers.anyString())).thenReturn(mockColumns); - List result = StreamingJobUtils.getColumns(jdbcClient, database, table, primaryKeys); + Mockito.when(sourceClient.getColumns(ArgumentMatchers.anyString(), ArgumentMatchers.anyString())).thenReturn(mockColumns); + List result = StreamingJobUtils.getColumns(sourceClient, database, table, primaryKeys); // Verify varchar primary key column has length multiplied by 3 Column pkVarcharColumn = result.stream() @@ -260,20 +259,20 @@ public void testGetOceanBaseRemoteDbName() { } @Test - public void testGenerateCreateTableCmdsClosesJdbcClientOnFailure() { + public void testGenerateCreateTableCmdsClosesSourceClientOnFailure() throws Exception { Map properties = new HashMap<>(); try (MockedStatic utils = Mockito.mockStatic(StreamingJobUtils.class, Mockito.CALLS_REAL_METHODS)) { - utils.when(() -> StreamingJobUtils.getJdbcClient(DataSourceType.OCEANBASE, properties)) - .thenReturn(jdbcClient); + utils.when(() -> StreamingJobUtils.openSourceClient(DataSourceType.OCEANBASE, properties)) + .thenReturn(sourceClient); utils.when(() -> StreamingJobUtils.getRemoteDbName(DataSourceType.OCEANBASE, properties)) .thenReturn("test_db"); - Mockito.when(jdbcClient.getTablesNameList("test_db")).thenReturn(new ArrayList<>()); + Mockito.when(sourceClient.listTables("test_db")).thenReturn(new ArrayList<>()); Assertions.assertThrows(JobException.class, () -> StreamingJobUtils.generateCreateTableCmds( "target_db", DataSourceType.OCEANBASE, properties, new HashMap<>())); - Mockito.verify(jdbcClient).closeClient(); + Mockito.verify(sourceClient).close(); } } @@ -291,11 +290,11 @@ public void testGenerateCreateTableCmdsFindsMixedCasePrecreatedTargetWhenStoredL InternalCatalog internalCatalog = Mockito.mock(InternalCatalog.class); Mockito.when(env.getInternalCatalog()).thenReturn(internalCatalog); Mockito.when(internalCatalog.getDbNullable("target_db")).thenReturn(targetDatabase); - Mockito.when(jdbcClient.getTablesNameList("source_db")) + Mockito.when(sourceClient.listTables("source_db")) .thenReturn(Arrays.asList("source_table")); - Mockito.when(jdbcClient.getPrimaryKeys("source_db", "source_table")) + Mockito.when(sourceClient.getPrimaryKeys("source_db", "source_table")) .thenReturn(Arrays.asList("id")); - Mockito.when(jdbcClient.getColumnsFromJdbc("source_db", "source_table")) + Mockito.when(sourceClient.getColumns("source_db", "source_table")) .thenReturn(Arrays.asList( new Column("id", ScalarType.createType(PrimitiveType.INT)), new Column("unsupported_col", new ScalarType(PrimitiveType.UNSUPPORTED)))); @@ -306,8 +305,8 @@ public void testGenerateCreateTableCmdsFindsMixedCasePrecreatedTargetWhenStoredL MockedStatic utils = Mockito.mockStatic(StreamingJobUtils.class, Mockito.CALLS_REAL_METHODS)) { mockedEnv.when(Env::getCurrentEnv).thenReturn(env); - utils.when(() -> StreamingJobUtils.getJdbcClient(DataSourceType.POSTGRES, properties)) - .thenReturn(jdbcClient); + utils.when(() -> StreamingJobUtils.openSourceClient(DataSourceType.POSTGRES, properties)) + .thenReturn(sourceClient); Assertions.assertFalse(StreamingJobUtils.generateCreateTableCmds( "target_db", DataSourceType.POSTGRES, properties, new HashMap<>()) diff --git a/fe/fe-core/src/test/java/org/apache/doris/job/util/StreamingSourceClientTest.java b/fe/fe-core/src/test/java/org/apache/doris/job/util/StreamingSourceClientTest.java new file mode 100644 index 00000000000000..bf2b4f411a101a --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/job/util/StreamingSourceClientTest.java @@ -0,0 +1,235 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + + +package org.apache.doris.job.util; + +import org.apache.doris.catalog.Column; +import org.apache.doris.connector.ConnectorFactory; +import org.apache.doris.connector.ConnectorPluginManager; +import org.apache.doris.connector.spi.Connector; +import org.apache.doris.connector.spi.ConnectorColumn; +import org.apache.doris.connector.spi.ConnectorContext; +import org.apache.doris.connector.spi.ConnectorMetadata; +import org.apache.doris.connector.spi.ConnectorPassthroughSqlOps; +import org.apache.doris.connector.spi.ConnectorProvider; +import org.apache.doris.connector.spi.ConnectorQueryResult; +import org.apache.doris.connector.spi.ConnectorSession; +import org.apache.doris.connector.spi.ConnectorTableSchema; +import org.apache.doris.connector.spi.ConnectorType; +import org.apache.doris.connector.spi.handle.ConnectorTableHandle; +import org.apache.doris.job.common.DataSourceType; +import org.apache.doris.job.exception.JobException; + +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * {@link StreamingSourceClient} against a recording fake of the connector plugin that serves streaming + * sources: it must reach that plugin through the engine's connector factory with the job's own source + * properties, hand back the connector's schema as Doris columns, and fail loud when the plugin is absent + * or cannot run probe queries. + */ +public class StreamingSourceClientTest { + + /** A table handle the fake connector hands out; the client must treat it as opaque. */ + private static final class Handle implements ConnectorTableHandle { + final String db; + final String table; + + Handle(String db, String table) { + this.db = db; + this.table = table; + } + } + + /** Records what the engine asked of it and answers a fixed one-table database. */ + private static final class RecordingMetadata implements ConnectorMetadata, ConnectorPassthroughSqlOps { + final List calls = new ArrayList<>(); + final List lastParams = new ArrayList<>(); + + @Override + public List listTableNames(ConnectorSession session, String dbName) { + calls.add("listTableNames:" + dbName); + return Collections.singletonList("t"); + } + + @Override + public Optional getTableHandle(ConnectorSession session, String dbName, + String tableName) { + calls.add("getTableHandle:" + dbName + "." + tableName); + return "t".equals(tableName) ? Optional.of(new Handle(dbName, tableName)) : Optional.empty(); + } + + @Override + public ConnectorTableSchema getTableSchema(ConnectorSession session, ConnectorTableHandle handle) { + calls.add("getTableSchema:" + ((Handle) handle).db + "." + ((Handle) handle).table); + return new ConnectorTableSchema("t", Arrays.asList( + new ConnectorColumn("id", ConnectorType.of("BIGINT"), "", false, null, true), + new ConnectorColumn("name", ConnectorType.of("VARCHAR", 20, -1), "", true, null, true)), + "FAKE", Collections.emptyMap()); + } + + @Override + public List getPrimaryKeys(ConnectorSession session, ConnectorTableHandle handle) { + calls.add("getPrimaryKeys:" + ((Handle) handle).db + "." + ((Handle) handle).table); + return Collections.singletonList("id"); + } + + @Override + public ConnectorQueryResult executeQuery(ConnectorSession session, String sql, List params) { + calls.add("executeQuery:" + sql); + lastParams.clear(); + lastParams.addAll(params); + return new ConnectorQueryResult(Collections.singletonList("v"), + Collections.singletonList(Collections.singletonList(Boolean.TRUE))); + } + } + + private static class RecordingConnector implements Connector { + final ConnectorMetadata metadata; + int closes; + + RecordingConnector(ConnectorMetadata metadata) { + this.metadata = metadata; + } + + @Override + public ConnectorMetadata getMetadata(ConnectorSession session) { + return metadata; + } + + @Override + public void close() { + closes++; + } + } + + private RecordingMetadata metadata; + private RecordingConnector connector; + private Map receivedProps; + private String receivedCatalogName; + + @BeforeEach + void setUp() { + metadata = new RecordingMetadata(); + connector = new RecordingConnector(metadata); + ConnectorPluginManager manager = new ConnectorPluginManager(); + manager.registerProvider(new ConnectorProvider() { + @Override + public String getType() { + return DataSourceType.MYSQL.connectorType(); + } + + @Override + public Connector create(Map properties, ConnectorContext context) { + receivedProps = properties; + receivedCatalogName = context.getCatalogName(); + return connector; + } + }); + ConnectorFactory.initPluginManager(manager); + } + + @AfterEach + void tearDown() { + ConnectorFactory.initPluginManager(new ConnectorPluginManager()); + } + + private static Map sourceProps() { + Map props = new HashMap<>(); + props.put("jdbc_url", "jdbc:mysql://h:3306"); + props.put("user", "u"); + props.put("password", "p"); + props.put("database", "db"); + props.put("include_tables", "t"); + return props; + } + + @Test + public void opensTheSourcePluginWithTheJobsPropertiesAndClosesIt() throws Exception { + try (StreamingSourceClient client = StreamingSourceClient.open(DataSourceType.MYSQL, sourceProps())) { + Assertions.assertEquals(sourceProps(), receivedProps, + "the job's source properties are handed to the plugin as they are"); + Assertions.assertEquals("MYSQL", receivedCatalogName, + "the source type names the temporary connector, as the old client did"); + Assertions.assertEquals(Collections.singletonList("t"), client.listTables("db")); + Assertions.assertTrue(client.tableExists("db", "t")); + Assertions.assertFalse(client.tableExists("db", "nope")); + } + Assertions.assertEquals(1, connector.closes, "close() releases the temporary connector"); + } + + @Test + public void columnsAndPrimaryKeysComeFromTheConnectorSchema() throws Exception { + try (StreamingSourceClient client = StreamingSourceClient.open(DataSourceType.MYSQL, sourceProps())) { + List columns = client.getColumns("db", "t"); + Assertions.assertEquals(2, columns.size()); + Assertions.assertEquals("id", columns.get(0).getName()); + Assertions.assertTrue(columns.get(0).getType().isBigIntType()); + Assertions.assertFalse(columns.get(0).isAllowNull()); + Assertions.assertEquals("name", columns.get(1).getName()); + Assertions.assertEquals(20, columns.get(1).getType().getLength()); + Assertions.assertEquals(Collections.singletonList("id"), client.getPrimaryKeys("db", "t")); + + // The handle is looked up once per table and reused by the schema and key reads. + Assertions.assertEquals(1, Collections.frequency(metadata.calls, "getTableHandle:db.t")); + + JobException e = Assertions.assertThrows(JobException.class, () -> client.getColumns("db", "nope")); + Assertions.assertTrue(e.getMessage().contains("db.nope does not exist"), e.getMessage()); + } + } + + @Test + public void probeQueriesBindTheirParameters() throws Exception { + try (StreamingSourceClient client = StreamingSourceClient.open(DataSourceType.MYSQL, sourceProps())) { + ConnectorQueryResult result = client.executeQuery("SELECT active FROM s WHERE n = ?", + Collections.singletonList("slot")); + Assertions.assertEquals(Boolean.TRUE, result.getRows().get(0).get(0)); + Assertions.assertEquals(Collections.singletonList("slot"), metadata.lastParams); + } + } + + @Test + public void missingPluginFailsLoud() { + ConnectorFactory.initPluginManager(new ConnectorPluginManager()); + JobException e = Assertions.assertThrows(JobException.class, + () -> StreamingSourceClient.open(DataSourceType.POSTGRES, sourceProps())); + Assertions.assertTrue(e.getMessage().contains("requires the 'jdbc' connector plugin"), e.getMessage()); + } + + @Test + public void sourceWithoutPassthroughSqlCannotBeProbed() throws Exception { + connector = new RecordingConnector(new ConnectorMetadata() { + }); + try (StreamingSourceClient client = StreamingSourceClient.open(DataSourceType.MYSQL, sourceProps())) { + JobException e = Assertions.assertThrows(JobException.class, + () -> client.executeQuery("SELECT 1", Collections.emptyList())); + Assertions.assertTrue(e.getMessage().contains("does not support probe queries"), e.getMessage()); + } + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/CdcStreamTableValuedFunctionTest.java b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/CdcStreamTableValuedFunctionTest.java index 17f683fbd7b313..747e79e93afd79 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/CdcStreamTableValuedFunctionTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/CdcStreamTableValuedFunctionTest.java @@ -20,11 +20,11 @@ import org.apache.doris.catalog.Column; import org.apache.doris.catalog.PrimitiveType; import org.apache.doris.common.AnalysisException; -import org.apache.doris.datasource.jdbc.client.JdbcClient; import org.apache.doris.job.cdc.DataSourceConfigKeys; import org.apache.doris.job.cdc.request.FetchRecordRequest; import org.apache.doris.job.common.DataSourceType; import org.apache.doris.job.util.StreamingJobUtils; +import org.apache.doris.job.util.StreamingSourceClient; import com.fasterxml.jackson.databind.ObjectMapper; import org.junit.jupiter.api.Assertions; @@ -86,16 +86,16 @@ public void testMysqlJdbcUrlIsNormalizedInPayload() throws Exception { } private List getTableColumns(Map properties) throws Exception { - JdbcClient jdbcClient = Mockito.mock(JdbcClient.class); + StreamingSourceClient sourceClient = Mockito.mock(StreamingSourceClient.class); List sourceColumns = new ArrayList<>(); sourceColumns.add(new Column("id", PrimitiveType.INT)); - Mockito.when(jdbcClient.isTableExist("test_db", "test_table")).thenReturn(true); - Mockito.when(jdbcClient.getColumnsFromJdbc("test_db", "test_table")).thenReturn(sourceColumns); + Mockito.when(sourceClient.tableExists("test_db", "test_table")).thenReturn(true); + Mockito.when(sourceClient.getColumns("test_db", "test_table")).thenReturn(sourceColumns); try (MockedStatic utils = Mockito.mockStatic(StreamingJobUtils.class)) { - utils.when(() -> StreamingJobUtils.getJdbcClient( + utils.when(() -> StreamingJobUtils.openSourceClient( Mockito.eq(DataSourceType.MYSQL), Mockito.anyMap())) - .thenReturn(jdbcClient); + .thenReturn(sourceClient); utils.when(() -> StreamingJobUtils.getRemoteDbName( Mockito.eq(DataSourceType.MYSQL), Mockito.anyMap())) .thenReturn("test_db"); From d06845ab00d8d5e4d33b64195db03646d192f850 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 16 Sep 2026 02:51:24 +0800 Subject: [PATCH 3/4] [refactor](fe) Turn JdbcResource into a deprecated persistence shell and drop dead JDBC plan/rule types MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: Part 3: `JdbcResource` carried the last JDBC logic in fe-core — url normalization, dialect constants, driver-jar resolution, white list, secure path, checksum and cloud download — although a resource no longer backs anything (`CREATE CATALOG ... WITH RESOURCE` is disallowed by default and reads nothing from a JDBC resource). It is now a `@Deprecated` shell: the Gson tag, the persisted `configs` field, the property list and defaults and the `SHOW RESOURCES` rows are unchanged, `CREATE RESOURCE type=jdbc` keeps working, and property validation plus the driver checksum are done by the jdbc connector plugin exactly as for a JDBC catalog. The stored `jdbc_url` is no longer rewritten. Also removed: the empty `JdbcTransactionManager`, the unreferenced `LOGICAL/PHYSICAL_JDBC_*` `PlanType` values and `*_JDBC_*` `RuleType` values, and the `"jdbc"` source-name gate around the schema-load debug point in `PluginDrivenExternalTable` (the debug point now applies to any plugin table). ### Release note `CREATE RESOURCE ... type=jdbc` is deprecated; use `CREATE CATALOG ... "type"="jdbc"`. It keeps working, validated by the jdbc connector plugin, and `SHOW RESOURCES` shows the `jdbc_url` as written instead of a normalized form. ### Check List (For Author) - Test: Unit Test - JdbcResourceTest (rewritten for the shell: defaults, replay, row count, connector validation, missing plugin, password masking), GrantResourcePrivilegeCommandTest, RevokeResourcePrivilegeCommandTest, CreateResourceCommandTest, PluginDrivenInsertExecutorTest. - Behavior changed: Yes (see release note) - Does this need documentation: Yes (deprecate CREATE RESOURCE type=jdbc) Co-Authored-By: Claude Opus 5 --- .../apache/doris/catalog/JdbcResource.java | 499 +++--------------- .../plugin/PluginDrivenExternalTable.java | 7 +- .../apache/doris/nereids/rules/RuleType.java | 4 - .../doris/nereids/trees/plans/PlanType.java | 5 - .../transaction/JdbcTransactionManager.java | 42 -- .../doris/catalog/JdbcResourceTest.java | 347 ++++-------- 6 files changed, 167 insertions(+), 737 deletions(-) delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/transaction/JdbcTransactionManager.java diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java index 0141d33d276d8c..31f827b7a91bc5 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java @@ -17,17 +17,15 @@ package org.apache.doris.catalog; - import org.apache.doris.common.AnalysisException; -import org.apache.doris.common.Config; import org.apache.doris.common.DdlException; -import org.apache.doris.common.EnvUtils; import org.apache.doris.common.FeConstants; -import org.apache.doris.common.plugin.CloudPluginDownloader; -import org.apache.doris.common.plugin.CloudPluginDownloader.PluginType; import org.apache.doris.common.proc.BaseProcResult; import org.apache.doris.common.util.TimeUtils; -import org.apache.doris.common.util.Util; +import org.apache.doris.connector.ConnectorFactory; +import org.apache.doris.connector.DefaultConnectorContext; +import org.apache.doris.connector.DefaultConnectorValidationContext; +import org.apache.doris.connector.spi.Connector; import org.apache.doris.datasource.CatalogProperty; import org.apache.doris.datasource.ExternalCatalog; @@ -37,70 +35,33 @@ import com.google.common.collect.Lists; import com.google.common.collect.Maps; import com.google.gson.annotations.SerializedName; -import org.apache.commons.codec.binary.Hex; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; -import java.io.File; import java.io.IOException; -import java.io.InputStream; -import java.net.URI; -import java.net.URISyntaxException; -import java.nio.file.Path; -import java.nio.file.Paths; -import java.security.MessageDigest; -import java.security.NoSuchAlgorithmException; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.List; +import java.util.Locale; import java.util.Map; -import java.util.Objects; /** - * External JDBC Catalog resource for external table query. - *

    - * create external resource jdbc_mysql - * properties ( - * "type"="jdbc", - * "user"="root", - * "password"="123456", - * "jdbc_url"="jdbc:mysql://127.0.0.1:3306/test", - * "driver_url"="http://127.0.0.1:8888/mysql-connector-java-5.1.47.jar", - * "driver_class"="com.mysql.jdbc.Driver" - * ); - *

    - * DROP RESOURCE "jdbc_mysql"; + * The legacy JDBC resource: {@code CREATE RESOURCE ... PROPERTIES ("type"="jdbc", ...)}. + * + *

    A resource no longer backs anything: {@code CREATE CATALOG ... WITH RESOURCE} is disallowed by default + * and, where allowed, reads nothing from a JDBC resource but its type. What is left is a named, grantable + * property bag that {@code SHOW RESOURCES} lists. The class stays for two reasons: metadata images and edit + * logs that hold one must keep replaying (the Gson tag and the persisted {@code configs} field are the + * contract), and {@code CREATE RESOURCE type=jdbc} keeps working for deployments that still script it. + * New work should use {@code CREATE CATALOG ... "type"="jdbc"}.

    + * + *

    Nothing here knows JDBC. Property validation and the driver-jar checksum are done by the jdbc connector + * plugin, exactly as for a JDBC catalog; the property list and defaults below are only this object's own + * persisted schema, kept so that a resource shows the same rows it always showed.

    + * + * @deprecated Use JDBC Catalog instead. */ +@Deprecated public class JdbcResource extends Resource { private static final Logger LOG = LogManager.getLogger(JdbcResource.class); - public static final String JDBC_MYSQL = "jdbc:mysql"; - public static final String JDBC_MARIADB = "jdbc:mariadb"; - public static final String JDBC_POSTGRESQL = "jdbc:postgresql"; - public static final String JDBC_ORACLE = "jdbc:oracle"; - public static final String JDBC_SQLSERVER = "jdbc:sqlserver"; - public static final String JDBC_CLICKHOUSE = "jdbc:clickhouse"; - public static final String JDBC_SAP_HANA = "jdbc:sap"; - public static final String JDBC_TRINO = "jdbc:trino"; - public static final String JDBC_PRESTO = "jdbc:presto"; - public static final String JDBC_OCEANBASE = "jdbc:oceanbase"; - public static final String JDBC_DB2 = "jdbc:db2"; - public static final String JDBC_GBASE = "jdbc:gbase"; - - public static final String MYSQL = "MYSQL"; - public static final String POSTGRESQL = "POSTGRESQL"; - public static final String ORACLE = "ORACLE"; - public static final String SQLSERVER = "SQLSERVER"; - public static final String CLICKHOUSE = "CLICKHOUSE"; - public static final String SAP_HANA = "SAP_HANA"; - public static final String TRINO = "TRINO"; - public static final String PRESTO = "PRESTO"; - public static final String OCEANBASE = "OCEANBASE"; - public static final String OCEANBASE_ORACLE = "OCEANBASE_ORACLE"; - public static final String DB2 = "DB2"; - public static final String GBASE = "GBASE"; - - public static final String JDBC_PROPERTIES_PREFIX = "jdbc."; public static final String JDBC_URL = "jdbc_url"; public static final String USER = "user"; public static final String PASSWORD = "password"; @@ -116,8 +77,8 @@ public class JdbcResource extends Resource { public static final String CHECK_SUM = "checksum"; public static final String CREATE_TIME = "create_time"; public static final String TEST_CONNECTION = "test_connection"; - public static final String FUNCTION_RULES = "function_rules"; + /** Every property a resource persists: what the user may set, plus what creation fills in. */ private static final ImmutableList ALL_PROPERTIES = new ImmutableList.Builder().add( JDBC_URL, USER, @@ -164,8 +125,6 @@ public class JdbcResource extends Resource { OPTIONAL_PROPERTIES_DEFAULT_VALUE.put(CatalogProperty.ENABLE_MAPPING_TIMESTAMP_TZ, "false"); } - // timeout for both connection and read. 10 seconds is long enough. - private static final int HTTP_TIMEOUT_MS = 10000; @SerializedName(value = "configs") private Map configs; @@ -188,7 +147,6 @@ public void modifyProperties(Map properties) throws DdlException for (String propertyKey : ALL_PROPERTIES) { replaceIfEffectiveValue(this.configs, propertyKey, properties.get(propertyKey)); } - this.configs.put(JDBC_URL, handleJdbcUrl(getProperty(JDBC_URL))); super.modifyProperties(properties); } @@ -209,6 +167,7 @@ protected void setProperties(ImmutableMap properties) throws Ddl Preconditions.checkState(properties != null); this.configs = Maps.newHashMap(properties); validateProperties(this.configs); + validateThroughConnector(this.configs); applyDefaultProperties(); String currentDateTime = TimeUtils.longToTimeString(System.currentTimeMillis()); configs.put(CREATE_TIME, currentDateTime); @@ -219,8 +178,7 @@ protected void setProperties(ImmutableMap properties) throws Ddl throw new DdlException("JdbcResource Missing " + property + " in properties"); } } - this.configs.put(JDBC_URL, handleJdbcUrl(getProperty(JDBC_URL))); - configs.put(CHECK_SUM, computeObjectChecksum(getProperty(DRIVER_URL))); + computeDriverChecksumThroughConnector(this.configs); } /** @@ -261,340 +219,6 @@ public String getProperty(String propertiesKey) { return configs.get(propertiesKey); } - public static String computeObjectChecksum(String driverPath) throws DdlException { - if (FeConstants.runningUnitTest) { - // skip checking checksum when running ut - return ""; - } - String fullDriverUrl = getFullDriverUrl(driverPath); - - try (InputStream inputStream = - Util.getInputStreamFromUrl(fullDriverUrl, null, HTTP_TIMEOUT_MS, HTTP_TIMEOUT_MS)) { - MessageDigest digest = MessageDigest.getInstance("MD5"); - byte[] buf = new byte[4096]; - int bytesRead = 0; - do { - bytesRead = inputStream.read(buf); - if (bytesRead < 0) { - break; - } - digest.update(buf, 0, bytesRead); - } while (true); - return Hex.encodeHexString(digest.digest()); - } catch (IOException e) { - throw new DdlException("compute driver checksum from url: " + driverPath - + " meet an IOException: " + e.getMessage()); - } catch (NoSuchAlgorithmException e) { - throw new DdlException("compute driver checksum from url: " + driverPath - + " could not find algorithm: " + e.getMessage()); - } - } - - private static void checkCloudWhiteList(String driverUrl) throws IllegalArgumentException { - // For compatibility with cloud mode, we use both `jdbc_driver_url_white_list` - // and jdbc_driver_secure_path to check whitelist - List cloudWhiteList = new ArrayList<>(Arrays.asList(Config.jdbc_driver_url_white_list)); - cloudWhiteList.removeIf(String::isEmpty); - if (!cloudWhiteList.isEmpty() && !cloudWhiteList.contains(driverUrl)) { - throw new IllegalArgumentException("Driver URL does not match any allowed paths" + driverUrl); - } - } - - public static String getFullDriverUrl(String driverUrl) throws IllegalArgumentException { - if (!(driverUrl.startsWith("file://") || driverUrl.startsWith("http://") - || driverUrl.startsWith("https://") || driverUrl.matches("^[^:/]+\\.jar$"))) { - throw new IllegalArgumentException("Invalid driver URL format. Supported formats are: " - + "file://xxx.jar, http://xxx.jar, https://xxx.jar, or xxx.jar (without prefix)."); - } - - URI uri; - try { - uri = new URI(driverUrl); - } catch (URISyntaxException e) { - // Fail closed: an unparsable URL must never be silently accepted, otherwise the - // allowed-path check below could be bypassed by a malformed URL. - LOG.warn("invalid jdbc driver url: {}", driverUrl, e); - throw new IllegalArgumentException("Invalid driver URL: " + driverUrl); - } - - String schema = uri.getScheme(); - checkCloudWhiteList(driverUrl); - if (schema == null && !driverUrl.startsWith("/")) { - // A scheme-less driver_url is a plain jar file name resolved under jdbc_drivers_dir. This - // shared resolver is also on the lazy load path of pre-existing catalogs (Iceberg/Paimon/ - // legacy JDBC consumers call it directly, with no create/alter or replay context), so it - // deliberately applies no new restriction here: an unmodified historical catalog must keep - // resolving exactly as before. The mandatory bare-name grammar is enforced only when a - // catalog is created or altered, in JdbcDorisConnector.checkDriverUrlSecurityRule. - return checkAndReturnDefaultDriverUrl(driverUrl); - } - - // "*" or an empty/blank value means allow all (the documented, backward-compatible contract). - String securePath = Config.jdbc_driver_secure_path; - if (securePath == null || securePath.trim().isEmpty() || "*".equals(securePath.trim())) { - return driverUrl; - } - - if (!isDriverUrlAllowed(driverUrl, uri)) { - throw new IllegalArgumentException("Driver URL does not match any allowed paths: " + driverUrl); - } - return driverUrl; - } - - /** - * Check whether {@code driverUrl} falls under one of the semicolon-separated prefixes configured in - * {@link Config#jdbc_driver_secure_path}. Matching is structural (component-based) rather than a raw string - * prefix, so that neither prefix confusion ({@code /opt/drivers} vs {@code /opt/drivers-evil}) nor path - * traversal ({@code /opt/drivers/../etc}) can slip a driver outside the allowed location. - */ - private static boolean isDriverUrlAllowed(String driverUrl, URI uri) { - String scheme = uri.getScheme(); - List allowedPaths = new ArrayList<>(); - for (String p : Config.jdbc_driver_secure_path.split(";")) { - String trimmed = p.trim(); - if (!trimmed.isEmpty()) { - allowedPaths.add(trimmed); - } - } - if ("http".equalsIgnoreCase(scheme) || "https".equalsIgnoreCase(scheme)) { - URI candidate = uri.normalize(); - return allowedPaths.stream().anyMatch(allowed -> remoteUrlMatches(candidate, allowed)); - } - // Only file:// reaches here; bare absolute paths and bare "*.jar" are handled earlier. - // A local file URL must carry no authority, query or fragment. Otherwise validation (which - // looks only at URI.getPath()) and the consumers (URLClassLoader / checksum, which act on the - // whole original URL) would address different objects — e.g. "file://attacker/dir/x.jar" is - // fetched from a remote authority, and "file:///dir/x.jar?evil" maps to a sibling file. - String authority = uri.getRawAuthority(); - if ((authority != null && !authority.isEmpty()) - || uri.getRawQuery() != null || uri.getRawFragment() != null) { - return false; - } - Path candidate = toLocalPath(driverUrl).normalize(); - return allowedPaths.stream() - .map(allowed -> toLocalPath(allowed).normalize()) - .anyMatch(candidate::startsWith); - } - - /** - * Turn a {@code file://} URL or a plain filesystem path into a {@link Path} for structural comparison. - * A {@code file://} URL is decoded exactly once via {@link URI#getPath()} so that percent-encoded - * segments (e.g. {@code %2e%2e}) are resolved into the same representation the driver-loading - * consumers ({@code URL.openStream} / {@code URLClassLoader}) will use; otherwise an encoded parent - * segment would survive normalization and escape the allowed directory. - */ - private static Path toLocalPath(String pathOrUrl) { - if (pathOrUrl.startsWith("file:")) { - try { - String decoded = new URI(pathOrUrl).getPath(); - if (decoded != null && !decoded.isEmpty()) { - return Paths.get(decoded); - } - } catch (URISyntaxException ignored) { - // fall through to literal stripping below - } - int sep = pathOrUrl.indexOf("//"); - return Paths.get(sep >= 0 ? pathOrUrl.substring(sep + 2) : pathOrUrl.substring("file:".length())); - } - return Paths.get(pathOrUrl); - } - - /** - * Structural match for remote (http/https) driver URLs: scheme, host and port must be equal, and the - * candidate path must sit under the allowed path (component-based). A bare path prefix (no scheme) can - * never authorize a remote URL. - */ - private static boolean remoteUrlMatches(URI candidate, String allowedPath) { - URI base; - try { - base = new URI(allowedPath).normalize(); - } catch (URISyntaxException e) { - return false; - } - if (base.getScheme() == null) { - return false; - } - // Scheme/host/port and the path prefix must match, and the resource-selecting components - // (user-info and query) that the checksum/classloader consumers act on must match exactly too, - // otherwise e.g. ".../download?id=approved" would authorize ".../download?id=evil". - return base.getScheme().equalsIgnoreCase(candidate.getScheme()) - && base.getHost() != null && base.getHost().equalsIgnoreCase(candidate.getHost()) - && base.getPort() == candidate.getPort() - && Objects.equals(base.getUserInfo(), candidate.getUserInfo()) - && Objects.equals(base.getRawQuery(), candidate.getRawQuery()) - && pathIsUnder(candidate.getPath(), base.getPath()); - } - - private static boolean pathIsUnder(String candidatePath, String basePath) { - Path candidate = Paths.get(candidatePath == null || candidatePath.isEmpty() ? "/" : candidatePath).normalize(); - Path base = Paths.get(basePath == null || basePath.isEmpty() ? "/" : basePath).normalize(); - return candidate.startsWith(base); - } - - private static String checkAndReturnDefaultDriverUrl(String driverUrl) { - final String defaultDriverUrl = EnvUtils.getDorisHome() + "/plugins/jdbc_drivers"; - final String defaultOldDriverUrl = EnvUtils.getDorisHome() + "/jdbc_drivers"; - if (Config.jdbc_drivers_dir.equals(defaultDriverUrl)) { - // If true, which means user does not set `jdbc_drivers_dir` and use the default one. - // Because in new version, we change the default value of `jdbc_drivers_dir` - // from `DORIS_HOME/jdbc_drivers` to `DORIS_HOME/plugins/jdbc_drivers`, - // so we need to check the old default dir for compatibility. - String targetPath = defaultDriverUrl + "/" + driverUrl; - File targetFile = new File(targetPath); - String oldTargetPath = defaultOldDriverUrl + "/" + driverUrl; - File oldTargetFile = new File(oldTargetPath); - if (targetFile.exists()) { - // File exists in new default directory - return "file://" + targetPath; - } else if (oldTargetFile.exists()) { - // File exists in old default directory - return "file://" + oldTargetPath; - } else if (Config.isCloudMode()) { - // Cloud mode: download from cloud to default directory - try { - String downloadedPath = CloudPluginDownloader.downloadFromCloud( - PluginType.JDBC_DRIVERS, driverUrl, targetPath); - return "file://" + downloadedPath; - } catch (Exception e) { - LOG.warn("failed to download jdbc driver url: " + driverUrl, e); - throw new RuntimeException("Cannot download JDBC driver from cloud: " + driverUrl - + ". Please retry later or check your driver has been uploaded to cloud. Error: " - + Util.getRootCauseMessage(e)); - } - } else { - // File does not exist in both new and old default directory - throw new RuntimeException("JDBC driver file does not exist: " + driverUrl); - } - } else { - // Return user specified driver url directly. - return "file://" + Config.jdbc_drivers_dir + "/" + driverUrl; - } - } - - public static String parseDbType(String url) throws DdlException { - if (url.startsWith(JDBC_MYSQL) || url.startsWith(JDBC_MARIADB)) { - return MYSQL; - } else if (url.startsWith(JDBC_POSTGRESQL)) { - return POSTGRESQL; - } else if (url.startsWith(JDBC_ORACLE)) { - return ORACLE; - } else if (url.startsWith(JDBC_SQLSERVER)) { - return SQLSERVER; - } else if (url.startsWith(JDBC_CLICKHOUSE)) { - return CLICKHOUSE; - } else if (url.startsWith(JDBC_SAP_HANA)) { - return SAP_HANA; - } else if (url.startsWith(JDBC_TRINO)) { - return TRINO; - } else if (url.startsWith(JDBC_PRESTO)) { - return PRESTO; - } else if (url.startsWith(JDBC_OCEANBASE)) { - return OCEANBASE; - } else if (url.startsWith(JDBC_DB2)) { - return DB2; - } else if (url.startsWith(JDBC_GBASE)) { - return GBASE; - } - throw new DdlException("Unsupported jdbc database type, please check jdbcUrl: " + url); - } - - public static String handleJdbcUrl(String jdbcUrl) throws DdlException { - // delete all space in jdbcUrl - String newJdbcUrl = jdbcUrl.replaceAll(" ", ""); - String dbType = parseDbType(newJdbcUrl); - if (dbType.equals(MYSQL) || dbType.equals(OCEANBASE)) { - // `yearIsDateType` is a parameter of JDBC, and the default is true. - // We force the use of `yearIsDateType=false` - newJdbcUrl = checkAndSetJdbcBoolParam(dbType, newJdbcUrl, "yearIsDateType", "true", "false"); - // MySQL Types and Return Values for GetColumnTypeName and GetColumnClassName - // are presented in https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-reference-type-conversions.html - // When mysql's tinyint stores non-0 or 1, we need to read the data correctly, - // so we need tinyInt1isBit=false - newJdbcUrl = checkAndSetJdbcBoolParam(dbType, newJdbcUrl, "tinyInt1isBit", "true", "false"); - // set useUnicode and characterEncoding to false and utf-8 - newJdbcUrl = checkAndSetJdbcBoolParam(dbType, newJdbcUrl, "useUnicode", "false", "true"); - newJdbcUrl = checkAndSetJdbcBoolParam(dbType, newJdbcUrl, "rewriteBatchedStatements", "false", "true"); - newJdbcUrl = checkAndSetJdbcParam(dbType, newJdbcUrl, "characterEncoding", "utf-8"); - if (dbType.equals(OCEANBASE)) { - // set useCursorFetch to true - newJdbcUrl = checkAndSetJdbcBoolParam(dbType, newJdbcUrl, "useCursorFetch", "false", "true"); - } - } - if (dbType.equals(POSTGRESQL)) { - newJdbcUrl = checkAndSetJdbcBoolParam(dbType, newJdbcUrl, "reWriteBatchedInserts", "false", "true"); - } - if (dbType.equals(SQLSERVER)) { - if (Config.force_sqlserver_jdbc_encrypt_false) { - newJdbcUrl = checkAndSetJdbcBoolParam(dbType, newJdbcUrl, "encrypt", "true", "false"); - } - newJdbcUrl = checkAndSetJdbcBoolParam(dbType, newJdbcUrl, "useBulkCopyForBatchInsert", "false", "true"); - } - return newJdbcUrl; - } - - /** - * Check jdbcUrl param, if the param is not set, set it to the expected value. - * If the param is set to an unexpected value, replace it with the expected value. - * If the param is set to the expected value, do nothing. - * - * @param jdbcUrl - * @param params - * @param unexpectedVal - * @param expectedVal - * @return - */ - private static String checkAndSetJdbcBoolParam(String dbType, String jdbcUrl, String params, String unexpectedVal, - String expectedVal) { - String delimiter = getDelimiter(jdbcUrl, dbType); - String unexpectedParams = params + "=" + unexpectedVal; - String expectedParams = params + "=" + expectedVal; - - if (jdbcUrl.contains(expectedParams)) { - return jdbcUrl; - } else if (jdbcUrl.contains(unexpectedParams)) { - jdbcUrl = jdbcUrl.replaceAll(unexpectedParams, expectedParams); - } else { - if (!jdbcUrl.endsWith(delimiter)) { - jdbcUrl += delimiter; - } - jdbcUrl += expectedParams; - } - return jdbcUrl; - } - - /** - * Check jdbcUrl param, if the param is set, do thing. - * If the param is not set, set it to expected value. - * - * @param jdbcUrl - * @param params - * @return - */ - private static String checkAndSetJdbcParam(String dbType, String jdbcUrl, String params, String expectedVal) { - String delimiter = getDelimiter(jdbcUrl, dbType); - String expectedParams = params + "=" + expectedVal; - - if (jdbcUrl.contains(expectedParams)) { - return jdbcUrl; - } else { - if (!jdbcUrl.endsWith(delimiter)) { - jdbcUrl += delimiter; - } - jdbcUrl += expectedParams; - } - return jdbcUrl; - } - - private static String getDelimiter(String jdbcUrl, String dbType) { - if (dbType.equals(SQLSERVER) || dbType.equals(DB2)) { - return ";"; - } else if (jdbcUrl.contains("?")) { - return "&"; - } else { - return "?"; - } - } - public static String getDefaultPropertyValue(String propertyName) { return OPTIONAL_PROPERTIES_DEFAULT_VALUE.getOrDefault(propertyName, ""); } @@ -607,44 +231,59 @@ public static void validateProperties(Map properties) throws Ddl } } - public static void checkBooleanProperty(String propertyName, String propertyValue) throws DdlException { - if (!propertyValue.equalsIgnoreCase("true") && !propertyValue.equalsIgnoreCase("false")) { - throw new DdlException(propertyName + " must be true or false"); - } + /** The connector type the plugin that validates this resource answers to: the resource type's name. */ + private String connectorType() { + return type.name().toLowerCase(Locale.ROOT); } - public static void checkDatabaseListProperties(String onlySpecifiedDatabase, - Map includeDatabaseList, Map excludeDatabaseList) throws DdlException { - if (!onlySpecifiedDatabase.equalsIgnoreCase("true")) { - if ((includeDatabaseList != null && !includeDatabaseList.isEmpty()) || (excludeDatabaseList != null - && !excludeDatabaseList.isEmpty())) { - throw new DdlException( - "include_database_list and exclude_database_list " - + "cannot be set when only_specified_database is false"); - } + /** + * The value rules of the connector that serves a JDBC catalog — required keys, booleans, connection-pool + * bounds, the driver_url grammar — applied to the resource's properties, so a resource is held to + * exactly what a catalog is held to and the engine keeps no copy of those rules. + */ + private void validateThroughConnector(Map properties) throws DdlException { + String connectorType = connectorType(); + if (!ConnectorFactory.findProvider(connectorType, properties).isPresent()) { + throw new DdlException("JDBC resource requires the '" + connectorType + + "' connector plugin, which is not installed"); + } + try { + ConnectorFactory.validateProperties(connectorType, properties); + } catch (IllegalArgumentException e) { + throw new DdlException(e.getMessage(), e); } } - public static void checkConnectionPoolProperties(int minSize, int maxSize, int maxWaitTime, int maxLifeTime) - throws DdlException { - if (minSize < 0) { - throw new DdlException("connection_pool_min_size must be greater than or equal to 0"); - } - if (maxSize < 1) { - throw new DdlException("connection_pool_max_size must be greater than or equal to 1"); - } - if (maxSize < minSize) { - throw new DdlException( - "connection_pool_max_size must be greater than or equal to connection_pool_min_size"); - } - if (maxWaitTime < 0) { - throw new DdlException("connection_pool_max_wait_time must be greater than or equal to 0"); + /** + * Resolves the driver jar and records its checksum under {@link #CHECK_SUM}, through the same + * pre-creation validation a JDBC catalog runs. The BE connectivity test that validation may request is + * left unsent: a resource never tested connectivity. Skipped under unit tests, where no driver jar exists + * (the checksum was likewise not computed there before). + */ + private void computeDriverChecksumThroughConnector(Map properties) throws DdlException { + if (FeConstants.runningUnitTest) { + properties.put(CHECK_SUM, ""); + return; } - if (maxWaitTime > 30000) { - throw new DdlException("connection_pool_max_wait_time must be less than or equal to 30000"); + Connector connector = ConnectorFactory.createConnector(connectorType(), properties, + DefaultConnectorContext.forCatalogCreationValidation(name, -1L, properties)); + if (connector == null) { + throw new DdlException("JDBC resource requires the '" + connectorType() + + "' connector plugin, which is not installed"); } - if (maxLifeTime < 150000) { - throw new DdlException("connection_pool_max_life_time must be greater than or equal to 150000"); + try { + connector.preCreateValidation(new DefaultConnectorValidationContext(-1L, + new CatalogProperty(null, properties))); + } catch (DdlException e) { + throw e; + } catch (Exception e) { + throw new DdlException(e.getMessage(), e); + } finally { + try { + connector.close(); + } catch (IOException e) { + LOG.warn("Failed to close the connector that validated resource {}", name, e); + } } } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java index dd0bc658e86330..68c527d5a4fb49 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java @@ -477,9 +477,10 @@ public boolean supportsExternalMetadataPreload() { @Override public Optional initSchema() { PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; - // Keep the JDBC schema delay debug point available for manual regression verification. - if ("jdbc".equalsIgnoreCase(pluginCatalog.getType()) - && DebugPointUtil.isEnable("PluginDrivenExternalTable.initSchema.sleep")) { + // Schema-load delay debug point for regression tests of non-blocking schema refresh; it applies to + // any plugin-driven table (the engine gates nothing by source name), and only the test that needs + // it turns it on. + if (DebugPointUtil.isEnable("PluginDrivenExternalTable.initSchema.sleep")) { long sleepMs = DebugPointUtil.getDebugParamOrDefault( "PluginDrivenExternalTable.initSchema.sleep", "sleepMs", 0L); if (sleepMs > 0) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java index e67c77d78611bf..0371654c2edce4 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java @@ -40,7 +40,6 @@ public enum RuleType { BINDING_INSERT_BLACKHOLE_SINK(RuleTypeClass.REWRITE), BINDING_INSERT_HIVE_TABLE(RuleTypeClass.REWRITE), BINDING_INSERT_MAX_COMPUTE_TABLE(RuleTypeClass.REWRITE), - BINDING_INSERT_JDBC_TABLE(RuleTypeClass.REWRITE), BINDING_INSERT_CONNECTOR_TABLE(RuleTypeClass.REWRITE), BINDING_INSERT_TARGET_TABLE(RuleTypeClass.REWRITE), BINDING_INSERT_DICTIONARY_TABLE(RuleTypeClass.REWRITE), @@ -336,7 +335,6 @@ public enum RuleType { FILE_SCAN_PARTITION_PRUNE(RuleTypeClass.REWRITE), PUSH_FILTER_INTO_SCHEMA_SCAN(RuleTypeClass.REWRITE), - PUSH_CONJUNCTS_INTO_JDBC_SCAN(RuleTypeClass.REWRITE), PUSH_CONJUNCTS_INTO_ODBC_SCAN(RuleTypeClass.REWRITE), PUSH_CONJUNCTS_INTO_ES_SCAN(RuleTypeClass.REWRITE), PUSH_DOWN_VIRTUAL_COLUMNS_INTO_OLAP_SCAN(RuleTypeClass.REWRITE), @@ -550,7 +548,6 @@ public enum RuleType { LOGICAL_OLAP_SCAN_TO_PHYSICAL_OLAP_SCAN_RULE(RuleTypeClass.IMPLEMENTATION), LOGICAL_SCHEMA_SCAN_TO_PHYSICAL_SCHEMA_SCAN_RULE(RuleTypeClass.IMPLEMENTATION), LOGICAL_FILE_SCAN_TO_PHYSICAL_FILE_SCAN_RULE(RuleTypeClass.IMPLEMENTATION), - LOGICAL_JDBC_SCAN_TO_PHYSICAL_JDBC_SCAN_RULE(RuleTypeClass.IMPLEMENTATION), LOGICAL_ODBC_SCAN_TO_PHYSICAL_ODBC_SCAN_RULE(RuleTypeClass.IMPLEMENTATION), LOGICAL_ES_SCAN_TO_PHYSICAL_ES_SCAN_RULE(RuleTypeClass.IMPLEMENTATION), LOGICAL_WORK_TABLE_REFERENCE_TO_PHYSICAL_WORK_TABLE_REFERENCE(RuleTypeClass.IMPLEMENTATION), @@ -561,7 +558,6 @@ public enum RuleType { RuleTypeClass.IMPLEMENTATION), LOGICAL_EXTERNAL_ROW_LEVEL_MERGE_SINK_TO_PHYSICAL_EXTERNAL_ROW_LEVEL_MERGE_SINK_RULE( RuleTypeClass.IMPLEMENTATION), - LOGICAL_JDBC_TABLE_SINK_TO_PHYSICAL_JDBC_TABLE_SINK_RULE(RuleTypeClass.IMPLEMENTATION), LOGICAL_CONNECTOR_TABLE_SINK_TO_PHYSICAL_CONNECTOR_TABLE_SINK_RULE(RuleTypeClass.IMPLEMENTATION), LOGICAL_RESULT_SINK_TO_PHYSICAL_RESULT_SINK_RULE(RuleTypeClass.IMPLEMENTATION), LOGICAL_FILE_SINK_TO_PHYSICAL_FILE_SINK_RULE(RuleTypeClass.IMPLEMENTATION), diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/PlanType.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/PlanType.java index 5c0fc387261c9f..453cf53cd7a584 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/PlanType.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/PlanType.java @@ -33,7 +33,6 @@ public enum PlanType { LOGICAL_FILE_SCAN, LOGICAL_EMPTY_RELATION, LOGICAL_ES_SCAN, - LOGICAL_JDBC_SCAN, LOGICAL_ODBC_SCAN, LOGICAL_OLAP_SCAN, LOGICAL_TEST_SCAN, @@ -51,7 +50,6 @@ public enum PlanType { LOGICAL_MAX_COMPUTE_TABLE_SINK, LOGICAL_EXTERNAL_ROW_LEVEL_DELETE_SINK, LOGICAL_EXTERNAL_ROW_LEVEL_MERGE_SINK, - LOGICAL_JDBC_TABLE_SINK, LOGICAL_CONNECTOR_TABLE_SINK, LOGICAL_RESULT_SINK, LOGICAL_BLACKHOLE_SINK, @@ -60,7 +58,6 @@ public enum PlanType { LOGICAL_UNBOUND_HIVE_TABLE_SINK, LOGICAL_UNBOUND_ICEBERG_TABLE_SINK, LOGICAL_UNBOUND_MAX_COMPUTE_TABLE_SINK, - LOGICAL_UNBOUND_JDBC_TABLE_SINK, LOGICAL_UNBOUND_CONNECTOR_TABLE_SINK, LOGICAL_UNBOUND_RESULT_SINK, LOGICAL_UNBOUND_DICTIONARY_SINK, @@ -109,7 +106,6 @@ public enum PlanType { PHYSICAL_EMPTY_RELATION, PHYSICAL_ES_SCAN, PHYSICAL_FILE_SCAN, - PHYSICAL_JDBC_SCAN, PHYSICAL_ODBC_SCAN, PHYSICAL_ONE_ROW_RELATION, PHYSICAL_OLAP_SCAN, @@ -123,7 +119,6 @@ public enum PlanType { PHYSICAL_MAX_COMPUTE_TABLE_SINK, PHYSICAL_EXTERNAL_ROW_LEVEL_DELETE_SINK, PHYSICAL_EXTERNAL_ROW_LEVEL_MERGE_SINK, - PHYSICAL_JDBC_TABLE_SINK, PHYSICAL_CONNECTOR_TABLE_SINK, PHYSICAL_RESULT_SINK, PHYSICAL_BLACKHOLE_SINK, diff --git a/fe/fe-core/src/main/java/org/apache/doris/transaction/JdbcTransactionManager.java b/fe/fe-core/src/main/java/org/apache/doris/transaction/JdbcTransactionManager.java deleted file mode 100644 index a0a1cc28803d4e..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/transaction/JdbcTransactionManager.java +++ /dev/null @@ -1,42 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.transaction; - -import org.apache.doris.common.UserException; - -public class JdbcTransactionManager implements TransactionManager { - @Override - public long begin() { - return 0; - } - - @Override - public void commit(long id) throws UserException { - - } - - @Override - public void rollback(long id) { - - } - - @Override - public Transaction getTransaction(long id) { - return null; - } -} diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/JdbcResourceTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/JdbcResourceTest.java index fce17458111ea5..a414b371e9a2bd 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/catalog/JdbcResourceTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/JdbcResourceTest.java @@ -17,10 +17,12 @@ package org.apache.doris.catalog; -import org.apache.doris.common.Config; import org.apache.doris.common.DdlException; import org.apache.doris.common.FeConstants; import org.apache.doris.common.UserException; +import org.apache.doris.connector.ConnectorFactory; +import org.apache.doris.connector.ConnectorPluginManager; +import org.apache.doris.connector.jdbc.JdbcConnectorProvider; import org.apache.doris.mysql.privilege.AccessControllerManager; import org.apache.doris.mysql.privilege.PrivPredicate; import org.apache.doris.nereids.trees.plans.commands.CreateResourceCommand; @@ -30,12 +32,14 @@ import com.google.common.collect.ImmutableMap; import com.google.common.collect.Maps; +import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import org.mockito.MockedStatic; import org.mockito.Mockito; +import java.util.List; import java.util.Map; public class JdbcResourceTest { @@ -47,6 +51,11 @@ public class JdbcResourceTest { @BeforeEach public void setUp() { FeConstants.runningUnitTest = true; + // A JDBC resource is validated by the jdbc connector plugin, the way a JDBC catalog is; the plugin + // manager is a static singleton shared across the fork, so start from one holding exactly that provider. + ConnectorPluginManager manager = new ConnectorPluginManager(); + manager.registerProvider(new JdbcConnectorProvider()); + ConnectorFactory.initPluginManager(manager); jdbcProperties = Maps.newHashMap(); jdbcProperties.put("type", "jdbc"); jdbcProperties.put("user", "postgres"); @@ -57,8 +66,12 @@ public void setUp() { jdbcProperties.put("checksum", "20c8228267b6c9ce620fddb39467d3eb"); } - @Test - public void testJdbcResourceCreateWithDefaultProperties() throws UserException { + @AfterEach + public void tearDown() { + ConnectorFactory.initPluginManager(new ConnectorPluginManager()); + } + + private void createResource(String name, Map properties) throws UserException { try (MockedStatic mockedEnv = Mockito.mockStatic(Env.class)) { Env env = Mockito.mock(Env.class); EditLog editLog = Mockito.mock(EditLog.class); @@ -66,27 +79,29 @@ public void testJdbcResourceCreateWithDefaultProperties() throws UserException { mockedEnv.when(Env::getCurrentEnv).thenReturn(env); Mockito.when(env.getEditLog()).thenReturn(editLog); Mockito.when(env.getAccessManager()).thenReturn(accessManager); - Mockito.when(accessManager.checkGlobalPriv(Mockito.nullable(ConnectContext.class), Mockito.eq(PrivPredicate.ADMIN))) - .thenReturn(true); - - jdbcProperties.remove("checksum"); - + Mockito.when(accessManager.checkGlobalPriv(Mockito.nullable(ConnectContext.class), + Mockito.eq(PrivPredicate.ADMIN))).thenReturn(true); CreateResourceCommand createResourceCommand = new CreateResourceCommand( - new CreateResourceInfo(true, false, "jdbc_resource_pg_14", - ImmutableMap.copyOf(jdbcProperties))); + new CreateResourceInfo(true, false, name, ImmutableMap.copyOf(properties))); createResourceCommand.getInfo().validate(); resourceMgr.createResource(createResourceCommand); + } + } - JdbcResource jdbcResource = (JdbcResource) resourceMgr.getResource("jdbc_resource_pg_14"); + @Test + public void testJdbcResourceCreateWithDefaultProperties() throws UserException { + jdbcProperties.remove("checksum"); + createResource("jdbc_resource_pg_14", jdbcProperties); - // Verify the default properties were applied during the replay - Map properties = jdbcResource.getCopiedProperties(); - Assertions.assertEquals("1", properties.get("connection_pool_min_size")); - Assertions.assertEquals("30", properties.get("connection_pool_max_size")); - Assertions.assertEquals("1800000", properties.get("connection_pool_max_life_time")); - Assertions.assertEquals("5000", properties.get("connection_pool_max_wait_time")); - Assertions.assertEquals("false", properties.get("connection_pool_keep_alive")); - } + JdbcResource jdbcResource = (JdbcResource) resourceMgr.getResource("jdbc_resource_pg_14"); + + // Verify the default properties were applied on creation + Map properties = jdbcResource.getCopiedProperties(); + Assertions.assertEquals("1", properties.get("connection_pool_min_size")); + Assertions.assertEquals("30", properties.get("connection_pool_max_size")); + Assertions.assertEquals("1800000", properties.get("connection_pool_max_life_time")); + Assertions.assertEquals("5000", properties.get("connection_pool_max_wait_time")); + Assertions.assertEquals("false", properties.get("connection_pool_keep_alive")); } @Test @@ -163,253 +178,79 @@ public void testJdbcResourceReplayWithModifiedAfterSetDefaultProperties() throws } @Test - public void testHandleJdbcUrlForMySql() throws DdlException { - String inputUrl = "jdbc:mysql://127.0.0.1:3306/test"; - String resultUrl = JdbcResource.handleJdbcUrl(inputUrl); - - // Check if the result URL contains the necessary delimiters for MySQL - Assertions.assertTrue(resultUrl.contains("?")); - Assertions.assertTrue(resultUrl.contains("&")); + public void testCreateShowsTheSameRowsAsBefore() throws UserException { + // 21 persisted keys plus the checksum the connector records (blank under unit tests, where the + // driver jar does not exist): the row count SHOW RESOURCES has always shown for a JDBC resource. + jdbcProperties.remove("checksum"); + createResource("jdbc_resource_rows", jdbcProperties); + JdbcResource resource = (JdbcResource) resourceMgr.getResource("jdbc_resource_rows"); + Map properties = resource.getCopiedProperties(); + Assertions.assertEquals(22, properties.size(), properties.toString()); + Assertions.assertTrue(properties.containsKey(JdbcResource.CHECK_SUM)); + Assertions.assertNotNull(properties.get(JdbcResource.CREATE_TIME)); + // The url is stored as the user wrote it; nothing in the engine rewrites JDBC urls any more. + Assertions.assertEquals("jdbc:postgresql://127.0.0.1:5432/postgres?currentSchema=doris_test", + properties.get(JdbcResource.JDBC_URL)); } @Test - public void testHandleJdbcUrlForSqlServerWithoutParams() throws DdlException { - String inputUrl = "jdbc:sqlserver://127.0.0.1:1433;databaseName=doris_test"; - String resultUrl = JdbcResource.handleJdbcUrl(inputUrl); - - // Ensure that the result URL for SQL Server doesn't have '?' or '&' - Assertions.assertFalse(resultUrl.contains("?")); - Assertions.assertFalse(resultUrl.contains("&")); + public void testCreateIsValidatedByTheJdbcConnector() { + // The connector's rules apply to a resource exactly as to a catalog: a required key, a pool bound, + // the driver_url grammar. The engine holds no copy of these rules. + jdbcProperties.remove("checksum"); + jdbcProperties.remove("driver_class"); + DdlException missing = Assertions.assertThrows(DdlException.class, + () -> createResource("jdbc_resource_bad", jdbcProperties)); + Assertions.assertTrue(missing.getMessage().contains("driver_class"), missing.getMessage()); - // Ensure the result URL still contains ';' - Assertions.assertTrue(resultUrl.contains(";")); - } - - @Test - public void testHandleJdbcUrlForSqlServerWithParams() throws DdlException { - String inputUrl - = "jdbc:sqlserver://127.0.0.1:1433;encrypt=false;databaseName=doris_test;trustServerCertificate=false"; - String resultUrl = JdbcResource.handleJdbcUrl(inputUrl); - - // Ensure that the result URL for SQL Server doesn't have '?' or '&' - Assertions.assertFalse(resultUrl.contains("?")); - Assertions.assertFalse(resultUrl.contains("&")); - - // Ensure the result URL still contains ';' - Assertions.assertTrue(resultUrl.contains(";")); - } - - @Test - public void testValidDriverUrls() { - String fileUrl = "file://path/to/driver.jar"; - Assertions.assertDoesNotThrow(() -> { - String result = JdbcResource.getFullDriverUrl(fileUrl); - Assertions.assertEquals(fileUrl, result); - }); - - String httpUrl = "http://example.com/driver.jar"; - Assertions.assertDoesNotThrow(() -> { - String result = JdbcResource.getFullDriverUrl(httpUrl); - Assertions.assertEquals(httpUrl, result); - }); - - String httpsUrl = "https://example.com/driver.jar"; - Assertions.assertDoesNotThrow(() -> { - String result = JdbcResource.getFullDriverUrl(httpsUrl); - Assertions.assertEquals(httpsUrl, result); - }); - - String jarFile = "driver.jar"; - Assertions.assertThrows(RuntimeException.class, () -> { - JdbcResource.getFullDriverUrl(jarFile); - }); - } - - @Test - public void testInvalidDriverUrls() { - String invalidUrl1 = "/mnt/path/to/driver.jar"; - Assertions.assertThrows(IllegalArgumentException.class, () -> { - JdbcResource.getFullDriverUrl(invalidUrl1); - }); - - String invalidUrl2 = "ftp://example.com/driver.jar"; - Assertions.assertThrows(IllegalArgumentException.class, () -> { - JdbcResource.getFullDriverUrl(invalidUrl2); - }); - - String invalidUrl3 = ""; - Assertions.assertThrows(IllegalArgumentException.class, () -> { - JdbcResource.getFullDriverUrl(invalidUrl3); - }); - - String invalidUrl4 = "example.com/driver"; - Assertions.assertThrows(IllegalArgumentException.class, () -> { - JdbcResource.getFullDriverUrl(invalidUrl4); - }); - } - - @Test - public void testSecurePathRejectsPrefixConfusion() { - String saved = Config.jdbc_driver_secure_path; - try { - Config.jdbc_driver_secure_path = "file:///opt/doris/jdbc_drivers"; - // A directory that merely shares a string prefix must NOT be allowed. - Assertions.assertThrows(IllegalArgumentException.class, () -> - JdbcResource.getFullDriverUrl("file:///opt/doris/jdbc_drivers-evil/x.jar")); - } finally { - Config.jdbc_driver_secure_path = saved; - } - } - - @Test - public void testSecurePathRejectsPathTraversal() { - String saved = Config.jdbc_driver_secure_path; - try { - Config.jdbc_driver_secure_path = "file:///opt/doris/jdbc_drivers"; - Assertions.assertThrows(IllegalArgumentException.class, () -> - JdbcResource.getFullDriverUrl("file:///opt/doris/jdbc_drivers/../../etc/x.jar")); - } finally { - Config.jdbc_driver_secure_path = saved; - } - } - - @Test - public void testSecurePathAllowsPathUnderAllowedDir() { - String saved = Config.jdbc_driver_secure_path; - try { - Config.jdbc_driver_secure_path = "file:///opt/doris/jdbc_drivers"; - String url = "file:///opt/doris/jdbc_drivers/sub/x.jar"; - Assertions.assertDoesNotThrow(() -> Assertions.assertEquals(url, JdbcResource.getFullDriverUrl(url))); - } finally { - Config.jdbc_driver_secure_path = saved; - } - } - - @Test - public void testSecurePathRejectsHostConfusion() { - String saved = Config.jdbc_driver_secure_path; - try { - Config.jdbc_driver_secure_path = "http://good.com/"; - Assertions.assertThrows(IllegalArgumentException.class, () -> - JdbcResource.getFullDriverUrl("http://good.com.evil.com/x.jar")); - } finally { - Config.jdbc_driver_secure_path = saved; - } - } - - @Test - public void testSecurePathAllowsRemoteUnderAllowedHost() { - String saved = Config.jdbc_driver_secure_path; - try { - Config.jdbc_driver_secure_path = "http://good.com/drivers"; - String url = "http://good.com/drivers/x.jar"; - Assertions.assertDoesNotThrow(() -> Assertions.assertEquals(url, JdbcResource.getFullDriverUrl(url))); - } finally { - Config.jdbc_driver_secure_path = saved; - } - } - - @Test - public void testSecurePathWildcardAllowsAll() { - String saved = Config.jdbc_driver_secure_path; - try { - Config.jdbc_driver_secure_path = "*"; - String url = "file:///any/where/x.jar"; - Assertions.assertDoesNotThrow(() -> Assertions.assertEquals(url, JdbcResource.getFullDriverUrl(url))); - } finally { - Config.jdbc_driver_secure_path = saved; - } - } - - @Test - public void testSecurePathRejectsEncodedTraversal() { - String saved = Config.jdbc_driver_secure_path; - try { - Config.jdbc_driver_secure_path = "file:///opt/doris/jdbc_drivers"; - // %2e%2e decodes to "..", which must be resolved the same way the classloader resolves it. - Assertions.assertThrows(IllegalArgumentException.class, () -> - JdbcResource.getFullDriverUrl("file:///opt/doris/jdbc_drivers/%2e%2e/%2e%2e/etc/x.jar")); - } finally { - Config.jdbc_driver_secure_path = saved; - } - } - - @Test - public void testSecurePathRejectsRemoteQueryMismatch() { - String saved = Config.jdbc_driver_secure_path; - try { - Config.jdbc_driver_secure_path = "http://good.com/drivers"; - // A query-bearing URL must not be authorized by a query-less allowed prefix. - Assertions.assertThrows(IllegalArgumentException.class, () -> - JdbcResource.getFullDriverUrl("http://good.com/drivers/x.jar?id=evil")); - } finally { - Config.jdbc_driver_secure_path = saved; - } - } - - @Test - public void testSecurePathRejectsRemoteUserInfoMismatch() { - String saved = Config.jdbc_driver_secure_path; - try { - Config.jdbc_driver_secure_path = "http://good.com/drivers"; - Assertions.assertThrows(IllegalArgumentException.class, () -> - JdbcResource.getFullDriverUrl("http://user@good.com/drivers/x.jar")); - } finally { - Config.jdbc_driver_secure_path = saved; - } - } - - @Test - public void testSchemelessLegacyCharsAccepted() { - // The shared resolver is on the lazy load path of pre-existing catalogs, so it applies no new - // restriction: a bare name using historically valid characters (e.g. '+') must keep resolving, - // so an unmodified historical catalog is not broken after upgrade. The stricter bare-name - // grammar applies only when a catalog is created or altered (enforced in the JDBC connector). - String savedDir = Config.jdbc_drivers_dir; - try { - Config.jdbc_drivers_dir = "/opt/doris/jdbc_drivers"; - Assertions.assertEquals("file:///opt/doris/jdbc_drivers/legacy+patched.jar", - JdbcResource.getFullDriverUrl("legacy+patched.jar")); - } finally { - Config.jdbc_drivers_dir = savedDir; - } + jdbcProperties.put("driver_class", "org.postgresql.Driver"); + jdbcProperties.put("connection_pool_max_size", "0"); + DdlException pool = Assertions.assertThrows(DdlException.class, + () -> createResource("jdbc_resource_bad", jdbcProperties)); + Assertions.assertTrue(pool.getMessage().contains("connection_pool_max_size"), pool.getMessage()); + + jdbcProperties.put("connection_pool_max_size", "10"); + jdbcProperties.put("driver_url", "../escape.jar"); + DdlException traversal = Assertions.assertThrows(DdlException.class, + () -> createResource("jdbc_resource_bad", jdbcProperties)); + Assertions.assertTrue(traversal.getMessage().contains("driver_url"), traversal.getMessage()); } @Test - public void testSecurePathRejectsFileAuthority() { - String saved = Config.jdbc_driver_secure_path; - try { - Config.jdbc_driver_secure_path = "file:///opt/doris/jdbc_drivers"; - // A non-local authority makes consumers fetch a remote object though the path matches. - Assertions.assertThrows(IllegalArgumentException.class, () -> - JdbcResource.getFullDriverUrl("file://attacker.example/opt/doris/jdbc_drivers/evil.jar")); - } finally { - Config.jdbc_driver_secure_path = saved; - } + public void testUnknownPropertyIsRejected() { + jdbcProperties.remove("checksum"); + jdbcProperties.put("no_such_property", "x"); + DdlException e = Assertions.assertThrows(DdlException.class, + () -> createResource("jdbc_resource_bad", jdbcProperties)); + Assertions.assertTrue(e.getMessage().contains("no_such_property"), e.getMessage()); } @Test - public void testSecurePathRejectsFileQuery() { - String saved = Config.jdbc_driver_secure_path; - try { - Config.jdbc_driver_secure_path = "file:///opt/doris/jdbc_drivers"; - Assertions.assertThrows(IllegalArgumentException.class, () -> - JdbcResource.getFullDriverUrl("file:///opt/doris/jdbc_drivers/x.jar?evil")); - } finally { - Config.jdbc_driver_secure_path = saved; - } + public void testCreateWithoutTheJdbcPluginFailsLoud() { + jdbcProperties.remove("checksum"); + ConnectorFactory.initPluginManager(new ConnectorPluginManager()); + DdlException e = Assertions.assertThrows(DdlException.class, + () -> createResource("jdbc_resource_noplugin", jdbcProperties)); + Assertions.assertTrue(e.getMessage().contains("connector plugin"), e.getMessage()); } @Test - public void testEmptySecurePathAllowsAll() { - String saved = Config.jdbc_driver_secure_path; - try { - // Empty means allow-all, same as "*" (backward-compatible contract). - Config.jdbc_driver_secure_path = ""; - String url = "file:///opt/doris/jdbc_drivers/x.jar"; - Assertions.assertDoesNotThrow(() -> Assertions.assertEquals(url, JdbcResource.getFullDriverUrl(url))); - } finally { - Config.jdbc_driver_secure_path = saved; + public void testProcNodeDataMasksThePassword() throws UserException { + jdbcProperties.remove("checksum"); + jdbcProperties.put("password", "secret"); + createResource("jdbc_resource_masked", jdbcProperties); + JdbcResource resource = (JdbcResource) resourceMgr.getResource("jdbc_resource_masked"); + org.apache.doris.common.proc.BaseProcResult result = new org.apache.doris.common.proc.BaseProcResult(); + resource.getProcNodeData(result); + boolean sawPassword = false; + for (List row : result.getRows()) { + if (row.get(2).equals(JdbcResource.PASSWORD)) { + sawPassword = true; + Assertions.assertEquals("", row.get(3), "the password row must be blanked"); + } + Assertions.assertEquals("jdbc", row.get(1)); } + Assertions.assertTrue(sawPassword); + Assertions.assertEquals(22, result.getRows().size()); } } From 5dbcecc3a909c6872fafa87dc8f1ee4aa243136a Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 16 Sep 2026 02:51:24 +0800 Subject: [PATCH 4/4] [chore](fe) Gate fe-core against JDBC data-source access and drop its HikariCP dependency ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: Part 4: keep the JDBC implementation from drifting back into fe-core. `build-support/check-fe-core-jdbc-free.sh` runs in fe-core's `validate` phase next to the metadata-funnel gate and fails the build on an import of `org.apache.doris.datasource.jdbc.*`, `com.zaxxer.hikari.*` or a `java.sql` connection/statement type in fe-core main sources; `httpv2/` (the FE's HTTP SQL gateway, JDBC to the FE itself) is the only exemption. Self-test in `build-support/tests/`. fe-core's HikariCP dependency goes with the deleted clients (the plugin bundles its own), and the fe-connector docs list the new gate. ### Release note None ### Check List (For Author) - Test: Unit Test - build-support/tests/run.sh (all gate self-tests), fe-core validate phase. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5 --- build-support/check-fe-core-jdbc-free.sh | 92 +++++++++++++ build-support/tests/test-fe-core-jdbc-free.sh | 128 ++++++++++++++++++ fe/fe-connector/AGENTS.md | 10 +- fe/fe-connector/README.md | 6 +- fe/fe-core/pom.xml | 37 +++-- 5 files changed, 259 insertions(+), 14 deletions(-) create mode 100755 build-support/check-fe-core-jdbc-free.sh create mode 100755 build-support/tests/test-fe-core-jdbc-free.sh diff --git a/build-support/check-fe-core-jdbc-free.sh b/build-support/check-fe-core-jdbc-free.sh new file mode 100755 index 00000000000000..aa9330799152bd --- /dev/null +++ b/build-support/check-fe-core-jdbc-free.sh @@ -0,0 +1,92 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# +# Arch gate: fe-core carries no JDBC data-source implementation. +# +# Invariant: every JDBC dialect client, type mapping, driver-jar policy and connection pool lives in +# fe/fe-connector (the jdbc plugin and the shared spi policy), never in fe-core. fe-core reaches a JDBC +# source only through the connector SPI (PluginDrivenExternalCatalog for catalogs, StreamingSourceClient +# for streaming jobs). A dialect fix that lands in fe-core again would silently fork the implementation +# — the exact drift this gate exists to prevent — with no compile error and no test failure. +# +# Forbidden in fe-core main sources: +# 1. import org.apache.doris.datasource.jdbc.* — the deleted legacy client package; it must not return. +# 2. import com.zaxxer.hikari.* — a connection pool is a connector concern. +# 3. import java.sql. — Connection, DriverManager, Driver, Statement, +# PreparedStatement, ResultSet, ResultSetMetaData, DatabaseMetaData, DataSource, SQLException. +# The value types (java.sql.Timestamp/Date/Time/Types) are not JDBC access and stay allowed. +# +# Exempt (kept silent): +# - org/apache/doris/httpv2/** — the FE's HTTP SQL gateway, which talks to the FE ITSELF over the +# MySQL protocol through a JDBC driver. That is not an external data source. +# - A comment line that merely names an import — never executable. +# +# Self-test: build-support/tests/test-fe-core-jdbc-free.sh. +# +# Usage: +# build-support/check-fe-core-jdbc-free.sh # default root fe/fe-core +# build-support/check-fe-core-jdbc-free.sh # supplied root +# +# Exit code: +# 0 — no forbidden imports +# 1 — at least one found (offending lines printed) +# 2 — search root not found + +set -e + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +DEFAULT_ROOT="${SCRIPT_DIR}/../fe/fe-core" +ROOT="${1:-${DEFAULT_ROOT}}" + +if [ ! -d "${ROOT}" ]; then + echo "check-fe-core-jdbc-free: search root not found: ${ROOT}" >&2 + exit 2 +fi + +# The FE's own HTTP SQL gateway: JDBC to the FE itself, exempt by path. +EXEMPT_DIR='org/apache/doris/httpv2/' + +# The three forbidden import shapes (anchored to an import statement, so a comment or a string that +# names the package is not matched). +LEGACY_PKG='^[[:space:]]*import[[:space:]]+(static[[:space:]]+)?org\.apache\.doris\.datasource\.jdbc\.' +POOL='^[[:space:]]*import[[:space:]]+(static[[:space:]]+)?com\.zaxxer\.hikari\.' +JDBC_ACCESS='^[[:space:]]*import[[:space:]]+(static[[:space:]]+)?java\.sql\.(Connection|DriverManager|Driver|Statement|PreparedStatement|CallableStatement|ResultSet|ResultSetMetaData|DatabaseMetaData|DataSource|SQLException)([[:space:].;]|$)' + +CANDIDATES=$(grep -rEn "${LEGACY_PKG}|${POOL}|${JDBC_ACCESS}" "${ROOT}/src/main/java" 2>/dev/null || true) + +RESULT="" +if [ -n "${CANDIDATES}" ]; then + while IFS= read -r line; do + [ -z "${line}" ] && continue + file="${line%%:*}" + case "${file}" in *"${EXEMPT_DIR}"*) continue ;; esac + RESULT="${RESULT}${line}"$'\n' + done <<< "${CANDIDATES}" +fi +RESULT=$(printf '%s' "${RESULT}" | sed '/^$/d') + +if [ -n "${RESULT}" ]; then + echo "JDBC data-source access in fe-core (must live in fe/fe-connector and be reached through the SPI):" >&2 + echo "${RESULT}" >&2 + echo "" >&2 + echo "fe-core must not import org.apache.doris.datasource.jdbc.*, com.zaxxer.hikari.* or the" >&2 + echo "java.sql connection/statement types. A JDBC source is reached through the connector SPI" >&2 + echo "(PluginDrivenExternalCatalog / StreamingSourceClient); dialect logic belongs in" >&2 + echo "fe/fe-connector/fe-connector-jdbc. Only ${EXEMPT_DIR} (JDBC to the FE itself) is exempt." >&2 + exit 1 +fi diff --git a/build-support/tests/test-fe-core-jdbc-free.sh b/build-support/tests/test-fe-core-jdbc-free.sh new file mode 100755 index 00000000000000..b19a17aefe4f81 --- /dev/null +++ b/build-support/tests/test-fe-core-jdbc-free.sh @@ -0,0 +1,128 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# +# Self-test for build-support/check-fe-core-jdbc-free.sh. +# +# The gate exits 0 on the real (already-clean) tree, so a controlled RED/GREEN fixture is the only way +# to prove it catches what it must and to lock the behavior against silent regression. +# Each seeded case targets one gate property: +# RED — import of the deleted legacy client package (the core violation) +# RED — import of the HikariCP pool (pool is a connector concern) +# RED — import of java.sql.Connection / a static import of DriverManager.getConnection +# SILENT — java.sql.Timestamp / java.sql.Types (value types, not JDBC access) +# SILENT — the same forbidden imports under httpv2/ (JDBC to the FE itself, exempt by path) +# SILENT — a comment or string that names a forbidden package (never an import) +# SILENT — a class whose simple name merely starts with a forbidden one (java.sql.Statement vs +# a hypothetical java.sql.StatementEvent is matched on the whole simple name) +# Plus: exit 0 on a clean tree. +# +# Usage: bash build-support/tests/test-fe-core-jdbc-free.sh # exit 0 = pass, 1 = fail + +set -u + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +GATE="${SCRIPT_DIR}/../check-fe-core-jdbc-free.sh" + +FX="$(mktemp -d)" +trap 'rm -rf "${FX}"' EXIT + +SRC="${FX}/src/main/java/org/apache/doris" +mkdir -p "${SRC}/job" "${SRC}/httpv2/util" "${SRC}/common" + +# RED: the legacy package, the pool, and a JDBC connection type. +cat > "${SRC}/job/LegacyClientUser.java" <<'EOF2' +package org.apache.doris.job; +import org.apache.doris.datasource.jdbc.client.JdbcClient; +public class LegacyClientUser { +} +EOF2 + +cat > "${SRC}/job/PoolUser.java" <<'EOF2' +package org.apache.doris.job; +import com.zaxxer.hikari.HikariDataSource; +public class PoolUser { +} +EOF2 + +cat > "${SRC}/job/ConnectionUser.java" <<'EOF2' +package org.apache.doris.job; +import java.sql.Connection; +import static java.sql.DriverManager.getConnection; +public class ConnectionUser { +} +EOF2 + +# SILENT: value types, comments and strings, and a longer simple name. +cat > "${SRC}/common/ValueTypes.java" <<'EOF2' +package org.apache.doris.common; +import java.sql.Timestamp; +import java.sql.Types; +import java.sql.StatementEvent; +public class ValueTypes { + // import java.sql.Connection; is only mentioned here + String s = "import org.apache.doris.datasource.jdbc.client.JdbcClient"; +} +EOF2 + +# SILENT: the FE's own HTTP SQL gateway is exempt by path. +cat > "${SRC}/httpv2/util/StatementSubmitter.java" <<'EOF2' +package org.apache.doris.httpv2.util; +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.ResultSet; +public class StatementSubmitter { +} +EOF2 + +FAILED=0 +fail() { echo "FAIL: $1"; FAILED=1; } + +# ---- run 1: mixed fixture -> exactly the RED imports flagged ---- +OUT="$(bash "${GATE}" "${FX}" 2>&1)"; EC=$? +REPORTED="$(printf '%s\n' "${OUT}" | grep -E "^${FX}.*:[0-9]+:" || true)" +N="$(printf '%s\n' "${REPORTED}" | grep -c 'import' || true)" + +[ "${EC}" -eq 1 ] || fail "expected exit 1 (violations present), got ${EC}" +[ "${N}" -eq 4 ] || fail "expected exactly 4 reported violations, got ${N}"$'\n'"${REPORTED}" + +must_report() { printf '%s\n' "${REPORTED}" | grep -qF "$1" || fail "violation NOT reported: $1"; } +must_report 'job/LegacyClientUser.java:' +must_report 'job/PoolUser.java:' +must_report 'import java.sql.Connection;' +must_report 'import static java.sql.DriverManager.getConnection;' + +must_not_report() { + printf '%s\n' "${REPORTED}" | grep -qF "$1" && fail "should NOT be reported: $1" || true +} +must_not_report 'common/ValueTypes.java:' +must_not_report 'httpv2/util/StatementSubmitter.java:' + +# ---- run 2: remove the RED cases -> clean tree exits 0 ---- +rm -f "${SRC}/job/LegacyClientUser.java" "${SRC}/job/PoolUser.java" "${SRC}/job/ConnectionUser.java" +bash "${GATE}" "${FX}" >/dev/null 2>&1; EC=$? +[ "${EC}" -eq 0 ] || fail "expected exit 0 on the clean fixture, got ${EC}" + +# ---- run 3: a missing root is reported as such, not as clean ---- +bash "${GATE}" "${FX}/does-not-exist" >/dev/null 2>&1; EC=$? +[ "${EC}" -eq 2 ] || fail "expected exit 2 for a missing search root, got ${EC}" + +if [ "${FAILED}" -ne 0 ]; then + echo "test-fe-core-jdbc-free: FAILED" + exit 1 +fi +echo "test-fe-core-jdbc-free: OK" diff --git a/fe/fe-connector/AGENTS.md b/fe/fe-connector/AGENTS.md index 7741dc44a9c196..ed72869cbe0835 100644 --- a/fe/fe-connector/AGENTS.md +++ b/fe/fe-connector/AGENTS.md @@ -33,7 +33,7 @@ mvn -f fe/pom.xml -pl :fe-connector-spi -am test \ ## Machine-Checked Obligations -Two architecture gates run in the `validate` phase (scripts and their +Three architecture gates run in the `validate` phase (scripts and their self-tests live in `build-support/` and `build-support/tests/`): 1. **Forbidden imports** — `build-support/check-fe-connector-imports.sh`, @@ -46,6 +46,14 @@ self-tests live in `build-support/` and `build-support/tests/`): `PluginDrivenMetadata` may call `Connector#getMetadata`; exempt call sites carry a `getMetadata-funnel-exempt` marker, and deleting a marker auto-tightens the gate. +3. **No JDBC access in fe-core** — `build-support/check-fe-core-jdbc-free.sh`, + wired into fe-core's `pom.xml`. fe-core main sources must not import + `org.apache.doris.datasource.jdbc.*`, `com.zaxxer.hikari.*` or the + `java.sql` connection/statement types; a JDBC source is reached through + the SPI (`PluginDrivenExternalCatalog`, `StreamingSourceClient`), and + dialect logic belongs in `fe-connector-jdbc`. The driver-jar policy every + connector applies is `DriverUrlPolicy` in fe-connector-spi. Only + `httpv2/` (JDBC to the FE itself) is exempt. **Changing the shared SPI surface (fe-connector-spi):** regenerate BOTH recorded baselines in the SAME commit — `connector-metadata-methods.txt` diff --git a/fe/fe-connector/README.md b/fe/fe-connector/README.md index b455e1c0bfcda9..c658ab9b501aaa 100644 --- a/fe/fe-connector/README.md +++ b/fe/fe-connector/README.md @@ -319,9 +319,9 @@ String, convert inside `of()`, log a warning) explicitly. you touch fe-connector-spi, run that module's tests — a consumer-only test run will not catch a stale baseline. - **Architecture gates** run in the `validate` phase of every FE build: the - forbidden-import gate for this directory and the metadata-funnel gate for - fe-core. Scripts and their self-tests live in `build-support/` and - `build-support/tests/`. + forbidden-import gate for this directory, and the metadata-funnel and + JDBC-free gates for fe-core. Scripts and their self-tests live in + `build-support/` and `build-support/tests/`. - **End-to-end**: docker environments under `docker/thirdparties/docker-compose/`, suites under `regression-test/suites/external_table_p0` and `external_table_p2`. diff --git a/fe/fe-core/pom.xml b/fe/fe-core/pom.xml index 656b9847c76147..dbb0fb1b270192 100644 --- a/fe/fe-core/pom.xml +++ b/fe/fe-core/pom.xml @@ -787,10 +787,6 @@ under the License. mariadb-java-client - - com.zaxxer - HikariCP - org.apache.kafka kafka-clients @@ -1158,12 +1154,20 @@ under the License. org.codehaus.mojo @@ -1183,6 +1187,19 @@ under the License. + + check-fe-core-jdbc-free + validate + + exec + + + ${project.basedir}/../../build-support/check-fe-core-jdbc-free.sh + + ${project.basedir} + + +