docs(bigquery-jdbc): add user guide with connection property and custom endpoint reference

java-bigquery-jdbc/DEVELOPMENT.md

@@ -0,0 +1,179 @@
+# BigQuery JDBC Developer & Contributor Guide
+
+This guide details the architectural design, core abstractions, coding principles, and testing workflows for developers contributing to the `google-cloud-bigquery-jdbc` module.
+
+---
+
+## Table of Contents
+1. [Core Architecture & Component Map](#1-core-architecture--component-map)
+2. [Developer Guardrails & Rules of Engagement](#2-developer-guardrails--rules-of-engagement)
+3. [Build & Test Playbook](#3-build--test-playbook)
+   - [Local Build Commands](#local-build-commands)
+   - [Running Unit Tests](#running-unit-tests)
+   - [Running Integration Tests](#running-integration-tests)
+   - [Dockerized Execution](#dockerized-execution)
+4. [Logging Architecture & Developer Conventions](#4-logging-architecture--developer-conventions)
+   - [Instantiating Loggers](#instantiating-loggers)
+   - [Developer Logging Rules & Conventions](#developer-logging-rules--conventions)
+5. [Pre-PR Checklist](#5-pre-pr-checklist)
+
+---
+
+## 1. Core Architecture & Component Map
+
+The driver is structured to provide high performance, zero-allocation MDC log tracing, strict JDBC compliance, and seamless execution over the Google Cloud BigQuery REST and Storage APIs.
+
+```mermaid
+graph TD
+    Client[Client Application / BI Tool] -->|DriverManager.getConnection| Driver[BigQueryDriver]
+    Driver -->|Parses URI & Options| UrlUtil[BigQueryJdbcUrlUtility]
+    Driver -->|Configures Logging| RootLogger[BigQueryJdbcRootLogger]
+    Driver -->|Creates| Conn[BigQueryConnection]
+    Conn -->|Dynamic Context Proxy| Proxy[BigQueryJdbcContextProxy]
+    Proxy -->|MDC Tracing| Mdc[BigQueryJdbcMdc]
+    Proxy -->|Delegates Exec| DirectConn[Client Session]
+    Conn -->|Type Mapping & Coercion| Coercion[BigQueryJdbcTypeMappings & BigQueryCoercion]
+    Conn -->|REST / Storage API| BQSDK[google-cloud-bigquery]
+```
+
+### Key Abstractions
+
+- **`BigQueryDriver`**: JDBC entry point registered with `java.sql.DriverManager`. Intercepts `jdbc:bigquery://` URLs, initializes early logger state, and instantiates `BigQueryConnection`.
+- **`BigQueryConnection`**: Represents an active BigQuery session, holding dataset defaults, connection configuration maps, and transaction/session state (`EnableSession=true`, `session_id`).
+- **`BigQueryJdbcUrlUtility`**: Parses and validates connection string parameters using a bounded LRU parse cache (`PARSE_CACHE`) to avoid heavy allocations during frequent connection creation.
+- **`BigQueryJdbcContextProxy`**: A dynamic proxy layer (`java.lang.reflect.Proxy`) wrapping JDBC statements, connections, and metadata. Intercepts calls to propagate ThreadLocal MDC parameters (`connectionId`) across execution threads and enforce state validation (`checkClosed()`).
+- **`BigQueryJdbcTypeMappings` & `BigQueryCoercion`**: Centralized mapping logic handling standard JDBC-to-BigQuery SQL type mappings (`StandardSQLTypeName`) and object coercions (`Date`, `Timestamp`, `BigDecimal`, etc.).
+- **`BigQueryArrowResultSet`**: Custom result set implementation accelerating large query result retrieval via the BigQuery Storage Read API gRPC stream.
+
+---
+
+## 2. Developer Guardrails & Rules of Engagement
+
+> [!IMPORTANT]
+> **Adhere strictly to the following guardrails when making code changes:**
+
+1. **Visibility Principle**: Always default to the most restrictive access level (`private`, package-private, or `@InternalApi`). Do **NOT** expose classes or methods as `public` unless strictly required by standard JDBC interfaces.
+2. **Explicit Class Imports**: Always write explicit `import` statements. Do **NOT** use wildcard star imports or inline fully qualified class names (e.g., use `import java.math.BigDecimal;` instead of `java.math.BigDecimal` inline).
+3. **Logger Preference**: Always prefer `BigQueryJdbcCustomLogger` over `java.util.logging.Logger`. Format strings using `String.format(...)` before logging, as `BigQueryJdbcRootLogger` evaluates `record.getMessage()` directly.
+4. **Exception Handling**: Always throw exceptions from the `com.google.cloud.bigquery.exception` package (`BigQueryJdbcException`, `BigQueryJdbcSqlSyntaxErrorException`, `BigQueryConversionException`).
+5. **No Mocking of Final JDK Classes**: Do **NOT** mock final JDK types (`BigDecimal`, `LocalDate`, `Instant`, `UUID`) with Mockito. Mocking final JDK classes is unstable and can cause JVM crashes under JDK 21+. Always construct real instances in unit tests.
+
+---
+
+## 3. Build & Test Playbook
+
+Builds and test tasks are managed via the module [Makefile](Makefile).
+
+### Local Build Commands
+
+```bash
+# Build & install module locally
+make install
+
+# Clean project target directory
+make clean
+
+# Format code and check linter compliance
+make lint
+```
+
+### Running Unit Tests
+
+```bash
+# Run all unit tests
+make unittest
+
+# Run a specific unit test class
+make unittest test=BigQueryPreparedStatementTest
+
+# Run a specific unit test method
+make unittest test=BigQueryPreparedStatementTest#testSetObjectWithTemporalTypes
+```
+
+### Running Integration Tests
+
+> [!WARNING]
+> Integration tests connect to real GCP BigQuery resources and require valid GCP credentials.
+
+```bash
+# Set GCP service account credentials
+export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json
+
+# Run a specific integration test
+make integration-test test=ITBigQueryJDBCTest#testValidServiceAccountAuthenticationOAuthPvtKey
+```
+
+### Dockerized Execution
+
+If local Java/Maven environments are not available, use the dockerized environment:
+
+```bash
+# Start an interactive shell session inside Docker container
+make docker-session
+
+# Run unit tests inside Docker
+make docker-unittest
+```
+
+---
+
+## 4. Logging Architecture & Developer Conventions
+
+The driver uses a custom logging subsystem built on top of `java.util.logging`: `BigQueryJdbcCustomLogger` and `BigQueryJdbcRootLogger`.
+
+### Instantiating Loggers
+
+- **For Instance Components** (`BigQueryConnection`, `BigQueryStatement`, `BigQueryDatabaseMetaData`):
+  Use `this.toString()` to include instance identity in logger output:
+  ```java
+  private final BigQueryJdbcCustomLogger LOG = new BigQueryJdbcCustomLogger(this.toString());
+  ```
+- **For Static / Utility Components** (`BigQueryJdbcUrlUtility`, `BigQueryJdbcTypeMappings`):
+  Use the class name:
+  ```java
+  private static final BigQueryJdbcCustomLogger LOG = 
+      new BigQueryJdbcCustomLogger(BigQueryJdbcTypeMappings.class.getName());
+  ```
+
+### Developer Logging Rules & Conventions
+
+1. **Method Entry / Exit Tracing**:
+   Methods at `FINER` level must log entrance and exit points:
+   ```java
+   public ResultSet executeQuery(String sql) throws SQLException {
+     LOG.finer("++enter++");
+     try {
+       // ... execution logic ...
+       return rs;
+     } finally {
+       LOG.finer("++exit++");
+     }
+   }
+   ```
+2. **Format Placeholders (Zero Allocation)**:
+   Avoid string concatenation in log calls. Use formatting placeholders or `Supplier<String>` lambdas to prevent unneeded string allocation when the log level is disabled:
+   ```java
+   // Recommended: Use printf-style formatting
+   LOG.fine("Executing query on dataset: %s, table: %s", datasetId, tableId);
+
+   // Recommended: Use supplier lambda for expensive calculations
+   LOG.fine(() -> "Parsed properties: " + complexObject.toDebugString());
+   ```
+3. **Caller Inference & MDC Propagation**:
+   - `BigQueryJdbcCustomLogger` automatically wraps log records in `BigQueryJdbcLogRecord`, which inspects the stack trace to accurately infer caller class and method names.
+   - `BigQueryJdbcMdc` maintains `connectionId` in a `ThreadLocal` context. When logging from proxy or worker threads, always ensure MDC context is preserved or propagated via `BigQueryJdbcContextProxy`.
+
+---
+
+## 5. Pre-PR Checklist
+
+Before submitting a Pull Request:
+
+- [ ] All new classes and methods use the narrowest possible visibility scope (`private` or package-private).
+- [ ] No inline fully qualified names or wildcard star imports are present.
+- [ ] All logger instances use `BigQueryJdbcCustomLogger`.
+- [ ] Method entrance/exit logging (`++enter++` / `++exit++`) is included for complex internal routines.
+- [ ] All method changes and feature additions are covered by corresponding JUnit 5 tests.
+- [ ] Unit tests pass cleanly without Mockito `UnnecessaryStubbingException` warnings.
+- [ ] All `Statement`, `ResultSet`, or `DatabaseMetaData` objects returned by public entry points are properly wrapped via `BigQueryJdbcContextProxy.wrap()`.
+- [ ] Code formatting and linting pass via `make lint`.

java-bigquery-jdbc/README.MD

@@ -7,6 +7,8 @@ Java idiomatic client for [BigQuery JDBC][product-docs].
 
 - [Product Documentation][product-docs]
 - [Client Library Documentation][javadocs]
+- [Driver User Guide](docs/USER_GUIDE.md)
+- [Storage APIs Deep-Dive Guide](docs/STORAGE_APIS.md)
 
 
 ## Quickstart

java-bigquery-jdbc/docs/STORAGE_APIS.md

@@ -0,0 +1,69 @@
+# BigQuery Storage APIs Deep Dive & Tuning Guide
+
+This document provides architectural details, property matrices, activation criteria, and workload tuning scenarios for the **BigQuery Storage Read API** and **BigQuery Storage Write API** integrated into the BigQuery JDBC Driver.
+
+---
+
+## 1. High-Throughput Storage Read API (HTAPI)
+
+The Storage Read API streams query result sets over high-speed gRPC channels using Apache Arrow format, bypassing standard REST JSON serialization for large datasets.
+
+### Property Reference
+
+| Property Name | Connection Parameter | Default Value | Functional Role |
+| :--- | :--- | :---: | :--- |
+| **`EnableHighThroughputAPI`** | `EnableHighThroughputAPI=true` | `false` | **Master Toggle**: Must be `true` to enable Read API evaluation. |
+| **`HighThroughputMinTableSize`** | `HighThroughputMinTableSize=10000` | `10000` | **Minimum Row Threshold**: Minimum total rows (`totalRows`) required. |
+| **`HighThroughputActivationRatio`** | `HighThroughputActivationRatio=2` | `2` | **Page Ratio Threshold**: `totalRows / MaxResults` ratio required. |
+| **`MaxResults`** | `MaxResults=10000` | `10000` | **Page Size**: Controls rows per page in standard REST calls. |
+
+### Activation Criteria & Fallback Mechanics
+
+When `EnableHighThroughputAPI=true` is set, the driver transparently switches to the Storage Read API if all of the following conditions are met:
+
+1. **Master Toggle**: `EnableHighThroughputAPI=true` is set.
+2. **Minimum Row Threshold**: The query returns at least `HighThroughputMinTableSize` rows (default: $\ge 10,000$ rows).
+3. **Multiple Response Pages**: The result set spans more than one page (total rows exceed `MaxResults`). If all rows fit on page 1, standard REST is used to avoid unnecessary gRPC stream setup.
+4. **Activation Ratio Test**: The ratio of total rows to page size ($\frac{\text{totalRows}}{\text{MaxResults}}$) exceeds `HighThroughputActivationRatio` (default: $> 2$).
+
+> [!NOTE]
+> **Automatic Permission Fallback**: If `EnableHighThroughputAPI=true` is set but the connecting principal lacks the `BigQuery Read Session User` IAM role, the driver catches the `PERMISSION_DENIED` status and automatically falls back to standard REST JSON pagination.
+
+### Workload Scenarios Matrix
+
+| Workload Scenario | `EnableHighThroughputAPI` | `HighThroughputMinTableSize` | `HighThroughputActivationRatio` | `MaxResults` | Execution Mechanism | Use Case |
+| :--- | :---: | :---: | :---: | :---: | :--- | :--- |
+| **Standard REST (Default)** | `false` | `10000` (ignored) | `2` (ignored) | `10000` | REST JSON Pagination | Small/medium queries; standard REST security policies. |
+| **Default Production Extractions** | `true` | `10000` | `2` | `10000` | gRPC Storage Read API (for results $> 20,000$ rows) | Standard analytical reports and ETL extracts. |
+| **Aggressive Streaming** | `true` | `100` | `0` | `50` | gRPC Storage Read API (for results $\ge 100$ rows) | High-speed streaming for smaller analytical datasets. |
+| **Bulk ETL Analytics** | `true` | `50000` | `5` | `10000` | gRPC Storage Read API (for results $> 50,000$ rows) | Large multi-gigabyte dataset extractions. |
+
+---
+
+## 2. Storage Write API (SWA)
+
+The Storage Write API streams high-throughput bulk insertions over gRPC channels for `PreparedStatement.executeBatch()` calls.
+
+### Property Reference
+
+| Property Name | Connection Parameter | Default Value | Functional Role |
+| :--- | :--- | :---: | :--- |
+| **`EnableWriteAPI`** | `EnableWriteAPI=true` | `false` | **Master Toggle**: Must be `true` to enable Storage Write API streaming. |
+| **`SWA_ActivationRowCount`** | `SWA_ActivationRowCount=3` | `3` | **Activation Threshold**: Minimum batch size added via `addBatch()` required to trigger SWA. |
+| **`SWA_AppendRowCount`** | `SWA_AppendRowCount=1000` | `1000` | **Chunk Size**: Maximum rows per gRPC append payload before flushing. |
+
+### Activation Criteria & Fallback Mechanics
+
+When `EnableWriteAPI=true` is set, the driver evaluates the batch size during `PreparedStatement.executeBatch()`:
+
+- **At or Above Threshold ($\ge \text{SWA\_ActivationRowCount}$)**: The driver opens a gRPC Storage Write stream and appends batch records in payload chunks governed by `SWA_AppendRowCount`.
+- **Below Threshold ($< \text{SWA\_ActivationRowCount}$)**: The driver uses standard SQL DML (`INSERT INTO ...`) to avoid gRPC stream overhead for tiny batches.
+
+### Workload Scenarios Matrix
+
+| Workload Scenario | `EnableWriteAPI` | `SWA_ActivationRowCount` | `SWA_AppendRowCount` | Execution Mechanism | Use Case |
+| :--- | :---: | :---: | :---: | :--- | :--- |
+| **Standard SQL DML (Default)** | `false` | `3` (ignored) | `1000` (ignored) | Concatenated REST SQL DML | Small transactional DML; standard SQL compatibility. |
+| **Default High-Throughput ETL** | `true` | `3` | `1000` | gRPC SWA stream (batches $\ge 3$, flushes per 1,000 rows) | Standard batch loader applications (Spring Batch, Spark). |
+| **Real-Time Micro-Batching** | `true` | `1` | `100` | gRPC SWA stream (batches $\ge 1$, flushes per 100 rows) | High-frequency streaming events (Kafka/Flink consumers). |
+| **High-Volume Bulk Ingestion** | `true` | `100` | `5000` | gRPC SWA stream (batches $\ge 100$, flushes per 5,000 rows) | Large nightly bulk ETL loading millions of records. |