diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/enterprise/iceberg.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/enterprise/iceberg.md new file mode 100644 index 000000000..213d78cb2 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/enterprise/iceberg.md @@ -0,0 +1,270 @@ +--- +keywords: [Iceberg, REST catalog, pyiceberg, Spark, parquet, 数据导出, 开放表格式] +description: 通过 Apache Iceberg REST catalog 使用 pyiceberg、Spark、Trino、DuckDB 等引擎读取 GreptimeDB 表——无需连接器，也无需复制数据。 +--- + +# Iceberg 导出 + +GreptimeDB Enterprise 可以通过 [Apache Iceberg](https://iceberg.apache.org/) REST catalog 暴露其表，使得外部查询引擎——pyiceberg、Spark、Trino、DuckDB 以及任何兼容 Iceberg 的客户端——都能使用标准的 Iceberg API 直接从对象存储读取 GreptimeDB 数据。 + +这是一种**只读导出**，既不是数据拷贝，也不是双写。GreptimeDB 仍按原有方式写入数据；Iceberg 集成只是发布元数据，让其他引擎能够找到并解读这些相同的文件。 + +## 工作原理 + +关键在于：GreptimeDB 本身就已将 SST（sorted string table）数据文件以 Parquet 格式存储在对象存储中。Iceberg 集成**不会**重新写入、复制或导出数据，而是发布指向现有 Parquet 文件的 Iceberg **元数据**——manifest、manifest-list 以及 table-metadata snapshot。任何支持 Iceberg 的引擎都能通过 REST catalog 读取这些文件。 + +整体流程如下图所示： + +```mermaid +flowchart LR + APP["客户端应用
(MySQL / PostgreSQL 协议)"] + + subgraph GT["GreptimeDB Enterprise"] + direction TB + FE["Frontend
Iceberg REST catalog (/v1/iceberg)"] + DN["Datanode / standalone
(写入端)"] + end + + subgraph OS["对象存储
(S3 / OSS / GCS / Azure Blob)"] + DATA["Parquet SST 数据文件"] + META["Iceberg 元数据
(manifests、snapshots)
位于 warehouse_root 下"] + end + + ENG["pyiceberg / Spark / Trino / DuckDB
(任何兼容 Iceberg 的引擎)"] + + APP -->|"写入"| FE + FE --> DN + DN -->|"在 flush / compaction 时
写入 SST 文件"| DATA + DN -->|"发布元数据、
提交新的 snapshot"| META + ENG -->|"1. 加载表元数据"| FE + ENG -->|"2. 直接读取 Parquet 文件"| DATA + META -.->|"指向"| DATA +``` + +这项工作分布在 GreptimeDB 已有的进程之间： + +- **Datanode / standalone（写入端）。** 每当写入 SST 文件时——包括 flush、compaction、批量写入和 truncate——GreptimeDB 都会将当前存活的 Parquet 文件集合转换为 Iceberg manifest 条目，并提交一个新的 Iceberg snapshot。Iceberg 元数据写入到 datanode 已使用的同一对象存储 bucket 下的 `warehouse_root` 前缀中。 +- **Frontend（catalog 服务端）。** 它实现了 [Iceberg REST Catalog API](https://iceberg.apache.org/docs/1.6.0/api/#rest-catalog-specification)，挂载在 `/v1/iceberg`，并向客户端提供每张表当前的元数据。 + +由于数据文件从不被复制，因此没有额外的存储开销，也没有写入路径上的重复——导出纯粹是在 GreptimeDB 已写入的数据旁附加的元数据。 + +### GreptimeDB 概念与 Iceberg 的对应关系 + +| GreptimeDB | Iceberg | +| ---------- | ------- | +| Schema（数据库，例如 `public`） | Namespace | +| Table（表） | Table | +| Column（列） | Field（字段） | +| 时间索引列 | 一个 `timestamptz` 字段 | + +来自 `ALTER TABLE`（新增 / 重命名 / 删除列）的 schema 变更会反映到 Iceberg schema 中，无需重启。 + +### 类型映射 + +GreptimeDB 列类型到 Iceberg 类型的映射如下： + +| GreptimeDB 类型 | Iceberg 类型 | +| --------------- | ------------ | +| `boolean` | `boolean` | +| `int8`、`int16`、`int32` | `int` | +| `int64` | `long` | +| `float32` | `float` | +| `float64` | `double` | +| `string` | `string` | +| `binary` | `binary` | +| `date` | `date` | +| `timestamp`（任意精度） | `timestamptz` | +| Prometheus native histogram（struct） | `struct`（包含 `list` 子字段） | +| `uint8`、`uint16`、`uint32`、`uint64` | `long` — **在 Spark 中不可读**（见下文） | +| `list`、`dictionary`、`json`、`interval`、`duration`、`time`、任意用户 `struct` | `string`（有损降级） | + +## 配置 + +Iceberg 集成是一个企业版插件。在**写入进程（datanode 或 standalone）**和**frontend**的 `[[plugins]]` 配置段中都添加一个 `iceberg_manifest` 条目即可启用： + +- **Datanode / standalone** 运行写入 hook，负责发布 Iceberg 元数据。 +- **Frontend** 运行 REST catalog，负责提供这些元数据。 + +两者必须引用**相同的** `warehouse_root`，以便 catalog 读取到写入端发布的内容。 + +```toml +## Iceberg manifest 导出会发布 Iceberg 格式的元数据（snapshot、 +## manifest、manifest-list），使外部引擎（pyiceberg、Spark、Trino、 +## DuckDB 等）可以通过 Iceberg REST catalog 查询 GreptimeDB 表。 +[[plugins]] +iceberg_manifest = { warehouse_root = "iceberg_warehouse" } +``` + +可选配置项： + +| 选项 | 默认值 | 说明 | +| ---- | ------ | ---- | +| `warehouse_root` | `"iceberg_warehouse"` | Iceberg 元数据在 datanode 对象存储 bucket 内的存放路径前缀。 | +| `enable_incremental` | `true` | 为 `true`（默认）时，每次 flush / compaction / truncate 都会自动发布一个新 snapshot。设为 `false` 可关闭自动发布，改为按需通过 rebuild 接口生成元数据；drop/GC 清理仍会正常运行。 | + +`warehouse_root` 是 GreptimeDB 已使用的对象存储 bucket 内的一个路径前缀，会被并入 store 的 `root`（例如 `s3://///`）。 + +支持的对象存储后端包括 S3、OSS、GCS 和 Azure Blob。 + +启用后，REST catalog 可通过以下地址访问： + +``` +http://:/v1/iceberg +``` + +例如，frontend 监听默认的 HTTP 端口 `4000` 时，catalog 基础 URI 为 `http://localhost:4000/v1/iceberg`，warehouse 名称为 `greptime`（默认值）。 + +## 让表可被读取 + +GreptimeDB 在 flush 时发布 Iceberg 元数据，因此新写入的数据会在下一次 flush（或 compaction）后才出现在导出中。若要立即暴露新写入的行，可手动 flush 表： + +```sql +-- 通过 MySQL 或 PostgreSQL 协议执行 +admin flush_table('your_table'); +``` + +元数据是异步发布的；flush 返回后不久，该表即可通过 REST catalog 查询。已经 flush 过的现有表会被自动导出。 + +## 使用 pyiceberg 读取 + +本示例使用 [pyiceberg](https://py.iceberg.apache.org/) 配合 pyarrow，通过 REST catalog 读取一张 GreptimeDB 表。你需要安装 `pyiceberg` 和 `pyarrow`，并准备好支撑 GreptimeDB 部署的对象存储的凭据（此处以 S3 为例）。 + +```python +from pyiceberg.catalog.rest import RestCatalog + +# GreptimeDB frontend 暴露的 REST catalog 端点。 +# `warehouse` 即 catalog 名称（默认为 "greptime"）。 +catalog = RestCatalog( + name="greptime", + uri="http://localhost:4000/v1/iceberg", + prefix="greptime", + # 对象存储凭据，供 pyiceberg 读取 Parquet 数据文件。 + # 请与你的 GreptimeDB 部署使用的后端保持一致。 + **{ + "s3.endpoint": "https://s3.us-east-1.amazonaws.com", + "s3.access-key-id": "YOUR_ACCESS_KEY", + "s3.secret-access-key": "YOUR_SECRET_KEY", + "s3.region": "us-east-1", + }, +) + +# 列出 namespace（= GreptimeDB schema）和表。 +print(catalog.list_namespaces()) # 例如 [('public',)] +print(catalog.list_tables("public")) # 例如 [('public', 'my_table')] + +# 加载表并用 pyarrow 扫描。 +table = catalog.load_table(("public", "my_table")) +print(table.schema()) # Iceberg schema（GreptimeDB 列 → 字段） + +arrow_table = table.scan().to_arrow() +print(arrow_table.num_rows, "rows") +print(arrow_table.to_pandas().head()) +``` + +## 使用 Spark 读取 + +本示例配置 Spark SQL 使用 GreptimeDB Iceberg REST catalog，并对一张 GreptimeDB 表执行查询。你需要将 Iceberg Spark runtime 和 AWS bundle JAR 放到 Spark classpath 上（版本需与你的 Spark/Scala 版本匹配——此处以 Spark 4.x 配 Iceberg 1.11.0 为例）。 + +**1. `spark-defaults.conf`** + +```properties +spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions +spark.sql.defaultCatalog greptime +spark.sql.catalog.greptime org.apache.iceberg.spark.SparkCatalog +spark.sql.catalog.greptime.type rest +spark.sql.catalog.greptime.uri http://localhost:4000/v1/iceberg +spark.sql.catalog.greptime.warehouse greptime +# S3FileIO 从 GreptimeDB 使用的同一对象存储读取 Parquet 数据文件。 +spark.sql.catalog.greptime.io-impl org.apache.iceberg.aws.s3.S3FileIO +spark.sql.catalog.greptime.client.region us-east-1 +spark.sql.catalog.greptime.s3.endpoint https://s3.us-east-1.amazonaws.com +spark.sql.catalog.greptime.s3.path-style-access false +spark.sql.catalog.greptime.s3.access-key-id YOUR_ACCESS_KEY +spark.sql.catalog.greptime.s3.secret-access-key YOUR_SECRET_KEY +``` + +将两个 Iceberg JAR 放到 classpath 上，启动 `spark-sql`（或运行 `spark-submit` 任务）： + +```bash +spark-sql \ + --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \ + --jars iceberg-spark-runtime-4.1_2.13-1.11.0.jar,iceberg-aws-bundle-1.11.0.jar +``` + +如果使用带自定义端点和 path-style 寻址的 S3 兼容存储（例如 MinIO、Garage 或本地测试部署），请把 `spark.sql.catalog.greptime.s3.endpoint` 设为该存储的 URL，并把 `spark.sql.catalog.greptime.s3.path-style-access` 设为 `true`。 + +**2. 在 GreptimeDB 中建表并写入数据** + +通过 MySQL 或 PostgreSQL 协议连接 GreptimeDB，建表，然后 flush 以发布 Iceberg 元数据： + +```sql +CREATE TABLE demo ( + ts TIMESTAMP(6) NOT NULL, -- TIME INDEX + host STRING, + region STRING, + cpu DOUBLE, + mem FLOAT, + status INT, + TIME INDEX (ts) +); + +INSERT INTO demo VALUES + ('2024-01-01 00:00:00', 'h1', 'us', 12.5, 4096.0, 200), + ('2024-01-01 01:00:00', 'h1', 'us', 88.8, 7000.5, 200), + ('2024-01-01 02:00:00', 'h2', 'eu', 55.0, 5500.0, 503); + +-- 为刚写入的行发布 Iceberg 元数据。 +admin flush_table('demo'); +``` + +**3. 从 Spark SQL 查询** + +```sql +-- catalog 是 `greptime`，namespace 是 GreptimeDB schema `public`。 +SHOW TABLES IN greptime.public; + +SELECT count(*) FROM greptime.public.demo; + +SELECT host, round(avg(cpu), 1) AS avg_cpu +FROM greptime.public.demo +WHERE ts >= '2024-01-01 00:00:00' +GROUP BY host +ORDER BY host; +``` + +该表完全可查询：时间范围过滤、聚合、join、窗口函数和排序都能像在 Spark 中使用任何其他 Iceberg 表一样正常工作。 + +## 限制 + +### 只读、单一 snapshot + +- **只读。** 你可以通过 Iceberg 读取 GreptimeDB 表，但无法通过 Iceberg catalog 向 GreptimeDB 回写数据。GreptimeDB 始终是唯一的写入方。 +- **仅保留最新 snapshot（不支持时间旅行）。** Compaction 会物理删除旧的数据文件，因此不会保留历史 snapshot。Iceberg 表始终反映当前存活的数据；你无法查询历史 snapshot 或回滚。 +- **没有完整的 schema 历史。** 当前 schema 始终会暴露，但过去的 schema 版本不会被保留，因此你无法重建一张表在过去某个时间点的结构。 + +### 数据类型与 Spark + +对于常见类型（boolean、有符号整数、浮点数、string、binary、date、timestamp），GreptimeDB 类型可以干净地映射到 Iceberg。需要注意以下几点，尤其是在 Spark 中： + +- **将时间索引声明为 `TIMESTAMP(6)`。** GreptimeDB 默认的 `TIMESTAMP` 是毫秒精度，但 Iceberg schema 将该列声明为 `timestamptz`（微秒）。若列是毫秒精度，Spark 的 Parquet row-group 统计信息过滤会用微秒谓词去比较毫秒的文件统计，可能错误地丢弃 row-group，导致 `>`、`=` 和范围查询出错。将时间索引声明为 `TIMESTAMP(6)` 可使磁盘上的 Parquet 微秒精度与 schema 一致，所有比较运算符即可正确工作。（秒/毫秒值本身仍被正确存储；该问题纯粹出在基于文件统计的谓词下推。） +- **有损的类型降级。** `list`、`dictionary`、`json`、`interval`、`duration`、`time` 以及任意用户 `struct` 类型会被导出为 Iceberg `string`，而非结构化类型，因此它们的内部结构无法通过 Iceberg 查询。 +- **无符号整数在 Spark 中不可读。** GreptimeDB 的无符号整数列（`uint8`、`uint16`、`uint32`、`uint64`）虽然声明为 Iceberg `long`，但底层 Parquet 文件以无符号物理类型存储，Spark 无法读取。任何触及无符号整数列的扫描在 Spark 中都会失败。如果你打算通过 Iceberg 查询某张表，请避免使用无符号类型，或将这些值以有符号类型存储。 + +### Metric 引擎表 + +- **仅导出物理 metric 表。** 逻辑 metric 表不会通过 Iceberg 暴露；请直接查询物理表。 +- **稀疏主键编码（metric 引擎默认）。** 所有 tag 列被折叠进单一的 `__primary_key` binary 列。要还原各个 tag 值（逻辑 table id、tsid 和 label），你必须按 metric 引擎的稀疏编解码器解码该 blob——Iceberg 将其暴露为不透明的 `binary`。metric 内部的 `__table_id` 和 `__tsid` 列**不会**被导出，因此你无法通过 Iceberg 将 metric 名称解析为物理 table id。 +- **Prometheus native histogram** 被导出为带 list 子字段的 Iceberg `struct`，因此每个 histogram 值会以嵌套 struct 的形式读取，而非标量。 + +### 运维说明 + +- **元数据异步发布。** 新写入的行会在下一次 flush 或 compaction 之后出现在 Iceberg 中；可手动 flush 表（`admin flush_table('')`）以立即暴露它们。 +- **旧 Iceberg 元数据会被回收**，与数据文件一起由 GreptimeDB 正常的 compaction 和 GC 处理——无需单独维护。 +- **Rebuild / 对账。** 如果 Iceberg 导出与 GreptimeDB 的真实状态出现偏差（发布失败、损坏，或在启用集成之前创建的表），运维人员可以从权威的存活 SST 集合重建一张表的 Iceberg 元数据： + + ```bash + curl -X POST \ + "http://localhost:4000/v1/iceberg/v1/greptime/namespaces/public/tables/
/rebuild" + ``` + + rebuild 会替换（而非合并）当前 snapshot。重建出的条目不携带列统计信息（读取正确性不受影响；只是谓词下推 / 扫描规划会降级，直到下一次 flush 或 compaction）。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/enterprise/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/enterprise/overview.md index 86a97e414..648d40d29 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/enterprise/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/enterprise/overview.md @@ -36,6 +36,7 @@ GreptimeDB Enterprise 包括以下高级功能， - [Trigger](./trigger.md)：定时查询和检测预配置的规则，可触发外部 webhook，兼容 Prometheus AlertManager。 - [内置用户管理](./user.md)：内置 RBAC 和细粒度 ACL，确保数据安全和隔离。 - Flow 的可靠性功能。 +- [Iceberg 元数据](./iceberg.md)：通过 Apache Iceberg REST catalog 暴露 GreptimeDB 表，使 pyiceberg、Spark、Trino、DuckDB 等引擎可以直接从对象存储读取数据。 ## 发布说明 diff --git a/versioned_docs/version-1.2/enterprise/iceberg.md b/versioned_docs/version-1.2/enterprise/iceberg.md new file mode 100644 index 000000000..1a03ba862 --- /dev/null +++ b/versioned_docs/version-1.2/enterprise/iceberg.md @@ -0,0 +1,321 @@ +--- +keywords: [Iceberg, REST catalog, pyiceberg, Spark, parquet, data export, open table format] +description: Read GreptimeDB tables through an Apache Iceberg REST catalog with pyiceberg, Spark, Trino, DuckDB, and other engines — no connector or data copy required. +--- + +# Iceberg Export + +GreptimeDB Enterprise can expose its tables through an [Apache Iceberg](https://iceberg.apache.org/) REST catalog, +so external query engines — pyiceberg, Spark, Trino, DuckDB, and any other Iceberg-compatible client — can read +GreptimeDB data directly from object storage using standard Iceberg APIs. + +This is a **read-only export**, not a copy or a dual-write. GreptimeDB keeps writing data the way it always has; +the Iceberg integration simply publishes the metadata that lets other engines find and interpret those same files. + +## How it works + +The key idea is that GreptimeDB already stores its SST (sorted string table) data files as Parquet in object storage. +The Iceberg integration does **not** re-write, duplicate, or export the data. Instead it publishes Iceberg +**metadata** — manifests, manifest-lists, and table-metadata snapshots — that point at the existing Parquet files. +Any engine that speaks Iceberg can then read those files through the REST catalog. + +The following diagram shows how the pieces fit together: + +```mermaid +flowchart LR + APP["Client applications
(MySQL / PostgreSQL protocol)"] + + subgraph GT["GreptimeDB Enterprise"] + direction TB + FE["Frontend
Iceberg REST catalog (/v1/iceberg)"] + DN["Datanode / standalone
(the writer)"] + end + + subgraph OS["Object storage
(S3 / OSS / GCS / Azure Blob)"] + DATA["Parquet SST data files"] + META["Iceberg metadata
(manifests, snapshots)
under warehouse_root"] + end + + ENG["pyiceberg / Spark / Trino / DuckDB
(any Iceberg-compatible engine)"] + + APP -->|"writes"| FE + FE --> DN + DN -->|"writes SST files on
flush / compaction"| DATA + DN -->|"publishes metadata,
commits a new snapshot"| META + ENG -->|"1. load table metadata"| FE + ENG -->|"2. read Parquet files directly"| DATA + META -.->|"points at"| DATA +``` + +This is split across the GreptimeDB processes you already run: + +- **Datanode / standalone (the writer).** Whenever SST files are written — on flush, compaction, bulk ingestion, + and truncate — GreptimeDB translates the live set of Parquet files into Iceberg manifest entries and commits a + new Iceberg snapshot. The Iceberg metadata is written under a `warehouse_root` prefix inside the same + object-storage bucket the datanode already uses. +- **Frontend (the catalog server).** It implements the [Iceberg REST Catalog API](https://iceberg.apache.org/docs/1.6.0/api/#rest-catalog-specification), + mounted at `/v1/iceberg`, and serves each table's current metadata to clients. + +Because the data files are never duplicated, there is no extra storage cost and no write-path duplication — the +export is pure metadata laid down beside the data GreptimeDB already writes. + +### How GreptimeDB concepts map to Iceberg + +| GreptimeDB | Iceberg | +| ---------- | ------- | +| Schema (database, e.g. `public`) | Namespace | +| Table | Table | +| Column | Field | +| Time index column | A `timestamptz` field | + +Schema changes from `ALTER TABLE` (added / renamed / dropped columns) are reflected in the Iceberg schema without +any restart. + +### Type mapping + +GreptimeDB column types map to Iceberg types as follows: + +| GreptimeDB type | Iceberg type | +| --------------- | ------------ | +| `boolean` | `boolean` | +| `int8`, `int16`, `int32` | `int` | +| `int64` | `long` | +| `float32` | `float` | +| `float64` | `double` | +| `string` | `string` | +| `binary` | `binary` | +| `date` | `date` | +| `timestamp` (any precision) | `timestamptz` | +| Prometheus native histogram (struct) | `struct` (with `list` sub-fields) | +| `uint8`, `uint16`, `uint32`, `uint64` | `long` — **not readable in Spark** (see below) | +| `list`, `dictionary`, `json`, `interval`, `duration`, `time`, arbitrary `struct` | `string` (lossy fallback) | + +## Configuration + +The Iceberg integration is an enterprise plugin. Enable it by adding an `iceberg_manifest` entry to the `[[plugins]]` +section of **both** the data-writing process (datanode or standalone) **and** the frontend: + +- The **datanode / standalone** runs the writer hook that publishes Iceberg metadata. +- The **frontend** runs the REST catalog that serves it. + +Both must reference the **same** `warehouse_root` so the catalog reads exactly the metadata the writer publishes. + +```toml +## Iceberg manifest export publishes Iceberg-format metadata (snapshots, +## manifests, manifest-lists) so external engines (pyiceberg, Spark, Trino, +## DuckDB, ...) can read GreptimeDB tables through the Iceberg REST catalog. +[[plugins]] +iceberg_manifest = { warehouse_root = "iceberg_warehouse" } +``` + +The options are: + +| Option | Default | Description | +| ------ | ------- | ----------- | +| `warehouse_root` | `"iceberg_warehouse"` | Path prefix, inside the datanode's object-storage bucket, where Iceberg metadata is stored. | +| `enable_incremental` | `true` | When `true` (the default), a new snapshot is published automatically on every flush / compaction / truncate. Set `false` to disable automatic publication and generate metadata on demand through the rebuild interface instead; drop/GC cleanup still runs. | + +The `warehouse_root` is a path prefix inside the object-storage bucket GreptimeDB already uses, folded into the +store's `root` (e.g. `s3://///`). + +The supported object-storage backends are S3, OSS, GCS, and Azure Blob. + +Once enabled, the REST catalog is available at: + +``` +http://:/v1/iceberg +``` + +For example, with a frontend listening on the default HTTP port `4000`, the catalog base URI is +`http://localhost:4000/v1/iceberg` and the warehouse name is `greptime` (the default). + +## Make a table readable + +GreptimeDB publishes Iceberg metadata on flush, so the export of recently written data appears after the next flush +(or compaction). To expose freshly-ingested rows immediately, flush the table manually: + +```sql +-- via the MySQL or PostgreSQL protocol +admin flush_table('your_table'); +``` + +The metadata is published asynchronously; the table becomes queryable through the REST catalog shortly after the +flush returns. Existing, already-flushed tables are exported automatically. + +## Reading with pyiceberg + +This example uses [pyiceberg](https://py.iceberg.apache.org/) with pyarrow to read a GreptimeDB table through the +REST catalog. You will need `pyiceberg` and `pyarrow` installed, plus credentials for the object storage that backs +your GreptimeDB deployment (S3 is shown here). + +```python +from pyiceberg.catalog.rest import RestCatalog + +# The REST catalog endpoint exposed by the GreptimeDB frontend. +# `warehouse` is the catalog name (the default catalog is "greptime"). +catalog = RestCatalog( + name="greptime", + uri="http://localhost:4000/v1/iceberg", + prefix="greptime", + # Object-storage credentials so pyiceberg can read the Parquet data files. + # Match the backend your GreptimeDB deployment uses. + **{ + "s3.endpoint": "https://s3.us-east-1.amazonaws.com", + "s3.access-key-id": "YOUR_ACCESS_KEY", + "s3.secret-access-key": "YOUR_SECRET_KEY", + "s3.region": "us-east-1", + }, +) + +# List namespaces (= GreptimeDB schemas) and tables. +print(catalog.list_namespaces()) # e.g. [('public',)] +print(catalog.list_tables("public")) # e.g. [('public', 'my_table')] + +# Load a table and scan it with pyarrow. +table = catalog.load_table(("public", "my_table")) +print(table.schema()) # the Iceberg schema (GreptimeDB columns → fields) + +arrow_table = table.scan().to_arrow() +print(arrow_table.num_rows, "rows") +print(arrow_table.to_pandas().head()) +``` + +## Reading with Spark + +This example configures Spark SQL to use the GreptimeDB Iceberg REST catalog and run queries over a GreptimeDB +table. You need the Iceberg Spark runtime and the AWS bundle JARs on the Spark classpath (versions must match your +Spark/Scala version — Spark 4.x with Iceberg 1.11.0 is shown). + +**1. `spark-defaults.conf`** + +```properties +spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions +spark.sql.defaultCatalog greptime +spark.sql.catalog.greptime org.apache.iceberg.spark.SparkCatalog +spark.sql.catalog.greptime.type rest +spark.sql.catalog.greptime.uri http://localhost:4000/v1/iceberg +spark.sql.catalog.greptime.warehouse greptime +# S3FileIO reads the Parquet data files from the same object storage GreptimeDB uses. +spark.sql.catalog.greptime.io-impl org.apache.iceberg.aws.s3.S3FileIO +spark.sql.catalog.greptime.client.region us-east-1 +spark.sql.catalog.greptime.s3.endpoint https://s3.us-east-1.amazonaws.com +spark.sql.catalog.greptime.s3.path-style-access false +spark.sql.catalog.greptime.s3.access-key-id YOUR_ACCESS_KEY +spark.sql.catalog.greptime.s3.secret-access-key YOUR_SECRET_KEY +``` + +Start `spark-sql` (or a `spark-submit` job) with the two Iceberg JARs on the classpath: + +```bash +spark-sql \ + --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \ + --jars iceberg-spark-runtime-4.1_2.13-1.11.0.jar,iceberg-aws-bundle-1.11.0.jar +``` + +For an S3-compatible store with a custom endpoint and path-style addressing (for example MinIO, Garage, or a local +test deployment), set `spark.sql.catalog.greptime.s3.endpoint` to the store URL and +`spark.sql.catalog.greptime.s3.path-style-access true`. + +**2. Create and populate a table in GreptimeDB** + +Connect to GreptimeDB over the MySQL or PostgreSQL protocol and create a table, then flush it so the Iceberg +metadata is published: + +```sql +CREATE TABLE demo ( + ts TIMESTAMP(6) NOT NULL, -- TIME INDEX + host STRING, + region STRING, + cpu DOUBLE, + mem FLOAT, + status INT, + TIME INDEX (ts) +); + +INSERT INTO demo VALUES + ('2024-01-01 00:00:00', 'h1', 'us', 12.5, 4096.0, 200), + ('2024-01-01 01:00:00', 'h1', 'us', 88.8, 7000.5, 200), + ('2024-01-01 02:00:00', 'h2', 'eu', 55.0, 5500.0, 503); + +-- Publish Iceberg metadata for the rows just written. +admin flush_table('demo'); +``` + +**3. Query it from Spark SQL** + +```sql +-- The catalog is `greptime`, the namespace is the GreptimeDB schema `public`. +SHOW TABLES IN greptime.public; + +SELECT count(*) FROM greptime.public.demo; + +SELECT host, round(avg(cpu), 1) AS avg_cpu +FROM greptime.public.demo +WHERE ts >= '2024-01-01 00:00:00' +GROUP BY host +ORDER BY host; +``` + +The table is fully queryable: time-range filters, aggregates, joins, window functions, and ordering all work as +with any other Iceberg table in Spark. + +## Limitations + +### Read-only, single snapshot + +- **Read-only.** You can read GreptimeDB tables through Iceberg, but you cannot write back to GreptimeDB through + the Iceberg catalog. GreptimeDB remains the sole writer. +- **Only the latest snapshot (no time travel).** Compaction physically deletes old data files, so historical + snapshots are not retained. The Iceberg table always reflects the current live data; you cannot query previous + snapshots or roll back. +- **No full schema history.** The current schema is always exposed, but past schema versions are not retained, so + you cannot reconstruct what a table looked like at an earlier point in time. + +### Data types and Spark + +GreptimeDB types map cleanly to Iceberg for the common cases (booleans, signed integers, floats, strings, binary, +date, timestamp). A few things to be aware of, especially in Spark: + +- **Declare the time index as `TIMESTAMP(6)`.** GreptimeDB's default `TIMESTAMP` is millisecond precision, but the + Iceberg schema declares the column as `timestamptz` (microsecond). With a millisecond column, Spark's Parquet + row-group statistics filtering compares microsecond predicates against millisecond file stats and can incorrectly + drop row groups for `>`, `=`, and range queries. Declaring the time index as `TIMESTAMP(6)` makes the on-disk + Parquet microsecond precision match the schema, and all comparison operators work correctly. (Second/millisecond + values are still stored correctly; the issue is purely predicate pushdown against file statistics.) +- **Lossy type fallbacks.** `list`, `dictionary`, `json`, `interval`, `duration`, `time`, and arbitrary user + `struct` types are exported as Iceberg `string` rather than a structured type, so their internal structure is not + queryable through Iceberg. +- **Unsigned integers are not readable in Spark.** GreptimeDB unsigned integer columns (`uint8`, `uint16`, + `uint32`, `uint64`) are declared as Iceberg `long`, but the underlying Parquet files store them with unsigned + physical types, which Spark cannot read. Any scan that touches an unsigned-integer column fails in Spark. If you + intend to query a table through Iceberg, avoid unsigned types or store the values as signed types instead. + +### Metric engine tables + +- **Only physical metric tables are exported.** Logical metric tables are not exposed through Iceberg; query the + physical table directly. +- **Sparse primary-key encoding (the metric-engine default).** All tag columns are folded into a single + `__primary_key` binary column. To recover individual tag values (the logical table id, tsid, and labels) you must + decode that blob according to the metric-engine sparse codec — Iceberg exposes it as opaque `binary`. The + metric-internal `__table_id` and `__tsid` columns are **not** exported, so you cannot resolve a metric name to its + physical table id through Iceberg. +- **Prometheus native histograms** are exported as an Iceberg `struct` with list sub-fields, so each histogram + value is read as a nested struct rather than a scalar. + +### Operational notes + +- **Metadata is published asynchronously.** Freshly written rows appear in Iceberg after the next flush or + compaction; flush a table manually with `admin flush_table('
')` to expose them immediately. +- **Old Iceberg metadata is garbage-collected** alongside the data files by GreptimeDB's normal compaction and + GC — no separate maintenance is required. +- **Rebuild / reconcile.** If the Iceberg export ever diverges from GreptimeDB's ground truth (a failed publish, + corruption, or tables created before the integration was enabled), an operator can rebuild a table's Iceberg + metadata from scratch from the authoritative live SST set: + + ```bash + curl -X POST \ + "http://localhost:4000/v1/iceberg/v1/greptime/namespaces/public/tables/
/rebuild" + ``` + + The rebuild replaces (not merges) the current snapshot. Rebuilt entries carry empty column statistics (read + correctness is unaffected; only predicate pushdown / scan planning degrades until the next flush or compaction). diff --git a/versioned_docs/version-1.2/enterprise/overview.md b/versioned_docs/version-1.2/enterprise/overview.md index b617d7f35..224cc5aa1 100644 --- a/versioned_docs/version-1.2/enterprise/overview.md +++ b/versioned_docs/version-1.2/enterprise/overview.md @@ -45,6 +45,8 @@ which are described in detail in the documentation in this section: - [Built-in User Management](./user.md): Built-in RBAC and fine-grained ACLs for data security and isolation. - Reliability features for Flow. +- [Iceberg Metadata](./iceberg.md): Expose GreptimeDB tables through an Apache Iceberg REST catalog + so pyiceberg, Spark, Trino, DuckDB, and other engines can read the data directly from object storage. ## Release Notes diff --git a/versioned_sidebars/version-1.2-sidebars.json b/versioned_sidebars/version-1.2-sidebars.json index 983fd9a25..649aa9cac 100644 --- a/versioned_sidebars/version-1.2-sidebars.json +++ b/versioned_sidebars/version-1.2-sidebars.json @@ -545,6 +545,7 @@ "enterprise/read-replicas/query-read-replicas" ] }, + "enterprise/iceberg", "enterprise/trigger", "enterprise/soft-drop", "enterprise/user",