diff --git a/.gitignore b/.gitignore
index 96028f1992..2b5e33dd3a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -35,3 +35,6 @@ static/skills/
# Used by Codex to store temporary artifacts
tmp/
+
+# Local MCP server configuration. Contains connection credentials.
+/.mcp.json
diff --git a/docs/contributor-guide/datanode/data-persistence-indexing.md b/docs/contributor-guide/datanode/data-persistence-indexing.md
index 6403ffc2df..1fa5d6d258 100644
--- a/docs/contributor-guide/datanode/data-persistence-indexing.md
+++ b/docs/contributor-guide/datanode/data-persistence-indexing.md
@@ -5,19 +5,23 @@ description: Explanation of data persistence and indexing in GreptimeDB, includi
# Data Persistence and Indexing
-Similar to all LSMT-like storage engines, data in MemTables is persisted to durable storage, for example, the local disk file system or object storage service. GreptimeDB adopts [Apache Parquet][1] as its persistent file format.
+Like other LSM-tree storage engines, GreptimeDB persists data from memtables to durable storage such as a local filesystem or object storage. It uses [Apache Parquet][1] as the persistent file format.
## SST File Format
Parquet is an open source columnar format that provides fast data querying and has already been adopted by many projects, such as Delta Lake.
-Parquet has a hierarchical structure like "row groups-columns-data pages". Data in a Parquet file is horizontally partitioned into row groups, in which all values of the same column are stored together to form a data page. Data page is the minimal storage unit. This structure greatly improves performance.
+Parquet organizes data as row groups, column chunks, and pages. A row group contains one column chunk for each column, and each column chunk contains one or more pages. Pages are the units of encoding and compression; column chunks are the I/O units for reading selected columns.
First, clustering data by column makes file scanning more efficient, especially when only a few columns are queried, which is very common in analytical systems.
-Second, data of the same column tends to be homogeneous which helps with compression when apply techniques like dictionary and Run-Length Encoding (RLE).
+Second, values within a column tend to be similar, which improves compression with techniques such as dictionary encoding and run-length encoding (RLE).
-
+The following diagram from the Apache Parquet specification also shows the physical file layout: column chunks are stored by row group, while file metadata and its length are written in the footer.
+
+
+
+*Source: Apache Parquet [FileLayout.gif](https://github.com/apache/parquet-format/blob/master/doc/images/FileLayout.gif). Copyright 2014 The Apache Software Foundation, licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).*
## Data Persistence
@@ -27,17 +31,17 @@ When the size of data buffered in MemTables reaches that threshold, GreptimeDB w
## Indexing Data in SST Files
-Apache Parquet file format provides inherent statistics in headers of column chunks and data pages, which are used for pruning and skipping.
+Parquet stores row-group column statistics such as minimum, maximum, and null count in each column chunk's metadata. Page metadata and optional column indexes can provide finer-grained statistics.
-
+
-For example, in the above Parquet file, if you want to filter rows where `name` = `Emily`, you can easily skip row group 0 because the max value for `name` field is `Charlie`. This statistical information reduces IO operations.
+For example, a query filtering for `name` = `Emily` can skip row group 0 because the maximum `name` value is `Charlie`. This avoids reading that row group.
## Index Files
-For each SST file, GreptimeDB not only maintains an internal index but also generates a separate file to store the index structures specific to that SST file.
+When an SST has one or more configured index outputs, GreptimeDB writes them to a Puffin file associated with that SST. An SST with no applicable index does not need a Puffin file.
-The index files utilize the [Puffin][3] format, which offers significant flexibility, allowing for the storage of additional metadata and supporting a broader range of index structures.
+Puffin provides a container for index blobs and their metadata, allowing different index structures to share one file.
@@ -57,13 +61,13 @@ The inverted index enables GreptimeDB to skip data segments that do not meet que
-For instance, the query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches the regex `.*users`, and `status` matches the regex `4...`. It then scans these data segments to produce the final results that meet all conditions, significantly reducing the number of IO operations.
+The query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches `.*users`, and `status` matches `4..`. It scans only those segments before applying the remaining filters.
### Inverted Index Format
-
+
-GreptimeDB builds inverted indexes by column, with each inverted index consisting of an FST and multiple Bitmaps.
+GreptimeDB builds inverted indexes by column. Each column index contains a null bitmap, multiple posting bitmaps, and an FST. The blob footer records the offsets, sizes, and metadata needed to locate and decode the column indexes.
The FST (Finite State Transducer) enables GreptimeDB to store mappings from column values to Bitmap positions in a compact format and provides excellent search performance and supports complex search capabilities (such as regular expression matching). The Bitmaps maintain a list of data segment IDs, with each bit representing a data segment.
@@ -77,7 +81,7 @@ The number of rows in a data segment is controlled by the engine option `index.i
## Unified Data Access Layer: OpenDAL
-GreptimeDB uses [OpenDAL][2] to provide a unified data access layer, thus, the storage engine does not need to interact with different storage APIs, and data can be migrated to cloud-based storage like AWS S3 seamlessly.
+GreptimeDB uses [OpenDAL][2] to provide a common access layer for local filesystems and object stores. Changing the configured storage backend does not migrate existing data.
[1]: https://parquet.apache.org
[2]: https://github.com/datafuselabs/opendal
diff --git a/docs/contributor-guide/datanode/memtable.md b/docs/contributor-guide/datanode/memtable.md
new file mode 100644
index 0000000000..92e3eb58de
--- /dev/null
+++ b/docs/contributor-guide/datanode/memtable.md
@@ -0,0 +1,100 @@
+---
+keywords: [memtable, Mito engine, write buffer, flush, time partition, BulkMemtable]
+description: How Mito organizes mutable Region data in memtables and moves it into SST files.
+---
+
+# Memtable design
+
+A memtable is Mito's in-memory write buffer for a Region. It makes writes available to reads before a flush creates SST files. A Region version identifies the memtables and SST files that a scan may read. Together with a committed-sequence fence, it keeps the scan consistent while writes and flushes advance the current version.
+
+## Write and flush lifecycle
+
+For a normal WAL-backed write, Mito uses this order:
+
+```text
+write request
+ |
+ v
+WAL append -> mutable memtable -> publish committed sequence
+ |
+ freeze
+ v
+ immutable memtable -> SST write -> manifest edit
+```
+
+The Region worker assigns sequence numbers and a WAL entry ID before appending the mutation to the [write-ahead log](wal.md). If the WAL append fails, Mito does not update the memtable. After the memtable update succeeds, Mito publishes the committed sequence and the rows become visible to new reads. A Region configured with `skip_wal` omits the WAL append, but keeps the same memtable and visibility ordering.
+
+A flush freezes the mutable memtables and installs a new mutable set before starting the background SST write. New writes therefore continue without changing the frozen data. The flush writes the immutable memtables to SST files, then persists a manifest edit containing the files and the flushed WAL and sequence checkpoints. Only after that edit is durable does Mito remove the flushed memtables from the current Region version. If the flush fails, the immutable memtables remain available for a later attempt.
+
+## Region versions and time partitions
+
+Each Region has one mutable `TimePartitions` container, which can hold more than one memtable:
+
+```text
+Region version
+├─ mutable TimePartitions
+│ ├─ [t0, t1) -> memtable
+│ └─ [t1, t2) -> memtable
+├─ immutable memtables
+└─ SST files
+```
+
+Mito routes each row to a partition by its time-index value. Partition ranges are half-open and aligned to a fixed duration. The duration follows the Region's compaction time window; Mito uses one day until a compaction window is available. An out-of-order write can create an earlier partition alongside the latest one.
+
+Freezing a Region freezes all mutable time partitions together. Mito moves their memtables to the immutable list and creates a new `TimePartitions` container. A failed flush can leave more than one generation of immutable memtables, so reads and later flushes must not assume that the list contains a single item.
+
+## Memtable implementations
+
+Mito selects a memtable implementation from the Region's SST format, primary-key encoding, and memtable options:
+
+```text
+flat SST format (the default) or sparse primary-key encoding -> BulkMemtable
+memtable.type=bulk -> BulkMemtable, and forces flat SST
+primary_key SST with dense encoding (legacy) -> a legacy implementation
+```
+
+With the default engine configuration, a Region without an explicit SST format uses `flat`, so `BulkMemtable` is the normal path and the rest of this page describes it. The rules exist to prevent incompatible combinations: flat format or sparse primary-key encoding requires `BulkMemtable`, and explicitly selecting the bulk implementation forces flat format.
+
+### BulkMemtable
+
+`BulkMemtable` stores writes as parts in the flat Arrow layout instead of inserting rows into per-series buffers:
+
+```text
+BulkMemtable
+├─ unordered_part
+│ └─ small BulkPart batches
+└─ parts
+ ├─ BulkPart (Arrow RecordBatch)
+ ├─ MultiBulkPart (raw RecordBatches)
+ └─ EncodedBulkPart (in-memory Parquet)
+```
+
+Small parts accumulate in `unordered_part`; larger parts enter `parts` directly. Background memtable compaction merge-sorts eligible parts into a `MultiBulkPart` or encodes them as an `EncodedBulkPart`. Scans use part statistics to prune ranges, and flush can write encoded ranges to SST without decoding and encoding the rows again. For the design rationale and performance results, see [Scaling Time Series to Millions of Cardinalities: GreptimeDB's Flat Format](https://greptime.com/blogs/2025-12-22-flat-format).
+
+### Legacy implementations
+
+Regions on the legacy `primary_key` SST format with dense primary-key encoding still use `TimeSeriesMemtable`, which groups rows by encoded primary key rather than storing flat parts. A Region with no primary-key columns gets `SimpleBulkMemtable` from the same builder. Both are compatibility code for existing tables and may be removed once the `primary_key` format is retired; new work targets the bulk and flat path.
+
+The removed `partition_tree` memtable is not a third implementation. The option parser accepts `memtable.type=partition_tree` for compatibility, but it does not recreate that implementation. The Region uses the bulk and flat path.
+
+## Read snapshots
+
+A scan obtains the Region version and committed sequence together from `VersionControl`. It selects the version before applying the sequence fence. Reading the sequence separately before the version could pair that sequence with a later version after flush or compaction removes an older input, producing an incomplete snapshot.
+
+The selected version supplies mutable memtables, immutable memtables, and SST files. Mito first prunes sources by time range, then asks each memtable for ranges using the scan's projection, predicate, and sequence bounds. The scan merges the resulting ranges with SST ranges and applies the same ordering, deletion, and merge semantics across all sources. References held by the scan keep an older memtable alive even after a newer Region version removes it.
+
+## Memory pressure
+
+Each memtable tracks its estimated heap allocation through the engine's write-buffer manager. Freezing a memtable removes its allocation from the mutable-memory count, but total usage includes the allocation until all references to that memtable are released. The mutable-memory count therefore tracks data that can still accept writes, while total usage continues to include memory retained by active scans.
+
+The global write-buffer limit causes workers to select Regions for flush. If memory remains above the configured limits, Mito stalls writes and can reject them at a higher threshold. An optional per-Region limit applies the same pressure to one hot Region without stalling unrelated Regions. Periodic, manual, and Region lifecycle operations can also request a flush.
+
+## Constraints for changes
+
+Changes to memtable code must preserve these properties:
+
+- For a WAL-backed Region, append to the WAL before installing rows in a memtable. Publish the committed sequence only after installation succeeds.
+- Keep frozen memtables readable and retryable until the SST files and manifest edit are durable.
+- Obtain the Region version and committed sequence from the same `VersionControl` snapshot; never read the sequence separately before the version.
+- Preserve the ordering and metadata that scans and flushes need to apply the same deletion, deduplication, and merge rules across memtable and SST ranges.
+- Charge allocations to the write-buffer manager and release them only when the underlying memory can no longer be referenced.
diff --git a/docs/contributor-guide/datanode/metric-engine.md b/docs/contributor-guide/datanode/metric-engine.md
index 064872ce14..fde3c178a3 100644
--- a/docs/contributor-guide/datanode/metric-engine.md
+++ b/docs/contributor-guide/datanode/metric-engine.md
@@ -7,9 +7,9 @@ description: Overview of the Metric engine in GreptimeDB, its concepts, architec
## Overview
-The `Metric` engine is a component of GreptimeDB, and it's an implementation of the storage engine. It mainly targets scenarios with a large number of small tables for observable metrics.
+The `Metric` engine stores workloads with many small metric tables.
-Its main feature is to use synthetic physical wide tables to store a large amount of small table data, achieving effects such as reuse of the same column and metadata. This reduces storage overhead for small tables and improves columnar compression efficiency. The concept of a table becomes even more lightweight under the `Metric` engine.
+It maps those logical tables onto shared physical wide tables so they can reuse columns and metadata. This reduces per-table storage overhead and improves columnar compression.
## Concepts
@@ -19,7 +19,7 @@ The `Metric` engine introduces two new concepts: "logical table" and "physical t
A logical table refers to user-defined tables. Just like any other ordinary table, its definition includes the name of the table, column definitions, index definitions etc. All operations such as queries or write-ins by users are based on these logical tables. Users don't need to worry about differences between logical and ordinary tables during usage.
-From an implementation standpoint, a logical table is virtual; it doesn't directly read or write physical data but maps read/write requests into corresponding requests for physical tables in order to implement data storage and querying.
+A logical table is virtual. The engine maps its read and write requests to the corresponding physical table instead of storing data for it directly.
### Physical Table
@@ -29,16 +29,14 @@ A physical table is a table that actually stores data, possessing several physic
The main design architecture of the `Metric` engine is as follows:
-
+
-In the current version implementation, the `Metric` engine reuses the `Mito` engine to achieve storage and query capabilities for physical data. It also provides access to both physical tables and logical tables simultaneously.
+The `Metric` engine delegates physical storage and queries to the `Mito` engine. Each physical Region group contains a data Region, which stores rows from its mapped logical tables, and a metadata Region, which stores the logical-table and logical-column mappings.
-Regarding partitioning, logical tables have identical partition rules and Region distribution as physical tables. This makes sense because the data of logical tables are directly stored in physical tables, so their partition rules are consistent.
+Logical tables associated with the same physical table share its partition layout. During writes, the engine records the logical table identity with each row. During reads, it adds a logical-table filter before scanning the physical Region.
-Concerning routing metadata, the routing address of a logical table is a logical address - what its corresponding physical table is - then through this physical table for secondary routing to obtain the real physical address. This indirect routing method can significantly reduce the number of metadata modifications required when Region migration scheduling occurs in Metric engines.
+A logical table's route stores only the ID of its physical table; the physical table route resolves that to the Datanodes holding the Regions. Because logical routes do not name peers, migrating a physical Region rewrites one physical route instead of every logical route that maps to it.
-Operationally speaking, The `Metric` engine supports standard DML operations (INSERT, DELETE, SELECT) on logical tables. However, it only supports limited operations on physical tables to prevent misoperations - for example, writing directly to a physical table is prohibited as it could affect user's logical table data. Generally speaking, users can consider that they have read-only access to these physical tables.
+Logical tables support normal INSERT, DELETE, and SELECT operations. Direct writes to a physical Region are rejected because they would bypass the logical-table mapping; querying a physical table remains supported.
-To improve performance during simultaneous DDL (Data Definition Language) operations on many tables, the 'Metric' engine has introduced some batch DDL operations. These batch DDL operations can merge lots of DDL actions into one request thereby reducing queries and modifications times for metadata thus enhancing performance. This feature is particularly beneficial in scenarios such as the automatic creation requests brought about by large amounts of metrics during Prometheus Remote Write cold start-up, as well as the modification requests for numerous route-tables mentioned earlier during migration of many physical regions.
-
-Apart from physical data regions belonging to physical tables, the 'Metric' engine creates an additional metadata region physically for each individual physical data region used in storing some metadata needed by itself while maintaining mapping and other states. This metadata includes the mapping relationship between logical tables and physical tables, the mapping relationship between logical columns and physical columns etc.
+Batch DDL operations reduce metadata work when many logical tables are created or updated together, such as during Prometheus Remote Write auto-creation or physical Region migration.
diff --git a/docs/contributor-guide/datanode/overview.md b/docs/contributor-guide/datanode/overview.md
index d0afe21b34..a21c112faa 100644
--- a/docs/contributor-guide/datanode/overview.md
+++ b/docs/contributor-guide/datanode/overview.md
@@ -7,28 +7,26 @@ description: Overview of Datanode in GreptimeDB, its responsibilities, component
## Introduction
-`Datanode` is mainly responsible for storing the actual data for GreptimeDB. As we know, in GreptimeDB,
-a `table` can have one or more `Region`s, and `Datanode` is responsible for managing the reading and writing
-of these `Region`s. `Datanode` is not aware of `table` and can be considered as a `region server`. Therefore,
-`Frontend` and `Metasrv` operate `Datanode` at the granularity of `Region`.
+A Datanode stores and processes Region data. A table can contain multiple Regions, but the Datanode does not own table-level routing. Frontend sends data requests by Region, while Metasrv controls Region placement and lifecycle.
-
+This boundary lets the same Region server host different storage engines without exposing their implementation to Frontend or Metasrv.
+
+
## Components
-A `Datanode` contains all the components needed for a `region server`. Here we list some of the vital parts:
-
-- A gRPC service is provided for reading and writing region data, and `Frontend` uses this service
- to read and write data from `Datanode`s.
-- An HTTP service, through which you can obtain metrics, configuration information, etc., of the current node.
-- `Heartbeat Task` is used to send heartbeat to the `Metasrv`. The heartbeat plays a crucial role in the
- distributed architecture of GreptimeDB and serves as a basic communication channel for distributed coordination.
- The upstream heartbeat messages contain important information such as the workload of a `Region`. If the
- `Metasrv `has made scheduling(such as `Region` migration) decisions, it will send instructions to the
- `Datanode` via downstream heartbeat messages.
-- The `Datanode` does not parse user SQL or perform distributed planning. The user's query requests for one or
- more `Table`s will be transformed into `Region` query requests in the `Frontend`. The `Datanode` is responsible
- for executing these `Region` query plans with its local query engine.
-- A `Region Manager` is used to manage all `Region`s on a `Datanode`.
-- GreptimeDB supports a pluggable multi-engine architecture, with existing engines including `File Engine` and
- `Mito Engine`.
+The main components are:
+
+- The Region server tracks open Regions and dispatches reads, writes, and lifecycle requests to the engine registered for each Region.
+- `Mito` is the primary time-series Region engine. `Metric` maps many logical metric Regions onto shared Mito Regions, and `File` exposes external files through the Region interface.
+- The local query engine executes Region query plans. It does not parse client SQL or perform cluster-wide planning.
+- The heartbeat task reports node and Region state to Metasrv and receives instructions such as open, close, upgrade, downgrade, and migration steps.
+- gRPC carries Region requests to the Datanode. HTTP exposes node diagnostics such as metrics and configuration.
+
+## Region Request Lifecycle
+
+For a Mito write, the Region server selects Mito from the Region metadata. Mito appends the mutation to the WAL, applies it to a memtable, and later flushes the memtable to SST files. A Metric write is first rewritten with the logical-table identity and then delegated to its physical Mito Region.
+
+For a read, the local query engine executes the Region plan against a table provider backed by the Region engine. A Mito scan takes an immutable Region version, reads the relevant memtables and SST files, merges and deduplicates rows, and returns a stream of Arrow record batches.
+
+Region ownership can change without restarting the Datanode. Metasrv sends lifecycle instructions over the heartbeat stream; the Region server applies them to the engine and reports the new Region role and statistics in subsequent heartbeats.
diff --git a/docs/contributor-guide/datanode/python-scripts.md b/docs/contributor-guide/datanode/python-scripts.md
deleted file mode 100644
index 98909142a6..0000000000
--- a/docs/contributor-guide/datanode/python-scripts.md
+++ /dev/null
@@ -1,35 +0,0 @@
----
-keywords: [Python scripts, data analysis, CPython backend, RustPython interpreter, RecordBatch]
-description: Guide on using Python scripts for data analysis in GreptimeDB, including backend options and setup instructions.
----
-
-# Python Scripts
-
-## Introduction
-
-Python scripts are methods for analyzing data in GreptimeDB,
-by running it in the database directly instead of fetching all the data from the database and running it locally.
-This approach saves a lot of data transfer costs.
-The image below depicts how the script works.
-The `RecordBatch` (which is basically a column in a table with type and nullability metadata)
-can come from anywhere in the database,
-and the returned `RecordBatch` can be annotated in Python grammar to indicate its metadata,
-such as type or nullability.
-The script will do its best to convert the returned object to a `RecordBatch`,
-whether it is a Python list, a `RecordBatch` computed from parameters,
-or a constant (which is extended to the same length as the input arguments).
-
-
-
-## Two optional backends
-
-### CPython Backend powered by PyO3
-
-This backend is powered by [PyO3](https://pyo3.rs/v0.18.1/), enabling the use of your favourite Python libraries (such as NumPy, Pandas, etc.) and allowing Conda to manage your Python environment.
-
-But using it also involves some complications. You must set up the correct Python shared library, which can be a bit challenging. In general, you just need to install the `python-dev` package. However, if you are using Homebrew to install Python on macOS, you must create a proper soft link to `Library/Frameworks/Python.framework`. Detailed instructions on using PyO3 crate with different Python Version can be found [here](https://pyo3.rs/v0.18.1/building_and_distribution#configuring-the-python-version)
-
-### Embedded RustPython Interpreter
-
-An experiment [python interpreter](https://github.com/RustPython/RustPython) to run
-the coprocessor script, it supports Python 3.10 grammar. You can use all the very Python syntax, see [User Guide/Python Coprocessor](/user-guide/python-scripts/overview.md) for more!
diff --git a/docs/contributor-guide/datanode/query-engine.md b/docs/contributor-guide/datanode/query-engine.md
index 5b83a30613..91f307aa59 100644
--- a/docs/contributor-guide/datanode/query-engine.md
+++ b/docs/contributor-guide/datanode/query-engine.md
@@ -7,51 +7,30 @@ description: Overview of GreptimeDB's query engine, its architecture, data repre
## Introduction
-GreptimeDB's query engine is built on [Apache DataFusion][1] (subproject under [Apache
-Arrow][2]), a brilliant query engine written in Rust. It provides a set of well functional components from
-logical plan, physical plan and the execution runtime. Below explains how each component is orchestrated and their positions during execution.
+GreptimeDB's query engine is built on [Apache DataFusion][1]. DataFusion supplies the logical and physical plan interfaces, optimizer framework, and execution runtime. GreptimeDB adds planners for its query languages, storage-aware optimizer rules, custom plan nodes, and distributed execution.
-
+DDL and other control-plane operations are dispatched by the statement executor. The query engine receives plans for data processing, including the input side of operations such as `INSERT ... SELECT`.
-The entry point is the logical plan, which is used as the general intermediate representation of a
-query or execution logic etc. Two noticeable sources of logical plan are from: 1. the user query, like
-SQL through SQL parser and planner; 2. the Frontend's distributed query, which is explained in details in the following section.
+## Query Lifecycle
-Next is the physical plan, or the execution plan. Unlike the logical plan which is a big
-enumeration containing all the logical plan variants (except the special extension plan node), the
-physical plan is in fact a trait that defines a group of methods invoked during
-execution. All data processing logics are packed in corresponding structures that
-implement the trait. They are the actual operations performed on the data, like
-aggregator `MIN` or `AVG`, and table scan `SELECT ... FROM`.
+1. The SQL, PromQL, or log-query planner resolves tables through the catalog and produces a DataFusion logical plan. GreptimeDB plan extensions represent operations that DataFusion does not provide directly.
+2. DataFusion analyzer and optimizer rules run together with GreptimeDB rules. These rules normalize expressions and types, rewrite time-range operations, push projections and filters toward scans, and introduce distributed plan nodes when required.
+3. The physical planner converts the optimized logical plan into streaming operators. GreptimeDB then applies physical rules for scan parallelism, ordering, and distributed execution.
+4. Execution pulls Arrow record batches through the physical plan. Storage scans receive the projection and predicates, and downstream operators consume the resulting stream without materializing the complete result first.
-The optimization phase which improves execution performance by transforming both logical and physical plans, is now all based on rules. It is also called, "Rule Based Optimization". Some of the rules are DataFusion native and others are customized in Greptime DB. In the future, we plan to add more
-rules and leverage the data statistics for Cost Based Optimization/CBO.
-
-The last phase "execute" is a verb, stands for the procedure that reads data from storage, performs
-calculations and generates the expected results. Although it's more abstract than previously mentioned concepts, you can just
-simply imagine it as executing a Rust async function. And it's indeed a future (stream).
-
-`EXPLAIN [VERBOSE] ` is very useful if you want to see how your SQL is represented in the logical or physical plan.
+Use [`EXPLAIN`](/reference/sql/explain.md) to inspect the logical and physical plans. `EXPLAIN ANALYZE` also executes the plan and reports runtime metrics.
## Data Representation
-GreptimeDB uses [Apache Arrow][2] as the in-memory data representation. It's column-oriented, in
-cross-platform format, and also contains many high-performance data operators. These features
-make it easy to share data in many different environments and implement calculation logic.
+GreptimeDB uses [Apache Arrow][2] record batches as its in-memory data representation. A record batch contains equal-length column arrays and a schema. Query operators exchange streams of these batches, which keeps the execution path columnar from Region scans through result encoding.
## Indexing
-In time series data, there are two important dimensions: timestamp and tag columns (or like
-primary key in a general relational database). GreptimeDB groups data in time buckets, so it's efficient
-to locate and extract data within the expected time range at a very low cost. The mainly used persistent file format [Apache Parquet][3] in GreptimeDB helps a lot -- it
-provides multi-level indices and filters that make it easy to prune data during querying. In the future, we
-will make more use of this feature, and develop our separated index to handle more complex use cases.
+Index construction and persistent index formats belong to the storage engine. The query layer supplies predicates and projections to a scan; Mito then uses time ranges, Parquet statistics, and indexes to avoid reading data that cannot match. See [Data Persistence and Indexing](./data-persistence-indexing.md).
## Distributed Execution
-Covered in [Distributed Querying][6].
+In distributed mode, the Frontend plans the cluster-wide query and Datanodes execute Region-local subplans. [`MergeScan`](../frontend/distributed-querying.md) is the boundary between those stages.
-[1]: https://github.com/apache/arrow-datafusion
+[1]: https://datafusion.apache.org/
[2]: https://arrow.apache.org/
-[3]: https://parquet.apache.org
-[6]: ../frontend/distributed-querying.md
diff --git a/docs/contributor-guide/datanode/storage-engine.md b/docs/contributor-guide/datanode/storage-engine.md
index c220c72208..36fef62833 100644
--- a/docs/contributor-guide/datanode/storage-engine.md
+++ b/docs/contributor-guide/datanode/storage-engine.md
@@ -7,7 +7,7 @@ description: Overview of the storage engine in GreptimeDB, its architecture, com
## Introduction
-The `storage engine` is responsible for storing the data of the database. Mito, based on [LSMT][1] (Log-structured Merge-tree), is the storage engine we use by default. We have made significant optimizations for handling time-series data scenarios, so mito engine is not suitable for general purposes.
+Mito is GreptimeDB's default storage engine. It uses an [LSM tree][1] and is designed for time-series workloads rather than as a general-purpose embedded storage engine.
## Architecture
@@ -23,9 +23,9 @@ The architecture is the same as a traditional LSMT engine:
media.
- Log records of the WAL can be stored on the local disk, or in a remote log service such as
Kafka (remote WAL) that implements the `Log Store` API.
-- Memtables:
- - Data is written into the `active memtable`, aka `mutable memtable` first.
- - When a `mutable memtable` is full, it will be changed to a `read-only memtable`, aka `immutable memtable`.
+- [Memtables](memtable.md):
+ - Mito routes rows by time index into mutable memtables.
+ - A flush freezes the mutable memtables, installs a new mutable set for writes, and writes the frozen memtables to SST files.
- SST
- The full name of SST, aka SSTable is `Sorted String Table`.
- `Immutable memtable` is flushed to persistent storage and produces an SST file.
@@ -103,7 +103,9 @@ Each Parquet SST is split into row groups, the unit that Parquet can read or ski
Mito supports two SST formats: `flat` and `primary_key`. `flat` is the default for new tables and works well across primary-key cardinalities, including high-cardinality keys. `primary_key` is the legacy format kept for compatibility with older tables. See [SST format](/reference/sql/create.md#create-a-table-with-sst-format) and the [table design guide](/user-guide/deployments-administration/performance-tuning/design-table.md#sst-format) for more details.
-
+
+
+An SST may span more than one compaction time window.
## Scan Pruning
diff --git a/docs/contributor-guide/datanode/wal.md b/docs/contributor-guide/datanode/wal.md
index 4ecb19ef02..8f898d91d6 100644
--- a/docs/contributor-guide/datanode/wal.md
+++ b/docs/contributor-guide/datanode/wal.md
@@ -7,30 +7,26 @@ description: Introduction to Write-Ahead Logging (WAL) in GreptimeDB, its purpos
## Introduction
-Our storage engine is inspired by the Log-structured Merge Tree (LSMT). Mutating operations are
-applied to a MemTable instead of persisting to disk, which significantly improves performance but
-also brings durability-related issues, especially when the Datanode crashes unexpectedly. Similar
-to all LSMT-like storage engines, GreptimeDB uses a write-ahead log (WAL) to ensure data durability
-and is safe from crashing.
+Mito buffers writes in [memtables](memtable.md) before flushing them to SST files. It first appends each Region's mutations to the write-ahead log (WAL), so data that has not reached an SST can be recovered.
-WAL is an append-only file group. All `INSERT` and `DELETE` operations are transformed into
-operation entries and then appended to WAL. Once operation entries are persisted to the underlying
-file, the operation can be further applied to MemTable.
+The WAL uses a common log-store abstraction with local raft-engine and remote Kafka providers.
-When the Datanode restarts, operation entries in WAL are replayed to reconstruct the correct
-in-memory state.
+## Write and Recovery Cycle
-
+The order of a normal write is:
+
+1. The Region worker assigns sequence numbers and a WAL entry ID.
+2. It appends the mutations to the WAL. If the append fails, the mutations are not applied to the memtable.
+3. After the append succeeds, Mito writes the mutations to the memtable and publishes the new committed sequence.
+4. A flush writes immutable SST files and persists a manifest edit containing the new files and `flushed_entry_id`.
+5. After the manifest edit is durable, WAL entries through `flushed_entry_id` are marked obsolete. The log store may reclaim them later.
+
+The manifest is the recovery boundary. On a normal reopen, Mito rebuilds the Region from the manifest and replays WAL entries starting at `flushed_entry_id + 1`. Region transitions may supply a later replay checkpoint, but they never replay entries before the persisted flush boundary.
## Namespace
-Namespace of WAL is used to separate entries from different tables (different regions). Append and
-read operations must provide a Namespace. Currently, region ID is used as the Namespace, because
-each region has a MemTable that needs to be reconstructed when Datanode restarts.
+WAL entries are isolated by Region, not by table. Each append and read identifies a Region namespace so one Region can be replayed or truncated independently. The local raft-engine provider uses the Region ID as its namespace ID. Kafka keeps Region identity within the provider's topic-backed log.
## Synchronous/Asynchronous flush
-By default, appending to WAL is asynchronous, which means the writer will not wait until entries are
-flushed to disk. This setting provides higher performance, but may lose data when running host shutdown unexpectedly. In the other hand, synchronous flush provides higher durability at the cost of performance.
-
-In v0.4 version, the new region worker architecture can use batching to alleviate the overhead of sync flush.
+For the local raft-engine provider, `sync_write` controls whether an append waits for the log to be synced to durable storage. It defaults to `false`. Asynchronous writes reduce latency but can lose recently acknowledged entries if the host fails before buffered data is synced. Kafka WAL durability is controlled by its producer and cluster settings instead of this local option.
diff --git a/docs/contributor-guide/flownode/arrangement.md b/docs/contributor-guide/flownode/arrangement.md
index aed75af777..8b472ea316 100644
--- a/docs/contributor-guide/flownode/arrangement.md
+++ b/docs/contributor-guide/flownode/arrangement.md
@@ -5,6 +5,8 @@ description: Details on the arrangement component in Flownode, which stores stat
# Arrangement
+This page describes state used by Flownode's legacy streaming mode. Batching mode does not use an Arrangement.
+
Arrangement stores the state in the dataflow's process. It stores the streams of update flows for further querying and updating.
The arrangement essentially stores key-value pairs with timestamps to mark their change time.
diff --git a/docs/contributor-guide/flownode/batching_mode.md b/docs/contributor-guide/flownode/batching_mode.md
index 37a695ce9c..aa8099d1ad 100644
--- a/docs/contributor-guide/flownode/batching_mode.md
+++ b/docs/contributor-guide/flownode/batching_mode.md
@@ -9,13 +9,13 @@ This guide provides a brief overview of the batching mode in `flownode`. It's in
## Overview
-The batching mode in `flownode` is designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. This is in contrast to the original streaming mode, now deprecated, where data was processed as it arrived.
+The batching mode in `flownode` is designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. This is in contrast to the legacy streaming path, which processes data as it arrives and is retained for compatibility but deprecated for new workloads.
The core idea is to:
1. Define a `flow` with a SQL query that aggregates data from a source table into a sink table.
2. The query typically includes a time window function (e.g., `date_bin`) on a timestamp column.
3. When new data is inserted into the source table, the system marks the corresponding time windows as "dirty."
-4. A background task periodically wakes up, identifies these dirty windows, and re-runs the aggregation query for those specific time ranges.
+4. A background task runs on its own cadence, consumes the pending dirty windows at its next evaluation, and re-runs the aggregation query for those time ranges.
5. The results are then inserted into the sink table, effectively updating the aggregated view.
## Architecture
@@ -39,15 +39,15 @@ A `BatchingTask` represents a single, independent data flow. Each task is associ
- **State (`TaskState`)**: This contains the dynamic, mutable state of the task, most importantly the `DirtyTimeWindows`.
- **Execution Loop**: The task runs an infinite loop (`start_executing_loop`) that:
1. Checks for a shutdown signal.
- 2. Waits for a scheduled interval or until it's woken up.
+ 2. Sleeps until its next evaluation time. A task with an evaluation schedule sleeps until the next scheduled time; an adaptive task sleeps for a polling interval derived from the time window size and the minimum refresh duration.
3. Generates a new query plan (`gen_insert_plan`) based on the current set of dirty time windows.
4. Executes the query (`execute_logical_plan`) against the database.
5. Cleans up the processed dirty windows.
### `TaskState` and `DirtyTimeWindows`
-- **`TaskState`**: This struct tracks the runtime state of a `BatchingTask`. It includes `dirty_time_windows`, which is crucial for determining what work needs to be done.
-- **`DirtyTimeWindows`**: This is a key data structure that keeps track of which time windows have received new data since the last query execution. It stores a set of non-overlapping time ranges. When a task's execution loop runs, it consults this structure to build a `WHERE` clause that filters the source table for only the dirty time windows.
+- **`TaskState`**: This struct tracks the runtime state of a `BatchingTask`, including the `dirty_time_windows` that determine its pending work.
+- **`DirtyTimeWindows`**: This data structure tracks which time windows have received new data since the last query execution. It stores a set of non-overlapping time ranges. The execution loop uses it to build a `WHERE` clause that selects only the dirty windows from the source table.
### `TimeWindowExpr`
@@ -56,15 +56,15 @@ The `TimeWindowExpr` is a helper utility for dealing with time window expression
- **Evaluation**: It can take a timestamp and evaluate the time window expression to determine the start and end of the window that the timestamp falls into.
- **Window Size**: It can also determine the size (duration) of the time window from the expression.
-This is essential for both marking windows as dirty and for generating the correct filter conditions when querying the source table.
+The same calculation is used to mark dirty windows and generate the source-table filters.
## Query Execution Flow
Here's a simplified step-by-step walkthrough of how a query is executed in batch mode:
1. **Data Ingestion**: New data is written to a source table.
-2. **Marking Dirty**: The `BatchingEngine` receives a notification about the new data. It uses the `TimeWindowExpr` associated with each relevant flow to determine which time windows are affected by the new data points. These windows are then added to the `DirtyTimeWindows` set in the corresponding `TaskState`.
-3. **Task Wake-up**: The `BatchingTask`'s execution loop wakes up, either due to its periodic schedule or because it was notified of a large backlog of dirty windows.
+2. **Marking Dirty**: The `BatchingEngine` receives a notification about the new data. It uses the `TimeWindowExpr` associated with each relevant flow to determine which time windows are affected by the new data points. These windows are then added to the `DirtyTimeWindows` set in the corresponding `TaskState`. Marking a window dirty does not wake the task.
+3. **Next Evaluation**: The `BatchingTask`'s execution loop reaches its next evaluation, either at a scheduled time or after its adaptive polling interval, and consumes the pending dirty windows.
4. **Plan Generation**: The task calls `gen_insert_plan`. This method:
- Inspects the `DirtyTimeWindows`.
- Generates a series of `OR`'d `WHERE` clauses (e.g., `(ts >= 't1' AND ts < 't2') OR (ts >= 't3' AND ts < 't4') ...`) that cover the dirty windows.
diff --git a/docs/contributor-guide/flownode/dataflow.md b/docs/contributor-guide/flownode/dataflow.md
index 000a65edb3..c876054d6d 100644
--- a/docs/contributor-guide/flownode/dataflow.md
+++ b/docs/contributor-guide/flownode/dataflow.md
@@ -1,17 +1,38 @@
---
-keywords: [dataflow module, SQL query transformation, execution plan, DAG, map and reduce operations]
-description: Explanation of the dataflow module in Flownode, its operations, internal data handling, and future enhancements.
+keywords: [Flownode, batching mode, streaming mode, dataflow, dirty time windows]
+description: How Flownode selects and runs its batching and legacy streaming execution paths.
---
# Dataflow
+Flownode has two internal execution paths:
+
+- **Batching mode** is the primary path for aggregation and TQL workloads. It evaluates queries over persisted source data and writes materialized results to a sink table.
+- **Streaming mode** is the legacy path retained for compatibility and deprecated for new workloads. It incrementally processes rows mirrored from Frontend as they arrive.
+
+Users do not select the mode directly. When a Flow is created, GreptimeDB chooses the path from the query and source-table properties. Aggregation, `DISTINCT`, and TQL queries use batching mode. Simple non-aggregation queries, and any Flow whose source table has `ttl = 'instant'`, currently use streaming mode. A Flow deferred because its source table does not yet exist starts as a pending batching Flow.
+
+## Batching mode
+
+Batching mode reuses GreptimeDB's query engine instead of maintaining an operator graph for every incoming row. For a time-windowed Flow, its main loop is:
+
+1. A source-table write marks the affected time windows as dirty.
+2. A `BatchingTask` runs on its evaluation schedule or adaptive polling cadence and collects the pending dirty windows at that evaluation. Marking a window dirty does not wake the task.
+3. The task adds time predicates for those windows to the Flow query and asks Frontend to execute it against the source tables.
+4. The query result is inserted into the sink table, updating the materialized result for windows that were evaluated.
+5. Successfully processed windows are removed from the dirty set. Failed work remains available for a later evaluation.
+
+Flows with an evaluation interval but without a time-window expression run the complete query on each scheduled evaluation. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Flownode Batching Mode Developer Guide](./batching_mode.md) for the task and dirty-window components.
+
+## Streaming mode
+
The `dataflow` module (see `flow::compute` module) is the core computing module of `flow`.
It takes a SQL query and transforms it into flow's internal execution plan.
This execution plan is then rendered into an actual dataflow, which is essentially a directed acyclic graph (DAG) of functions with input and output ports.
-The dataflow is triggered to run when needed.
+New row changes drive the graph incrementally.
-Currently, this dataflow only supports `map` and `reduce` operations. Support for `join` operations will be added in the future.
+The renderer supports map/filter/project and reduce operations. Join and union plan nodes exist, but their streaming renderers are not implemented.
Internally, the dataflow handles data in row format, using a tuple `(row, time, diff)`. Here, `row` represents the actual data being passed, which may contain multiple `Value` objects.
`time` is the system time which tracks the progress of the dataflow, and `diff` typically represents the insertion or deletion of the row (+1 or -1).
-Therefore, the tuple represents the insert/delete operation of the `row` at a given system `time`.
\ No newline at end of file
+Therefore, the tuple represents the insert/delete operation of the `row` at a given system `time`. Stateful operators keep indexed traces of these changes in an [Arrangement](./arrangement.md).
diff --git a/docs/contributor-guide/frontend/distributed-querying.md b/docs/contributor-guide/frontend/distributed-querying.md
index 21ee07d7e8..ca3822e113 100644
--- a/docs/contributor-guide/frontend/distributed-querying.md
+++ b/docs/contributor-guide/frontend/distributed-querying.md
@@ -5,29 +5,16 @@ description: Describes the process of distributed querying in GreptimeDB, focusi
# Distributed Querying
-Most steps of querying in frontend and datanode are identical. The only difference is that
-Frontend have a "special" step in planning phase to make the logical query plan distributed.
-Let's reference it as "dist planner" in the following text.
-
-The modified, distributed logical plan has multiple stages, each of them is executed in different
-server node.
+Frontend and Datanode use the same DataFusion-based query engine. In distributed mode, Frontend adds a planning step that separates work performed by Datanodes from work completed by Frontend.
## Dist Planner
-Planner will traverse the input logical plan, and split it into multiple stages by the "[commutativity
-rule](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md)".
+The distributed planner rewrites the logical plan. It pushes compatible operators toward table scans and wraps remote subplans in `MergeScan` nodes. Partition predicates are also used to prune Regions before the remote work is scheduled.
-This rule is under heavy development. At present it will consider things like:
-- whether the operator itself is commutative
-- how the partition rule is configured
-- etc...
+Whether an operator can be pushed down depends on the plan shape and the operator's properties. Unsupported parts remain on Frontend. The original design and its commutativity rules are described in the [distributed planner RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md).
## Dist Plan
-Except the first stage, which have to read data from files in storage. All other stages' leaf node
-are actually a gRPC call to its previous stage.
-
-Sub-plan in a stage is itself a complete logical plan, and can be executed independently without
-the follow up stages. The plan is encoded in [substrait format](https://substrait.io).
+A remote input is a complete logical subplan, not just a table scan. Frontend serializes the subplan in [Substrait](https://substrait.io) format and sends a Region-specific request to the Datanode that owns the data. The Datanode plans and executes it locally, then streams the result back. Frontend merges the remote streams and executes any operators that were not pushed down.
diff --git a/docs/contributor-guide/frontend/overview.md b/docs/contributor-guide/frontend/overview.md
index 3a2f63e7ae..18cabfc2b8 100644
--- a/docs/contributor-guide/frontend/overview.md
+++ b/docs/contributor-guide/frontend/overview.md
@@ -5,27 +5,47 @@ description: Overview of GreptimeDB's Frontend component - a stateless proxy ser
# Frontend
-The **Frontend** is a stateless service that serves as the entry point for client requests in GreptimeDB. It provides a unified interface for multiple database protocols and acts as a proxy that forwards read/write requests to appropriate Datanodes in the distributed system.
+Frontend is GreptimeDB's stateless request-orchestration service. The server layer terminates protocols and converts wire messages; Frontend supplies the database behavior behind those handlers, including permission checks, statement execution, routing, and distributed query planning.
+
+Frontend does not store table data. It caches catalog and route metadata obtained from Metasrv, and Metasrv invalidates those caches through heartbeat responses when metadata changes.
## Core Functions
-- **Protocol Support**: Multiple database protocols including SQL, PromQL, MySQL, and PostgreSQL. See [Protocols][1] for details
-- **Request Routing**: Routes requests to appropriate Datanodes based on metadata
-- **Query Distribution**: Splits distributed queries across multiple nodes
-- **Response Aggregation**: Combines results from multiple Datanodes
-- **Authorization**: Security and access control validation
+- Provide query and ingestion behavior for the supported [protocols][1].
+- Resolve catalogs, schemas, tables, and Region routes.
+- Validate permissions before executing a request.
+- Plan distributed queries and merge results from Datanodes.
+- Convert table-level writes and deletes into Region requests.
## Architecture
### Key Components
-- **Protocol Handlers**: Handle different database protocols
-- **Catalog Manager**: Caches metadata from Metasrv to enable efficient request routing and schema validation
-- **Dist Planner**: Converts logical plans to distributed execution plans
-- **Request Router**: Determines target Datanodes for each request
+
+- Protocol handlers adapt SQL, PromQL, gRPC ingestion, and observability protocols to Frontend's internal request interfaces.
+- The catalog and partition managers provide table metadata, partition rules, and Region routes.
+- The statement executor dispatches queries, DML, and DDL to their respective execution paths.
+- The distributed planner replaces table scans with `MergeScan` plans that can run across Datanodes.
### Request Flow
-
+The request path depends on the operation.
+
+#### Queries
+
+1. A protocol handler creates the query context and performs authentication and permission checks.
+2. The language-specific planner produces a logical plan. In distributed mode, the planner uses partition metadata to select Regions and constructs a distributed plan.
+3. Frontend sends Region subplans to the owning Datanodes. Datanodes execute them against local Region engines and return streams of Arrow record batches.
+4. Frontend runs the remaining operators, merges the streams, and formats the result for the client protocol.
+
+#### Writes and deletes
+
+1. Frontend validates the request against the table schema. Protocols that support schema-on-write may create a missing table or add columns before retrying the write.
+2. The partition rule assigns rows to Regions. Frontend builds one Region request per target and routes it to the current Region leader.
+3. The Datanode's Region server dispatches each request to the Region engine. In standalone mode, the same request is sent to an embedded Region server instead of over RPC.
+
+#### DDL
+
+The statement executor converts DDL into a task. In distributed mode, Metasrv runs that task as a persisted procedure, updates metadata, and coordinates Region operations on Datanodes. Standalone mode uses the same statement boundary with local implementations of the metadata and procedure services.
### Deployment
diff --git a/docs/contributor-guide/frontend/table-sharding.md b/docs/contributor-guide/frontend/table-sharding.md
index a60276d14e..beaece40c0 100644
--- a/docs/contributor-guide/frontend/table-sharding.md
+++ b/docs/contributor-guide/frontend/table-sharding.md
@@ -5,21 +5,15 @@ description: Explains how table data in GreptimeDB is sharded and distributed, i
# Table Sharding
-The sharding of stored data is essential to any distributed database. This document will describe how table's data in GreptimeDB is being sharded, and distributed.
+GreptimeDB shards a table into Regions. Partition expressions define which rows belong to each Region, while Region routes define which Datanode currently owns each Region.
## Partition
-For the syntax of creating a partitioned table, please refer to the [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) section in the User Guide.
+A partition is a logical row set described by an expression over one or more columns. The partition layout must cover the table's input domain so each row has one target Region. See [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) for the SQL syntax and supported expressions.
## Region
-The data within a table is logically split after creating partitions. You may ask the question "
-how are the data, after being logically partitioned, stored in the GreptimeDB? The answer is in "`Region`"s.
-
-Each region is corresponding to a partition, and stores the data in the partition. The regions are distributed among
-`Datanode`s. `Metasrv` manages the route information that maps regions to Datanodes.
-If the partition layout needs to change after table creation, GreptimeDB supports explicit
-[repartitioning](/user-guide/deployments-administration/manage-data/repartition.md) through split and merge operations.
+Each partition maps to one Region. Region IDs remain the storage and routing identity used by Frontend, Datanode, and Metasrv. Multiple Regions from the same table may be placed on one Datanode.
The relationship between partition and region can be viewed as the following diagram:
@@ -53,3 +47,14 @@ The relationship between partition and region can be viewed as the following dia
│ │
└──────────────────────────────────┘
Could be placed in one Datanode
+```
+
+## Routing and Pruning
+
+For writes, Frontend evaluates the partition rule for each row, groups rows by Region, and sends Region requests to the current leaders from the route table.
+
+For queries, the distributed planner compares query predicates with the partition expressions. It scans only Regions that can satisfy the predicates. If partition metadata is missing or cannot be interpreted safely, the planner falls back to all Regions rather than risk omitting data.
+
+## Changing the Partition Layout
+
+[Repartitioning](/user-guide/deployments-administration/manage-data/repartition.md) changes an existing layout through explicit split and merge operations. Metasrv runs the change as a persisted procedure, updates the Region routes and partition expressions, and invalidates stale table-route caches. New requests use the published layout after their Frontend refreshes that metadata.
diff --git a/docs/contributor-guide/getting-started.md b/docs/contributor-guide/getting-started.md
index b17184dfa8..7e8cca3ad8 100644
--- a/docs/contributor-guide/getting-started.md
+++ b/docs/contributor-guide/getting-started.md
@@ -15,14 +15,13 @@ At the moment, GreptimeDB supports Linux (both amd64 and arm64), macOS (both amd
### Build Dependencies
-- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) (optional)
+- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) (optional; needed to clone the repository, not to build it)
- C/C++ Toolchain: provides essential tools for compiling and linking. This is available either as `build-essential` on ubuntu or a similar name on other platforms.
-- Rust nightly toolchain ([guide][1])
- - Compile the source code
+- [Rustup][1]. The repository pins the required nightly toolchain in `rust-toolchain.toml`.
- Protobuf ([guide][2])
- Compile the proto file
- Note that the version needs to be >= 3.15. You can check it with `protoc --version`
-- Machine: Recommended memory is 16GB or more, or use the [mold](https://github.com/rui314/mold) tool to reduce memory usage during linking.
+- Machine: 16GB of memory or more is recommended. On a smaller machine, use [mold](https://github.com/rui314/mold) to reduce memory usage during linking.
[1]:
[2]:
diff --git a/docs/contributor-guide/how-to/how-to-write-sdk.md b/docs/contributor-guide/how-to/how-to-write-sdk.md
index 40d0bc3713..0fd47f94e4 100644
--- a/docs/contributor-guide/how-to/how-to-write-sdk.md
+++ b/docs/contributor-guide/how-to/how-to-write-sdk.md
@@ -1,21 +1,17 @@
---
keywords: [gRPC SDK, GreptimeDatabase, Handle, HandleRequests, GreptimeRequest, GreptimeResponse]
-description: Explains how to write a gRPC SDK for GreptimeDB, focusing on the GreptimeDatabase service, its methods, and the structure of requests and responses.
+description: Protocol contracts and error-handling requirements for a GreptimeDB gRPC ingestion SDK.
---
# How to write a gRPC SDK for GreptimeDB
-A GreptimeDB gRPC SDK only needs to handle the writes. The reads are standard SQL and PromQL, can be handled by any JDBC
-client or Prometheus client. This is also why GreptimeDB gRPC SDKs are all named
-like "`greptimedb-ingester-`". Please make sure your GreptimeDB SDK follow the same naming convention.
+GreptimeDB's public gRPC SDKs are ingestion clients. Queries normally use SQL or PromQL through their standard clients. A new SDK should therefore focus on writes and deletes unless it has a separate requirement, and follow the `greptimedb-ingester-` naming convention. See the [gRPC SDK overview](/user-guide/ingest-data/for-iot/grpc-sdks/overview.md) for the user-facing API.
## `GreptimeDatabase` Service
-GreptimeDB defines a custom gRPC service called `GreptimeDatabase`. All you need to do in your SDK are implement it. You
-can find its Protobuf
-definitions [here](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto).
+Generate client stubs from the [`GreptimeDatabase` Protobuf definition](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto). Do not maintain a handwritten copy of the messages or service definition.
-The service contains two RPC methods:
+The service provides a unary method and a client-streaming method:
```protobuf
service GreptimeDatabase {
@@ -25,13 +21,9 @@ service GreptimeDatabase {
}
```
-The `Handle` method is for unary call: when a `GreptimeRequest` is received and processed by a GreptimeDB
-server, it responds with a `GreptimeResponse` immediately.
+`Handle` returns one response for one request. It is the usual choice for an SDK's insert and delete APIs.
-The `HandleRequests` acts in
-a "[Client streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc)" style. It ingests a
-stream of `GreptimeRequest`, and handles them on the fly. After all the requests have been handled, it returns a
-summarized `GreptimeResponse`. Through `HandleRequests`, we can achieve a very high throughput of requests handling.
+`HandleRequests` is a [client-streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc). The server returns a cumulative response only after the client closes the request stream. An SDK that exposes streaming must document this acknowledgement boundary and bind the stream to one endpoint.
### `GreptimeRequest`
@@ -51,13 +43,13 @@ message GreptimeRequest {
}
```
-A `RequestHeader` is needed, it includes some context, authentication and others. The "oneof" field contains the request
-to the GreptimeDB server.
+A client must populate `RequestHeader` with the database context and authentication expected by the server. Set exactly one request variant.
-Note that we have two types of insertions, one is in the form of "column" (the `InsertRequests`), and the other is "
-row" (`RowInsertRequests`). It's generally recommended to use the "row" form, since it's more natural for insertions on
-a table, and easier to use. However, if there's a need to insert a large number of columns at once, or there're plenty
-of "null" values to insert, the "column" form is better to be used.
+The message also contains query and DDL variants used by internal callers. The public ingester API should not expose them: `GreptimeDatabase` does not return query result streams.
+
+GreptimeDB accepts row-oriented `RowInsertRequests` and column-oriented `InsertRequests`. Row-oriented requests are the default for public ingestion APIs. A column-native client may use the column form, but it must keep column lengths consistent and preserve null values, timestamp precision, data types, and column semantic types during conversion.
+
+Deletes have the same row-oriented and column-oriented distinction. Expose only the forms that the SDK can map without losing type information.
### `GreptimeResponse`
@@ -70,8 +62,18 @@ message GreptimeResponse {
}
```
-The `ResponseHeader` contains the response's status code, and error message (if there's any). The "oneof" response only
-contains the affected rows for now.
+On success, the response contains a successful header and `affected_rows`. Treat that value as the number acknowledged by the server, including the cumulative value returned when a request stream closes.
+
+Request failures are returned as a gRPC status. When present, the trailing metadata keys `x-greptime-err-code` and `x-greptime-err-retry-hint` carry GreptimeDB's error code and retry classification. Preserve the gRPC status and expose the GreptimeDB metadata rather than replacing them with a generic SDK error.
+
+## Retry and Delivery Semantics
+
+Retries must be bounded and observable. A unary request may be retried only when the failure is classified as retryable and the deadline still permits it. Do not retry cancellation or deadline-expiration errors.
+
+A lost response does not prove that the server rejected a write. Retrying such a request can insert duplicate rows unless the caller's data model makes the operation idempotent. Document this possibility and return the final error when delivery is ambiguous.
+
+Do not transparently retry a partially sent `HandleRequests` stream. The server may already have accepted some requests even though the client has not received the cumulative response. Close the failed stream and report the ambiguity to the caller.
+
+Keep Arrow Flight bulk ingestion separate from the `GreptimeDatabase` RPCs. Its batching and partial-acceptance behavior needs its own API contract.
-GreptimeDB has a lot of SDKs now, you can refer to
-them [here](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=) for some examples.
+Use the existing [GreptimeDB ingester repositories](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=) to compare public API conventions, but derive wire behavior from the current Protobuf definition and server contract.
diff --git a/docs/contributor-guide/metasrv/admin-api.md b/docs/contributor-guide/metasrv/admin-api.md
index b091b48b70..d1f2ee6dbe 100644
--- a/docs/contributor-guide/metasrv/admin-api.md
+++ b/docs/contributor-guide/metasrv/admin-api.md
@@ -1,6 +1,6 @@
---
-keywords: [admin api, health check, leader query, heartbeat, maintenance mode, RESTful API]
-description: Details the Admin API for Metasrv, including endpoints for health checks, leader queries, heartbeat data, maintenance mode, and Procedure Manager controls.
+keywords: [admin api, health check, leader query, heartbeat, maintenance mode, recovery mode, table id sequence]
+description: Details the Metasrv Admin API for status inspection, cluster controls, and metadata recovery.
---
# Admin API
@@ -9,16 +9,17 @@ description: Details the Admin API for Metasrv, including endpoints for health c
Note that all Admin API endpoints in this document listen on Metasrv's `HTTP_PORT`, which defaults to `4000`.
:::
-The Admin API provides a simple way to view and manage cluster information, including metasrv health detection, metasrv leader query, datanode heartbeat detection, maintenance mode, and Procedure Manager controls.
-
-The Admin API is an HTTP service that provides a set of RESTful APIs that can be called through HTTP requests. The Admin API is simple, user-friendly and safe.
+The Admin API exposes Metasrv status, cluster controls, and metadata recovery operations over HTTP. It does not provide authentication, and some endpoints change cluster behavior or metadata allocation. Deployments must protect the HTTP port with network-level controls.
This page covers the following APIs:
- /health
- /leader
- /heartbeat
+- /node-lease
- /maintenance
- /procedure-manager
+- /recovery
+- /sequence/table
All these APIs are under the parent resource `/admin`.
@@ -26,7 +27,7 @@ In the following sections, we assume that your metasrv instance is running on lo
## /health HTTP endpoint
-The `/health` endpoint accepts GET HTTP requests and you can use this endpoint to check the health of your metasrv instance.
+The `/health` endpoint accepts GET requests and returns `OK` when the HTTP service is running. It does not check whether this Metasrv is the leader or whether external dependencies are available.
### Definition
@@ -120,9 +121,17 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100'
]
```
+## /node-lease HTTP endpoint
+
+The `/node-lease` endpoint returns the current leases recorded for Datanodes. Use it when diagnosing whether Metasrv still considers a Datanode active.
+
+```bash
+curl -X GET http://localhost:4000/admin/node-lease
+```
+
## /maintenance HTTP endpoint
-Cluster Maintenance Mode is a safety feature in GreptimeDB that temporarily disables automatic cluster management operations. This mode is particularly useful during cluster upgrades, planned downtime, and any operation that might temporarily affect cluster stability. For more details, please refer to [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md).
+Maintenance mode temporarily disables automatic cluster management operations during upgrades, planned downtime, or similar work. See [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md) for its effect on the cluster.
The `/maintenance` endpoint supports the following HTTP requests:
@@ -155,3 +164,39 @@ The response body uses the following format:
"status": "running"
}
```
+
+## /recovery HTTP endpoints
+
+Recovery mode gates metadata repair endpoints such as manual table ID sequence changes. It is intended for recovery work, not routine maintenance.
+
+- `GET /admin/recovery/status`: query whether recovery mode is enabled.
+- `POST /admin/recovery/enable`: enable recovery mode.
+- `POST /admin/recovery/disable`: disable recovery mode.
+
+The response body uses the following format:
+
+```json
+{
+ "enabled": true
+}
+```
+
+Disable recovery mode after the repair is complete. Use [maintenance mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md) instead when the goal is to suspend automatic cluster operations during planned maintenance.
+
+## /sequence/table HTTP endpoints
+
+These endpoints inspect or repair the table ID sequence:
+
+- `GET /admin/sequence/table/next-id`: return the next table ID without allocating it.
+- `POST /admin/sequence/table/set-next-id`: advance the next table ID.
+
+Setting the sequence requires recovery mode. The new value must be greater than the current value; the API cannot move the sequence backwards. Recovery mode is an API precondition, not a DDL barrier. Follow [Manage table ID sequences](/user-guide/deployments-administration/maintenance/sequence-management.md) for the required cluster-wide procedure.
+
+```bash
+curl -X POST \
+ -H 'Content-Type: application/json' \
+ -d '{"next_table_id": 2048}' \
+ http://localhost:4000/admin/sequence/table/set-next-id
+```
+
+Changing this value affects IDs allocated to future tables.
diff --git a/docs/contributor-guide/metasrv/overview.md b/docs/contributor-guide/metasrv/overview.md
index 7aa052dd7e..6458b6380d 100644
--- a/docs/contributor-guide/metasrv/overview.md
+++ b/docs/contributor-guide/metasrv/overview.md
@@ -1,161 +1,101 @@
---
-keywords: [metasrv, metadata, request-router, load balancing, election, high availability, heartbeat]
-description: Provides an overview of the Metasrv service, its components, interactions with the Frontend, architecture, and key functionalities like distributed consensus and heartbeat management.
+keywords: [metasrv, metadata, routing, leader election, procedure, heartbeat]
+description: Overview of the metadata and coordination mechanisms provided by Metasrv.
---
# Metasrv
-
-
## What's in Metasrv
-- Store metadata (Catalog, Schema, Table, Region, etc.)
-- Request-Router. It tells the Frontend where to write and read data.
-- Load balancing for Datanode, determines who should handle new table creation requests, more precisely, it makes resource allocation decisions.
-- Election & High Availability, GreptimeDB is designed in a Leader-Follower architecture, only Leader nodes can write while Follower nodes can read, the number of Follower nodes is usually >= 1, and Follower nodes need to be able to switch to Leader quickly when Leader is not available.
-- Statistical data collection (reported via Heartbeats on each node), such as CPU, Load, number of Tables on the node, average/peak data read/write size, etc., can be used as the basis for distributed scheduling.
+Metasrv is the metadata and coordination service in a distributed GreptimeDB cluster. It does not sit on the data path. Its main responsibilities are:
+
+- storing Catalog, Schema, Table, Region, route, and node metadata;
+- choosing Datanodes for new Regions and maintaining table routes;
+- electing one Metasrv leader to coordinate metadata changes;
+- running recoverable procedures for DDL, Region migration, failover, and repartitioning;
+- tracking node leases and Region statistics through heartbeats;
+- broadcasting cache invalidations to Frontends, Datanodes, and Flownodes when metadata changes;
+- sending Region lifecycle instructions to Datanodes.
## How the Frontend interacts with Metasrv
-First, the routing table in Request-Router is in the following structure (note that this is only the logical structure, the actual storage structure varies, for example, endpoints may have dictionary compression).
+Frontend obtains table metadata and Region routes from Metasrv and caches them locally. Metadata-changing statements are sent to the Metasrv leader, while reads and writes use the cached routes to reach Datanodes directly.
+
+The control and data paths are separate:
+
+```text
+Frontend
+ |-- metadata lookup and DDL ------------> Metasrv leader
+ `-- Region reads and writes ------------> Datanode
+
+Metasrv leader
+ |-- Region lifecycle instructions ------> Datanode
+ `-- cache invalidations ----------------> Frontend / Datanode / Flownode
+Datanode
+ `-- heartbeat, lease renewal, Region stats -> Metasrv leader
```
- table_A
- table_name
- table_schema // for physical plan
- regions
- region_1
- mutate_endpoint
- select_endpoint_1, select_endpoint_2
- region_2
- mutate_endpoint
- select_endpoint_1, select_endpoint_2, select_endpoint_3
- region_xxx
- table_B
- ...
+
+In steady state, a table route records one leader peer and zero or more follower peers for each Region. The leader is the write target. Deployments with read-replica support can route reads to followers:
+
+```text
+Table route
+ |-- Region 0
+ | |-- leader -> Datanode A
+ | `-- followers -> Datanode B, Datanode C
+ `-- Region 1
+ `-- leader -> Datanode D
```
+Region migration or failover changes peer roles and can temporarily leave a Region without a leader. Frontend refreshes its cached route before sending subsequent reads or writes to the current peers.
+
### Create Table
-1. The Frontend sends `CREATE TABLE` requests to Metasrv.
-2. Plan the number of Regions according to the partition rules contained in the request.
-3. Check the global view of resources available to Datanodes (collected by Heartbeats) and assign one node to each region.
-4. The Frontend creates the table and stores the `Schema` to Metasrv after successful creation.
+1. Frontend submits the DDL request to the Metasrv leader.
+2. Metasrv derives Regions from the partition rules and [selects a Datanode for each Region](/contributor-guide/metasrv/selector.md).
+3. A persisted procedure creates the Regions and records the table and route metadata. If leadership changes, the procedure can resume from its persisted state.
+4. Metasrv notifies Frontends after the metadata change is committed so their caches can be refreshed.
### Insert
-1. The Frontend fetches the routes of the specified table from Metasrv. Note that the smallest routing unit is the route of the table (several regions), i.e., it contains the addresses of all regions of this table.
-2. The best practice is that the Frontend first fetches the routes from its local cache and forwards the request to the Datanode. If the route is no longer valid, then Datanode is obliged to return an `Invalid Route` error, and the Frontend re-fetches the latest data from Metasrv and updates its cache. Route information does not change frequently, thus, it's sufficient for Frontend uses the Lazy policy to maintain the cache.
-3. The Frontend processes a batch of writes that may contain multiple tables and multiple regions, so the Frontend needs to split user requests based on the 'route table'.
+Frontend resolves the table route, splits rows according to the partition rules, and sends each Region write to the corresponding Datanode. Route changes cause the cached metadata to be invalidated and fetched again from Metasrv.
### Select
-1. As with `Insert`, the Frontend first fetches the route table from the local cache.
-2. Unlike `Insert`, for `Select`, the Frontend needs to extract the read-only node (follower) from the route table, then dispatch the request to the leader or follower node depending on the priority.
-3. The distributed query engine in the Frontend distributes multiple sub-query tasks based on the routing information and aggregates the query results.
+Frontend uses table and Region metadata while planning the query. Predicates on partition columns prune Regions, and the distributed query engine sends work to the Datanodes that own the selected Regions. See [Distributed Querying](../frontend/distributed-querying.md).
## Metasrv Architecture
-
-
-## Distributed Consensus
+The main coordination paths are:
+
+```text
+Leader election
+ |
+ v
+Metasrv leader
+├─ DDL manager -> Procedure manager
+├─ Selector -> new Region placement
+├─ Heartbeat handler chain -> leases and Region statistics
+├─ Region supervisor -> Region migration procedures
+├─ Mailbox -> cache invalidations and Region instructions
+└─ Metadata managers -> KV backend
+```
-As you can see, Metasrv has a dependency on distributed consensus because:
+These mechanisms share metadata, but they have different failure boundaries. A process restart may discard caches and leader-local state; metadata and procedure state required for recovery must be durable.
-1. First, Metasrv has to elect a leader, Datanode only sends heartbeats to the leader, and we only use a single metasrv node to receive heartbeats, which makes it easy to do some calculations or scheduling accurately and quickly based on global information. As for how the Datanode connects to the leader, this is for MetaClient to decide (using a redirect, Heartbeat requests becomes a gRPC stream, and using redirect will be less error-prone than forwarding), and it is transparent to the Datanode.
-2. Second, Metasrv must provide an election API for Datanode to elect "write" and "read-only" nodes and help Datanode achieve high availability.
-3. Finally, `Metadata`, `Schema` and other data must be reliably and consistently stored on Metasrv. Therefore, consensus-based algorithms are the ideal approach for storing them.
+## Distributed Consensus
-For the first version of Metasrv, we choose Etcd as the consensus algorithm component (Metasrv is designed to consider adapting different implementations and even creating a new wheel) for the following reasons:
+Metasrv separates leader election from metadata storage. Only the elected Metasrv leader performs coordination and metadata-changing operations. Other Metasrv nodes direct clients to the current leader.
-1. Etcd provides exactly the API we need, such as `Watch`, `Election`, `KV`, etc.
-2. We only perform two tasks with distributed consensus: elections (using the `Watch` mechanism) and storing (a small amount of metadata), and neither of them requires us to customize our own state machine, nor do we need to customize our own state machine based on raft; the small amount of data also does not require multi-raft-group support.
-3. The initial version of Metasrv uses Etcd, which allows us to focus on the capabilities of Metasrv and not spend too much effort on distributed consensus algorithms, which improves the design of the system (avoiding coupling with consensus algorithms) and helps with rapid development at the beginning, as well as allows easy access to good consensus algorithm implementations in the future through good architectural designs.
+The key-value backend stores table metadata, routes, procedure state, and other information that must survive a leader change. Metasrv does not use this election to create leader and follower replicas for Datanode Regions; Region availability is managed through heartbeats, Region failure detection, and failover procedures.
## Heartbeat Management
-The primary means of communication between Datanode and Metasrv is the Heartbeat Request/Response Stream, and we want this to be the only way to communicate. This idea is inspired by the design of [TiKV PD](https://github.com/tikv/pd), and we have practical experience in [RheaKV](https://github.com/sofastack/sofa-jraft/tree/master/jraft-rheakv/rheakv-pd). The request sends its state, while Metasrv sends different scheduling instructions via Heartbeat Response.
-
-A heartbeat will probably carry the data listed below, but this is not the final design, and we are still discussing and exploring exactly which data should be mostly collected.
-
-```
-service Heartbeat {
- // Heartbeat, there may be many contents of the heartbeat, such as:
- // 1. Metadata to be registered to metasrv and discoverable by other nodes.
- // 2. Some performance metrics, such as Load, CPU usage, etc.
- // 3. The number of computing tasks being executed.
- rpc Heartbeat(stream HeartbeatRequest) returns (stream HeartbeatResponse) {}
-}
-
-message HeartbeatRequest {
- RequestHeader header = 1;
-
- // Self peer
- Peer peer = 2;
- // Leader node
- bool is_leader = 3;
- // Actually reported time interval
- TimeInterval report_interval = 4;
- // Node stat
- NodeStat node_stat = 5;
- // Region stats in this node
- repeated RegionStat region_stats = 6;
- // Follower nodes and stats, empty on follower nodes
- repeated ReplicaStat replica_stats = 7;
-}
-
-message NodeStat {
- // The read capacity units during this period
- uint64 rcus = 1;
- // The write capacity units during this period
- uint64 wcus = 2;
- // Table number in this node
- uint64 table_num = 3;
- // Region number in this node
- uint64 region_num = 4;
-
- double cpu_usage = 5;
- double load = 6;
- // Read disk I/O in the node
- double read_io_rate = 7;
- // Write disk I/O in the node
- double write_io_rate = 8;
-
- // Others
- map attrs = 100;
-}
-
-message RegionStat {
- uint64 region_id = 1;
- TableName table_name = 2;
- // The read capacity units during this period
- uint64 rcus = 3;
- // The write capacity units during this period
- uint64 wcus = 4;
- // Approximate region size
- uint64 approximate_size = 5;
- // Approximate number of rows
- uint64 approximate_rows = 6;
-
- // Others
- map attrs = 100;
-}
-
-message ReplicaStat {
- Peer peer = 1;
- bool in_sync = 2;
- bool is_learner = 3;
-}
-```
-
-## Central Nervous System (CNS)
-
-We are to build an algorithmic system, which relies on real-time and historical heartbeat data from each node, should make some smarter scheduling decisions and send them to Metasrv's Autoadmin unit, which distributes the scheduling decisions, either by the Datanode itself or more likely by the PaaS platform.
-
-## Abstraction of Workloads
+Datanodes maintain heartbeat streams to the Metasrv leader. Heartbeat requests report node identity, lease information, Region statistics, and other state used for placement and supervision. Responses carry control messages such as Region lifecycle instructions and cache invalidations.
-The level of workload abstraction determines the efficiency and quality of the scheduling strategy generated by Metasrv such as resource allocation.
+A heartbeat drives two independent mechanisms, and a change to heartbeat timing affects both:
-DynamoDB defines RCUs & WCUs (Read Capacity Units / Write Capacity Units), explaining that a RCU is a read request of 4KB data, and a WCU is a write request of 1KB data. When using RCU and WCU to describe workloads, it's easier to achieve performance measurability and get more informative resource preallocation because we can abstract different hardware capabilities as a combination of RCU and WCU.
+- **Node lease.** The keep-lease handler renews the sending Datanode's lease. Selectors and the `/node-lease` endpoint use these leases to decide whether a Datanode is still active.
+- **Region failure detection.** The Region supervisor keeps a per-Region Phi Accrual detector over heartbeat arrival intervals. Its verdict is independent of lease expiry.
-However, GreptimeDB still faces a more complex situation than DynamoDB, in particular, RCU doesn't fit to describe GreptimeDB's read workloads which require a lot of computation. We are working on that.
+A failure verdict submits a failover migration only when Region failover is enabled; it is disabled by default and requires remote WAL unless explicitly allowed on local WAL. Maintenance mode also suppresses failover. See [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md) for the prerequisites and how to enable it.
diff --git a/docs/contributor-guide/metasrv/selector.md b/docs/contributor-guide/metasrv/selector.md
index 790ffb849b..23190cd6a9 100644
--- a/docs/contributor-guide/metasrv/selector.md
+++ b/docs/contributor-guide/metasrv/selector.md
@@ -7,32 +7,28 @@ description: Describes the different types of selectors in the Metasrv service,
## Introduction
-What is the `Selector`? As its name suggests, it allows users to select specific items from a given `namespace` and `context`. There is a related trait, also named `Selector`, whose definition can be found [below][0].
-
-[0]: https://github.com/GreptimeTeam/greptimedb/blob/main/src/meta-srv/src/selector.rs
-
-There is a specific scenario in `Metasrv` service. When a request to create a table is sent to the `Metasrv` service, it creates a routing table (the details of table creation will not be described here). The `Metasrv` service needs to select the appropriate `Datanode` list when creating a routing table.
+When a table is created, Metasrv uses a `Selector` to choose Datanodes for its Regions. Selection uses the current node leases and, depending on the selector, Region statistics.
## Selector Type
The `Metasrv` service currently offers the following types of `Selectors`:
-### LeasebasedSelector
+### LeaseBasedSelector
-`LeasebasedSelector` randomly selects from all available (in lease) `Datanode`s, its characteristic is simplicity and fast.
+`LeaseBasedSelector` randomly selects from Datanodes with valid leases.
### LoadBasedSelector
The `LoadBasedSelector` load value is determined by the number of regions on each `Datanode`, fewer regions indicate lower load, and `LoadBasedSelector` prioritizes selecting low-load `Datanodes`.
### RoundRobinSelector [default]
-`RoundRobinSelector` selects `Datanode`s in a round-robin fashion. It is recommended and the default option in most cases. If you're unsure which to choose, it's usually the right choice.
+`RoundRobinSelector` selects `Datanode`s in a round-robin fashion. It is the default and recommended choice for most deployments.
## Configuration
You can configure the `Selector` by its name when starting the `Metasrv` service.
-- LeasebasedSelector: `lease_based` or `LeaseBased`
+- LeaseBasedSelector: `lease_based` or `LeaseBased`
- LoadBasedSelector: `load_based` or `LoadBased`
- RoundRobinSelector: `round_robin` or `RoundRobin`
diff --git a/docs/contributor-guide/overview.md b/docs/contributor-guide/overview.md
index 875b4b1e16..6e6ec669c9 100644
--- a/docs/contributor-guide/overview.md
+++ b/docs/contributor-guide/overview.md
@@ -5,9 +5,7 @@ description: Overview of GreptimeDB's architecture, key components, and how they
# Contributor Guide
-DeepWiki provides a detailed and clear explanation of GreptimeDB's architecture and implementation. Highly recommended:
-
-[https://deepwiki.com/GreptimeTeam/greptimedb](https://deepwiki.com/GreptimeTeam/greptimedb)
+This guide explains the internal design of GreptimeDB for contributors. Start with [Getting Started](/contributor-guide/getting-started.md) to build and run it from source. Submission requirements, including the CLA, license headers, formatting, and the checks a pull request must pass, are maintained in the source repository's [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md).
## Architecture
@@ -18,8 +16,13 @@ For more details on each component, see the following guides:
- [frontend][1]
- [datanode][2]
- [metasrv][3]
+- [flownode][4]
[1]: /contributor-guide/frontend/overview.md
[2]: /contributor-guide/datanode/overview.md
[3]: /contributor-guide/metasrv/overview.md
+[4]: /contributor-guide/flownode/overview.md
+
+## Additional reference
+[DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb) provides an automatically generated walkthrough of the GreptimeDB repository. It can help when exploring an unfamiliar area, but it is a secondary reference: verify version-sensitive behavior against the source code.
diff --git a/docs/contributor-guide/tests/integration-test.md b/docs/contributor-guide/tests/integration-test.md
index 5d5f6cb1a5..bc52ee4854 100644
--- a/docs/contributor-guide/tests/integration-test.md
+++ b/docs/contributor-guide/tests/integration-test.md
@@ -7,7 +7,14 @@ description: Guide on writing and running integration tests in GreptimeDB, cover
## Introduction
-Integration testing is written with Rust test harness (`#[test]`), unlike unit testing, they are placed separately
-[here](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration).
-It covers scenarios involving multiple components, in which one typical case is HTTP/gRPC-related features. You can check
-its [documentation](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md) for more information.
+Integration tests cover behavior that crosses crate or service boundaries, such as HTTP and gRPC handling, distributed components, or external storage. They use Rust's test harness and live in the [`tests-integration`](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration) package.
+
+Run the package with:
+
+```shell
+cargo nextest run -p tests-integration
+```
+
+Some cases require environment variables or fixtures for external services. Follow the package's [setup instructions](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md) before running those cases.
+
+Use an integration test when a crate-level test or a Sqlness case cannot exercise the required boundary. Keep isolated logic in unit tests so that failures remain fast to reproduce.
diff --git a/docs/contributor-guide/tests/overview.md b/docs/contributor-guide/tests/overview.md
index feeefe2b6d..6fe48b2c32 100644
--- a/docs/contributor-guide/tests/overview.md
+++ b/docs/contributor-guide/tests/overview.md
@@ -5,4 +5,13 @@ description: Overview of the testing methods used in GreptimeDB to ensure its be
# Tests
-Our team has conducted lots of tests to ensure the behaviours of `GreptimeDB` . This chapter will introduce several significant methods used to test `GreptimeDB`, and how to work with them.
+Choose the narrowest test that exercises the behavior you changed:
+
+| Test type | Use it for | Typical command |
+| --- | --- | --- |
+| [Unit test](unit-test.md) | Logic contained within one crate or component | `cargo nextest run -p ` |
+| [Sqlness test](sqlness-test.md) | SQL, protocol, planner, execution, and end-to-end regressions | `cargo sqlness bare -t ` |
+| [Integration test](integration-test.md) | Behavior that crosses components or requires external services | `cargo nextest run -p tests-integration` |
+| [Compatibility test](https://github.com/GreptimeTeam/greptimedb/blob/main/tests/compatibility/README.md) | Reading data or metadata written by an older release | `cargo sqlness compat --from-version ` |
+
+Run `make test` when a change needs the full Rust workspace test suite. Changes to persisted metadata, WAL records, SST files, or wire formats should also include a compatibility test when an older release may have produced the input.
diff --git a/docs/contributor-guide/tests/sqlness-test.md b/docs/contributor-guide/tests/sqlness-test.md
index b5fdcf2b0b..0262ff6851 100644
--- a/docs/contributor-guide/tests/sqlness-test.md
+++ b/docs/contributor-guide/tests/sqlness-test.md
@@ -7,42 +7,34 @@ description: Instructions for running SQL tests in GreptimeDB using the `sqlness
## Introduction
-SQL is an important user interface for `GreptimeDB`. We have a separate test suite for it (named `sqlness`).
+Sqlness is GreptimeDB's end-to-end regression suite for SQL and protocol behavior. A case sends statements to a running GreptimeDB instance and compares the output with a checked-in result file.
## Sqlness manual
### Case file
-Sqlness has two types of file
+Each case uses two files:
- `.sql`: test input, SQL only
- `.result`: expected test output, SQL and its results
-The `.result` file is the expected execution output. If you see `.result` files changed,
-it means the test gets a different result and indicates it may fail. You should
-check the change logs to solve the problem.
-
-You only need to write test SQL in the `.sql` file, and run the test.
+Write the input in the `.sql` file and run the test to generate or update `.result`. Review every result diff: accept it only when the behavior change is intended.
### Case organization
-The root dir of input cases is `tests/cases`. It contains several sub-directories stand for different test
-modes. E.g., `standalone/` contains all the tests to run under `greptimedb standalone start` mode.
+Input cases live under `tests/cases`. The first directory level selects an environment. For example, `standalone/` runs against a standalone GreptimeDB instance.
-Under the first level of sub-directory (e.g. the `cases/standalone`), you can organize your cases as you like.
-Sqlness walks through every file recursively and runs them.
+Within an environment, group a new case with the feature it exercises. Sqlness discovers case files recursively.
## Run the test
-Unlike other tests, this harness is in a binary target form. You can run it with
+Run the suite with:
```shell
-cargo run --bin sqlness-runner bare
+cargo sqlness bare
```
-It automatically finishes the following procedures: compile `GreptimeDB`, start it, grab tests and feed it to
-the server, then collect and compare the results. You only need to check whether any `.result` files changed.
-If not, congratulations, the test is passed 🥳!
+The command builds and starts GreptimeDB, runs the selected cases, and compares their output. A changed `.result` file is part of the review, not proof that the new output is correct.
### Run a specific test
diff --git a/docs/contributor-guide/tests/unit-test.md b/docs/contributor-guide/tests/unit-test.md
index 82e30bf5fa..183a72aab4 100644
--- a/docs/contributor-guide/tests/unit-test.md
+++ b/docs/contributor-guide/tests/unit-test.md
@@ -8,25 +8,26 @@ description: Guide on writing and running unit tests in GreptimeDB using Rust's
## Introduction
Unit tests are embedded into the codebase, usually placed next to the logic being tested.
-They are written using Rust's `#[test]` attribute and can run with `cargo nextest run`.
+They are written using Rust's `#[test]` attribute. GreptimeDB uses [`cargo-nextest`](https://nexte.st/) as its primary Rust test runner.
-The default test runner ships with `cargo` is not supported in GreptimeDB codebase. It's recommended
-to use [`nextest`](https://nexte.st/) instead. You can install it with
+Install it with:
```shell
cargo install cargo-nextest --locked
```
-And run the tests (here the `--workspace` is not necessary)
+Run the package you changed first:
```shell
-cargo nextest run
+cargo nextest run -p
```
-Notes if your Rust is installed via `rustup`, be sure to install `nextest` with `cargo` rather
-than the package manager like `homebrew`. Otherwise it will mess up your local environment.
+Use a test name or nextest filter to narrow the run further while developing. Before submitting a change with broad effects, run the full workspace suite:
+
+```shell
+make test
+```
## Coverage
-Our continuous integration (CI) jobs have a "coverage checking" step. It will report how many
-codes are covered by unit tests. Please add the necessary unit test to your patch.
+CI reports unit-test coverage. Add tests for changed behavior and failure cases that could otherwise regress; coverage percentage alone is not the goal.
diff --git a/docs/reference/sql/create.md b/docs/reference/sql/create.md
index 4eccb986d0..29a9cf666f 100644
--- a/docs/reference/sql/create.md
+++ b/docs/reference/sql/create.md
@@ -26,7 +26,7 @@ If the `db_name` database already exists, then GreptimeDB has the following beha
The database can also carry options similar to the `CREATE TABLE` statement by using the `WITH` keyword. The following options are available for databases:
- `ttl` - Time-To-Live for data in all tables within the database (cannot be set to `instant`)
-- `memtable.type` - Type of memtable (`time_series`, `partition_tree`)
+- `memtable.type` - Type of memtable (`bulk`, `time_series`)
- `append_mode` - Whether tables in the database should be append-only (`true`/`false`)
- `merge_mode` - Strategy for merging duplicate rows (`last_row`, `last_non_null`)
- `skip_wal` - Whether to disable Write-Ahead-Log for tables in the database (`'true'`/`'false'`)
@@ -74,7 +74,7 @@ Create a database with multiple options, including append mode and custom memtab
```sql
CREATE DATABASE test WITH (
ttl='30d',
- 'memtable.type'='partition_tree',
+ 'memtable.type'='bulk',
'append_mode'='true'
);
```
@@ -154,7 +154,7 @@ Users can add table options by using `WITH`. The valid options contain the follo
| `compaction.twcs.trigger_file_num` | Number of files in a specific time window to trigger a compaction | String value, such as '8'. Only available when `compaction.type` is `twcs`. You can refer to this [document](https://cassandra.apache.org/doc/latest/cassandra/managing/operating/compaction/twcs.html) to learn more about the `twcs` compaction strategy. |
| `compaction.twcs.time_window` | Compaction time window | String value, such as '1d' for 1 day. The table usually partitions rows into different time windows by their timestamps. Only available when `compaction.type` is `twcs`. |
| `compaction.twcs.max_output_file_size` | Maximum allowed output file size for TWCS compaction | String value, such as '1GB', '512MB'. Sets the maximum size for files produced by TWCS compaction. Only available when `compaction.type` is `twcs`. |
-| `memtable.type` | Type of the memtable. | String value, supports `time_series`, `partition_tree`. |
+| `memtable.type` | Type of the memtable | String value: `bulk` or `time_series`. If omitted, Mito selects the implementation from the SST format; the default flat format uses `bulk`. Setting `bulk` forces `sst_format=flat`, and flat SSTs use the bulk implementation even if `time_series` is specified. The legacy value `partition_tree` is accepted for compatibility and maps to the bulk and flat path. |
| `append_mode` | Whether the table is append-only | String value. Default is 'false', which removes duplicate rows by primary keys and timestamps according to the `merge_mode`. Setting it to 'true' to enable append mode and create an append-only table which keeps duplicate rows. |
| `merge_mode` | The strategy to merge duplicate rows | String value. Only available when `append_mode` is 'false'. Default is `last_row`, which keeps the last row for the same primary key and timestamp. Setting it to `last_non_null` to keep the last non-null field for the same primary key and timestamp. |
| `sst_format` | The format of SST files | String value, supports `primary_key`, `flat`. Default is `flat`. `flat` is recommended for tables which have a large number of unique primary keys. |
diff --git a/docs/user-guide/deployments-administration/configuration.md b/docs/user-guide/deployments-administration/configuration.md
index a8b99a3014..5684faaac0 100644
--- a/docs/user-guide/deployments-administration/configuration.md
+++ b/docs/user-guide/deployments-administration/configuration.md
@@ -603,20 +603,9 @@ create_on_compaction = "auto"
apply_on_query = "auto"
mem_threshold_on_create = "64M"
intermediate_path = ""
-
-[region_engine.mito.memtable]
-type = "time_series"
```
-The `mito` engine provides an experimental memtable which optimizes for write performance and memory efficiency under large amounts of time-series. Its read performance might not as fast as the default `time_series` memtable.
-
-```toml
-[region_engine.mito.memtable]
-type = "partition_tree"
-index_max_keys_per_shard = 8192
-data_freeze_threshold = 32768
-fork_dictionary_bytes = "1GiB"
-```
+Mito selects the memtable implementation for each Region according to its table options and SST format. When `default_flat_format` is `true`, Regions without an explicit `sst_format` use flat SSTs and the bulk memtable. Configure `memtable.type` as a database or table option; `[region_engine.mito.memtable]` is not an engine setting. See [table options](/reference/sql/create.md#table-options).
Available options:
@@ -651,7 +640,7 @@ Available options:
| `scan_memory_on_exhausted` | String | `fail` | Behavior when scan memory is exhausted. Options: `fail` (fail fast), `wait` or `wait()` (wait for memory). |
| `min_compaction_interval` | String | `0m` | Minimum time interval between two compactions. Set to "0m" (default) to allow compactions to run immediately without restriction. |
| `schedule_compaction_after_edit` | Bool | `true` | Whether to allow scheduling a compaction after a successful region edit. Setting this to `true` is a necessary but not sufficient condition for scheduling compaction after a region edit. Other constraints, such as `min_compaction_interval`, may still prevent compaction from being scheduled. Setting this to `false` guarantees that compaction will not be scheduled after a region edit. |
-| `default_flat_format` | Bool | `true` | Whether to enable flat format as the default SST format. |
+| `default_flat_format` | Bool | `true` | Whether Regions without an explicit `sst_format` use flat SSTs. Flat SSTs use the bulk memtable. |
| `experimental_series_scan_v2` | Bool | `true` | Whether to enable the experimental two-phase mode for series scans of metric engine physical regions. Set to `false` to use the legacy mode. Other series scans also use the legacy mode. |
| `scan_parallelism` | Integer | `0` | (Deprecated, use `max_concurrent_scan_files` instead) Legacy option for scan parallelism. |
| `index` | -- | -- | The options for index in Mito engine. |
@@ -668,10 +657,6 @@ Available options:
| `inverted_index.apply_on_query` | String | `auto` | Whether to apply the index on query - `auto`: automatically - `disable`: never |
| `inverted_index.mem_threshold_on_create` | String | `64M` | Memory threshold for performing an external sort during index creation. Setting to empty will disable external sorting, forcing all sorting operations to happen in memory. |
| `inverted_index.intermediate_path` | String | `""` | File system path to store intermediate files for external sorting (default `{data_home}/index_intermediate`). |
-| `memtable.type` | String | `time_series` | Memtable type. - `time_series`: time-series memtable - `partition_tree`: partition tree memtable (experimental) |
-| `memtable.index_max_keys_per_shard` | Integer | `8192` | The max number of keys in one shard. Only available for `partition_tree` memtable. |
-| `memtable.data_freeze_threshold` | Integer | `32768` | The max rows of data inside the actively writing buffer in one shard. Only available for `partition_tree` memtable. |
-| `memtable.fork_dictionary_bytes` | String | `1GiB` | Max dictionary bytes. Only available for `partition_tree` memtable. |
The `metric` engine is optimized for handling metrics data with a large number of small tables.
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md
index e0ef9e8e54..e0ce16721d 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md
@@ -5,19 +5,23 @@ description: 介绍了 GreptimeDB 的数据持久化和索引机制，包括 SST
# 数据持久化与索引
-与所有类似 LSMT 的存储引擎一样，MemTables 中的数据被持久化到耐久性存储，例如本地磁盘文件系统或对象存储服务。GreptimeDB 采用 [Apache Parquet][1] 作为其持久文件格式。
+与其他 LSM-tree 存储引擎类似，GreptimeDB 将 memtable 中的数据持久化到本地文件系统或对象存储，并使用 [Apache Parquet][1] 作为持久化文件格式。
## SST 文件格式
Parquet 是一种提供快速数据查询的开源列式存储格式，已经被许多项目采用，例如 Delta Lake。
-Parquet 具有层次结构，类似于“行组 - 列-数据页”。Parquet 文件中的数据被水平分区为行组（row group），在其中相同列的所有值一起存储以形成数据页（data pages）。数据页是最小的存储单元。这种结构极大地提高了性能。
+Parquet 按 row group、column chunk 和 page 组织数据。每个 row group 为每一列保存一个 column chunk，每个 column chunk 再包含一个或多个 page。Page 是编码和压缩单元，读取指定列时则以 column chunk 为 I/O 单元。
首先，数据按列聚集，这使得文件扫描更加高效，特别是当查询只涉及少数列时，这在分析系统中非常常见。
-其次，相同列的数据往往是同质的（比如具备近似的值），这有助于在采用字典和 Run-Length Encoding（RLE）等技术进行压缩。
+其次，同一列中的值通常比较相似，有利于字典编码和 Run-Length Encoding（RLE）等压缩技术发挥作用。
-
+下面这张来自 Apache Parquet 规范的图进一步展示了物理文件布局：column chunk 按 row group 写入，文件元数据及其长度则保存在 footer 中。
+
+
+
+*来源：Apache Parquet [FileLayout.gif](https://github.com/apache/parquet-format/blob/master/doc/images/FileLayout.gif)。Copyright 2014 The Apache Software Foundation，依据 [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) 使用。*
## 数据持久化
@@ -26,18 +30,18 @@ GreptimeDB 提供了 `region_engine.mito.global_write_buffer_size` 的配置项
## SST 文件中的索引数据
-Apache Parquet 文件格式在列块和数据页的头部提供了内置的统计信息，用于剪枝和跳过。
+Parquet 在每个 column chunk 的元数据中保存 row group 级列统计信息，例如最小值、最大值和 null 数量。Page 元数据和可选的 column index 可以提供粒度更细的统计信息。
-
+
-例如，在上述 Parquet 文件中，如果你想要过滤 `name` 等于 `Emily` 的行，你可以轻松跳过行组 0，因为 `name` 字段的最大值是 `Charlie`。这些统计信息减少了 IO 操作。
+例如，查询 `name` 等于 `Emily` 的行时，可以跳过 row group 0，因为其中 `name` 的最大值是 `Charlie`，无需读取该 row group。
## 索引文件
-对于每个 SST 文件，GreptimeDB 不但维护 SST 文件内部索引，还会单独生成一个文件用于存储针对该 SST 文件的索引结构。
+当一个 SST 存在已配置且适用的索引输出时，GreptimeDB 将这些索引写入与该 SST 关联的 Puffin 文件。没有适用索引的 SST 不需要生成 Puffin 文件。
-索引文件采用 [Puffin][3] 格式，这种格式具有较大的灵活性，能够存储更多的元数据，并支持更多的索引结构。
+Puffin 是索引 Blob 及其元数据的容器，使不同索引结构可以共用一个文件。
@@ -58,13 +62,13 @@ GreptimeDB 会将多种索引结构作为 Blob 存储在 Puffin 文件中，包
-例如，上述查询使用倒排索引来定位数据段，数据段满足条件：`job` 等于 `apiserver`，`handler` 符合正则匹配 `.*users` 及 `status` 符合正则匹配 `4..`，然后扫描这些数据段以产生满足所有条件的最终结果，从而显着减少 IO 操作的次数。
+上述查询使用倒排索引定位 `job` 等于 `apiserver`、`handler` 匹配 `.*users` 且 `status` 匹配 `4..` 的数据段。Mito 只扫描这些数据段，再应用剩余过滤条件。
### 倒排索引格式
-
+
-GreptimeDB 按列构建倒排索引，每个倒排索引包含一个 FST 和多个 Bitmap。
+GreptimeDB 按列构建倒排索引。每个列索引包含一个 null bitmap、多个 posting bitmap 和一个 FST。Blob footer 记录定位和解码各列索引所需的 offset、size 和元数据。
FST（Finite State Transducer）允许 GreptimeDB 以紧凑的格式存储列值到 Bitmap 位置的映射，并且提供了优秀的搜索性能和支持复杂搜索（例如正则表达式匹配）；Bitmap 则维护了数据段 ID 列表，每个位表示一个数据段。
@@ -80,7 +84,7 @@ GreptimeDB 把一个 SST 文件分割成多个索引数据段，每个数据段
## 统一数据访问层：OpenDAL
-GreptimeDB 使用 [OpenDAL][2] 提供统一的数据访问层，因此，存储引擎无需与不同的存储 API 交互，数据可以无缝迁移到基于云的存储，如 AWS S3。
+GreptimeDB 使用 [OpenDAL][2] 为本地文件系统和对象存储提供统一访问层。修改配置的存储 backend 不会迁移已有数据。
[1]: https://parquet.apache.org
[2]: https://github.com/datafuselabs/opendal
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md
new file mode 100644
index 0000000000..3905a1aecd
--- /dev/null
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md
@@ -0,0 +1,100 @@
+---
+keywords: [memtable, Mito engine, 写缓冲, flush, 时间分区, BulkMemtable]
+description: 介绍 Mito 如何在 memtable 中组织 Region 的可变数据，以及如何将这些数据写入 SST 文件。
+---
+
+# Memtable 设计
+
+Memtable 是 Mito 为每个 Region 维护的内存写缓冲。数据 flush 为 SST 文件前，读取可以先从 memtable 获取这些数据。Region version 确定 scan 可以读取的 memtable 和 SST 文件；配合 committed sequence 上限，scan 可以在写入和 flush 推进当前 version 时保持一致性。
+
+## 写入和 flush 生命周期
+
+对于使用 WAL 的常规写入，Mito 按以下顺序处理：
+
+```text
+写请求
+ |
+ v
+追加 WAL -> mutable memtable -> 发布 committed sequence
+ |
+ freeze
+ v
+ immutable memtable -> 写入 SST -> manifest edit
+```
+
+Region worker 先分配 sequence number 和 WAL entry ID，再将 mutation 追加到[预写日志](wal.md)。如果追加失败，Mito 不会更新 memtable。Memtable 更新成功后，Mito 发布 committed sequence，新的读取随后可以看到这些数据。配置了 `skip_wal` 的 Region 会跳过 WAL，但 memtable 更新和可见性顺序不变。
+
+Flush 在启动后台 SST 写入前，先冻结 mutable memtable 并安装一组新的 mutable memtable。后续写入可以继续进行，也不会修改已经冻结的数据。Flush 将 immutable memtable 写为 SST 文件，再持久化包含新文件、flushed WAL checkpoint 和 sequence checkpoint 的 manifest edit。只有 manifest edit 持久化成功后，Mito 才会从当前 Region version 中移除已 flush 的 memtable。Flush 失败时，immutable memtable 会保留，供后续任务重试。
+
+## Region version 和时间分区
+
+每个 Region 包含一个 mutable `TimePartitions` 容器，其中可以有多个 memtable：
+
+```text
+Region version
+├─ mutable TimePartitions
+│ ├─ [t0, t1) -> memtable
+│ └─ [t1, t2) -> memtable
+├─ immutable memtables
+└─ SST files
+```
+
+Mito 根据 time index 的值把每行数据路由到对应分区。分区使用左闭右开的时间范围，并按照固定时长对齐。该时长跟随 Region 的 compaction time window；在取得 compaction time window 前，Mito 使用一天作为初始值。乱序写入可能在最新分区之外创建更早的分区。
+
+冻结 Region 时，Mito 会同时冻结所有 mutable 时间分区，把其中的 memtable 移入 immutable 列表，再创建新的 `TimePartitions` 容器。Flush 失败可能留下多代 immutable memtable，因此读取和后续 flush 不能假定列表中只有一个对象。
+
+## Memtable 实现
+
+Mito 根据 Region 的 SST format、primary key encoding 和 memtable 选项选择实现：
+
+```text
+flat SST format（默认）或 sparse primary-key encoding -> BulkMemtable
+memtable.type=bulk -> BulkMemtable，并强制 flat SST
+primary_key SST + dense encoding（遗留） -> 遗留实现
+```
+
+使用默认 engine 配置时，没有显式指定 SST format 的 Region 会使用 `flat`，因此通常走 `BulkMemtable` 路径，本页其余内容也以它为准。这些规则用于排除不兼容的组合：flat format 或 sparse primary key encoding 必须使用 `BulkMemtable`；显式选择 bulk 实现则会强制使用 flat format。
+
+### BulkMemtable
+
+`BulkMemtable` 使用 flat Arrow 布局把写入保存为 part，而不是将数据行插入按时间序列组织的缓冲区：
+
+```text
+BulkMemtable
+├─ unordered_part
+│ └─ 小批量 BulkPart
+└─ parts
+ ├─ BulkPart (Arrow RecordBatch)
+ ├─ MultiBulkPart (未编码的 RecordBatch)
+ └─ EncodedBulkPart (内存中的 Parquet 数据)
+```
+
+小 part 先积累在 `unordered_part` 中，较大的 part 则直接进入 `parts`。后台 memtable compaction 对符合条件的 part 执行 merge sort，生成 `MultiBulkPart` 或编码为 `EncodedBulkPart`。Scan 利用 part 的统计信息裁剪 range；flush 可以将已编码的 range 写入 SST，无需再次解码和编码数据行。设计动机和性能数据见 [Scaling Time Series to Millions of Cardinalities: GreptimeDB's Flat Format](https://greptime.cn/blogs/2025-12-22-flat-format)。
+
+### 遗留实现
+
+使用遗留 `primary_key` SST format 且为 dense primary key encoding 的 Region 仍然使用 `TimeSeriesMemtable`，它按编码后的 primary key 对数据行分组，而不是保存 flat part。如果 Region 没有 primary key 列，同一个 builder 会创建 `SimpleBulkMemtable`。两者都是为已有表保留的兼容代码，`primary_key` format 退役后可能一并移除；新的工作应面向 bulk 和 flat 路径。
+
+已经删除的 `partition_tree` memtable 不是第三种实现。Option parser 仍接受 `memtable.type=partition_tree` 以兼容旧配置，但不会恢复该实现。Region 最终使用 bulk 和 flat 路径。
+
+## 读取快照
+
+Scan 通过一次 `VersionControl` 快照同时取得 Region version 和 committed sequence，并先确定 version，再应用 sequence 上限。如果单独读取 sequence 后再获取 version，flush 或 compaction 可能在两次读取之间移除旧输入，使 scan 得到不完整的快照。
+
+选定的 version 提供 mutable memtable、immutable memtable 和 SST 文件。Mito 先按照时间范围裁剪数据源，再使用 scan 的 projection、predicate 和 sequence range 从各 memtable 获取 range。Scan 将这些 range 与 SST range 合并，并在所有数据源上应用相同的排序、删除和 merge 语义。即使新的 Region version 已经移除某个 memtable，scan 持有的引用也会让该 memtable 存活到本次读取结束。
+
+## 内存压力
+
+每个 memtable 通过 engine 的 write-buffer manager 记录估算的堆内存分配量。冻结 memtable 后，这部分内存不再计入 mutable memory，但在所有引用释放前仍计入总用量。因此，mutable memory 只反映仍可接收写入的数据，总用量仍包含活跃 scan 保留的内存。
+
+全局 write-buffer 达到限制后，worker 会选择 Region 执行 flush。如果内存用量持续超过配置限制，Mito 会阻塞写入，并在达到更高阈值后拒绝写入。可选的 Region 级限制会单独约束热点 Region，避免其阻塞无关 Region。定期任务、手动请求和 Region 生命周期操作也可以触发 flush。
+
+## 修改约束
+
+修改 memtable 代码时必须保持以下性质：
+
+- 对于使用 WAL 的 Region，先追加 WAL，再把数据安装到 memtable；只有安装成功后才能发布 committed sequence。
+- SST 文件和 manifest edit 持久化前，冻结的 memtable 必须保持可读，并能在 flush 失败后重试。
+- 从同一个 `VersionControl` 快照取得 Region version 和 committed sequence，不得先单独读取 sequence 再获取 version。
+- 保留 scan 和 flush 在 memtable range 与 SST range 之间执行统一删除、去重和 merge 所需的排序及元数据。
+- 通过 write-buffer manager 记录内存分配，并且只在底层内存不再可能被引用时释放计数。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md
index c0cbf681b9..57baecf400 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md
@@ -7,9 +7,9 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述
## 概述
-`Metric` 引擎是 GreptimeDB 的一个组件，属于存储引擎的一种实现，主要针对可观测 metrics 等存在大量小表的场景。
+`Metric` 引擎用于存储包含大量小型指标表的负载。
-它的主要特点是利用合成的物理宽表来存储大量的小表数据，实现相同列复用和元数据复用等效果，从而达到减少小表的存储开销以及提高列式压缩效率等目标。表这一概念在 `Metric` 引擎下变得更更加轻量。
+它将这些逻辑表映射到共享的物理宽表，使其复用列和元数据，从而降低每张表的存储开销并改善列式压缩。
## 概念
@@ -18,7 +18,7 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述
### 逻辑表
逻辑表，即用户定义的表。与普通的表都完全一样，逻辑表的定义包括表的名称、列的定义、索引的定义等。用户的查询、写入等操作都是基于逻辑表进行的。用户在使用过程中不需要关心逻辑表和普通表的区别。
-从实现层面来说，逻辑表是一个虚拟的表，它并不直接读写物理的数据，而是通过将读写请求映射成对应物理表的请求来实现数据的存储与查询。
+逻辑表是虚拟表，本身不直接存储数据。Metric 引擎将其读写请求映射为对应物理表的请求。
### 物理表
物理表是真实存储数据的表，它拥有若干个由分区规则定义的物理 Region。
@@ -27,16 +27,14 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述
`Metric` 引擎的主要设计架构如下：
-
+
-在目前版本的实现中，`Metric` 引擎复用了 `Mito` 引擎来实现物理数据的存储及查询能力，并在此之上同时提供物理表与逻辑表的访问能力。
+`Metric` 引擎将物理存储和查询交给 `Mito` 引擎。每个物理 Region 组包含一个数据 Region 和一个元数据 Region：数据 Region 保存映射到该 Region 组的逻辑表数据，元数据 Region 保存逻辑表及逻辑列的映射。
-在分区方面，逻辑表拥有与物理表完全一致的分区规则及 Region 分布。这是非常自然的，因为逻辑表的数据直接存储在物理表中，所以分区规则也是一致的。
+关联到同一物理表的逻辑表使用相同的分区布局。写入时，Metric 引擎为每行数据记录逻辑表身份；读取时，它在扫描物理 Region 前增加逻辑表过滤条件。
-在路由元数据方面，逻辑表的路由地址为逻辑地址，即该逻辑表所对应的物理表是什么，而后通过该物理表进行二次路由取得真正的物理地址。这一间接路由方式能够显著减少 `Metric` 引擎的 Region 发生迁移调度时所需要修改的元数据数量。
+逻辑表的路由只保存所属物理表的 ID，再由物理表路由解析出持有 Region 的 Datanode。逻辑路由本身不记录 peer，因此迁移物理 Region 只需改写一条物理路由，而不必改写映射到它的每一条逻辑路由。
-在操作方面，`Metric` 引擎支持对逻辑表进行标准的 DML 操作（INSERT、DELETE、SELECT）。然而，对物理表的操作进行了有限的支持以防止误操作，例如禁止直接写入物理表等操作防止影响用户逻辑表的数据。总体上可以认为物理表是对用户只读的。
+逻辑表支持普通的 INSERT、DELETE 和 SELECT 操作。直接写入物理 Region 会绕过逻辑表映射，因此会被拒绝；物理表仍然可以查询。
-为了提升对大量表同时进行 DDL（Data Definition Language，数据操作语言）操作时性能，如 Prometheus Remote Write 冷启动时大量 metrics 带来的自动建表请求，以及前面提到的迁移物理 Region 时大量路由表的修改请求等，`Metric` 引擎引入了一些批量 DDL 操作。这些批量 DDL 操作能够将大量的 DDL 操作合并成一个请求，从而减少了元数据的查询及修改次数，提升了性能。
-
-除了物理表的物理数据 Region 之外，`Metric` 引擎还额外为每一个物理数据 Region 创建了一个物理的元数据 Region，用于存储 `Metric` 引擎自身为了维护映射等状态所需要的一些元数据。这些元数据包括逻辑表与物理表的映射关系，逻辑列与物理列的映射关系等等。
+批量 DDL 用于减少大量逻辑表同时创建或更新时的元数据操作，例如 Prometheus Remote Write 自动建表或物理 Region 迁移。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md
index 3f44a4543b..c7684eb165 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md
@@ -7,22 +7,26 @@ description: 介绍了 Datanode 的主要职责和组件，包括 gRPC 服务、
## Introduction
-`Datanode` 主要的职责是为 GreptimeDB 存储数据，我们知道在 GreptimeDB 中一个 `table` 可以有一个或者多个 `Region`,
-而 `Datanode` 的职责便是管理这些 `Region` 的读写。`Datanode` 不感知 `table`，可以认为它是一个 `region server`。
-所以 `Frontend` 和 `Metasrv` 按照 `Region` 粒度来操作 `Datanode`。
+Datanode 存储并处理 Region 数据。一张表可以包含多个 Region，但 Datanode 不负责表级路由。Frontend 按 Region 发送数据请求，Metasrv 则控制 Region 的放置和生命周期。
-
+这个边界使同一个 Region server 可以承载不同的存储引擎，而不向 Frontend 或 Metasrv 暴露引擎实现。
+
+
## Components
-一个 datanode 包含了 region server 所需的全部组件。这里列出了比较重要的部分：
-
-- 一个 gRPC 服务来提供对 `Region` 数据的读写，`Frontend` 便是使用这个服务来从 `Datanode` 读写数据。
-- 一个 HTTP 服务，可以通过它来获得当前节点的 metrics、配置信息等
-- `Heartbeat Task` 用来向 `Metasrv` 发送心跳，心跳在 GreptimeDB 的分布式架构中发挥着至关重要的作用，
- 是分布式协调和调度的基础通信通道，心跳的上行消息中包含了重要信息比如 `Region` 的负载，如果 `Metasrv` 做出了调度
- 决定（比如 Region 转移），它会通过心跳的下行消息发送指令到 `Datanode`
-- `Datanode` 不负责解析用户 SQL 或进行分布式规划，用户对一个或多个 `Table` 的查询请求会在 `Frontend` 中被转换为
- `Region` 查询请求，`Datanode` 负责用本地 query engine 执行这些 `Region` 查询计划
-- 一个 `Region Manager` 用来管理 `Datanode` 上的所有 `Region`s
-- GreptimeDB 支持可插拔的多引擎架构，目前已有的 engine 包括 `File Engine` 和 `Mito Engine`
+Datanode 包含以下主要组件：
+
+- Region server 记录已打开的 Region，并把读写和生命周期请求分发给该 Region 注册的 engine。
+- `Mito` 是主要的时序 Region engine。`Metric` 将多个逻辑指标 Region 映射到共享的 Mito Region，`File` 通过 Region 接口访问外部文件。
+- 本地 query engine 执行 Region 查询计划。它不解析客户端 SQL，也不进行集群级规划。
+- Heartbeat task 向 Metasrv 上报节点和 Region 状态，并接收 open、close、upgrade、downgrade 和迁移步骤等指令。
+- gRPC 承载发往 Datanode 的 Region 请求；HTTP 提供 metrics 和配置等节点诊断信息。
+
+## Region 请求生命周期
+
+Mito 写入到达 Region server 后，Region server 根据 Region 元数据选择 Mito。Mito 将 mutation 追加到 WAL，写入 memtable，并在之后把 memtable flush 为 SST 文件。Metric 写入会先补充逻辑表标识，再委托给对应的物理 Mito Region。
+
+读取时，本地 query engine 在 Region engine 提供的 table provider 上执行 Region 计划。Mito scan 获取不可变的 Region version，读取相关 memtable 和 SST 文件，合并并去重数据，最后返回 Arrow record batch 流。
+
+Region 所有权可以在不重启 Datanode 的情况下改变。Metasrv 通过心跳流下发生命周期指令；Region server 将指令应用到对应 engine，并在后续心跳中上报新的 Region role 和统计信息。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/python-scripts.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/python-scripts.md
deleted file mode 100644
index 831278e80d..0000000000
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/python-scripts.md
+++ /dev/null
@@ -1,30 +0,0 @@
----
-keywords: [Python 脚本, 数据分析, CPython, RustPython]
-description: 介绍了在 GreptimeDB 中使用 Python 脚本进行数据分析的两种后端实现：CPython 和嵌入式 RustPython 解释器。
----
-
-# Python 脚本
-
-## 简介
-
-Python 脚本是分析本地数据库中的数据的便捷方式，
-通过将脚本直接在数据库内运行而不是从数据库拉取数据的方式，可以节省大量的数据传输时间。
-下图描述了 Python 脚本的工作原理。
-`RecordBatch`（基本上是表中的一列，带有类型和元数据）可以来自数据库中的任何地方，
-而返回的 `RecordBatch` 可以用 Python 语法注释以指示其元数据，例如类型或空。
-脚本将尽其所能将返回的对象转换为 `RecordBatch`，无论它是 Python 列表、从参数计算出的 `RecordBatch` 还是常量（它被扩展到与输入参数相同的长度）。
-
-
-
-## 两种可选的后端
-
-### CPython 后端
-
-该后端由 [PyO3](https://pyo3.rs/v0.18.1/) 提供支持，可以使用您最喜欢的 Python 库（如 NumPy、Pandas 等），并允许 Conda 管理您的 Python 环境。
-
-但是使用它也涉及一些复杂性。您必须设置正确的 Python 共享库，这可能有点棘手。一般来说，您只需要安装 `python-dev` 包。但是，如果您使用 Homebrew 在 macOS 上安装 Python，则必须创建一个适当的软链接到 `Library/Frameworks/Python.framework`。有关使用 PyO3 crate 与不同 Python 版本的详细说明，请参见 [这里](https://pyo3.rs/v0.18.1/building_and_distribution#configuring-the-python-version)
-
-### 嵌入式 RustPython 解释器
-
-可以运行脚本的实验性 [python 解释器](https://github.com/RustPython/RustPython)，它支持 Python 3.10 语法。您可以使用所有的 Python 语法，更多信息请参见 [Python 脚本的用户指南](/user-guide/python-scripts/overview.md).
-
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/query-engine.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/query-engine.md
index 62d7563bb2..84445759d4 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/query-engine.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/query-engine.md
@@ -7,33 +7,33 @@ description: 介绍了 GreptimeDB 的查询引擎架构，基于 Apache DataFusi
## 介绍
-GreptimeDB 的查询引擎是基于[Apache DataFusion][1]（属于[Apache Arrow][2]的子项目）构建的，它是一个用 Rust 编写的出色的查询引擎。它提供了一整套功能齐全的组件，从逻辑计划、物理计划到执行运行时。下面将解释每个组件如何被整合在一起，以及在执行过程中它们的位置。
+GreptimeDB 的查询引擎基于 [Apache DataFusion][1]。DataFusion 提供逻辑计划、物理计划、优化器框架和执行运行时；GreptimeDB 在此基础上增加各查询语言的 planner、存储相关优化规则、自定义计划节点和分布式执行。
-
+DDL 和其他控制面操作由 statement executor 分发。Query engine 接收数据处理计划，包括 `INSERT ... SELECT` 等操作中读取输入数据的部分。
-入口点是逻辑计划，它被用作查询或执行逻辑等的通用中间表示。逻辑计划的两个主要来源是：1. 用户查询，例如通过 SQL 解析器和规划器的 SQL；2. Frontend 的分布式查询，这将在下一节中详细解释。
+## 查询生命周期
-接下来是物理计划，或称执行计划。与包含所有逻辑计划变体（除特殊扩展计划节点外）的大型枚举的逻辑计划不同，物理计划实际上是一个定义了在执行过程中调用的一组方法的特性。所有数据处理逻辑都包装在实现该特性的相应结构中。它们是对数据执行的实际操作，如聚合器 `MIN` 或 `AVG` ，以及表扫描 `SELECT ... FROM`。
+1. SQL、PromQL 或日志查询 planner 通过 catalog 解析表，并生成 DataFusion logical plan。DataFusion 不直接支持的操作由 GreptimeDB plan extension 表示。
+2. DataFusion 的 analyzer 和 optimizer rule 与 GreptimeDB rule 共同运行。这些规则规范化表达式和类型、改写时间范围操作、将 projection 和 filter 下推到 scan，并在需要时引入分布式计划节点。
+3. Physical planner 将优化后的 logical plan 转换为流式 operator。GreptimeDB 随后应用 scan 并行度、排序和分布式执行相关的 physical rule。
+4. 执行阶段通过 physical plan 拉取 Arrow record batch。存储 scan 接收 projection 和 predicate，下游 operator 消费数据流，无需先物化完整结果。
-优化阶段通过转换逻辑计划和物理计划来提高执行性能，现在全部基于规则。它也被称为“基于规则的优化”。一些规则是 DataFusion 原生的，其他一些是在 GreptimeDB 中自定义的。在未来，我们计划添加更多规则，并利用数据统计进行基于成本的优化 (CBO)。
-
-最后一个阶段"执行"是一个动词，代表从存储读取数据、进行计算并生成预期结果的过程。虽然它比之前提到的概念更抽象，但你可以简单地将它想象为执行一个 Rust 异步函数，并且它确实是一个异步流。
-
-当你想知道 SQL 是如何通过逻辑计划或物理计划中表示时，`EXPLAIN [VERBOSE] ` 是非常有用的。
+使用 [`EXPLAIN`](/reference/sql/explain.md) 查看逻辑和物理计划。`EXPLAIN ANALYZE` 还会执行计划并报告运行时指标。
## 数据表示
-GreptimeDB 使用 [Apache Arrow][2]作为内存中的数据表示格式。它是面向列的，以跨平台格式，也包含许多高性能的基础操作。这些特性使得在许多不同的环境中共享数据和实现计算逻辑变得容易。
+GreptimeDB 使用 [Apache Arrow][2] record batch 作为内存数据表示。一个 record batch 包含等长的列数组和 schema。查询 operator 交换这些 batch 组成的数据流，使 Region scan 到结果编码的执行路径保持列式处理。
## 索引
-在时序数据中，有两个重要的维度：时间戳和标签列（或者类似于关系数据库中的主键）。GreptimeDB 将数据分组到时间桶中，因此能在非常低的成本下定位和提取预期时间范围内的数据。GreptimeDB 中主要使用的持久文件格式 [Apache Parquet][3] 提供了多级索引和过滤器，使得在查询过程中很容易修剪数据。在未来，我们将更多地利用这个特性，并开发我们的分离索引来处理更复杂的用例。
+索引构建和持久化格式属于存储引擎。查询层向 scan 提供 predicate 和 projection，Mito 再利用时间范围、Parquet 统计信息和索引跳过不可能匹配的数据。参见[数据持久化与索引](./data-persistence-indexing.md)。
+
+
-## 分布式查询
+## 分布式执行
-参考 [Distributed Querying][6].
+分布式模式下，Frontend 规划集群级查询，Datanode 执行 Region 本地子计划。[`MergeScan`][6] 是两个阶段之间的边界。
-[1]: https://github.com/apache/arrow-datafusion
+[1]: https://datafusion.apache.org/
[2]: https://arrow.apache.org/
-[3]: https://parquet.apache.org
[6]: ../frontend/distributed-querying.md
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md
index 67f98216fe..d3abb58cfa 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md
@@ -7,7 +7,7 @@ description: 详细介绍了 GreptimeDB 的存储引擎架构、数据模型和
## 概述
-`存储引擎` 负责存储数据库的数据。Mito 是我们默认使用的存储引擎，基于 [LSMT][1]（Log-structured Merge-tree）。我们针对处理时间序列数据的场景做了很多优化，因此 mito 这个存储引擎并不适用于通用用途。
+Mito 是 GreptimeDB 的默认存储引擎，基于 [LSM tree][1]，面向时间序列负载设计，而不是通用的嵌入式存储引擎。
## 架构
下图展示了存储引擎的架构和处理数据的流程。
@@ -20,9 +20,9 @@ description: 详细介绍了 GreptimeDB 的存储引擎架构、数据模型和
- 为尚未刷盘的数据提供高持久性保证。
- 基于 `LogStore` API 实现，不关心底层存储介质。
- WAL 的日志记录可以存储在本地磁盘上，也可以存储在实现了 `LogStore` API 的远程日志服务中，例如 Kafka（remote WAL）。
-- Memtable
- - 数据首先写入 `active memtable`，又称 `mutable memtable`。
- - 当 `mutable memtable` 已满时，它将变为只读的 `immutable memtable`。
+- [Memtable](memtable.md)
+ - Mito 根据 time index 将数据行写入 mutable memtable。
+ - Flush 冻结 mutable memtable，安装一组新的 mutable memtable 以接收写入，再将冻结的 memtable 写为 SST 文件。
- SST
- SST 的全名为有序字符串表（`Sorted String Table`）。
- `immutable memtable` 刷到持久存储后形成一个 SST 文件。
@@ -100,7 +100,9 @@ Mito 会按 primary key 对行分组，并按时间排序，因此 SST 中的数
Mito 支持两种 SST 格式：`flat` 和 `primary_key`。`flat` 是新表的默认格式，适用于各种 primary key 基数，包括高基数 key。`primary_key` 是为了兼容旧表而保留的遗留格式。更多详情请参考 [SST format](/reference/sql/create.md#创建指定-sst-格式的表) 和[表设计指南](/user-guide/deployments-administration/performance-tuning/design-table.md#sst-格式)。
-
+
+
+一个 SST 可能跨越多个 compaction time window。
## 扫描裁剪
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/wal.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/wal.md
index 529adcdf50..36e0689430 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/wal.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/wal.md
@@ -9,20 +9,26 @@ description: 介绍了 GreptimeDB 的预写日志（WAL）机制，包括其命
## 介绍
-我们的存储引擎受到了日志结构合并树（Log-structured Merge Tree，LSMT）的启发。对数据的变更操作直接应用于 MemTable 而不是持久化到磁盘上的数据页，这显著提高了性能，但也带来了持久化相关的问题，特别是在 Datanode 意外崩溃时。与所有类似 LSMT 的存储引擎一样，GreptimeDB 使用预写日志（Write-Ahead Log，WAL）来确保数据被可靠地持久化，并且保证崩溃时的数据完整性。
+Mito 在将数据 flush 为 SST 文件前，先在 [memtable](memtable.md) 中缓冲写入。每个 Region 的 mutation 会先追加到预写日志（WAL），从而恢复尚未进入 SST 的数据。
-预写日志是一个仅提供追加写的文件组。所有的 INSERT 和 DELETE 操作都被转换为操作日志，然后追加到 WAL。一旦操作日志被持久化到底层文件，该操作才可以进一步应用到 MemTable。
+WAL 通过统一的 log-store 抽象访问，可以使用本地 raft-engine 或远端 Kafka。
-当数据节点重新启动时，WAL 中的操作条目将被重放，以重建正确的 MemTable 状态。
+## 写入与恢复流程
-
+正常写入遵循以下顺序：
+
+1. Region worker 分配 sequence number 和 WAL entry ID。
+2. 将 mutation 追加到 WAL。追加失败时，不会把 mutation 写入 memtable。
+3. WAL 追加成功后，Mito 将 mutation 写入 memtable，并发布新的 committed sequence。
+4. Flush 将不可变 memtable 写为 SST 文件，并持久化包含新文件和 `flushed_entry_id` 的 manifest edit。
+5. Manifest edit 持久化后，`flushed_entry_id` 及以前的 WAL entry 被标记为 obsolete；log store 可以稍后再回收物理空间。
+
+Manifest 是恢复边界。正常重新打开 Region 时，Mito 根据 manifest 重建 Region，并从 `flushed_entry_id + 1` 开始重放 WAL。Region 状态切换可以指定更晚的 replay checkpoint，但不会重放早于已持久化 flush 边界的 entry。
## 命名空间
-WAL 的命名空间用于区分来自不同 region 的条目。追加和读取操作必须提供一个命名空间。目前，region ID 被用作命名空间，因为每个 region 都有一个在数据节点重新启动时需要重构的 MemTable。
+WAL entry 按 Region 隔离，而不是按表隔离。追加和读取都需要指定 Region namespace，使单个 Region 可以独立重放或截断。本地 raft-engine 使用 Region ID 作为 namespace ID；Kafka provider 则在基于 topic 的日志中保留 Region 标识。
## 同步/异步刷盘
-默认情况下，WAL 的追加写是异步的，这意味着写入方不会等待操作日志被刷入到磁盘并持久化。这个默认设置提供了更高的性能，但在服务器意外关闭时可能会丢失数据。另一方面，同步刷新提供了更高的可靠性，但其代价是性能更低。
-
-在 v0.4 版本中，新的 region worker 架构可以使用批处理来减轻同步刷盘的开销。
+对于本地 raft-engine，`sync_write` 控制追加写是否等待日志同步到持久化存储，默认值为 `false`。异步写入延迟较低，但主机在缓冲数据同步前故障时，可能丢失最近确认的 entry。Kafka WAL 的持久性由 producer 和集群配置决定，不受这个本地选项控制。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/arrangement.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/arrangement.md
index dd3b6de090..7b35b50259 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/arrangement.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/arrangement.md
@@ -5,6 +5,8 @@ description: 描述了 Arrangement 在数据流进程中的状态存储功能，
# Arrangement
+本页介绍 Flownode 旧 streaming 模式使用的状态结构；batching 模式不使用 Arrangement。
+
Arrangement 存储数据流进程中的状态，存储 flow 的更新流（stream）以供进一步查询和更新。
Arrangement 本质上存储的是带有时间戳的键值对。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md
index bec092b0af..8e2ecdb80b 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md
@@ -9,13 +9,13 @@ description: Flownode 批处理模式概述，这是持续数据聚合当前使
## 概述
-`flownode` 中的批处理模式专为持续数据聚合而设计。它在离散的、微小的时间窗口上周期性地执行用户定义的 SQL 查询。这与原始的流处理模式形成对比；流处理模式现在已经废弃，在该模式下数据会在到达时即被处理。
+`flownode` 中的批处理模式专为持续数据聚合而设计。它在离散的小时间窗口上周期性执行用户定义的 SQL 查询。旧 streaming 路径则在数据到达时进行处理，目前仅为兼容已有 workload 而保留，不推荐新 workload 使用。
其核心思想是：
1. 定义一个带有 SQL 查询的 `flow`，该查询将数据从源表聚合到目标表。
2. 查询通常在时间戳列上包含一个时间窗口函数（例如 `date_bin`）。
3. 当新数据插入源表时，系统会将相应的时间窗口标记为“脏”(dirty)。
-4. 一个后台任务会周期性地唤醒，识别这些脏窗口，并为那些特定的时间范围重新运行聚合查询。
+4. 一个后台任务按自身的节奏运行，在下一次求值时取出待处理的脏窗口，并对这些时间范围重新运行聚合查询。
5. 然后将结果插入到目标表中，从而有效地更新聚合视图。
## 架构
@@ -39,15 +39,15 @@ description: Flownode 批处理模式概述，这是持续数据聚合当前使
- **状态 (`TaskState`)**: 包含任务的动态、可变状态，最重要的是 `DirtyTimeWindows`。
- **执行循环**: 任务运行一个无限循环 (`start_executing_loop`)，该循环：
1. 检查关闭信号。
- 2. 等待一个预定的时间间隔或直到被唤醒。
+ 2. 睡眠到下一次求值时间。设置了求值调度的任务睡眠到下一个调度时间点；自适应任务则按时间窗口大小和最小刷新间隔计算出的轮询间隔睡眠。
3. 基于当前的脏时间窗口集合生成一个新的查询计划 (`gen_insert_plan`)。
4. 对数据库执行查询 (`execute_logical_plan`)。
5. 清理已处理的脏窗口。
### `TaskState` 和 `DirtyTimeWindows`
-- **`TaskState`**: 此结构体跟踪 `BatchingTask` 的运行时状态。它包括 `dirty_time_windows`，这对于确定需要完成哪些操作至关重要。
-- **`DirtyTimeWindows`**: 这是一个关键的数据结构，用于跟踪自上次查询执行以来哪些时间窗口接收到了新数据。它存储一组不重叠的时间范围。当任务的执行循环运行时，它会参考此结构来构建一个 `WHERE` 子句，该子句仅过滤源表中的脏时间窗口。
+- **`TaskState`**: 此结构体跟踪 `BatchingTask` 的运行时状态，包括用于确定待处理工作的 `dirty_time_windows`。
+- **`DirtyTimeWindows`**: 此数据结构跟踪上次查询执行后接收到新数据的时间窗口，并保存一组不重叠的时间范围。执行循环根据它构造 `WHERE` 子句，只从源表选择脏窗口。
### `TimeWindowExpr`
@@ -56,15 +56,15 @@ description: Flownode 批处理模式概述，这是持续数据聚合当前使
- **求值**: 它可以接受一个时间戳并对时间窗口表达式求值，以确定该时间戳所属窗口的开始和结束。
- **窗口大小**: 它还可以从表达式中确定时间窗口的大小（持续时间）。
-这对于标记窗口为脏以及在查询源表时生成正确的过滤条件都至关重要。
+标记脏窗口和生成源表过滤条件使用同一套计算。
## 查询执行流程
以下是批处理模式下查询执行的简化分步演练：
1. **数据摄取**: 新数据被写入源表。
-2. **标记为脏**: `BatchingEngine` 收到有关新数据的通知。它使用与每个相关 flow 关联的 `TimeWindowExpr` 来确定哪些时间窗口受到新数据点的影响。然后将这些窗口添加到相应 `TaskState` 中的 `DirtyTimeWindows` 集合中。
-3. **任务唤醒**: `BatchingTask` 的执行循环被唤醒，原因可能是其周期性调度，也可能是因为它被通知有大量积压的脏窗口。
+2. **标记为脏**: `BatchingEngine` 收到有关新数据的通知。它使用与每个相关 flow 关联的 `TimeWindowExpr` 来确定哪些时间窗口受到新数据点的影响。然后将这些窗口添加到相应 `TaskState` 中的 `DirtyTimeWindows` 集合中。标记脏窗口不会唤醒任务。
+3. **下一次求值**: `BatchingTask` 的执行循环在调度时间点或自适应轮询间隔结束后进入下一次求值，取出待处理的脏窗口。
4. **计划生成**: 任务调用 `gen_insert_plan`。此方法：
- 检查 `DirtyTimeWindows`。
- 生成一系列 `OR` 连接的 `WHERE` 子句（例如 `(ts >= 't1' AND ts < 't2') OR (ts >= 't3' AND ts < 't4') ...`），覆盖所有脏窗口。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md
index 9d07922542..95def5e844 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md
@@ -1,18 +1,39 @@
---
-keywords: [Dataflow, SQL 查询, 执行计划, 数据流, map, reduce]
-description: 解释了 Dataflow 模块的核心计算功能，包括 SQL 查询转换、内部执行计划、数据流的触发运行和支持的操作。
+keywords: [Flownode, batching mode, streaming mode, Dataflow, 脏时间窗口]
+description: 介绍 Flownode 如何选择并运行 batching 和旧 streaming 两条执行路径。
---
# 数据流
+Flownode 内部有两条执行路径：
+
+- **Batching mode** 是聚合和 TQL workload 的主要执行路径。它查询已经持久化的 source 数据，并将物化结果写入 sink table。
+- **Streaming mode** 是为兼容已有 workload 而保留的旧执行路径，不推荐新 workload 使用。Frontend 会把新到达的行同步给它进行增量处理。
+
+用户不能直接选择执行模式。创建 Flow 时，GreptimeDB 根据查询和 source table 的属性选择执行路径。聚合、`DISTINCT` 和 TQL 查询使用 batching mode；简单的非聚合查询，以及任何 source table 使用 `ttl = 'instant'` 的 Flow，目前仍使用 streaming mode。如果 source table 尚不存在并选择延迟创建，Flow 会先成为 pending batching Flow。
+
+## Batching mode
+
+Batching mode 复用 GreptimeDB 的查询引擎，不需要为每一行输入维护一张算子图。对于基于时间窗口的 Flow，主循环如下：
+
+1. Source table 收到写入后，把受影响的时间窗口标记为 dirty。
+2. `BatchingTask` 按求值调度或自适应轮询节奏运行，并在该次求值时收集待处理的 dirty window。标记 dirty window 不会唤醒任务。
+3. 任务把这些窗口转换成时间谓词，加入 Flow 查询，再请求 Frontend 查询 source table。
+4. 查询结果写入 sink table，更新已重新计算窗口对应的物化结果。
+5. 成功处理的窗口从 dirty set 中移除；执行失败的工作仍可在后续调度中处理。
+
+设置了 evaluation interval、但查询中没有时间窗口表达式的 Flow，会在每次调度时执行完整查询。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见 [Flownode 批处理模式开发者指南](./batching_mode.md)。
+
+## Streaming mode
+
Dataflow 模块（参见 `flow::compute` 模块）是 `flow` 的核心计算模块。
它接收 SQL 查询并将其转换为 `flow` 的内部执行计划。
然后，该执行计划被转化为实际的数据流，而数据流本质上是一个由带有输入和输出端口的函数组成的有向无环图（DAG）。
-数据流会在需要时被触发运行。
+新到达的行变更会增量驱动这张图执行。
-目前该数据流只支持 `map`和 `reduce` 操作，未来将添加对 `join` 等操作的支持。
+Renderer 支持 map/filter/project 和 reduce 操作。执行计划中已经有 join 和 union 节点，但 streaming renderer 尚未实现它们。
在内部，数据流使用 `tuple(row, time, diff)` 以行格式处理数据。
这里 `row` 表示实际传递的数据，可能包含多个 `value` 对象。
`time` 是系统时间，用于跟踪数据流的进度，`diff` 通常表示行的插入或删除（+1 或 -1）。
-因此，`tuple` 表示给定系统时间的 `row` 的插入/删除操作。
+因此，`tuple` 表示给定系统时间的 `row` 的插入/删除操作。有状态算子通过 [Arrangement](./arrangement.md) 保存这些变更的索引 trace。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/distributed-querying.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/distributed-querying.md
index 612174662b..e6d0c35160 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/distributed-querying.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/distributed-querying.md
@@ -1,41 +1,20 @@
---
-keywords: [分布式查询, 查询拆分, 查询合并, TableScan, 物理计划]
-description: 介绍 GreptimeDB 中的分布式查询方法，包括查询的拆分和合并过程，以及 TableScan 节点的作用。
+keywords: [分布式查询, 逻辑计划, MergeScan, Substrait, Region 裁剪]
+description: 介绍 GreptimeDB 如何把逻辑查询计划划分为 Frontend 和 Datanode 上的执行任务。
---
# 分布式查询
-我们知道在 GreptimeDB 中数据是如何分布的（参见“[表分片][1]”），那么如何查询呢？在 GreptimeDB 中，分布式查询非常简单。简单来说，我们只需将查询拆分为子查询，每个子查询负责查询表数据的一个部分，然后将所有结果合并为最终结果。这是一种典型的“拆分 - 合并”方法。具体来说，让我们从查询到达 `frontend` 开始。
+Frontend 和 Datanode 使用同一套基于 DataFusion 的查询引擎。在分布式模式下，Frontend 会增加一个规划步骤，将 Datanode 上执行的工作与 Frontend 上完成的工作分开。
-当查询到达 `frontend` 时，它首先被解析为 SQL 抽象语法树（AST）。我们遍历 AST，并从中生成逻辑计划。顾名思义，逻辑计划只是如何“逻辑地”执行查询的“提示”，它不能被直接运行，因此我们进一步从中生成可执行的物理计划。物理计划是一种类似树形的数据结构，每个节点实际上表示查询的执行方法。一旦我们从上到下运行物理计划树，结果数据将从叶子到根流动，被合并或计算。最终，我们在根节点的输出处得到了查询的结果。
+
-到目前为止，这只是一个典型的“volcano”查询执行模型，你可以在几乎每个 SQL 数据库中看到这种模型。那么“分布式”是在哪里发生的呢？这全部发生在一个名为“TableScan”的物理计划节点中。TableScan 是物理计划树中的一个叶子节点，它负责扫描表的数据（就像它的名称所暗示的）。当 `frontend` 即将扫描表时，它首先需要根据每个 `region` 的数据范围将表扫描拆分为较小的扫描。
+## 分布式规划
-[1]: ./table-sharding.md
+分布式规划器重写逻辑计划，把可以下推的算子移向表扫描，并用 `MergeScan` 节点包装远端子计划。分区列上的谓词还会在任务调度前用于裁剪 Region。
-表的所有 `region` 都有它们存储数据的范围。以下表为例：
+算子能否下推取决于计划形态和算子本身的性质。不支持的部分会保留在 Frontend。初始设计及交换律规则参见[分布式规划器 RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md)。
-```sql
-CREATE TABLE my_table (
- a INT,
- others STRING,
- ts TIMESTAMP TIME INDEX,
-)
-PARTITION ON COLUMNS (a) (
- a < 10,
- a >= 10 AND a < 20,
- a >= 20
-);
-```
+## 分布式计划
-`my_table` 表创建时被设定了 3 个分区。在 GreptimeDB 的当前实现中，将为该表创建 3 个 `region`（分区与 `region` 的比例为 1:1）。这 3 个区域将分别包含以下范围："[-∞, 10)", "[10, 20)" 和 "[20, +∞)"。例如，如果提供了值 "42"，我们将搜索这些范围，并找到包含该值的相应的 `region`（在此示例中为第 3 个 `region`）。
-
-对于查询，我们使用“过滤器”来查找 `region`。 "过滤器"是 "WHERE" 子句中的条件。例如，查询 `SELECT * FROM my_table WHERE a < 10 AND others = 'x'`，其“过滤器”为“a < 10 AND others = 'x'”。然后我们检查这些范围，找出包含满足过滤器条件的值的所有 `region`。
-
-> 如果某个查询没有任何过滤器，则将其视为全表扫描。
-
-找到所需的区域后，我们只需在其中组装子扫描。通过这种方式，我们将查询拆分为子查询，每个子查询都获取表数据的一部分。子查询在 `datanode` 中执行，并在 `frontend` 中等待完成。它们的结果将合并为表扫描请求的最终返回。
-
-下面这张图片总结了分布式查询执行的过程：
-
-
+远端输入是完整的逻辑子计划，并不局限于表扫描。Frontend 使用 [Substrait](https://substrait.io) 序列化子计划，再向持有相应数据的 Datanode 发送 Region 级请求。Datanode 在本地规划并执行子计划，将结果流返回 Frontend。Frontend 合并远端数据流，并执行没有下推的算子。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/overview.md
index ee48da3ce3..6befdc55a9 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/overview.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/overview.md
@@ -5,31 +5,51 @@ description: GreptimeDB Frontend 组件概述 - 为客户端请求提供服务
# Frontend
-**Frontend** 是一个无状态服务，作为 GreptimeDB 中客户端请求的入口点。它为多种数据库协议提供统一接口，并充当代理，将读写请求转发到分布式系统中的相应 Datanode。
+Frontend 是 GreptimeDB 中负责请求编排的无状态服务。Server 层负责终止协议并转换线上消息；Frontend 为这些协议处理器提供数据库行为，包括权限检查、语句执行、路由和分布式查询规划。
+
+Frontend 不存储表数据。它缓存从 Metasrv 获取的 catalog 和路由元数据；元数据发生变化时，Metasrv 通过心跳响应通知 Frontend 失效相应缓存。
## 核心功能
-- **协议支持**：支持多种数据库协议，包括 SQL、PromQL、MySQL 和 PostgreSQL。详见[协议][1]
-- **请求路由**：基于元数据将请求路由到相应的 Datanode
-- **查询分发**：将分布式查询拆分到多个节点
-- **响应聚合**：合并来自多个 Datanode 的结果
-- **认证授权**：安全和访问控制验证
+- 为支持的[协议][1]提供查询和写入行为。
+- 解析 catalog、schema、table 和 Region 路由。
+- 在执行请求前完成权限检查。
+- 规划分布式查询并合并 Datanode 返回的结果。
+- 将表级写入和删除转换为 Region 请求。
## 架构
### 关键组件
-- **协议处理器**：处理不同的数据库协议
-- **目录管理器**：缓存来自 Metasrv 的元数据以实现高效的请求路由和 Schema 校验
-- **分布式规划器**：将逻辑计划转换为分布式执行计划
-- **请求路由器**：为每个请求确定目标 Datanodes
+
+- 协议处理器将 SQL、PromQL、gRPC 写入和可观测性协议转换为 Frontend 的内部请求接口。
+- Catalog manager 和 partition manager 提供表元数据、分区规则和 Region 路由。
+- Statement executor 将查询、DML 和 DDL 分发到各自的执行路径。
+- 分布式规划器把表扫描替换为可跨 Datanode 执行的 `MergeScan` 计划。
### 请求流程
-
+不同操作会走不同的请求路径。
+
+#### 查询
+
+1. 协议处理器创建查询上下文，并完成认证和权限检查。
+2. 对应查询语言的 planner 生成逻辑计划。分布式模式下，planner 根据分区元数据选择 Region 并生成分布式计划。
+3. Frontend 将 Region 子计划发送到对应 Datanode。Datanode 在本地 Region engine 上执行，并返回 Arrow record batch 流。
+4. Frontend 执行剩余算子、合并数据流，再按客户端协议编码结果。
+
+#### 写入和删除
+
+1. Frontend 根据表 schema 校验请求。支持 schema-on-write 的协议可以先创建缺失的表或新增列，再重试写入。
+2. 分区规则把每一行分配给 Region。Frontend 为各目标 Region 构造请求，并路由到当前 Region leader。
+3. Datanode 的 Region server 将请求分发到对应的 Region engine。单机模式下，请求直接发送给内嵌的 Region server。
+
+#### DDL
+
+Statement executor 将 DDL 转换为 task。分布式模式下，Metasrv 以持久化 procedure 执行 task、更新元数据，并协调 Datanode 上的 Region 操作。单机模式复用相同的语句边界，但使用本地元数据和 procedure 实现。
### 部署
-下图是 GreptimeDB 在云上的一个典型的部署。`Frontend` 实例组成了一个集群处理来自客户端的请求：
+下图展示了 GreptimeDB 的一种云上部署。多个 Frontend 实例共同处理客户端请求：
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/table-sharding.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/table-sharding.md
index 63a8ac10a6..cf39afe27b 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/table-sharding.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/table-sharding.md
@@ -5,21 +5,17 @@ description: 介绍 GreptimeDB 中表数据的分片方法，包括分区和 Reg
# 表分片
-对于任何分布式数据库来说，数据的分片都是必不可少的。本文将描述 GreptimeDB 中的表数据如何进行分片。
+GreptimeDB 将一张表分为多个 Region。分区表达式定义每行数据属于哪个 Region，Region 路由则定义当前由哪个 Datanode 持有该 Region。
## 分区
-有关创建分区表的语法，请参阅用户指南中的[表分片](/user-guide/deployments-administration/manage-data/table-sharding.md)部分。
+分区是由一个或多个列上的表达式描述的逻辑行集合。分区布局需要覆盖表的输入域，使每一行都能找到唯一的目标 Region。SQL 语法和支持的表达式参见[表分片](/user-guide/deployments-administration/manage-data/table-sharding.md)。
## Region
-在创建分区后，表中的数据被逻辑上分割。你可能会问："在 GreptimeDB 中，被逻辑上分区的数据是如何存储的？" 答案是保存在 `Region` 当中。
-
-每个 `Region` 对应一个分区，并保存分区的数据。所有的 `Region` 分布在各个 `Datanode` 之中。
-`Metasrv` 管理 `Region` 到 `Datanode` 的路由信息。如果建表后需要调整分区布局，
-GreptimeDB 支持通过显式的 [repartition](/user-guide/deployments-administration/manage-data/repartition.md) 操作拆分或合并分区。
+每个分区对应一个 Region。Region ID 是 Frontend、Datanode 和 Metasrv 用于存储和路由的标识。同一张表的多个 Region 可以放在同一个 Datanode 上。
分区和 Region 的关系参见下图：
@@ -54,3 +50,13 @@ GreptimeDB 支持通过显式的 [repartition](/user-guide/deployments-administr
└──────────────────────────────────┘
可以放在同一个 Datanode 之中
```
+
+## 路由与剪枝
+
+写入时，Frontend 对每行数据计算分区规则，按 Region 分组，再根据路由表把 Region 请求发送到当前 leader。
+
+查询时，分布式 planner 将查询谓词与分区表达式比较，只扫描可能满足谓词的 Region。如果分区元数据缺失或无法安全解释，planner 会退化为扫描所有 Region，避免漏掉数据。
+
+## 调整分区布局
+
+[Repartition](/user-guide/deployments-administration/manage-data/repartition.md) 通过显式的 split 或 merge 调整已有布局。Metasrv 以持久化 procedure 执行变更，更新 Region 路由和分区表达式，并使旧的表路由缓存失效。Frontend 刷新到新元数据后，后续请求使用新的布局。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/getting-started.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/getting-started.md
index e5aa2ca855..999f696ed0 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/getting-started.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/getting-started.md
@@ -15,14 +15,13 @@ description: 介绍如何在本地环境中从源代码编译和运行 GreptimeD
### 构建依赖项
-- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line)（可选）
+- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line)（可选；克隆仓库需要，构建本身不需要）
- C/C++ 工具链：提供编译和链接的基本工具。在 Ubuntu 上，这可用作 `build-essential`。在其他平台上，也有类似的命令。
-- Rust nightly 工具链（[指南][1]）
- - 编译源代码
+- [Rustup][1]。仓库通过 `rust-toolchain.toml` 指定所需的 nightly 工具链。
- Protobuf（[指南][2]）
- 编译 proto 文件
- 请注意，版本需要 >= 3.15。你可以使用 `protoc --version` 检查它。
-- 机器：建议内存在 16GB 以上 或者 使用[mold](https://github.com/rui314/mold)工具以降低链接时的内存使用。
+- 机器：建议 16GB 以上内存。内存较小时，可使用 [mold](https://github.com/rui314/mold) 降低链接阶段的内存占用。
[1]:
[2]:
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-write-sdk.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-write-sdk.md
index 28ff757763..916aec36f3 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-write-sdk.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-write-sdk.md
@@ -1,21 +1,17 @@
---
keywords: [gRPC SDK, GreptimeDatabase, GreptimeRequest, GreptimeResponse, 插入请求]
-description: 介绍如何为 GreptimeDB 开发一个 gRPC SDK，包括 GreptimeDatabase 服务的定义、GreptimeRequest 和 GreptimeResponse 的结构。
+description: 介绍 GreptimeDB gRPC 写入 SDK 需要遵守的协议契约和错误处理要求。
---
# 如何为 GreptimeDB 开发一个 gRPC SDK
-GreptimeDB 的 gRPC SDK 只需要处理写请求即可。读请求是标准 SQL 或 PromQL，可以由任何 JDBC 客户端或 Prometheus
-客户端处理。这也是为什么所有的 GreptimeDB SDK 都命名为 "`greptimedb-ingester-`"。请确保你的 GreptimeDB SDK
-遵循相同的命名约定。
+GreptimeDB 的公开 gRPC SDK 是写入客户端。查询通常通过标准 SQL 或 PromQL 客户端完成。除非有单独需求，新 SDK 应聚焦写入和删除，并遵循 `greptimedb-ingester-` 命名约定。面向用户的 API 参见 [gRPC SDK 概述](/user-guide/ingest-data/for-iot/grpc-sdks/overview.md)。
## `GreptimeDatabase` 服务
-GreptimeDB 自定义了一个 gRPC 服务：`GreptimeDatabase`
-。你只需要实现这个服务即可。你可以在[这里](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto)
-找到它的 Protobuf 定义。
+从 [`GreptimeDatabase` Protobuf 定义](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto)生成客户端 stub，不要在 SDK 中手写一份 message 或 service 定义。
-`GreptimeDatabase` 有 2 个 RPC 方法：
+该 service 提供一个 unary method 和一个 client-streaming method：
```protobuf
service GreptimeDatabase {
@@ -25,13 +21,9 @@ service GreptimeDatabase {
}
```
-`Handle` 方法是一个 unary 调用：当 GreptimeDB 服务接收到一个 `GreptimeRequest` 请求后，它立刻处理该请求并返回一个相应的
-`GreptimeResponse`。
+`Handle` 对一个请求返回一个响应，是 SDK insert 和 delete API 通常使用的方法。
-`HandleRequests` 方法则是一个 "[Client Streaming RPC][3]" 方式的调用。
-它可以接受一个连续的 `GreptimeRequest` 请求流，持续地发给 GreptimeDB 服务。
-GreptimeDB 服务会在收到流中的每个请求时立刻进行处理，并最终（流结束时）返回一个总结性的 `GreptimeResponse`。
-通过 `HandleRequests`，我们可以获得一个非常高的请求吞吐量。
+`HandleRequests` 是 [client-streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc)。客户端关闭请求流后，服务端才返回累计响应。SDK 如果暴露 streaming API，需要明确这个确认边界，并将一个 stream 绑定到一个 endpoint。
### `GreptimeRequest`
@@ -51,11 +43,13 @@ message GreptimeRequest {
}
```
-`RequestHeader` 是必需，它包含了一些上下文，鉴权和其他信息。"oneof" 的字段包含了发往 GreptimeDB 服务的请求。
+客户端需要在 `RequestHeader` 中填写服务端要求的 database context 和认证信息，并且只能设置一个 request variant。
-注意我们有两种类型的插入请求，一种是以 "列" 的形式（`InsertRequests`），另一种是以 "行" 的形式（`RowInsertRequests`
-）。通常我们建议使用 "行" 的形式，因为它对于表的插入更自然，更容易使用。但是，如果需要一次插入大量列，或者有大量的 "null"
-值需要插入，那么最好使用 "列" 的形式。
+该 message 还包含供内部调用者使用的 query 和 DDL variant。公开 ingester API 不应暴露它们，因为 `GreptimeDatabase` 不返回 query result stream。
+
+GreptimeDB 同时接受行式 `RowInsertRequests` 和列式 `InsertRequests`。公开写入 API 默认使用行式请求。面向列的客户端可以使用列式请求，但转换过程中必须保持列长度一致，并保留 null、时间戳精度、数据类型和列 semantic type。
+
+删除同样区分行式和列式。SDK 只应暴露能够在不丢失类型信息的前提下完成映射的形式。
### `GreptimeResponse`
@@ -68,6 +62,18 @@ message GreptimeResponse {
}
```
-`ResponseHeader` 包含了返回值的状态码，以及错误信息（如果有的话）。"oneof" 的字段目前只有 "affected rows"。
+成功响应包含 success header 和 `affected_rows`。该值表示服务端确认的行数；关闭请求流时返回的是累计值。
+
+请求失败通过 gRPC status 返回。Trailing metadata 中的 `x-greptime-err-code` 和 `x-greptime-err-retry-hint` 在存在时分别提供 GreptimeDB error code 和 retry classification。SDK 应保留 gRPC status 并暴露这些 GreptimeDB metadata，不能用一个通用 SDK error 将其覆盖。
+
+## 重试与交付语义
+
+重试次数必须有上限，并且对调用者可见。只有错误被标记为 retryable 且 deadline 仍允许时，才能重试 unary request。Cancellation 和 deadline expiration 不应重试。
+
+响应丢失不代表服务端拒绝了写入。除非调用者的数据模型保证操作幂等，重试这类请求可能插入重复行。SDK 需要说明这一点，并在交付结果不确定时返回最终错误。
+
+不要自动重试只发送了一部分的 `HandleRequests` stream。即使客户端尚未收到累计响应，服务端也可能已经接受了部分请求。此时应关闭失败的 stream，并将不确定状态返回给调用者。
+
+Arrow Flight bulk ingestion 与 `GreptimeDatabase` RPC 应使用不同的 API。它的 batching 和 partial acceptance 需要独立的契约。
-GreptimeDB 现在有很多 SDK，你可以参考[这里](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=)获取一些示例。
+可以参考现有 [GreptimeDB ingester 仓库](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=)的公开 API 约定，但线上行为应以当前 Protobuf 定义和服务端契约为准。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/admin-api.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/admin-api.md
index 3ac3d44f3b..bd28206c6f 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/admin-api.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/admin-api.md
@@ -1,20 +1,25 @@
---
-keywords: [Admin API, 健康检查, leader 查询, 心跳检测, 维护模式]
-description: 介绍 Metasrv 的 Admin API，包括健康检查、leader 查询、心跳检测、维护模式和 Procedure Manager 控制等功能。
+keywords: [Admin API, 健康检查, leader 查询, 心跳检测, 维护模式, 恢复模式, table id sequence]
+description: 介绍 Metasrv 用于状态检查、集群控制和元数据恢复的 Admin API。
---
# Admin API
-Admin 提供了一种简单的方法来查看和管理集群信息，包括 metasrv 健康检测、metasrv leader 查询、数据节点心跳检测、维护模式和 Procedure Manager 控制。
+:::tip
+本页所有 Admin API 都监听 Metasrv 的 `HTTP_PORT`，默认值为 `4000`。
+:::
-Admin API 是一个 HTTP 服务，提供一组可以通过 HTTP 请求调用的 RESTful API。Admin API 简单、用户友好且安全。
+Admin API 通过 HTTP 提供 Metasrv 状态、集群控制和元数据恢复操作。该 API 不提供认证，且部分端点会改变集群行为或元数据分配，部署时必须通过网络策略保护 HTTP 端口。
本页介绍以下 API：
- /health
- /leader
- /heartbeat
+- /node-lease
- /maintenance
- /procedure-manager
+- /recovery
+- /sequence/table
所有这些 API 都在父资源 `/admin` 下。
@@ -22,7 +27,7 @@ Admin API 是一个 HTTP 服务，提供一组可以通过 HTTP 请求调用的
## /health HTTP 端点
-`/health` 端点接受 GET HTTP 请求，你可以使用此端点检查你的 metasrv 实例的健康状况。
+`/health` 端点接受 GET 请求。HTTP 服务运行时返回 `OK`，但不会检查当前 Metasrv 是否为 leader，也不会检查外部依赖是否可用。
### 定义
@@ -116,9 +121,17 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100'
]
```
+## /node-lease HTTP 端点
+
+`/node-lease` 返回 Metasrv 当前记录的 Datanode lease，可用于判断 Metasrv 是否仍将某个 Datanode 视为存活。
+
+```bash
+curl -X GET http://localhost:4000/admin/node-lease
+```
+
## /maintenance HTTP 端点
-集群维护模式是 GreptimeDB 中的一项安全功能，它可以临时禁用自动集群管理操作。此模式在集群升级、计划停机以及任何可能暂时影响集群稳定性的操作期间特别有用。有关更多详细信息，请参阅[集群维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。
+维护模式在升级、计划停机等操作期间临时禁用自动集群管理。它对集群的具体影响参见[集群维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。
`/maintenance` 端点支持以下 HTTP 请求：
@@ -151,3 +164,39 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100'
"status": "running"
}
```
+
+## /recovery HTTP 端点
+
+Recovery mode 控制手动修改 table ID sequence 等元数据修复端点。它只用于恢复工作，不用于常规维护。
+
+- `GET /admin/recovery/status`：查询 recovery mode 是否开启。
+- `POST /admin/recovery/enable`：开启 recovery mode。
+- `POST /admin/recovery/disable`：关闭 recovery mode。
+
+响应体格式如下：
+
+```json
+{
+ "enabled": true
+}
+```
+
+修复完成后应关闭 recovery mode。如果只是计划暂停自动集群操作，应使用[维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。
+
+## /sequence/table HTTP 端点
+
+这些端点用于检查或修复 table ID sequence：
+
+- `GET /admin/sequence/table/next-id`：返回下一个 table ID，但不执行分配。
+- `POST /admin/sequence/table/set-next-id`：推进下一个 table ID。
+
+设置 sequence 前必须开启 recovery mode。新值必须大于当前值，不能通过该 API 回退 sequence。Recovery mode 只是该 API 的前置条件，不能阻止 DDL。执行该操作时，必须遵循[管理 Table ID Sequence](/user-guide/deployments-administration/maintenance/sequence-management.md)中的完整集群操作流程。
+
+```bash
+curl -X POST \
+ -H 'Content-Type: application/json' \
+ -d '{"next_table_id": 2048}' \
+ http://localhost:4000/admin/sequence/table/set-next-id
+```
+
+该操作会影响后续新表分配到的 ID。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md
index 2e3c525baa..e9f6144f26 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md
@@ -1,162 +1,101 @@
---
-keywords: [Metasrv, 元数据存储, 请求路由, 负载均衡, 高可用性]
-description: 介绍 Metasrv 的功能、架构和与前端的交互方式。
+keywords: [Metasrv, 元数据, 路由, Leader 选举, Procedure, 心跳]
+description: 介绍 Metasrv 提供的元数据及集群协调机制。
---
# Metasrv
-
-
## Metasrv 包含什么
-- 存储元数据（Catalog, Schema, Table, Region 等）
-- 请求路由器。它告诉前端在哪里写入和读取数据。
-- 数据节点的负载均衡，决定谁应该处理新的表创建请求，更准确地说，它做出资源分配决策。
-- 选举与高可用性，GreptimeDB 设计为 Leader-Follower 架构，只有 leader 节点可以写入，而 follower 节点可以读取，follower 节点的数量通常 >= 1，当 leader 不可用时，follower 节点需要能够快速切换为 leader。
-- 统计数据收集（通过每个节点上的心跳报告），如 CPU、负载、节点上的表数量、平均/峰值数据读写大小等，可用作分布式调度的基础。
+Metasrv 是 GreptimeDB 分布式集群中的元数据和协调服务，不参与数据读写链路。它主要负责：
+
+- 存储 Catalog、Schema、Table、Region、路由和节点元数据；
+- 为新 Region 选择 Datanode，并维护表路由；
+- 选举一个 Metasrv leader 负责协调元数据变更；
+- 通过可恢复的 Procedure 执行 DDL、Region 迁移、故障转移和重分区；
+- 通过心跳维护节点租约和 Region 统计信息；
+- 元数据变更时向 Frontend、Datanode 和 Flownode 广播缓存失效；
+- 向 Datanode 下发 Region 生命周期指令。
## 前端如何与 Metasrv 交互
-首先，请求路由器中的路由表结构如下（注意这只是逻辑结构，实际存储结构可能不同，例如端点可能有字典压缩）。
-
-```txt
- table_A
- table_name
- table_schema // 用于物理计划
- regions
- region_1
- mutate_endpoint
- select_endpoint_1, select_endpoint_2
- region_2
- mutate_endpoint
- select_endpoint_1, select_endpoint_2, select_endpoint_3
- region_xxx
- table_B
- ...
-```
+Frontend 从 Metasrv 获取表元数据和 Region 路由，并缓存在本地。修改元数据的语句会发送给 Metasrv leader；普通读写则使用缓存的路由直接访问 Datanode。
-### 创建表
+控制链路和数据链路相互分离：
-1. 前端发送 `CREATE TABLE` 请求到 Metasrv。
-2. 根据请求中包含的分区规则规划 Region 数量。
-3. 检查数据节点可用资源的全局视图（通过心跳收集）并为每个 Region 分配一个节点。
-4. 前端创建表并在成功创建后将 `Schema` 存储到 Metasrv。
+```text
+Frontend
+ |-- 元数据查询和 DDL -------------------> Metasrv leader
+ `-- Region 读写 ------------------------> Datanode
-### `Insert`
+Metasrv leader
+ |-- Region 生命周期指令 ----------------> Datanode
+ `-- 缓存失效 --------------------------> Frontend / Datanode / Flownode
-1. 前端从 Metasrv 获取指定表的路由。注意，最小的路由单元是表的路由（多个 Region），即包含该表所有 Region 的地址。
-2. 最佳实践是前端首先从本地缓存中获取路由并将请求转发到数据节点。如果路由不再有效，则数据节点有义务返回 `Invalid Route` 错误，前端重新从 Metasrv 获取最新数据并更新其缓存。路由信息不经常变化，因此，前端使用惰性策略维护缓存是足够的。
-3. 前端处理可能包含多个表和多个 Region 的一批写入，因此前端需要根据“路由表”拆分用户请求。
+Datanode
+ `-- 心跳、租约续期和 Region 统计信息 ----> Metasrv leader
+```
-### `Select`
+在稳定状态下，表路由为每个 Region 记录一个 leader peer 和零个或多个 follower peer。Leader 是写入目标；支持只读副本的部署可以把读取路由到 follower：
-1. 与 `Insert` 类似，前端首先从本地缓存中获取路由表。
-2. 与 `Insert` 不同，对于 `Select`，前端需要从路由表中提取只读节点（follower），然后根据优先级将请求分发到 leader 或 follower 节点。
-3. 前端的分布式查询引擎根据路由信息分发多个子查询任务并聚合查询结果。
+```text
+Table route
+ |-- Region 0
+ | |-- leader -> Datanode A
+ | `-- followers -> Datanode B, Datanode C
+ `-- Region 1
+ `-- leader -> Datanode D
+```
-## Metasrv 架构
+Region 迁移或故障转移会改变 peer 角色，并可能使 Region 暂时没有 leader。Frontend 刷新缓存路由后，再把后续读写发送给当前 peer。
-
+### 创建表
-## 分布式共识
+1. Frontend 向 Metasrv leader 提交 DDL 请求。
+2. Metasrv 根据分区规则确定 Region，并[为每个 Region 选择 Datanode](/contributor-guide/metasrv/selector.md)。
+3. 持久化的 Procedure 创建 Region，并写入表元数据和路由。发生 leader 切换后，Procedure 可以从已保存的状态继续执行。
+4. 元数据提交后，Metasrv 通知 Frontend 刷新相关缓存。
-如你所见，Metasrv 依赖于分布式共识，因为：
+### `Insert`
-1. 首先，Metasrv 必须选举一个 leader，数据节点只向 leader 发送心跳，我们只使用单个 Metasrv 节点接收心跳，这使得基于全局信息进行一些计算或调度变得容易且快速。至于数据节点如何连接到 leader，这由 MetaClient 决定（使用重定向，心跳请求变为 gRPC 流，使用重定向比转发更不容易出错），这对数据节点是透明的。
-2. 其次，Metasrv 必须为数据节点提供选举 API，以选举“写入”和“只读”节点，并帮助数据节点实现高可用性。
-3. 最后，`Metadata`、`Schema` 和其他数据必须在 Metasrv 上可靠且一致地存储。因此，基于共识的算法是存储它们的理想方法。
+Frontend 解析表路由，按照分区规则拆分数据行，再把各 Region 的写入发送到对应 Datanode。路由发生变化时，相关缓存会失效，Frontend 随后从 Metasrv 重新获取元数据。
-对于 Metasrv 的第一个版本，我们选择 Etcd 作为共识算法组件（Metasrv 设计时考虑适应不同的实现，甚至创建一个新的轮子），原因如下：
+### `Select`
-1. Etcd 提供了我们需要的 API，例如 `Watch`、`Election`、`KV` 等。
-2. 我们只执行两个分布式共识任务：选举（使用 `Watch` 机制）和存储（少量元数据），这两者都不需要我们定制自己的状态机，也不需要基于 raft 定制自己的状态机；少量数据也不需要多 raft 组支持。
-3. Metasrv 的初始版本使用 Etcd，使我们能够专注于 Metasrv 的功能，而不需要在分布式共识算法上花费太多精力，这提高了系统设计（避免与共识算法耦合）并有助于初期的快速开发，同时通过良好的架构设计，未来可以轻松接入优秀的共识算法实现。
+Frontend 在查询规划期间使用表和 Region 元数据。分区列上的谓词用于裁剪 Region，分布式查询引擎再把任务发送给持有这些 Region 的 Datanode。参见[分布式查询](../frontend/distributed-querying.md)。
-## 心跳管理
+## Metasrv 架构
-数据节点与 Metasrv 之间的主要通信方式是心跳请求/响应流，我们希望这是唯一的通信方式。这个想法受到 [TiKV PD](https://github.com/tikv/pd) 设计的启发，我们在 [RheaKV](https://github.com/sofastack/sofa-jraft/tree/master/jraft-rheakv/rheakv-pd) 中有实际经验。请求发送其状态，而 Metasrv 通过心跳响应发送不同的调度指令。
+主要协调路径如下：
+
+```text
+Leader election
+ |
+ v
+Metasrv leader
+├─ DDL manager -> Procedure manager
+├─ Selector -> 新 Region 的放置
+├─ Heartbeat handler chain -> 租约和 Region 统计信息
+├─ Region supervisor -> Region 迁移 Procedure
+├─ Mailbox -> 缓存失效和 Region 指令
+└─ Metadata managers -> KV backend
+```
-心跳可能携带以下数据，但这不是最终设计，我们仍在讨论和探索究竟应该收集哪些数据。
+这些机制共享元数据，但故障边界不同。进程重启可以丢弃缓存和 leader 本地状态；恢复所需的元数据和 Procedure 状态必须持久化。
-```
-service Heartbeat {
- // 心跳，心跳可能有很多内容，例如：
- // 1. 要注册到 Metasrv 并可被其他节点发现的元数据。
- // 2. 一些性能指标，例如负载、CPU 使用率等。
- // 3. 正在执行的计算任务数量。
- rpc Heartbeat(stream HeartbeatRequest) returns (stream HeartbeatResponse) {}
-}
-
-message HeartbeatRequest {
- RequestHeader header = 1;
-
- // 自身节点
- Peer peer = 2;
- // leader 节点
- bool is_leader = 3;
- // 实际报告时间间隔
- TimeInterval report_interval = 4;
- // 节点状态
- NodeStat node_stat = 5;
- // 此节点中的 Region 状态
- repeated RegionStat region_stats = 6;
- // follower 节点和状态，在 follower 节点上为空
- repeated ReplicaStat replica_stats = 7;
-}
-
-message NodeStat {
- // 此期间的读取容量单位
- uint64 rcus = 1;
- // 此期间的写入容量单位
- uint64 wcus = 2;
- // 此节点中的表数量
- uint64 table_num = 3;
- // 此节点中的 Region 数量
- uint64 region_num = 4;
-
- double cpu_usage = 5;
- double load = 6;
- // 节点中的读取磁盘 I/O
- double read_io_rate = 7;
- // 节点中的写入磁盘 I/O
- double write_io_rate = 8;
-
- // 其他
- map attrs = 100;
-}
-
-message RegionStat {
- uint64 region_id = 1;
- TableName table_name = 2;
- // 此期间的读取容量单位
- uint64 rcus = 3;
- // 此期间的写入容量单位
- uint64 wcus = 4;
- // 近似 Region 大小
- uint64 approximate_size = 5;
- // 近似行数
- uint64 approximate_rows = 6;
-
- // 其他
- map attrs = 100;
-}
-
-message ReplicaStat {
- Peer peer = 1;
- bool in_sync = 2;
- bool is_learner = 3;
-}
-```
+## 分布式共识
-## Central Nervous System (CNS)
+Metasrv 将 leader 选举与元数据存储分开。只有选出的 Metasrv leader 执行协调和元数据变更操作，其他 Metasrv 节点会把 client 引导到当前 leader。
-我们要构建一个算法系统，该系统依赖于每个节点的实时和历史心跳数据，应该做出一些更智能的调度决策并将其发送到 Metasrv 的 Autoadmin 单元，该单元分发调度决策，由数据节点本身或更可能由 PaaS 平台执行。
+Key-value backend 保存表元数据、路由、Procedure 状态以及其他必须跨 leader 切换保留的信息。Metasrv 不使用这套选举为 Datanode Region 创建读写副本；Region 可用性由心跳、Region 故障检测和故障转移 Procedure 管理。
-## 工作负载抽象
+## 心跳管理
-工作负载抽象的级别决定了 Metasrv 生成的调度策略（如资源分配）的效率和质量。
+Datanode 与 Metasrv leader 保持心跳流。心跳请求报告节点身份、租约、Region 统计信息以及放置和监控所需的其他状态；响应则携带 Region 生命周期指令、缓存失效等控制消息。
-DynamoDB 定义了 RCUs 和 WCUs（读取容量单位/写入容量单位），解释说 RCU 是一个 4KB 数据的读取请求，WCU 是一个 1KB 数据的写入请求。当使用 RCU 和 WCU 描述工作负载时，更容易实现性能可测量性并获得更有信息量的资源预分配，因为我们可以将不同的硬件能力抽象为 RCU 和 WCU 的组合。
+心跳驱动两套相互独立的机制，调整心跳周期会同时影响两者：
-然而，GreptimeDB 面临比 DynamoDB 更复杂的情况，特别是 RCU 不适合描述需要大量计算的 GreptimeDB 读取工作负载。我们正在努力解决这个问题。
+- **节点租约**：keep-lease handler 为发送心跳的 Datanode 续期。Selector 和 `/node-lease` 端点据此判断 Datanode 是否仍然存活。
+- **Region 故障检测**：Region supervisor 为每个 Region 维护一个基于心跳到达间隔的 Phi Accrual 检测器，其判定与租约是否过期无关。
+只有开启 Region 故障转移时，故障判定才会提交故障转移迁移。该功能默认关闭，并且要求使用 remote WAL，除非显式允许在本地 WAL 上执行。维护模式同样会抑制故障转移。前置条件和开启方式参见 [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md)。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/selector.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/selector.md
index 9048e0daa4..29b521212b 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/selector.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/selector.md
@@ -7,11 +7,7 @@ description: 介绍 Metasrv 中的 Selector，包括其类型和配置方法。
## 介绍
-什么是 `Selector`？顾名思义，它允许用户从给定的 `namespace` 和 `context` 中选择 `Item`s。有一个相关的 `trait`，也叫做 `Selector`，其定义可以在[这里][0]找到。
-
-[0]: https://github.com/GreptimeTeam/greptimedb/blob/main/src/meta-srv/src/selector.rs
-
-在 `Metasrv` 中存在一个特定的场景。当 `Frontend` 向 `Metasrv` 发送建表请求时，`Metasrv` 会创建一个路由表（表的创建细节不在这里赘述）。在创建路由表时，`Metasrv` 需要选择适当的 `Datanode`s，这时候就需要用到 `Selector`。
+建表时，Metasrv 使用 `Selector` 为各 Region 选择 Datanode。Selector 根据当前节点租约进行选择；部分实现还会使用 Region 统计信息。
@@ -19,22 +15,22 @@ description: 介绍 Metasrv 中的 Selector，包括其类型和配置方法。
`Metasrv` 目前提供以下几种类型的 `Selectors`:
-### LeasebasedSelector
+### LeaseBasedSelector
-`LeasebasedSelector` 从所有可用的（也就是在租约期间内）`Datanode` 中随机选择，其特点是简单和快速。
+`LeaseBasedSelector` 从租约有效的 Datanode 中随机选择。
### LoadBasedSelector
`LoadBasedSelector` 按照负载来选择，负载值则由每个 `Datanode` 上的 region 数量决定，较少的 region 表示较低的负载，`LoadBasedSelector` 优先选择低负载的 `Datanode`。
### RoundRobinSelector [默认选项]
-`RoundRobinSelector` 以轮询的方式选择 `Datanode`。在大多数情况下，这是默认的且推荐的选项。如果你不确定选择哪个，通常它就是正确的选择。
+`RoundRobinSelector` 以轮询方式选择 Datanode，是默认选项，也适用于大多数部署。
## 配置
您可以在启动 `Metasrv` 服务时通过名称配置 `Selector`。
-- LeasebasedSelector: `lease_based` 或 `LeaseBased`
+- LeaseBasedSelector: `lease_based` 或 `LeaseBased`
- LoadBasedSelector: `load_based` 或 `LoadBased`
- RoundRobinSelector: `round_robin` 或 `RoundRobin`
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/overview.md
index 9f82a72171..badd6ad62d 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/overview.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/overview.md
@@ -5,9 +5,7 @@ description: 介绍 GreptimeDB 的架构、关键概念和工作原理，包括
# 贡献者指南
-DeepWiki 对 GreptimeDB 的架构和实现进行了详细且清晰的描述，强烈推荐阅读：
-
-[https://deepwiki.com/GreptimeTeam/greptimedb](https://deepwiki.com/GreptimeTeam/greptimedb)
+本指南面向 GreptimeDB 贡献者，介绍理解内部实现所需的设计机制。从源码构建和运行参见[快速开始](/contributor-guide/getting-started.md)。提交要求（CLA、license header、代码格式，以及 PR 必须通过的检查）以源码仓库的 [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md) 为准。
## 架构
@@ -18,7 +16,13 @@ DeepWiki 对 GreptimeDB 的架构和实现进行了详细且清晰的描述，
- [frontend][1]
- [datanode][2]
- [metasrv][3]
+- [flownode][4]
[1]: /contributor-guide/frontend/overview.md
[2]: /contributor-guide/datanode/overview.md
[3]: /contributor-guide/metasrv/overview.md
+[4]: /contributor-guide/flownode/overview.md
+
+## 补充参考
+
+[DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb) 提供了自动生成的 GreptimeDB 源码导读，可用于了解不熟悉的模块。它属于辅助资料；涉及具体版本的行为时，仍应以对应源码为准。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/integration-test.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/integration-test.md
index 63ffa56bc0..767109f773 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/integration-test.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/integration-test.md
@@ -7,8 +7,14 @@ description: 介绍 GreptimeDB 的集成测试，包括测试范围和如何运
## 介绍
-集成测试使用 Rust 测试工具（`#[test]`）编写，与单元测试不同，它们被单独放置在
-[这里](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration)。
-它涵盖了涉及多个组件的场景，其中一个典型案例是与 HTTP/gRPC 相关的功能。你可以查看
-其[文档](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md)以获取更多信息。
+集成测试覆盖跨 crate 或服务边界的行为，例如 HTTP 和 gRPC 处理、分布式组件或外部存储。测试使用 Rust test harness，位于 [`tests-integration`](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration) package。
+运行命令如下：
+
+```shell
+cargo nextest run -p tests-integration
+```
+
+部分 case 依赖外部服务的环境变量或 fixture。运行前按照 package 的[准备说明](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md)配置环境。
+
+只有 crate 级测试或 Sqlness case 无法覆盖所需边界时才使用集成测试。隔离的逻辑仍放在单元测试中，便于快速复现失败。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/overview.md
index 81b6defa45..9ac8eb2ec1 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/overview.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/overview.md
@@ -5,5 +5,13 @@ description: GreptimeDB 的测试
# 测试
-我们的团队进行了大量测试，以确保 GreptimeDB 的行为。本章将介绍几种用于测试 GreptimeDB 的重要方法，以及如何使用它们。
+选择能够覆盖本次改动的最小测试范围：
+| 测试类型 | 适用场景 | 常用命令 |
+| --- | --- | --- |
+| [单元测试](unit-test.md) | 单个 crate 或组件内的逻辑 | `cargo nextest run -p ` |
+| [Sqlness 测试](sqlness-test.md) | SQL、协议、planner、执行和端到端回归 | `cargo sqlness bare -t ` |
+| [集成测试](integration-test.md) | 跨组件或依赖外部服务的行为 | `cargo nextest run -p tests-integration` |
+| [兼容性测试](https://github.com/GreptimeTeam/greptimedb/blob/main/tests/compatibility/README.md) | 读取旧版本写入的数据或元数据 | `cargo sqlness compat --from-version ` |
+
+需要运行完整 Rust workspace 测试时使用 `make test`。如果改动涉及持久化元数据、WAL record、SST 文件或线上协议，并且输入可能由旧版本产生，还需要补充兼容性测试。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/sqlness-test.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/sqlness-test.md
index d29b69e835..06503a434c 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/sqlness-test.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/sqlness-test.md
@@ -7,36 +7,34 @@ description: 介绍 GreptimeDB 的 Sqlness 测试，包括测试文件类型、
## 介绍
-SQL 是 `GreptimeDB` 的一个重要用户接口。我们为它提供了一个单独的测试套件（名为 `sqlness`）。
+Sqlness 是 GreptimeDB 针对 SQL 和协议行为的端到端回归测试。每个 case 向运行中的 GreptimeDB 发送语句，并将输出与仓库中的结果文件比较。
## Sqlness 手册
### 测试文件
-Sqlness 有两种类型的文件
+每个 case 使用两类文件：
- `.sql`：测试输入，仅包含 SQL
- `.result`：预期的测试输出，包含 SQL 和其结果
-`.result` 文件是预期的执行输出。如果 `.result` 文件发生变化，意味着测试结果不同，测试可能失败。你应该检查变更日志来解决问题。
-
-你只需要在 `.sql` 文件中编写测试 SQL，然后运行测试。
+在 `.sql` 文件中编写输入，运行测试后生成或更新 `.result`。必须检查每一处结果差异，只有行为变化符合预期时才能接受。
### 组织测试案例
-输入案例的根目录是 `tests/cases`。它包含几个子目录，代表不同的测试模式。例如，`standalone/` 包含所有在 `greptimedb standalone start` 模式下运行的测试。
+输入 case 位于 `tests/cases`。第一级目录选择运行环境，例如 `standalone/` 表示使用单机 GreptimeDB。
-在第一级子目录下（例如 `cases/standalone`），你可以随意组织你的测试案例。Sqlness 会递归地遍历每个文件并运行它们。
+在环境目录内，新 case 应与它覆盖的功能放在一起。Sqlness 会递归发现 case 文件。
## 运行测试
-与其他测试不同，这个测试工具是以二进制目标形式存在的。你可以用以下命令运行它
+运行命令如下：
```shell
-cargo run --bin sqlness-runner bare
+cargo sqlness bare
```
-它会自动完成以下步骤：编译 `GreptimeDB`，启动它，抓取测试并将其发送到服务器，然后收集和比较结果。你只需要检查是否有 `.result` 文件发生变化。如果没有，恭喜你，测试通过了 🥳！
+该命令会构建并启动 GreptimeDB、执行选中的 case，再比较输出。`.result` 发生变化只是待审查的结果，不代表新输出一定正确。
### 运行特定测试
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/unit-test.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/unit-test.md
index 79c73775b4..b63b29df94 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/unit-test.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/unit-test.md
@@ -7,22 +7,26 @@ description: 介绍 GreptimeDB 的单元测试，包括如何编写、运行和
## 介绍
-单元测试嵌入在代码库中，通常放置在被测试逻辑的旁边。它们使用 Rust 的 `#[test]` 属性编写，并可以使用 `cargo nextest run` 运行。
+单元测试通常放在被测逻辑旁边，使用 Rust 的 `#[test]` 属性编写。GreptimeDB 主要使用 [`cargo-nextest`](https://nexte.st/) 运行 Rust 测试。
-GreptimeDB 代码库不支持默认的 `cargo` 测试运行器。推荐使用 [`nextest`](https://nexte.st/)。你可以通过以下命令安装它：
+安装命令如下：
```shell
cargo install cargo-nextest --locked
```
-然后运行测试（这里 `--workspace` 不是必须的）
+开发时先运行本次修改的 package：
```shell
-cargo nextest run
+cargo nextest run -p
```
-注意，如果你的 Rust 是通过 `rustup` 安装的，请确保使用 `cargo` 安装 `nextest`，而不是像 `homebrew` 这样的包管理器，否则会弄乱你的本地环境。
+可以继续使用测试名称或 nextest filter 缩小范围。影响范围较广的改动在提交前运行完整 workspace 测试：
+
+```shell
+make test
+```
## 覆盖率
-我们的持续集成（CI）作业有一个“覆盖率检查”步骤。它会报告有多少代码被单元测试覆盖。请在你的补丁中添加必要的单元测试。
+CI 会报告单元测试覆盖率。测试应覆盖本次改变的行为和可能回归的失败路径，而不是只追求覆盖率数字。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference/sql/create.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference/sql/create.md
index 5e1d660bf0..660ca1751e 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/reference/sql/create.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference/sql/create.md
@@ -26,7 +26,7 @@ CREATE DATABASE [IF NOT EXISTS] db_name [WITH ]
数据库也可以通过使用 `WITH` 关键字配置与 `CREATE TABLE` 语句类似的选项。数据库支持以下选项：
- `ttl` - 数据库中所有表的数据存活时间（不能设置为 `instant`）
-- `memtable.type` - 内存表类型（`time_series`、`partition_tree`）
+- `memtable.type` - memtable 类型（`bulk`、`time_series`）
- `append_mode` - 数据库中的表是否为仅追加模式（`true`/`false`）
- `merge_mode` - 合并重复行的策略（`last_row`、`last_non_null`）
- `skip_wal` - 是否为数据库中的表禁用预写日志（`'true'`/`'false'`）
@@ -74,7 +74,7 @@ CREATE DATABASE test WITH (ttl='7d');
```sql
CREATE DATABASE test WITH (
ttl='30d',
- 'memtable.type'='partition_tree',
+ 'memtable.type'='bulk',
'append_mode'='true'
);
```
@@ -156,7 +156,7 @@ GreptimeDB 提供了丰富的索引实现来加速查询，请在[索引](/user-
| `compaction.twcs.trigger_file_num` | 某个窗口内触发 compaction 的最小文件数量阈值 | 字符串值，如 '8'。只在 `compaction.type` 为 `twcs` 时可用 |
| `compaction.twcs.time_window` | Compaction 时间窗口 | 字符串值，如 '1d' 表示 1 天。该表会根据时间戳将数据分区到不同的时间窗口中。只在 `compaction.type` 为 `twcs` 时可用 |
| `compaction.twcs.max_output_file_size` | TWCS compaction 的最大输出文件大小 | 字符串值，如 '1GB'、'512MB'。设置 TWCS compaction 产生的文件的最大大小。只在 `compaction.type` 为 `twcs` 时可用 |
-| `memtable.type` | memtable 的类型 | 字符串值，支持 `time_series`，`partition_tree` |
+| `memtable.type` | memtable 类型 | 字符串值：`bulk` 或 `time_series`。未设置时，Mito 根据 SST format 选择实现；默认的 flat format 使用 `bulk`。设置 `bulk` 会强制使用 `sst_format=flat`；使用 flat SST 时，即使设置了 `time_series`，Mito 也会选择 bulk 实现。旧值 `partition_tree` 仅为兼容保留，并映射到 bulk 和 flat 路径。 |
| `append_mode` | 该表是否时 append-only 的 | 字符串值。默认值为 'false'，根据 'merge_mode' 按主键和时间戳删除重复行。设置为 'true' 可以开启 append 模式和创建 append-only 表，保留所有重复的行 |
| `merge_mode` | 合并重复行的策略 | 字符串值。只有当 `append_mode` 为 'false' 时可用。默认值为 `last_row`，保留相同主键和时间戳的最后一行。设置为 `last_non_null` 则保留相同主键和时间戳的最后一个非空字段。 |
| `sst_format` | SST 文件的格式 | 字符串值，支持 `primary_key`，`flat`。默认为 `flat`。`flat` 格式建议用于具有高基数主键的表。 |
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/configuration.md b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/configuration.md
index 9b1d7d7b3b..f2d9ba3de1 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/configuration.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/configuration.md
@@ -607,20 +607,9 @@ create_on_compaction = "auto"
apply_on_query = "auto"
mem_threshold_on_create = "64M"
intermediate_path = ""
-
-[region_engine.mito.memtable]
-type = "time_series"
```
-此外，`mito` 也提供了一个实验性质的 memtable。该 memtable 主要优化大量时间序列下的写入性能和内存占用。其查询性能可能会不如默认的 `time_series` memtable。
-
-```toml
-[region_engine.mito.memtable]
-type = "partition_tree"
-index_max_keys_per_shard = 8192
-data_freeze_threshold = 32768
-fork_dictionary_bytes = "1GiB"
-```
+Mito 根据表选项和 SST format 为每个 Region 选择 memtable 实现。`default_flat_format` 为 `true` 时，没有显式设置 `sst_format` 的 Region 使用 flat SST 和 bulk memtable。`memtable.type` 是数据库或表选项，不是 `[region_engine.mito.memtable]` 引擎配置。详见[表选项](/reference/sql/create.md#表选项)。
以下是可供使用的选项
@@ -655,7 +644,7 @@ fork_dictionary_bytes = "1GiB"
| `scan_memory_on_exhausted` | 字符串 | `fail` | 扫描内存耗尽时的行为。选项：`fail`（快速失败），`wait` 或 `wait()`（等待内存）。 |
| `min_compaction_interval` | 字符串 | `0m` | 两次 compaction 之间的最小时间间隔。设为 "0m"（默认）允许 compactions 立即运行，无限制。 |
| `schedule_compaction_after_edit` | 布尔值 | `true` | 是否允许在成功的 region edit 之后调度 compaction。 设为 `true` 是在 region edit 后调度 compaction 的必要但不充分条件，`min_compaction_interval` 等其他约束仍可能阻止 compaction 被调度。 设为 `false` 则保证 region edit 后不会调度 compaction。 |
-| `default_flat_format` | 布尔值 | `true` | 是否启用 Flat 格式作为默认 SST 格式。 |
+| `default_flat_format` | 布尔值 | `true` | 没有显式设置 `sst_format` 的 Region 是否使用 flat SST。Flat SST 使用 bulk memtable。 |
| `experimental_series_scan_v2` | 布尔值 | `true` | 是否为 metric 引擎物理 region 的 series scan 启用实验性的 two-phase 模式。设为 `false` 时使用 legacy 模式，其他 series scan 也使用 legacy 模式。 |
| `scan_parallelism` | 整数 | `0` | （已弃用，请使用 `max_concurrent_scan_files`）旧版扫描并发度选项。 |
| `index` | -- | -- | Mito 引擎中索引的选项。 |
@@ -671,10 +660,6 @@ fork_dictionary_bytes = "1GiB"
| `inverted_index.apply_on_query` | 字符串 | `auto` | 是否在查询时使用索引 - `auto`: 自动 - `disable`: 从不 |
| `inverted_index.mem_threshold_on_create` | 字符串 | `64M` | 创建索引时如果超过该内存阈值则改为使用外部排序 设置为空会关闭外排，在内存中完成所有排序 |
| `inverted_index.intermediate_path` | 字符串 | `""` | 存放外排临时文件的路径 (默认 `{data_home}/index_intermediate`). |
-| `memtable.type` | 字符串 | `time_series` | Memtable type. - `time_series`: time-series memtable - `partition_tree`: partition tree memtable (实验性功能) |
-| `memtable.index_max_keys_per_shard` | 整数 | `8192` | 一个 shard 内的主键数 只对 `partition_tree` memtable 生效 |
-| `memtable.data_freeze_threshold` | 整数 | `32768` | 一个 shard 内写缓存可容纳的最大行数 只对 `partition_tree` memtable 生效 |
-| `memtable.fork_dictionary_bytes` | 字符串 | `1GiB` | 主键字典的大小 只对 `partition_tree` memtable 生效 |
`metric` 引擎针对包含大量小表的 metrics 数据进行了优化。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/data-persistence-indexing.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/data-persistence-indexing.md
index fbd986781f..6ace5b8d65 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/data-persistence-indexing.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/data-persistence-indexing.md
@@ -5,19 +5,23 @@ description: 介绍了 GreptimeDB 的数据持久化和索引机制，包括 SST
# 数据持久化与索引
-与所有类似 LSMT 的存储引擎一样，MemTables 中的数据被持久化到耐久性存储，例如本地磁盘文件系统或对象存储服务。GreptimeDB 采用 [Apache Parquet][1] 作为其持久文件格式。
+与其他 LSM-tree 存储引擎类似，GreptimeDB 将 memtable 中的数据持久化到本地文件系统或对象存储，并使用 [Apache Parquet][1] 作为持久化文件格式。
## SST 文件格式
Parquet 是一种提供快速数据查询的开源列式存储格式，已经被许多项目采用，例如 Delta Lake。
-Parquet 具有层次结构，类似于“行组 - 列-数据页”。Parquet 文件中的数据被水平分区为行组（row group），在其中相同列的所有值一起存储以形成数据页（data pages）。数据页是最小的存储单元。这种结构极大地提高了性能。
+Parquet 按 row group、column chunk 和 page 组织数据。每个 row group 为每一列保存一个 column chunk，每个 column chunk 再包含一个或多个 page。Page 是编码和压缩单元，读取指定列时则以 column chunk 为 I/O 单元。
首先，数据按列聚集，这使得文件扫描更加高效，特别是当查询只涉及少数列时，这在分析系统中非常常见。
-其次，相同列的数据往往是同质的（比如具备近似的值），这有助于在采用字典和 Run-Length Encoding（RLE）等技术进行压缩。
+其次，同一列中的值通常比较相似，有利于字典编码和 Run-Length Encoding（RLE）等压缩技术发挥作用。
-
+下面这张来自 Apache Parquet 规范的图进一步展示了物理文件布局：column chunk 按 row group 写入，文件元数据及其长度则保存在 footer 中。
+
+
+
+*来源：Apache Parquet [FileLayout.gif](https://github.com/apache/parquet-format/blob/master/doc/images/FileLayout.gif)。Copyright 2014 The Apache Software Foundation，依据 [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) 使用。*
## 数据持久化
@@ -26,18 +30,18 @@ GreptimeDB 提供了 `region_engine.mito.global_write_buffer_size` 的配置项
## SST 文件中的索引数据
-Apache Parquet 文件格式在列块和数据页的头部提供了内置的统计信息，用于剪枝和跳过。
+Parquet 在每个 column chunk 的元数据中保存 row group 级列统计信息，例如最小值、最大值和 null 数量。Page 元数据和可选的 column index 可以提供粒度更细的统计信息。
-
+
-例如，在上述 Parquet 文件中，如果你想要过滤 `name` 等于 `Emily` 的行，你可以轻松跳过行组 0，因为 `name` 字段的最大值是 `Charlie`。这些统计信息减少了 IO 操作。
+例如，查询 `name` 等于 `Emily` 的行时，可以跳过 row group 0，因为其中 `name` 的最大值是 `Charlie`，无需读取该 row group。
## 索引文件
-对于每个 SST 文件，GreptimeDB 不但维护 SST 文件内部索引，还会单独生成一个文件用于存储针对该 SST 文件的索引结构。
+当一个 SST 存在已配置且适用的索引输出时，GreptimeDB 将这些索引写入与该 SST 关联的 Puffin 文件。没有适用索引的 SST 不需要生成 Puffin 文件。
-索引文件采用 [Puffin][3] 格式，这种格式具有较大的灵活性，能够存储更多的元数据，并支持更多的索引结构。
+Puffin 是索引 Blob 及其元数据的容器，使不同索引结构可以共用一个文件。
@@ -58,13 +62,13 @@ GreptimeDB 会将多种索引结构作为 Blob 存储在 Puffin 文件中，包
-例如，上述查询使用倒排索引来定位数据段，数据段满足条件：`job` 等于 `apiserver`，`handler` 符合正则匹配 `.*users` 及 `status` 符合正则匹配 `4..`，然后扫描这些数据段以产生满足所有条件的最终结果，从而显着减少 IO 操作的次数。
+上述查询使用倒排索引定位 `job` 等于 `apiserver`、`handler` 匹配 `.*users` 且 `status` 匹配 `4..` 的数据段。Mito 只扫描这些数据段，再应用剩余过滤条件。
### 倒排索引格式
-
+
-GreptimeDB 按列构建倒排索引，每个倒排索引包含一个 FST 和多个 Bitmap。
+GreptimeDB 按列构建倒排索引。每个列索引包含一个 null bitmap、多个 posting bitmap 和一个 FST。Blob footer 记录定位和解码各列索引所需的 offset、size 和元数据。
FST（Finite State Transducer）允许 GreptimeDB 以紧凑的格式存储列值到 Bitmap 位置的映射，并且提供了优秀的搜索性能和支持复杂搜索（例如正则表达式匹配）；Bitmap 则维护了数据段 ID 列表，每个位表示一个数据段。
@@ -80,7 +84,7 @@ GreptimeDB 把一个 SST 文件分割成多个索引数据段，每个数据段
## 统一数据访问层：OpenDAL
-GreptimeDB 使用 [OpenDAL][2] 提供统一的数据访问层，因此，存储引擎无需与不同的存储 API 交互，数据可以无缝迁移到基于云的存储，如 AWS S3。
+GreptimeDB 使用 [OpenDAL][2] 为本地文件系统和对象存储提供统一访问层。修改配置的存储 backend 不会迁移已有数据。
[1]: https://parquet.apache.org
[2]: https://github.com/datafuselabs/opendal
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/metric-engine.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/metric-engine.md
index c0cbf681b9..57baecf400 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/metric-engine.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/metric-engine.md
@@ -7,9 +7,9 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述
## 概述
-`Metric` 引擎是 GreptimeDB 的一个组件，属于存储引擎的一种实现，主要针对可观测 metrics 等存在大量小表的场景。
+`Metric` 引擎用于存储包含大量小型指标表的负载。
-它的主要特点是利用合成的物理宽表来存储大量的小表数据，实现相同列复用和元数据复用等效果，从而达到减少小表的存储开销以及提高列式压缩效率等目标。表这一概念在 `Metric` 引擎下变得更更加轻量。
+它将这些逻辑表映射到共享的物理宽表，使其复用列和元数据，从而降低每张表的存储开销并改善列式压缩。
## 概念
@@ -18,7 +18,7 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述
### 逻辑表
逻辑表，即用户定义的表。与普通的表都完全一样，逻辑表的定义包括表的名称、列的定义、索引的定义等。用户的查询、写入等操作都是基于逻辑表进行的。用户在使用过程中不需要关心逻辑表和普通表的区别。
-从实现层面来说，逻辑表是一个虚拟的表，它并不直接读写物理的数据，而是通过将读写请求映射成对应物理表的请求来实现数据的存储与查询。
+逻辑表是虚拟表，本身不直接存储数据。Metric 引擎将其读写请求映射为对应物理表的请求。
### 物理表
物理表是真实存储数据的表，它拥有若干个由分区规则定义的物理 Region。
@@ -27,16 +27,14 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述
`Metric` 引擎的主要设计架构如下：
-
+
-在目前版本的实现中，`Metric` 引擎复用了 `Mito` 引擎来实现物理数据的存储及查询能力，并在此之上同时提供物理表与逻辑表的访问能力。
+`Metric` 引擎将物理存储和查询交给 `Mito` 引擎。每个物理 Region 组包含一个数据 Region 和一个元数据 Region：数据 Region 保存映射到该 Region 组的逻辑表数据，元数据 Region 保存逻辑表及逻辑列的映射。
-在分区方面，逻辑表拥有与物理表完全一致的分区规则及 Region 分布。这是非常自然的，因为逻辑表的数据直接存储在物理表中，所以分区规则也是一致的。
+关联到同一物理表的逻辑表使用相同的分区布局。写入时，Metric 引擎为每行数据记录逻辑表身份；读取时，它在扫描物理 Region 前增加逻辑表过滤条件。
-在路由元数据方面，逻辑表的路由地址为逻辑地址，即该逻辑表所对应的物理表是什么，而后通过该物理表进行二次路由取得真正的物理地址。这一间接路由方式能够显著减少 `Metric` 引擎的 Region 发生迁移调度时所需要修改的元数据数量。
+逻辑表的路由只保存所属物理表的 ID，再由物理表路由解析出持有 Region 的 Datanode。逻辑路由本身不记录 peer，因此迁移物理 Region 只需改写一条物理路由，而不必改写映射到它的每一条逻辑路由。
-在操作方面，`Metric` 引擎支持对逻辑表进行标准的 DML 操作（INSERT、DELETE、SELECT）。然而，对物理表的操作进行了有限的支持以防止误操作，例如禁止直接写入物理表等操作防止影响用户逻辑表的数据。总体上可以认为物理表是对用户只读的。
+逻辑表支持普通的 INSERT、DELETE 和 SELECT 操作。直接写入物理 Region 会绕过逻辑表映射，因此会被拒绝；物理表仍然可以查询。
-为了提升对大量表同时进行 DDL（Data Definition Language，数据操作语言）操作时性能，如 Prometheus Remote Write 冷启动时大量 metrics 带来的自动建表请求，以及前面提到的迁移物理 Region 时大量路由表的修改请求等，`Metric` 引擎引入了一些批量 DDL 操作。这些批量 DDL 操作能够将大量的 DDL 操作合并成一个请求，从而减少了元数据的查询及修改次数，提升了性能。
-
-除了物理表的物理数据 Region 之外，`Metric` 引擎还额外为每一个物理数据 Region 创建了一个物理的元数据 Region，用于存储 `Metric` 引擎自身为了维护映射等状态所需要的一些元数据。这些元数据包括逻辑表与物理表的映射关系，逻辑列与物理列的映射关系等等。
+批量 DDL 用于减少大量逻辑表同时创建或更新时的元数据操作，例如 Prometheus Remote Write 自动建表或物理 Region 迁移。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/overview.md
index 3f44a4543b..c7684eb165 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/overview.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/overview.md
@@ -7,22 +7,26 @@ description: 介绍了 Datanode 的主要职责和组件，包括 gRPC 服务、
## Introduction
-`Datanode` 主要的职责是为 GreptimeDB 存储数据，我们知道在 GreptimeDB 中一个 `table` 可以有一个或者多个 `Region`,
-而 `Datanode` 的职责便是管理这些 `Region` 的读写。`Datanode` 不感知 `table`，可以认为它是一个 `region server`。
-所以 `Frontend` 和 `Metasrv` 按照 `Region` 粒度来操作 `Datanode`。
+Datanode 存储并处理 Region 数据。一张表可以包含多个 Region，但 Datanode 不负责表级路由。Frontend 按 Region 发送数据请求，Metasrv 则控制 Region 的放置和生命周期。
-
+这个边界使同一个 Region server 可以承载不同的存储引擎，而不向 Frontend 或 Metasrv 暴露引擎实现。
+
+
## Components
-一个 datanode 包含了 region server 所需的全部组件。这里列出了比较重要的部分：
-
-- 一个 gRPC 服务来提供对 `Region` 数据的读写，`Frontend` 便是使用这个服务来从 `Datanode` 读写数据。
-- 一个 HTTP 服务，可以通过它来获得当前节点的 metrics、配置信息等
-- `Heartbeat Task` 用来向 `Metasrv` 发送心跳，心跳在 GreptimeDB 的分布式架构中发挥着至关重要的作用，
- 是分布式协调和调度的基础通信通道，心跳的上行消息中包含了重要信息比如 `Region` 的负载，如果 `Metasrv` 做出了调度
- 决定（比如 Region 转移），它会通过心跳的下行消息发送指令到 `Datanode`
-- `Datanode` 不负责解析用户 SQL 或进行分布式规划，用户对一个或多个 `Table` 的查询请求会在 `Frontend` 中被转换为
- `Region` 查询请求，`Datanode` 负责用本地 query engine 执行这些 `Region` 查询计划
-- 一个 `Region Manager` 用来管理 `Datanode` 上的所有 `Region`s
-- GreptimeDB 支持可插拔的多引擎架构，目前已有的 engine 包括 `File Engine` 和 `Mito Engine`
+Datanode 包含以下主要组件：
+
+- Region server 记录已打开的 Region，并把读写和生命周期请求分发给该 Region 注册的 engine。
+- `Mito` 是主要的时序 Region engine。`Metric` 将多个逻辑指标 Region 映射到共享的 Mito Region，`File` 通过 Region 接口访问外部文件。
+- 本地 query engine 执行 Region 查询计划。它不解析客户端 SQL，也不进行集群级规划。
+- Heartbeat task 向 Metasrv 上报节点和 Region 状态，并接收 open、close、upgrade、downgrade 和迁移步骤等指令。
+- gRPC 承载发往 Datanode 的 Region 请求；HTTP 提供 metrics 和配置等节点诊断信息。
+
+## Region 请求生命周期
+
+Mito 写入到达 Region server 后，Region server 根据 Region 元数据选择 Mito。Mito 将 mutation 追加到 WAL，写入 memtable，并在之后把 memtable flush 为 SST 文件。Metric 写入会先补充逻辑表标识，再委托给对应的物理 Mito Region。
+
+读取时，本地 query engine 在 Region engine 提供的 table provider 上执行 Region 计划。Mito scan 获取不可变的 Region version，读取相关 memtable 和 SST 文件，合并并去重数据，最后返回 Arrow record batch 流。
+
+Region 所有权可以在不重启 Datanode 的情况下改变。Metasrv 通过心跳流下发生命周期指令；Region server 将指令应用到对应 engine，并在后续心跳中上报新的 Region role 和统计信息。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/python-scripts.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/python-scripts.md
deleted file mode 100644
index 831278e80d..0000000000
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/python-scripts.md
+++ /dev/null
@@ -1,30 +0,0 @@
----
-keywords: [Python 脚本, 数据分析, CPython, RustPython]
-description: 介绍了在 GreptimeDB 中使用 Python 脚本进行数据分析的两种后端实现：CPython 和嵌入式 RustPython 解释器。
----
-
-# Python 脚本
-
-## 简介
-
-Python 脚本是分析本地数据库中的数据的便捷方式，
-通过将脚本直接在数据库内运行而不是从数据库拉取数据的方式，可以节省大量的数据传输时间。
-下图描述了 Python 脚本的工作原理。
-`RecordBatch`（基本上是表中的一列，带有类型和元数据）可以来自数据库中的任何地方，
-而返回的 `RecordBatch` 可以用 Python 语法注释以指示其元数据，例如类型或空。
-脚本将尽其所能将返回的对象转换为 `RecordBatch`，无论它是 Python 列表、从参数计算出的 `RecordBatch` 还是常量（它被扩展到与输入参数相同的长度）。
-
-
-
-## 两种可选的后端
-
-### CPython 后端
-
-该后端由 [PyO3](https://pyo3.rs/v0.18.1/) 提供支持，可以使用您最喜欢的 Python 库（如 NumPy、Pandas 等），并允许 Conda 管理您的 Python 环境。
-
-但是使用它也涉及一些复杂性。您必须设置正确的 Python 共享库，这可能有点棘手。一般来说，您只需要安装 `python-dev` 包。但是，如果您使用 Homebrew 在 macOS 上安装 Python，则必须创建一个适当的软链接到 `Library/Frameworks/Python.framework`。有关使用 PyO3 crate 与不同 Python 版本的详细说明，请参见 [这里](https://pyo3.rs/v0.18.1/building_and_distribution#configuring-the-python-version)
-
-### 嵌入式 RustPython 解释器
-
-可以运行脚本的实验性 [python 解释器](https://github.com/RustPython/RustPython)，它支持 Python 3.10 语法。您可以使用所有的 Python 语法，更多信息请参见 [Python 脚本的用户指南](/user-guide/python-scripts/overview.md).
-
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/query-engine.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/query-engine.md
index 62d7563bb2..84445759d4 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/query-engine.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/query-engine.md
@@ -7,33 +7,33 @@ description: 介绍了 GreptimeDB 的查询引擎架构，基于 Apache DataFusi
## 介绍
-GreptimeDB 的查询引擎是基于[Apache DataFusion][1]（属于[Apache Arrow][2]的子项目）构建的，它是一个用 Rust 编写的出色的查询引擎。它提供了一整套功能齐全的组件，从逻辑计划、物理计划到执行运行时。下面将解释每个组件如何被整合在一起，以及在执行过程中它们的位置。
+GreptimeDB 的查询引擎基于 [Apache DataFusion][1]。DataFusion 提供逻辑计划、物理计划、优化器框架和执行运行时；GreptimeDB 在此基础上增加各查询语言的 planner、存储相关优化规则、自定义计划节点和分布式执行。
-
+DDL 和其他控制面操作由 statement executor 分发。Query engine 接收数据处理计划，包括 `INSERT ... SELECT` 等操作中读取输入数据的部分。
-入口点是逻辑计划，它被用作查询或执行逻辑等的通用中间表示。逻辑计划的两个主要来源是：1. 用户查询，例如通过 SQL 解析器和规划器的 SQL；2. Frontend 的分布式查询，这将在下一节中详细解释。
+## 查询生命周期
-接下来是物理计划，或称执行计划。与包含所有逻辑计划变体（除特殊扩展计划节点外）的大型枚举的逻辑计划不同，物理计划实际上是一个定义了在执行过程中调用的一组方法的特性。所有数据处理逻辑都包装在实现该特性的相应结构中。它们是对数据执行的实际操作，如聚合器 `MIN` 或 `AVG` ，以及表扫描 `SELECT ... FROM`。
+1. SQL、PromQL 或日志查询 planner 通过 catalog 解析表，并生成 DataFusion logical plan。DataFusion 不直接支持的操作由 GreptimeDB plan extension 表示。
+2. DataFusion 的 analyzer 和 optimizer rule 与 GreptimeDB rule 共同运行。这些规则规范化表达式和类型、改写时间范围操作、将 projection 和 filter 下推到 scan，并在需要时引入分布式计划节点。
+3. Physical planner 将优化后的 logical plan 转换为流式 operator。GreptimeDB 随后应用 scan 并行度、排序和分布式执行相关的 physical rule。
+4. 执行阶段通过 physical plan 拉取 Arrow record batch。存储 scan 接收 projection 和 predicate，下游 operator 消费数据流，无需先物化完整结果。
-优化阶段通过转换逻辑计划和物理计划来提高执行性能，现在全部基于规则。它也被称为“基于规则的优化”。一些规则是 DataFusion 原生的，其他一些是在 GreptimeDB 中自定义的。在未来，我们计划添加更多规则，并利用数据统计进行基于成本的优化 (CBO)。
-
-最后一个阶段"执行"是一个动词，代表从存储读取数据、进行计算并生成预期结果的过程。虽然它比之前提到的概念更抽象，但你可以简单地将它想象为执行一个 Rust 异步函数，并且它确实是一个异步流。
-
-当你想知道 SQL 是如何通过逻辑计划或物理计划中表示时，`EXPLAIN [VERBOSE] ` 是非常有用的。
+使用 [`EXPLAIN`](/reference/sql/explain.md) 查看逻辑和物理计划。`EXPLAIN ANALYZE` 还会执行计划并报告运行时指标。
## 数据表示
-GreptimeDB 使用 [Apache Arrow][2]作为内存中的数据表示格式。它是面向列的，以跨平台格式，也包含许多高性能的基础操作。这些特性使得在许多不同的环境中共享数据和实现计算逻辑变得容易。
+GreptimeDB 使用 [Apache Arrow][2] record batch 作为内存数据表示。一个 record batch 包含等长的列数组和 schema。查询 operator 交换这些 batch 组成的数据流，使 Region scan 到结果编码的执行路径保持列式处理。
## 索引
-在时序数据中，有两个重要的维度：时间戳和标签列（或者类似于关系数据库中的主键）。GreptimeDB 将数据分组到时间桶中，因此能在非常低的成本下定位和提取预期时间范围内的数据。GreptimeDB 中主要使用的持久文件格式 [Apache Parquet][3] 提供了多级索引和过滤器，使得在查询过程中很容易修剪数据。在未来，我们将更多地利用这个特性，并开发我们的分离索引来处理更复杂的用例。
+索引构建和持久化格式属于存储引擎。查询层向 scan 提供 predicate 和 projection，Mito 再利用时间范围、Parquet 统计信息和索引跳过不可能匹配的数据。参见[数据持久化与索引](./data-persistence-indexing.md)。
+
+
-## 分布式查询
+## 分布式执行
-参考 [Distributed Querying][6].
+分布式模式下，Frontend 规划集群级查询，Datanode 执行 Region 本地子计划。[`MergeScan`][6] 是两个阶段之间的边界。
-[1]: https://github.com/apache/arrow-datafusion
+[1]: https://datafusion.apache.org/
[2]: https://arrow.apache.org/
-[3]: https://parquet.apache.org
[6]: ../frontend/distributed-querying.md
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/storage-engine.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/storage-engine.md
index 67f98216fe..1b0f0eb689 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/storage-engine.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/storage-engine.md
@@ -7,7 +7,7 @@ description: 详细介绍了 GreptimeDB 的存储引擎架构、数据模型和
## 概述
-`存储引擎` 负责存储数据库的数据。Mito 是我们默认使用的存储引擎，基于 [LSMT][1]（Log-structured Merge-tree）。我们针对处理时间序列数据的场景做了很多优化，因此 mito 这个存储引擎并不适用于通用用途。
+Mito 是 GreptimeDB 的默认存储引擎，基于 [LSM tree][1]，面向时间序列负载设计，而不是通用的嵌入式存储引擎。
## 架构
下图展示了存储引擎的架构和处理数据的流程。
@@ -21,8 +21,8 @@ description: 详细介绍了 GreptimeDB 的存储引擎架构、数据模型和
- 基于 `LogStore` API 实现，不关心底层存储介质。
- WAL 的日志记录可以存储在本地磁盘上，也可以存储在实现了 `LogStore` API 的远程日志服务中，例如 Kafka（remote WAL）。
- Memtable
- - 数据首先写入 `active memtable`，又称 `mutable memtable`。
- - 当 `mutable memtable` 已满时，它将变为只读的 `immutable memtable`。
+ - Mito 根据 time index 将数据行写入 mutable memtable。
+ - Flush 冻结 mutable memtable，安装一组新的 mutable memtable 以接收写入，再将冻结的 memtable 写为 SST 文件。
- SST
- SST 的全名为有序字符串表（`Sorted String Table`）。
- `immutable memtable` 刷到持久存储后形成一个 SST 文件。
@@ -100,7 +100,9 @@ Mito 会按 primary key 对行分组，并按时间排序，因此 SST 中的数
Mito 支持两种 SST 格式：`flat` 和 `primary_key`。`flat` 是新表的默认格式，适用于各种 primary key 基数，包括高基数 key。`primary_key` 是为了兼容旧表而保留的遗留格式。更多详情请参考 [SST format](/reference/sql/create.md#创建指定-sst-格式的表) 和[表设计指南](/user-guide/deployments-administration/performance-tuning/design-table.md#sst-格式)。
-
+
+
+一个 SST 可能跨越多个 compaction time window。
## 扫描裁剪
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/wal.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/wal.md
index 529adcdf50..805b294091 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/wal.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/wal.md
@@ -9,20 +9,26 @@ description: 介绍了 GreptimeDB 的预写日志（WAL）机制，包括其命
## 介绍
-我们的存储引擎受到了日志结构合并树（Log-structured Merge Tree，LSMT）的启发。对数据的变更操作直接应用于 MemTable 而不是持久化到磁盘上的数据页，这显著提高了性能，但也带来了持久化相关的问题，特别是在 Datanode 意外崩溃时。与所有类似 LSMT 的存储引擎一样，GreptimeDB 使用预写日志（Write-Ahead Log，WAL）来确保数据被可靠地持久化，并且保证崩溃时的数据完整性。
+Mito 在将数据 flush 为 SST 文件前，先在 memtable 中缓冲写入。每个 Region 的 mutation 会先追加到预写日志（WAL），从而恢复尚未进入 SST 的数据。
-预写日志是一个仅提供追加写的文件组。所有的 INSERT 和 DELETE 操作都被转换为操作日志，然后追加到 WAL。一旦操作日志被持久化到底层文件，该操作才可以进一步应用到 MemTable。
+WAL 通过统一的 log-store 抽象访问，可以使用本地 raft-engine 或远端 Kafka。
-当数据节点重新启动时，WAL 中的操作条目将被重放，以重建正确的 MemTable 状态。
+## 写入与恢复流程
-
+正常写入遵循以下顺序：
+
+1. Region worker 分配 sequence number 和 WAL entry ID。
+2. 将 mutation 追加到 WAL。追加失败时，不会把 mutation 写入 memtable。
+3. WAL 追加成功后，Mito 将 mutation 写入 memtable，并发布新的 committed sequence。
+4. Flush 将不可变 memtable 写为 SST 文件，并持久化包含新文件和 `flushed_entry_id` 的 manifest edit。
+5. Manifest edit 持久化后，`flushed_entry_id` 及以前的 WAL entry 被标记为 obsolete；log store 可以稍后再回收物理空间。
+
+Manifest 是恢复边界。正常重新打开 Region 时，Mito 根据 manifest 重建 Region，并从 `flushed_entry_id + 1` 开始重放 WAL。Region 状态切换可以指定更晚的 replay checkpoint，但不会重放早于已持久化 flush 边界的 entry。
## 命名空间
-WAL 的命名空间用于区分来自不同 region 的条目。追加和读取操作必须提供一个命名空间。目前，region ID 被用作命名空间，因为每个 region 都有一个在数据节点重新启动时需要重构的 MemTable。
+WAL entry 按 Region 隔离，而不是按表隔离。追加和读取都需要指定 Region namespace，使单个 Region 可以独立重放或截断。本地 raft-engine 使用 Region ID 作为 namespace ID；Kafka provider 则在基于 topic 的日志中保留 Region 标识。
## 同步/异步刷盘
-默认情况下，WAL 的追加写是异步的，这意味着写入方不会等待操作日志被刷入到磁盘并持久化。这个默认设置提供了更高的性能，但在服务器意外关闭时可能会丢失数据。另一方面，同步刷新提供了更高的可靠性，但其代价是性能更低。
-
-在 v0.4 版本中，新的 region worker 架构可以使用批处理来减轻同步刷盘的开销。
+对于本地 raft-engine，`sync_write` 控制追加写是否等待日志同步到持久化存储，默认值为 `false`。异步写入延迟较低，但主机在缓冲数据同步前故障时，可能丢失最近确认的 entry。Kafka WAL 的持久性由 producer 和集群配置决定，不受这个本地选项控制。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/arrangement.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/arrangement.md
index dd3b6de090..7b35b50259 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/arrangement.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/arrangement.md
@@ -5,6 +5,8 @@ description: 描述了 Arrangement 在数据流进程中的状态存储功能，
# Arrangement
+本页介绍 Flownode 旧 streaming 模式使用的状态结构；batching 模式不使用 Arrangement。
+
Arrangement 存储数据流进程中的状态，存储 flow 的更新流（stream）以供进一步查询和更新。
Arrangement 本质上存储的是带有时间戳的键值对。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/batching_mode.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/batching_mode.md
index 257cca80b9..f83c4e5da1 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/batching_mode.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/batching_mode.md
@@ -9,13 +9,13 @@ description: Flownode 批处理模式概述，一个为数据库提供持续数
## 概述
-`flownode` 中的批处理模式专为持续数据聚合而设计。它在离散的、微小的时间窗口上周期性地执行用户定义的 SQL 查询。这与数据在到达时即被处理的流处理模式形成对比。
+`flownode` 中的批处理模式专为持续数据聚合而设计。它在离散的小时间窗口上周期性执行用户定义的 SQL 查询。旧 streaming 路径则在数据到达时进行处理，目前仅为兼容已有 workload 而保留，不推荐新 workload 使用。
其核心思想是：
1. 定义一个带有 SQL 查询的 `flow`，该查询将数据从源表聚合到目标表。
2. 查询通常在时间戳列上包含一个时间窗口函数（例如 `date_bin`）。
3. 当新数据插入源表时，系统会将相应的时间窗口标记为“脏”(dirty)。
-4. 一个后台任务会周期性地唤醒，识别这些脏窗口，并为那些特定的时间范围重新运行聚合查询。
+4. 一个后台任务按自身的节奏运行，在下一次求值时取出待处理的脏窗口，并对这些时间范围重新运行聚合查询。
5. 然后将结果插入到目标表中，从而有效地更新聚合视图。
## 架构
@@ -39,15 +39,15 @@ description: Flownode 批处理模式概述，一个为数据库提供持续数
- **状态 (`TaskState`)**: 包含任务的动态、可变状态，最重要的是 `DirtyTimeWindows`。
- **执行循环**: 任务运行一个无限循环 (`start_executing_loop`)，该循环：
1. 检查关闭信号。
- 2. 等待一个预定的时间间隔或直到被唤醒。
+ 2. 睡眠到下一次求值时间。设置了求值调度的任务睡眠到下一个调度时间点；自适应任务则按时间窗口大小和最小刷新间隔计算出的轮询间隔睡眠。
3. 基于当前的脏时间窗口集合生成一个新的查询计划 (`gen_insert_plan`)。
4. 对数据库执行查询 (`execute_logical_plan`)。
5. 清理已处理的脏窗口。
### `TaskState` 和 `DirtyTimeWindows`
-- **`TaskState`**: 此结构体跟踪 `BatchingTask` 的运行时状态。它包括 `dirty_time_windows`，这对于确定需要完成哪些操作至关重要。
-- **`DirtyTimeWindows`**: 这是一个关键的数据结构，用于跟踪自上次查询执行以来哪些时间窗口接收到了新数据。它存储一组不重叠的时间范围。当任务的执行循环运行时，它会参考此结构来构建一个 `WHERE` 子句，该子句仅过滤源表中的脏时间窗口。
+- **`TaskState`**: 此结构体跟踪 `BatchingTask` 的运行时状态，包括用于确定待处理工作的 `dirty_time_windows`。
+- **`DirtyTimeWindows`**: 此数据结构跟踪上次查询执行后接收到新数据的时间窗口，并保存一组不重叠的时间范围。执行循环根据它构造 `WHERE` 子句，只从源表选择脏窗口。
### `TimeWindowExpr`
@@ -56,15 +56,15 @@ description: Flownode 批处理模式概述，一个为数据库提供持续数
- **求值**: 它可以接受一个时间戳并对时间窗口表达式求值，以确定该时间戳所属窗口的开始和结束。
- **窗口大小**: 它还可以从表达式中确定时间窗口的大小（持续时间）。
-这对于标记窗口为脏以及在查询源表时生成正确的过滤条件都至关重要。
+标记脏窗口和生成源表过滤条件使用同一套计算。
## 查询执行流程
以下是批处理模式下查询执行的简化分步演练：
1. **数据摄取**: 新数据被写入源表。
-2. **标记为脏**: `BatchingEngine` 收到有关新数据的通知。它使用与每个相关 flow 关联的 `TimeWindowExpr` 来确定哪些时间窗口受到新数据点的影响。然后将这些窗口添加到相应 `TaskState` 中的 `DirtyTimeWindows` 集合中。
-3. **任务唤醒**: `BatchingTask` 的执行循环被唤醒，原因可能是其周期性调度，也可能是因为它被通知有大量积压的脏窗口。
+2. **标记为脏**: `BatchingEngine` 收到有关新数据的通知。它使用与每个相关 flow 关联的 `TimeWindowExpr` 来确定哪些时间窗口受到新数据点的影响。然后将这些窗口添加到相应 `TaskState` 中的 `DirtyTimeWindows` 集合中。标记脏窗口不会唤醒任务。
+3. **下一次求值**: `BatchingTask` 的执行循环在调度时间点或自适应轮询间隔结束后进入下一次求值，取出待处理的脏窗口。
4. **计划生成**: 任务调用 `gen_insert_plan`。此方法：
- 检查 `DirtyTimeWindows`。
- 生成一系列 `OR` 连接的 `WHERE` 子句（例如 `(ts >= 't1' AND ts < 't2') OR (ts >= 't3' AND ts < 't4') ...`），覆盖所有脏窗口。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/dataflow.md
index 9d07922542..ee17b9c785 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/dataflow.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/dataflow.md
@@ -1,18 +1,39 @@
---
-keywords: [Dataflow, SQL 查询, 执行计划, 数据流, map, reduce]
-description: 解释了 Dataflow 模块的核心计算功能，包括 SQL 查询转换、内部执行计划、数据流的触发运行和支持的操作。
+keywords: [Flownode, batching mode, streaming mode, Dataflow, 脏时间窗口]
+description: 介绍 Flownode 如何选择并运行 batching 和旧 streaming 两条执行路径。
---
# 数据流
+Flownode 内部有两条执行路径：
+
+- **Batching mode** 是聚合和 TQL workload 的主要执行路径。它查询已经持久化的 source 数据，并将物化结果写入 sink table。
+- **Streaming mode** 是为兼容已有 workload 而保留的旧执行路径，不推荐新 workload 使用。Frontend 会把新到达的行同步给它进行增量处理。
+
+用户不能直接选择执行模式。创建 Flow 时，GreptimeDB 根据查询和 source table 的属性选择执行路径。聚合、`DISTINCT` 和 TQL 查询使用 batching mode；简单的非聚合查询，以及任何 source table 使用 `ttl = 'instant'` 的 Flow，目前仍使用 streaming mode。
+
+## Batching mode
+
+Batching mode 复用 GreptimeDB 的查询引擎，不需要为每一行输入维护一张算子图。对于基于时间窗口的 Flow，主循环如下：
+
+1. Source table 收到写入后，把受影响的时间窗口标记为 dirty。
+2. `BatchingTask` 按求值调度或自适应轮询节奏运行，并在该次求值时收集待处理的 dirty window。标记 dirty window 不会唤醒任务。
+3. 任务把这些窗口转换成时间谓词，加入 Flow 查询，再请求 Frontend 查询 source table。
+4. 查询结果写入 sink table，更新已重新计算窗口对应的物化结果。
+5. 成功处理的窗口从 dirty set 中移除；执行失败的工作仍可在后续调度中处理。
+
+设置了 evaluation interval、但查询中没有时间窗口表达式的 Flow，会在每次调度时执行完整查询。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见 [Flownode 批处理模式开发者指南](./batching_mode.md)。
+
+## Streaming mode
+
Dataflow 模块（参见 `flow::compute` 模块）是 `flow` 的核心计算模块。
它接收 SQL 查询并将其转换为 `flow` 的内部执行计划。
然后，该执行计划被转化为实际的数据流，而数据流本质上是一个由带有输入和输出端口的函数组成的有向无环图（DAG）。
-数据流会在需要时被触发运行。
+新到达的行变更会增量驱动这张图执行。
-目前该数据流只支持 `map`和 `reduce` 操作，未来将添加对 `join` 等操作的支持。
+Renderer 支持 map/filter/project 和 reduce 操作。执行计划中已经有 join 和 union 节点，但 streaming renderer 尚未实现它们。
在内部，数据流使用 `tuple(row, time, diff)` 以行格式处理数据。
这里 `row` 表示实际传递的数据，可能包含多个 `value` 对象。
`time` 是系统时间，用于跟踪数据流的进度，`diff` 通常表示行的插入或删除（+1 或 -1）。
-因此，`tuple` 表示给定系统时间的 `row` 的插入/删除操作。
+因此，`tuple` 表示给定系统时间的 `row` 的插入/删除操作。有状态算子通过 [Arrangement](./arrangement.md) 保存这些变更的索引 trace。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/distributed-querying.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/distributed-querying.md
index 612174662b..e6d0c35160 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/distributed-querying.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/distributed-querying.md
@@ -1,41 +1,20 @@
---
-keywords: [分布式查询, 查询拆分, 查询合并, TableScan, 物理计划]
-description: 介绍 GreptimeDB 中的分布式查询方法，包括查询的拆分和合并过程，以及 TableScan 节点的作用。
+keywords: [分布式查询, 逻辑计划, MergeScan, Substrait, Region 裁剪]
+description: 介绍 GreptimeDB 如何把逻辑查询计划划分为 Frontend 和 Datanode 上的执行任务。
---
# 分布式查询
-我们知道在 GreptimeDB 中数据是如何分布的（参见“[表分片][1]”），那么如何查询呢？在 GreptimeDB 中，分布式查询非常简单。简单来说，我们只需将查询拆分为子查询，每个子查询负责查询表数据的一个部分，然后将所有结果合并为最终结果。这是一种典型的“拆分 - 合并”方法。具体来说，让我们从查询到达 `frontend` 开始。
+Frontend 和 Datanode 使用同一套基于 DataFusion 的查询引擎。在分布式模式下，Frontend 会增加一个规划步骤，将 Datanode 上执行的工作与 Frontend 上完成的工作分开。
-当查询到达 `frontend` 时，它首先被解析为 SQL 抽象语法树（AST）。我们遍历 AST，并从中生成逻辑计划。顾名思义，逻辑计划只是如何“逻辑地”执行查询的“提示”，它不能被直接运行，因此我们进一步从中生成可执行的物理计划。物理计划是一种类似树形的数据结构，每个节点实际上表示查询的执行方法。一旦我们从上到下运行物理计划树，结果数据将从叶子到根流动，被合并或计算。最终，我们在根节点的输出处得到了查询的结果。
+
-到目前为止，这只是一个典型的“volcano”查询执行模型，你可以在几乎每个 SQL 数据库中看到这种模型。那么“分布式”是在哪里发生的呢？这全部发生在一个名为“TableScan”的物理计划节点中。TableScan 是物理计划树中的一个叶子节点，它负责扫描表的数据（就像它的名称所暗示的）。当 `frontend` 即将扫描表时，它首先需要根据每个 `region` 的数据范围将表扫描拆分为较小的扫描。
+## 分布式规划
-[1]: ./table-sharding.md
+分布式规划器重写逻辑计划，把可以下推的算子移向表扫描，并用 `MergeScan` 节点包装远端子计划。分区列上的谓词还会在任务调度前用于裁剪 Region。
-表的所有 `region` 都有它们存储数据的范围。以下表为例：
+算子能否下推取决于计划形态和算子本身的性质。不支持的部分会保留在 Frontend。初始设计及交换律规则参见[分布式规划器 RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md)。
-```sql
-CREATE TABLE my_table (
- a INT,
- others STRING,
- ts TIMESTAMP TIME INDEX,
-)
-PARTITION ON COLUMNS (a) (
- a < 10,
- a >= 10 AND a < 20,
- a >= 20
-);
-```
+## 分布式计划
-`my_table` 表创建时被设定了 3 个分区。在 GreptimeDB 的当前实现中，将为该表创建 3 个 `region`（分区与 `region` 的比例为 1:1）。这 3 个区域将分别包含以下范围："[-∞, 10)", "[10, 20)" 和 "[20, +∞)"。例如，如果提供了值 "42"，我们将搜索这些范围，并找到包含该值的相应的 `region`（在此示例中为第 3 个 `region`）。
-
-对于查询，我们使用“过滤器”来查找 `region`。 "过滤器"是 "WHERE" 子句中的条件。例如，查询 `SELECT * FROM my_table WHERE a < 10 AND others = 'x'`，其“过滤器”为“a < 10 AND others = 'x'”。然后我们检查这些范围，找出包含满足过滤器条件的值的所有 `region`。
-
-> 如果某个查询没有任何过滤器，则将其视为全表扫描。
-
-找到所需的区域后，我们只需在其中组装子扫描。通过这种方式，我们将查询拆分为子查询，每个子查询都获取表数据的一部分。子查询在 `datanode` 中执行，并在 `frontend` 中等待完成。它们的结果将合并为表扫描请求的最终返回。
-
-下面这张图片总结了分布式查询执行的过程：
-
-
+远端输入是完整的逻辑子计划，并不局限于表扫描。Frontend 使用 [Substrait](https://substrait.io) 序列化子计划，再向持有相应数据的 Datanode 发送 Region 级请求。Datanode 在本地规划并执行子计划，将结果流返回 Frontend。Frontend 合并远端数据流，并执行没有下推的算子。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/overview.md
index ee48da3ce3..6befdc55a9 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/overview.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/overview.md
@@ -5,31 +5,51 @@ description: GreptimeDB Frontend 组件概述 - 为客户端请求提供服务
# Frontend
-**Frontend** 是一个无状态服务，作为 GreptimeDB 中客户端请求的入口点。它为多种数据库协议提供统一接口，并充当代理，将读写请求转发到分布式系统中的相应 Datanode。
+Frontend 是 GreptimeDB 中负责请求编排的无状态服务。Server 层负责终止协议并转换线上消息；Frontend 为这些协议处理器提供数据库行为，包括权限检查、语句执行、路由和分布式查询规划。
+
+Frontend 不存储表数据。它缓存从 Metasrv 获取的 catalog 和路由元数据；元数据发生变化时，Metasrv 通过心跳响应通知 Frontend 失效相应缓存。
## 核心功能
-- **协议支持**：支持多种数据库协议，包括 SQL、PromQL、MySQL 和 PostgreSQL。详见[协议][1]
-- **请求路由**：基于元数据将请求路由到相应的 Datanode
-- **查询分发**：将分布式查询拆分到多个节点
-- **响应聚合**：合并来自多个 Datanode 的结果
-- **认证授权**：安全和访问控制验证
+- 为支持的[协议][1]提供查询和写入行为。
+- 解析 catalog、schema、table 和 Region 路由。
+- 在执行请求前完成权限检查。
+- 规划分布式查询并合并 Datanode 返回的结果。
+- 将表级写入和删除转换为 Region 请求。
## 架构
### 关键组件
-- **协议处理器**：处理不同的数据库协议
-- **目录管理器**：缓存来自 Metasrv 的元数据以实现高效的请求路由和 Schema 校验
-- **分布式规划器**：将逻辑计划转换为分布式执行计划
-- **请求路由器**：为每个请求确定目标 Datanodes
+
+- 协议处理器将 SQL、PromQL、gRPC 写入和可观测性协议转换为 Frontend 的内部请求接口。
+- Catalog manager 和 partition manager 提供表元数据、分区规则和 Region 路由。
+- Statement executor 将查询、DML 和 DDL 分发到各自的执行路径。
+- 分布式规划器把表扫描替换为可跨 Datanode 执行的 `MergeScan` 计划。
### 请求流程
-
+不同操作会走不同的请求路径。
+
+#### 查询
+
+1. 协议处理器创建查询上下文，并完成认证和权限检查。
+2. 对应查询语言的 planner 生成逻辑计划。分布式模式下，planner 根据分区元数据选择 Region 并生成分布式计划。
+3. Frontend 将 Region 子计划发送到对应 Datanode。Datanode 在本地 Region engine 上执行，并返回 Arrow record batch 流。
+4. Frontend 执行剩余算子、合并数据流，再按客户端协议编码结果。
+
+#### 写入和删除
+
+1. Frontend 根据表 schema 校验请求。支持 schema-on-write 的协议可以先创建缺失的表或新增列，再重试写入。
+2. 分区规则把每一行分配给 Region。Frontend 为各目标 Region 构造请求，并路由到当前 Region leader。
+3. Datanode 的 Region server 将请求分发到对应的 Region engine。单机模式下，请求直接发送给内嵌的 Region server。
+
+#### DDL
+
+Statement executor 将 DDL 转换为 task。分布式模式下，Metasrv 以持久化 procedure 执行 task、更新元数据，并协调 Datanode 上的 Region 操作。单机模式复用相同的语句边界，但使用本地元数据和 procedure 实现。
### 部署
-下图是 GreptimeDB 在云上的一个典型的部署。`Frontend` 实例组成了一个集群处理来自客户端的请求：
+下图展示了 GreptimeDB 的一种云上部署。多个 Frontend 实例共同处理客户端请求：
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/table-sharding.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/table-sharding.md
index 63a8ac10a6..cf39afe27b 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/table-sharding.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/table-sharding.md
@@ -5,21 +5,17 @@ description: 介绍 GreptimeDB 中表数据的分片方法，包括分区和 Reg
# 表分片
-对于任何分布式数据库来说，数据的分片都是必不可少的。本文将描述 GreptimeDB 中的表数据如何进行分片。
+GreptimeDB 将一张表分为多个 Region。分区表达式定义每行数据属于哪个 Region，Region 路由则定义当前由哪个 Datanode 持有该 Region。
## 分区
-有关创建分区表的语法，请参阅用户指南中的[表分片](/user-guide/deployments-administration/manage-data/table-sharding.md)部分。
+分区是由一个或多个列上的表达式描述的逻辑行集合。分区布局需要覆盖表的输入域，使每一行都能找到唯一的目标 Region。SQL 语法和支持的表达式参见[表分片](/user-guide/deployments-administration/manage-data/table-sharding.md)。
## Region
-在创建分区后，表中的数据被逻辑上分割。你可能会问："在 GreptimeDB 中，被逻辑上分区的数据是如何存储的？" 答案是保存在 `Region` 当中。
-
-每个 `Region` 对应一个分区，并保存分区的数据。所有的 `Region` 分布在各个 `Datanode` 之中。
-`Metasrv` 管理 `Region` 到 `Datanode` 的路由信息。如果建表后需要调整分区布局，
-GreptimeDB 支持通过显式的 [repartition](/user-guide/deployments-administration/manage-data/repartition.md) 操作拆分或合并分区。
+每个分区对应一个 Region。Region ID 是 Frontend、Datanode 和 Metasrv 用于存储和路由的标识。同一张表的多个 Region 可以放在同一个 Datanode 上。
分区和 Region 的关系参见下图：
@@ -54,3 +50,13 @@ GreptimeDB 支持通过显式的 [repartition](/user-guide/deployments-administr
└──────────────────────────────────┘
可以放在同一个 Datanode 之中
```
+
+## 路由与剪枝
+
+写入时，Frontend 对每行数据计算分区规则，按 Region 分组，再根据路由表把 Region 请求发送到当前 leader。
+
+查询时，分布式 planner 将查询谓词与分区表达式比较，只扫描可能满足谓词的 Region。如果分区元数据缺失或无法安全解释，planner 会退化为扫描所有 Region，避免漏掉数据。
+
+## 调整分区布局
+
+[Repartition](/user-guide/deployments-administration/manage-data/repartition.md) 通过显式的 split 或 merge 调整已有布局。Metasrv 以持久化 procedure 执行变更，更新 Region 路由和分区表达式，并使旧的表路由缓存失效。Frontend 刷新到新元数据后，后续请求使用新的布局。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/getting-started.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/getting-started.md
index e5aa2ca855..999f696ed0 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/getting-started.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/getting-started.md
@@ -15,14 +15,13 @@ description: 介绍如何在本地环境中从源代码编译和运行 GreptimeD
### 构建依赖项
-- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line)（可选）
+- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line)（可选；克隆仓库需要，构建本身不需要）
- C/C++ 工具链：提供编译和链接的基本工具。在 Ubuntu 上，这可用作 `build-essential`。在其他平台上，也有类似的命令。
-- Rust nightly 工具链（[指南][1]）
- - 编译源代码
+- [Rustup][1]。仓库通过 `rust-toolchain.toml` 指定所需的 nightly 工具链。
- Protobuf（[指南][2]）
- 编译 proto 文件
- 请注意，版本需要 >= 3.15。你可以使用 `protoc --version` 检查它。
-- 机器：建议内存在 16GB 以上 或者 使用[mold](https://github.com/rui314/mold)工具以降低链接时的内存使用。
+- 机器：建议 16GB 以上内存。内存较小时，可使用 [mold](https://github.com/rui314/mold) 降低链接阶段的内存占用。
[1]:
[2]:
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/how-to/how-to-write-sdk.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/how-to/how-to-write-sdk.md
index 28ff757763..31e2d45881 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/how-to/how-to-write-sdk.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/how-to/how-to-write-sdk.md
@@ -1,21 +1,17 @@
---
keywords: [gRPC SDK, GreptimeDatabase, GreptimeRequest, GreptimeResponse, 插入请求]
-description: 介绍如何为 GreptimeDB 开发一个 gRPC SDK，包括 GreptimeDatabase 服务的定义、GreptimeRequest 和 GreptimeResponse 的结构。
+description: 介绍 GreptimeDB gRPC 写入 SDK 需要遵守的协议契约和错误处理要求。
---
# 如何为 GreptimeDB 开发一个 gRPC SDK
-GreptimeDB 的 gRPC SDK 只需要处理写请求即可。读请求是标准 SQL 或 PromQL，可以由任何 JDBC 客户端或 Prometheus
-客户端处理。这也是为什么所有的 GreptimeDB SDK 都命名为 "`greptimedb-ingester-`"。请确保你的 GreptimeDB SDK
-遵循相同的命名约定。
+GreptimeDB 的公开 gRPC SDK 是写入客户端。查询通常通过标准 SQL 或 PromQL 客户端完成。除非有单独需求，新 SDK 应聚焦写入和删除，并遵循 `greptimedb-ingester-` 命名约定。面向用户的 API 参见 [gRPC SDK 概述](/user-guide/ingest-data/for-iot/grpc-sdks/overview.md)。
## `GreptimeDatabase` 服务
-GreptimeDB 自定义了一个 gRPC 服务：`GreptimeDatabase`
-。你只需要实现这个服务即可。你可以在[这里](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto)
-找到它的 Protobuf 定义。
+从 [`GreptimeDatabase` Protobuf 定义](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto)生成客户端 stub，不要在 SDK 中手写一份 message 或 service 定义。
-`GreptimeDatabase` 有 2 个 RPC 方法：
+该 service 提供一个 unary method 和一个 client-streaming method：
```protobuf
service GreptimeDatabase {
@@ -25,13 +21,9 @@ service GreptimeDatabase {
}
```
-`Handle` 方法是一个 unary 调用：当 GreptimeDB 服务接收到一个 `GreptimeRequest` 请求后，它立刻处理该请求并返回一个相应的
-`GreptimeResponse`。
+`Handle` 对一个请求返回一个响应，是 SDK insert 和 delete API 通常使用的方法。
-`HandleRequests` 方法则是一个 "[Client Streaming RPC][3]" 方式的调用。
-它可以接受一个连续的 `GreptimeRequest` 请求流，持续地发给 GreptimeDB 服务。
-GreptimeDB 服务会在收到流中的每个请求时立刻进行处理，并最终（流结束时）返回一个总结性的 `GreptimeResponse`。
-通过 `HandleRequests`，我们可以获得一个非常高的请求吞吐量。
+`HandleRequests` 是 [client-streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc)。客户端关闭请求流后，服务端才返回累计响应。SDK 如果暴露 streaming API，需要明确这个确认边界，并将一个 stream 绑定到一个 endpoint。
### `GreptimeRequest`
@@ -51,11 +43,13 @@ message GreptimeRequest {
}
```
-`RequestHeader` 是必需，它包含了一些上下文，鉴权和其他信息。"oneof" 的字段包含了发往 GreptimeDB 服务的请求。
+客户端需要在 `RequestHeader` 中填写服务端要求的 database context 和认证信息，并且只能设置一个 request variant。
-注意我们有两种类型的插入请求，一种是以 "列" 的形式（`InsertRequests`），另一种是以 "行" 的形式（`RowInsertRequests`
-）。通常我们建议使用 "行" 的形式，因为它对于表的插入更自然，更容易使用。但是，如果需要一次插入大量列，或者有大量的 "null"
-值需要插入，那么最好使用 "列" 的形式。
+该 message 还包含供内部调用者使用的 query 和 DDL variant。公开 ingester API 不应暴露它们，因为 `GreptimeDatabase` 不返回 query result stream。
+
+GreptimeDB 同时接受行式 `RowInsertRequests` 和列式 `InsertRequests`。公开写入 API 默认使用行式请求。面向列的客户端可以使用列式请求，但转换过程中必须保持列长度一致，并保留 null、时间戳精度、数据类型和列 semantic type。
+
+删除同样区分行式和列式。SDK 只应暴露能够在不丢失类型信息的前提下完成映射的形式。
### `GreptimeResponse`
@@ -68,6 +62,18 @@ message GreptimeResponse {
}
```
-`ResponseHeader` 包含了返回值的状态码，以及错误信息（如果有的话）。"oneof" 的字段目前只有 "affected rows"。
+成功响应包含 success header 和 `affected_rows`。该值表示服务端确认的行数；关闭请求流时返回的是累计值。
+
+请求失败通过 gRPC status 返回。Trailing metadata 中的 `x-greptime-err-code` 在存在时提供 GreptimeDB error code，错误文本则由 status message 携带。SDK 应保留 gRPC status 并暴露 GreptimeDB error code，不能用一个通用 SDK error 将其覆盖。
+
+## 重试与交付语义
+
+重试次数必须有上限，并且对调用者可见。只有错误被标记为 retryable 且 deadline 仍允许时，才能重试 unary request。Cancellation 和 deadline expiration 不应重试。
+
+响应丢失不代表服务端拒绝了写入。除非调用者的数据模型保证操作幂等，重试这类请求可能插入重复行。SDK 需要说明这一点，并在交付结果不确定时返回最终错误。
+
+不要自动重试只发送了一部分的 `HandleRequests` stream。即使客户端尚未收到累计响应，服务端也可能已经接受了部分请求。此时应关闭失败的 stream，并将不确定状态返回给调用者。
+
+Arrow Flight bulk ingestion 与 `GreptimeDatabase` RPC 应使用不同的 API。它的 batching 和 partial acceptance 需要独立的契约。
-GreptimeDB 现在有很多 SDK，你可以参考[这里](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=)获取一些示例。
+可以参考现有 [GreptimeDB ingester 仓库](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=)的公开 API 约定，但线上行为应以当前 Protobuf 定义和服务端契约为准。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/admin-api.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/admin-api.md
index 3ac3d44f3b..bd28206c6f 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/admin-api.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/admin-api.md
@@ -1,20 +1,25 @@
---
-keywords: [Admin API, 健康检查, leader 查询, 心跳检测, 维护模式]
-description: 介绍 Metasrv 的 Admin API，包括健康检查、leader 查询、心跳检测、维护模式和 Procedure Manager 控制等功能。
+keywords: [Admin API, 健康检查, leader 查询, 心跳检测, 维护模式, 恢复模式, table id sequence]
+description: 介绍 Metasrv 用于状态检查、集群控制和元数据恢复的 Admin API。
---
# Admin API
-Admin 提供了一种简单的方法来查看和管理集群信息，包括 metasrv 健康检测、metasrv leader 查询、数据节点心跳检测、维护模式和 Procedure Manager 控制。
+:::tip
+本页所有 Admin API 都监听 Metasrv 的 `HTTP_PORT`，默认值为 `4000`。
+:::
-Admin API 是一个 HTTP 服务，提供一组可以通过 HTTP 请求调用的 RESTful API。Admin API 简单、用户友好且安全。
+Admin API 通过 HTTP 提供 Metasrv 状态、集群控制和元数据恢复操作。该 API 不提供认证，且部分端点会改变集群行为或元数据分配，部署时必须通过网络策略保护 HTTP 端口。
本页介绍以下 API：
- /health
- /leader
- /heartbeat
+- /node-lease
- /maintenance
- /procedure-manager
+- /recovery
+- /sequence/table
所有这些 API 都在父资源 `/admin` 下。
@@ -22,7 +27,7 @@ Admin API 是一个 HTTP 服务，提供一组可以通过 HTTP 请求调用的
## /health HTTP 端点
-`/health` 端点接受 GET HTTP 请求，你可以使用此端点检查你的 metasrv 实例的健康状况。
+`/health` 端点接受 GET 请求。HTTP 服务运行时返回 `OK`，但不会检查当前 Metasrv 是否为 leader，也不会检查外部依赖是否可用。
### 定义
@@ -116,9 +121,17 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100'
]
```
+## /node-lease HTTP 端点
+
+`/node-lease` 返回 Metasrv 当前记录的 Datanode lease，可用于判断 Metasrv 是否仍将某个 Datanode 视为存活。
+
+```bash
+curl -X GET http://localhost:4000/admin/node-lease
+```
+
## /maintenance HTTP 端点
-集群维护模式是 GreptimeDB 中的一项安全功能，它可以临时禁用自动集群管理操作。此模式在集群升级、计划停机以及任何可能暂时影响集群稳定性的操作期间特别有用。有关更多详细信息，请参阅[集群维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。
+维护模式在升级、计划停机等操作期间临时禁用自动集群管理。它对集群的具体影响参见[集群维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。
`/maintenance` 端点支持以下 HTTP 请求：
@@ -151,3 +164,39 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100'
"status": "running"
}
```
+
+## /recovery HTTP 端点
+
+Recovery mode 控制手动修改 table ID sequence 等元数据修复端点。它只用于恢复工作，不用于常规维护。
+
+- `GET /admin/recovery/status`：查询 recovery mode 是否开启。
+- `POST /admin/recovery/enable`：开启 recovery mode。
+- `POST /admin/recovery/disable`：关闭 recovery mode。
+
+响应体格式如下：
+
+```json
+{
+ "enabled": true
+}
+```
+
+修复完成后应关闭 recovery mode。如果只是计划暂停自动集群操作，应使用[维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。
+
+## /sequence/table HTTP 端点
+
+这些端点用于检查或修复 table ID sequence：
+
+- `GET /admin/sequence/table/next-id`：返回下一个 table ID，但不执行分配。
+- `POST /admin/sequence/table/set-next-id`：推进下一个 table ID。
+
+设置 sequence 前必须开启 recovery mode。新值必须大于当前值，不能通过该 API 回退 sequence。Recovery mode 只是该 API 的前置条件，不能阻止 DDL。执行该操作时，必须遵循[管理 Table ID Sequence](/user-guide/deployments-administration/maintenance/sequence-management.md)中的完整集群操作流程。
+
+```bash
+curl -X POST \
+ -H 'Content-Type: application/json' \
+ -d '{"next_table_id": 2048}' \
+ http://localhost:4000/admin/sequence/table/set-next-id
+```
+
+该操作会影响后续新表分配到的 ID。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/overview.md
index 2e3c525baa..e9f6144f26 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/overview.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/overview.md
@@ -1,162 +1,101 @@
---
-keywords: [Metasrv, 元数据存储, 请求路由, 负载均衡, 高可用性]
-description: 介绍 Metasrv 的功能、架构和与前端的交互方式。
+keywords: [Metasrv, 元数据, 路由, Leader 选举, Procedure, 心跳]
+description: 介绍 Metasrv 提供的元数据及集群协调机制。
---
# Metasrv
-
-
## Metasrv 包含什么
-- 存储元数据（Catalog, Schema, Table, Region 等）
-- 请求路由器。它告诉前端在哪里写入和读取数据。
-- 数据节点的负载均衡，决定谁应该处理新的表创建请求，更准确地说，它做出资源分配决策。
-- 选举与高可用性，GreptimeDB 设计为 Leader-Follower 架构，只有 leader 节点可以写入，而 follower 节点可以读取，follower 节点的数量通常 >= 1，当 leader 不可用时，follower 节点需要能够快速切换为 leader。
-- 统计数据收集（通过每个节点上的心跳报告），如 CPU、负载、节点上的表数量、平均/峰值数据读写大小等，可用作分布式调度的基础。
+Metasrv 是 GreptimeDB 分布式集群中的元数据和协调服务，不参与数据读写链路。它主要负责：
+
+- 存储 Catalog、Schema、Table、Region、路由和节点元数据；
+- 为新 Region 选择 Datanode，并维护表路由；
+- 选举一个 Metasrv leader 负责协调元数据变更；
+- 通过可恢复的 Procedure 执行 DDL、Region 迁移、故障转移和重分区；
+- 通过心跳维护节点租约和 Region 统计信息；
+- 元数据变更时向 Frontend、Datanode 和 Flownode 广播缓存失效；
+- 向 Datanode 下发 Region 生命周期指令。
## 前端如何与 Metasrv 交互
-首先，请求路由器中的路由表结构如下（注意这只是逻辑结构，实际存储结构可能不同，例如端点可能有字典压缩）。
-
-```txt
- table_A
- table_name
- table_schema // 用于物理计划
- regions
- region_1
- mutate_endpoint
- select_endpoint_1, select_endpoint_2
- region_2
- mutate_endpoint
- select_endpoint_1, select_endpoint_2, select_endpoint_3
- region_xxx
- table_B
- ...
-```
+Frontend 从 Metasrv 获取表元数据和 Region 路由，并缓存在本地。修改元数据的语句会发送给 Metasrv leader；普通读写则使用缓存的路由直接访问 Datanode。
-### 创建表
+控制链路和数据链路相互分离：
-1. 前端发送 `CREATE TABLE` 请求到 Metasrv。
-2. 根据请求中包含的分区规则规划 Region 数量。
-3. 检查数据节点可用资源的全局视图（通过心跳收集）并为每个 Region 分配一个节点。
-4. 前端创建表并在成功创建后将 `Schema` 存储到 Metasrv。
+```text
+Frontend
+ |-- 元数据查询和 DDL -------------------> Metasrv leader
+ `-- Region 读写 ------------------------> Datanode
-### `Insert`
+Metasrv leader
+ |-- Region 生命周期指令 ----------------> Datanode
+ `-- 缓存失效 --------------------------> Frontend / Datanode / Flownode
-1. 前端从 Metasrv 获取指定表的路由。注意，最小的路由单元是表的路由（多个 Region），即包含该表所有 Region 的地址。
-2. 最佳实践是前端首先从本地缓存中获取路由并将请求转发到数据节点。如果路由不再有效，则数据节点有义务返回 `Invalid Route` 错误，前端重新从 Metasrv 获取最新数据并更新其缓存。路由信息不经常变化，因此，前端使用惰性策略维护缓存是足够的。
-3. 前端处理可能包含多个表和多个 Region 的一批写入，因此前端需要根据“路由表”拆分用户请求。
+Datanode
+ `-- 心跳、租约续期和 Region 统计信息 ----> Metasrv leader
+```
-### `Select`
+在稳定状态下，表路由为每个 Region 记录一个 leader peer 和零个或多个 follower peer。Leader 是写入目标；支持只读副本的部署可以把读取路由到 follower：
-1. 与 `Insert` 类似，前端首先从本地缓存中获取路由表。
-2. 与 `Insert` 不同，对于 `Select`，前端需要从路由表中提取只读节点（follower），然后根据优先级将请求分发到 leader 或 follower 节点。
-3. 前端的分布式查询引擎根据路由信息分发多个子查询任务并聚合查询结果。
+```text
+Table route
+ |-- Region 0
+ | |-- leader -> Datanode A
+ | `-- followers -> Datanode B, Datanode C
+ `-- Region 1
+ `-- leader -> Datanode D
+```
-## Metasrv 架构
+Region 迁移或故障转移会改变 peer 角色，并可能使 Region 暂时没有 leader。Frontend 刷新缓存路由后，再把后续读写发送给当前 peer。
-
+### 创建表
-## 分布式共识
+1. Frontend 向 Metasrv leader 提交 DDL 请求。
+2. Metasrv 根据分区规则确定 Region，并[为每个 Region 选择 Datanode](/contributor-guide/metasrv/selector.md)。
+3. 持久化的 Procedure 创建 Region，并写入表元数据和路由。发生 leader 切换后，Procedure 可以从已保存的状态继续执行。
+4. 元数据提交后，Metasrv 通知 Frontend 刷新相关缓存。
-如你所见，Metasrv 依赖于分布式共识，因为：
+### `Insert`
-1. 首先，Metasrv 必须选举一个 leader，数据节点只向 leader 发送心跳，我们只使用单个 Metasrv 节点接收心跳，这使得基于全局信息进行一些计算或调度变得容易且快速。至于数据节点如何连接到 leader，这由 MetaClient 决定（使用重定向，心跳请求变为 gRPC 流，使用重定向比转发更不容易出错），这对数据节点是透明的。
-2. 其次，Metasrv 必须为数据节点提供选举 API，以选举“写入”和“只读”节点，并帮助数据节点实现高可用性。
-3. 最后，`Metadata`、`Schema` 和其他数据必须在 Metasrv 上可靠且一致地存储。因此，基于共识的算法是存储它们的理想方法。
+Frontend 解析表路由，按照分区规则拆分数据行，再把各 Region 的写入发送到对应 Datanode。路由发生变化时，相关缓存会失效，Frontend 随后从 Metasrv 重新获取元数据。
-对于 Metasrv 的第一个版本，我们选择 Etcd 作为共识算法组件（Metasrv 设计时考虑适应不同的实现，甚至创建一个新的轮子），原因如下：
+### `Select`
-1. Etcd 提供了我们需要的 API，例如 `Watch`、`Election`、`KV` 等。
-2. 我们只执行两个分布式共识任务：选举（使用 `Watch` 机制）和存储（少量元数据），这两者都不需要我们定制自己的状态机，也不需要基于 raft 定制自己的状态机；少量数据也不需要多 raft 组支持。
-3. Metasrv 的初始版本使用 Etcd，使我们能够专注于 Metasrv 的功能，而不需要在分布式共识算法上花费太多精力，这提高了系统设计（避免与共识算法耦合）并有助于初期的快速开发，同时通过良好的架构设计，未来可以轻松接入优秀的共识算法实现。
+Frontend 在查询规划期间使用表和 Region 元数据。分区列上的谓词用于裁剪 Region，分布式查询引擎再把任务发送给持有这些 Region 的 Datanode。参见[分布式查询](../frontend/distributed-querying.md)。
-## 心跳管理
+## Metasrv 架构
-数据节点与 Metasrv 之间的主要通信方式是心跳请求/响应流，我们希望这是唯一的通信方式。这个想法受到 [TiKV PD](https://github.com/tikv/pd) 设计的启发，我们在 [RheaKV](https://github.com/sofastack/sofa-jraft/tree/master/jraft-rheakv/rheakv-pd) 中有实际经验。请求发送其状态，而 Metasrv 通过心跳响应发送不同的调度指令。
+主要协调路径如下：
+
+```text
+Leader election
+ |
+ v
+Metasrv leader
+├─ DDL manager -> Procedure manager
+├─ Selector -> 新 Region 的放置
+├─ Heartbeat handler chain -> 租约和 Region 统计信息
+├─ Region supervisor -> Region 迁移 Procedure
+├─ Mailbox -> 缓存失效和 Region 指令
+└─ Metadata managers -> KV backend
+```
-心跳可能携带以下数据，但这不是最终设计，我们仍在讨论和探索究竟应该收集哪些数据。
+这些机制共享元数据，但故障边界不同。进程重启可以丢弃缓存和 leader 本地状态；恢复所需的元数据和 Procedure 状态必须持久化。
-```
-service Heartbeat {
- // 心跳，心跳可能有很多内容，例如：
- // 1. 要注册到 Metasrv 并可被其他节点发现的元数据。
- // 2. 一些性能指标，例如负载、CPU 使用率等。
- // 3. 正在执行的计算任务数量。
- rpc Heartbeat(stream HeartbeatRequest) returns (stream HeartbeatResponse) {}
-}
-
-message HeartbeatRequest {
- RequestHeader header = 1;
-
- // 自身节点
- Peer peer = 2;
- // leader 节点
- bool is_leader = 3;
- // 实际报告时间间隔
- TimeInterval report_interval = 4;
- // 节点状态
- NodeStat node_stat = 5;
- // 此节点中的 Region 状态
- repeated RegionStat region_stats = 6;
- // follower 节点和状态，在 follower 节点上为空
- repeated ReplicaStat replica_stats = 7;
-}
-
-message NodeStat {
- // 此期间的读取容量单位
- uint64 rcus = 1;
- // 此期间的写入容量单位
- uint64 wcus = 2;
- // 此节点中的表数量
- uint64 table_num = 3;
- // 此节点中的 Region 数量
- uint64 region_num = 4;
-
- double cpu_usage = 5;
- double load = 6;
- // 节点中的读取磁盘 I/O
- double read_io_rate = 7;
- // 节点中的写入磁盘 I/O
- double write_io_rate = 8;
-
- // 其他
- map attrs = 100;
-}
-
-message RegionStat {
- uint64 region_id = 1;
- TableName table_name = 2;
- // 此期间的读取容量单位
- uint64 rcus = 3;
- // 此期间的写入容量单位
- uint64 wcus = 4;
- // 近似 Region 大小
- uint64 approximate_size = 5;
- // 近似行数
- uint64 approximate_rows = 6;
-
- // 其他
- map attrs = 100;
-}
-
-message ReplicaStat {
- Peer peer = 1;
- bool in_sync = 2;
- bool is_learner = 3;
-}
-```
+## 分布式共识
-## Central Nervous System (CNS)
+Metasrv 将 leader 选举与元数据存储分开。只有选出的 Metasrv leader 执行协调和元数据变更操作，其他 Metasrv 节点会把 client 引导到当前 leader。
-我们要构建一个算法系统，该系统依赖于每个节点的实时和历史心跳数据，应该做出一些更智能的调度决策并将其发送到 Metasrv 的 Autoadmin 单元，该单元分发调度决策，由数据节点本身或更可能由 PaaS 平台执行。
+Key-value backend 保存表元数据、路由、Procedure 状态以及其他必须跨 leader 切换保留的信息。Metasrv 不使用这套选举为 Datanode Region 创建读写副本；Region 可用性由心跳、Region 故障检测和故障转移 Procedure 管理。
-## 工作负载抽象
+## 心跳管理
-工作负载抽象的级别决定了 Metasrv 生成的调度策略（如资源分配）的效率和质量。
+Datanode 与 Metasrv leader 保持心跳流。心跳请求报告节点身份、租约、Region 统计信息以及放置和监控所需的其他状态；响应则携带 Region 生命周期指令、缓存失效等控制消息。
-DynamoDB 定义了 RCUs 和 WCUs（读取容量单位/写入容量单位），解释说 RCU 是一个 4KB 数据的读取请求，WCU 是一个 1KB 数据的写入请求。当使用 RCU 和 WCU 描述工作负载时，更容易实现性能可测量性并获得更有信息量的资源预分配，因为我们可以将不同的硬件能力抽象为 RCU 和 WCU 的组合。
+心跳驱动两套相互独立的机制，调整心跳周期会同时影响两者：
-然而，GreptimeDB 面临比 DynamoDB 更复杂的情况，特别是 RCU 不适合描述需要大量计算的 GreptimeDB 读取工作负载。我们正在努力解决这个问题。
+- **节点租约**：keep-lease handler 为发送心跳的 Datanode 续期。Selector 和 `/node-lease` 端点据此判断 Datanode 是否仍然存活。
+- **Region 故障检测**：Region supervisor 为每个 Region 维护一个基于心跳到达间隔的 Phi Accrual 检测器，其判定与租约是否过期无关。
+只有开启 Region 故障转移时，故障判定才会提交故障转移迁移。该功能默认关闭，并且要求使用 remote WAL，除非显式允许在本地 WAL 上执行。维护模式同样会抑制故障转移。前置条件和开启方式参见 [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md)。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/selector.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/selector.md
index 9048e0daa4..29b521212b 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/selector.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/selector.md
@@ -7,11 +7,7 @@ description: 介绍 Metasrv 中的 Selector，包括其类型和配置方法。
## 介绍
-什么是 `Selector`？顾名思义，它允许用户从给定的 `namespace` 和 `context` 中选择 `Item`s。有一个相关的 `trait`，也叫做 `Selector`，其定义可以在[这里][0]找到。
-
-[0]: https://github.com/GreptimeTeam/greptimedb/blob/main/src/meta-srv/src/selector.rs
-
-在 `Metasrv` 中存在一个特定的场景。当 `Frontend` 向 `Metasrv` 发送建表请求时，`Metasrv` 会创建一个路由表（表的创建细节不在这里赘述）。在创建路由表时，`Metasrv` 需要选择适当的 `Datanode`s，这时候就需要用到 `Selector`。
+建表时，Metasrv 使用 `Selector` 为各 Region 选择 Datanode。Selector 根据当前节点租约进行选择；部分实现还会使用 Region 统计信息。
@@ -19,22 +15,22 @@ description: 介绍 Metasrv 中的 Selector，包括其类型和配置方法。
`Metasrv` 目前提供以下几种类型的 `Selectors`:
-### LeasebasedSelector
+### LeaseBasedSelector
-`LeasebasedSelector` 从所有可用的（也就是在租约期间内）`Datanode` 中随机选择，其特点是简单和快速。
+`LeaseBasedSelector` 从租约有效的 Datanode 中随机选择。
### LoadBasedSelector
`LoadBasedSelector` 按照负载来选择，负载值则由每个 `Datanode` 上的 region 数量决定，较少的 region 表示较低的负载，`LoadBasedSelector` 优先选择低负载的 `Datanode`。
### RoundRobinSelector [默认选项]
-`RoundRobinSelector` 以轮询的方式选择 `Datanode`。在大多数情况下，这是默认的且推荐的选项。如果你不确定选择哪个，通常它就是正确的选择。
+`RoundRobinSelector` 以轮询方式选择 Datanode，是默认选项，也适用于大多数部署。
## 配置
您可以在启动 `Metasrv` 服务时通过名称配置 `Selector`。
-- LeasebasedSelector: `lease_based` 或 `LeaseBased`
+- LeaseBasedSelector: `lease_based` 或 `LeaseBased`
- LoadBasedSelector: `load_based` 或 `LoadBased`
- RoundRobinSelector: `round_robin` 或 `RoundRobin`
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/overview.md
index 9f82a72171..badd6ad62d 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/overview.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/overview.md
@@ -5,9 +5,7 @@ description: 介绍 GreptimeDB 的架构、关键概念和工作原理，包括
# 贡献者指南
-DeepWiki 对 GreptimeDB 的架构和实现进行了详细且清晰的描述，强烈推荐阅读：
-
-[https://deepwiki.com/GreptimeTeam/greptimedb](https://deepwiki.com/GreptimeTeam/greptimedb)
+本指南面向 GreptimeDB 贡献者，介绍理解内部实现所需的设计机制。从源码构建和运行参见[快速开始](/contributor-guide/getting-started.md)。提交要求（CLA、license header、代码格式，以及 PR 必须通过的检查）以源码仓库的 [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md) 为准。
## 架构
@@ -18,7 +16,13 @@ DeepWiki 对 GreptimeDB 的架构和实现进行了详细且清晰的描述，
- [frontend][1]
- [datanode][2]
- [metasrv][3]
+- [flownode][4]
[1]: /contributor-guide/frontend/overview.md
[2]: /contributor-guide/datanode/overview.md
[3]: /contributor-guide/metasrv/overview.md
+[4]: /contributor-guide/flownode/overview.md
+
+## 补充参考
+
+[DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb) 提供了自动生成的 GreptimeDB 源码导读，可用于了解不熟悉的模块。它属于辅助资料；涉及具体版本的行为时，仍应以对应源码为准。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/integration-test.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/integration-test.md
index 63ffa56bc0..767109f773 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/integration-test.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/integration-test.md
@@ -7,8 +7,14 @@ description: 介绍 GreptimeDB 的集成测试，包括测试范围和如何运
## 介绍
-集成测试使用 Rust 测试工具（`#[test]`）编写，与单元测试不同，它们被单独放置在
-[这里](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration)。
-它涵盖了涉及多个组件的场景，其中一个典型案例是与 HTTP/gRPC 相关的功能。你可以查看
-其[文档](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md)以获取更多信息。
+集成测试覆盖跨 crate 或服务边界的行为，例如 HTTP 和 gRPC 处理、分布式组件或外部存储。测试使用 Rust test harness，位于 [`tests-integration`](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration) package。
+运行命令如下：
+
+```shell
+cargo nextest run -p tests-integration
+```
+
+部分 case 依赖外部服务的环境变量或 fixture。运行前按照 package 的[准备说明](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md)配置环境。
+
+只有 crate 级测试或 Sqlness case 无法覆盖所需边界时才使用集成测试。隔离的逻辑仍放在单元测试中，便于快速复现失败。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/overview.md
index 81b6defa45..f38b625082 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/overview.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/overview.md
@@ -5,5 +5,12 @@ description: GreptimeDB 的测试
# 测试
-我们的团队进行了大量测试，以确保 GreptimeDB 的行为。本章将介绍几种用于测试 GreptimeDB 的重要方法，以及如何使用它们。
+选择能够覆盖本次改动的最小测试范围：
+| 测试类型 | 适用场景 | 常用命令 |
+| --- | --- | --- |
+| [单元测试](unit-test.md) | 单个 crate 或组件内的逻辑 | `cargo nextest run -p ` |
+| [Sqlness 测试](sqlness-test.md) | SQL、协议、planner、执行和端到端回归 | `cargo sqlness bare -t ` |
+| [集成测试](integration-test.md) | 跨组件或依赖外部服务的行为 | `cargo nextest run -p tests-integration` |
+
+需要运行完整 Rust workspace 测试时使用 `make test`。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/sqlness-test.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/sqlness-test.md
index d29b69e835..06503a434c 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/sqlness-test.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/sqlness-test.md
@@ -7,36 +7,34 @@ description: 介绍 GreptimeDB 的 Sqlness 测试，包括测试文件类型、
## 介绍
-SQL 是 `GreptimeDB` 的一个重要用户接口。我们为它提供了一个单独的测试套件（名为 `sqlness`）。
+Sqlness 是 GreptimeDB 针对 SQL 和协议行为的端到端回归测试。每个 case 向运行中的 GreptimeDB 发送语句，并将输出与仓库中的结果文件比较。
## Sqlness 手册
### 测试文件
-Sqlness 有两种类型的文件
+每个 case 使用两类文件：
- `.sql`：测试输入，仅包含 SQL
- `.result`：预期的测试输出，包含 SQL 和其结果
-`.result` 文件是预期的执行输出。如果 `.result` 文件发生变化，意味着测试结果不同，测试可能失败。你应该检查变更日志来解决问题。
-
-你只需要在 `.sql` 文件中编写测试 SQL，然后运行测试。
+在 `.sql` 文件中编写输入，运行测试后生成或更新 `.result`。必须检查每一处结果差异，只有行为变化符合预期时才能接受。
### 组织测试案例
-输入案例的根目录是 `tests/cases`。它包含几个子目录，代表不同的测试模式。例如，`standalone/` 包含所有在 `greptimedb standalone start` 模式下运行的测试。
+输入 case 位于 `tests/cases`。第一级目录选择运行环境，例如 `standalone/` 表示使用单机 GreptimeDB。
-在第一级子目录下（例如 `cases/standalone`），你可以随意组织你的测试案例。Sqlness 会递归地遍历每个文件并运行它们。
+在环境目录内，新 case 应与它覆盖的功能放在一起。Sqlness 会递归发现 case 文件。
## 运行测试
-与其他测试不同，这个测试工具是以二进制目标形式存在的。你可以用以下命令运行它
+运行命令如下：
```shell
-cargo run --bin sqlness-runner bare
+cargo sqlness bare
```
-它会自动完成以下步骤：编译 `GreptimeDB`，启动它，抓取测试并将其发送到服务器，然后收集和比较结果。你只需要检查是否有 `.result` 文件发生变化。如果没有，恭喜你，测试通过了 🥳！
+该命令会构建并启动 GreptimeDB、执行选中的 case，再比较输出。`.result` 发生变化只是待审查的结果，不代表新输出一定正确。
### 运行特定测试
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/unit-test.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/unit-test.md
index 79c73775b4..b63b29df94 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/unit-test.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/unit-test.md
@@ -7,22 +7,26 @@ description: 介绍 GreptimeDB 的单元测试，包括如何编写、运行和
## 介绍
-单元测试嵌入在代码库中，通常放置在被测试逻辑的旁边。它们使用 Rust 的 `#[test]` 属性编写，并可以使用 `cargo nextest run` 运行。
+单元测试通常放在被测逻辑旁边，使用 Rust 的 `#[test]` 属性编写。GreptimeDB 主要使用 [`cargo-nextest`](https://nexte.st/) 运行 Rust 测试。
-GreptimeDB 代码库不支持默认的 `cargo` 测试运行器。推荐使用 [`nextest`](https://nexte.st/)。你可以通过以下命令安装它：
+安装命令如下：
```shell
cargo install cargo-nextest --locked
```
-然后运行测试（这里 `--workspace` 不是必须的）
+开发时先运行本次修改的 package：
```shell
-cargo nextest run
+cargo nextest run -p
```
-注意，如果你的 Rust 是通过 `rustup` 安装的，请确保使用 `cargo` 安装 `nextest`，而不是像 `homebrew` 这样的包管理器，否则会弄乱你的本地环境。
+可以继续使用测试名称或 nextest filter 缩小范围。影响范围较广的改动在提交前运行完整 workspace 测试：
+
+```shell
+make test
+```
## 覆盖率
-我们的持续集成（CI）作业有一个“覆盖率检查”步骤。它会报告有多少代码被单元测试覆盖。请在你的补丁中添加必要的单元测试。
+CI 会报告单元测试覆盖率。测试应覆盖本次改变的行为和可能回归的失败路径，而不是只追求覆盖率数字。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/data-persistence-indexing.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/data-persistence-indexing.md
index e0ef9e8e54..e0ce16721d 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/data-persistence-indexing.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/data-persistence-indexing.md
@@ -5,19 +5,23 @@ description: 介绍了 GreptimeDB 的数据持久化和索引机制，包括 SST
# 数据持久化与索引
-与所有类似 LSMT 的存储引擎一样，MemTables 中的数据被持久化到耐久性存储，例如本地磁盘文件系统或对象存储服务。GreptimeDB 采用 [Apache Parquet][1] 作为其持久文件格式。
+与其他 LSM-tree 存储引擎类似，GreptimeDB 将 memtable 中的数据持久化到本地文件系统或对象存储，并使用 [Apache Parquet][1] 作为持久化文件格式。
## SST 文件格式
Parquet 是一种提供快速数据查询的开源列式存储格式，已经被许多项目采用，例如 Delta Lake。
-Parquet 具有层次结构，类似于“行组 - 列-数据页”。Parquet 文件中的数据被水平分区为行组（row group），在其中相同列的所有值一起存储以形成数据页（data pages）。数据页是最小的存储单元。这种结构极大地提高了性能。
+Parquet 按 row group、column chunk 和 page 组织数据。每个 row group 为每一列保存一个 column chunk，每个 column chunk 再包含一个或多个 page。Page 是编码和压缩单元，读取指定列时则以 column chunk 为 I/O 单元。
首先，数据按列聚集，这使得文件扫描更加高效，特别是当查询只涉及少数列时，这在分析系统中非常常见。
-其次，相同列的数据往往是同质的（比如具备近似的值），这有助于在采用字典和 Run-Length Encoding（RLE）等技术进行压缩。
+其次，同一列中的值通常比较相似，有利于字典编码和 Run-Length Encoding（RLE）等压缩技术发挥作用。
-
+下面这张来自 Apache Parquet 规范的图进一步展示了物理文件布局：column chunk 按 row group 写入，文件元数据及其长度则保存在 footer 中。
+
+
+
+*来源：Apache Parquet [FileLayout.gif](https://github.com/apache/parquet-format/blob/master/doc/images/FileLayout.gif)。Copyright 2014 The Apache Software Foundation，依据 [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) 使用。*
## 数据持久化
@@ -26,18 +30,18 @@ GreptimeDB 提供了 `region_engine.mito.global_write_buffer_size` 的配置项
## SST 文件中的索引数据
-Apache Parquet 文件格式在列块和数据页的头部提供了内置的统计信息，用于剪枝和跳过。
+Parquet 在每个 column chunk 的元数据中保存 row group 级列统计信息，例如最小值、最大值和 null 数量。Page 元数据和可选的 column index 可以提供粒度更细的统计信息。
-
+
-例如，在上述 Parquet 文件中，如果你想要过滤 `name` 等于 `Emily` 的行，你可以轻松跳过行组 0，因为 `name` 字段的最大值是 `Charlie`。这些统计信息减少了 IO 操作。
+例如，查询 `name` 等于 `Emily` 的行时，可以跳过 row group 0，因为其中 `name` 的最大值是 `Charlie`，无需读取该 row group。
## 索引文件
-对于每个 SST 文件，GreptimeDB 不但维护 SST 文件内部索引，还会单独生成一个文件用于存储针对该 SST 文件的索引结构。
+当一个 SST 存在已配置且适用的索引输出时，GreptimeDB 将这些索引写入与该 SST 关联的 Puffin 文件。没有适用索引的 SST 不需要生成 Puffin 文件。
-索引文件采用 [Puffin][3] 格式，这种格式具有较大的灵活性，能够存储更多的元数据，并支持更多的索引结构。
+Puffin 是索引 Blob 及其元数据的容器，使不同索引结构可以共用一个文件。
@@ -58,13 +62,13 @@ GreptimeDB 会将多种索引结构作为 Blob 存储在 Puffin 文件中，包
-例如，上述查询使用倒排索引来定位数据段，数据段满足条件：`job` 等于 `apiserver`，`handler` 符合正则匹配 `.*users` 及 `status` 符合正则匹配 `4..`，然后扫描这些数据段以产生满足所有条件的最终结果，从而显着减少 IO 操作的次数。
+上述查询使用倒排索引定位 `job` 等于 `apiserver`、`handler` 匹配 `.*users` 且 `status` 匹配 `4..` 的数据段。Mito 只扫描这些数据段，再应用剩余过滤条件。
### 倒排索引格式
-
+
-GreptimeDB 按列构建倒排索引，每个倒排索引包含一个 FST 和多个 Bitmap。
+GreptimeDB 按列构建倒排索引。每个列索引包含一个 null bitmap、多个 posting bitmap 和一个 FST。Blob footer 记录定位和解码各列索引所需的 offset、size 和元数据。
FST（Finite State Transducer）允许 GreptimeDB 以紧凑的格式存储列值到 Bitmap 位置的映射，并且提供了优秀的搜索性能和支持复杂搜索（例如正则表达式匹配）；Bitmap 则维护了数据段 ID 列表，每个位表示一个数据段。
@@ -80,7 +84,7 @@ GreptimeDB 把一个 SST 文件分割成多个索引数据段，每个数据段
## 统一数据访问层：OpenDAL
-GreptimeDB 使用 [OpenDAL][2] 提供统一的数据访问层，因此，存储引擎无需与不同的存储 API 交互，数据可以无缝迁移到基于云的存储，如 AWS S3。
+GreptimeDB 使用 [OpenDAL][2] 为本地文件系统和对象存储提供统一访问层。修改配置的存储 backend 不会迁移已有数据。
[1]: https://parquet.apache.org
[2]: https://github.com/datafuselabs/opendal
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/memtable.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/memtable.md
new file mode 100644
index 0000000000..3905a1aecd
--- /dev/null
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/memtable.md
@@ -0,0 +1,100 @@
+---
+keywords: [memtable, Mito engine, 写缓冲, flush, 时间分区, BulkMemtable]
+description: 介绍 Mito 如何在 memtable 中组织 Region 的可变数据，以及如何将这些数据写入 SST 文件。
+---
+
+# Memtable 设计
+
+Memtable 是 Mito 为每个 Region 维护的内存写缓冲。数据 flush 为 SST 文件前，读取可以先从 memtable 获取这些数据。Region version 确定 scan 可以读取的 memtable 和 SST 文件；配合 committed sequence 上限，scan 可以在写入和 flush 推进当前 version 时保持一致性。
+
+## 写入和 flush 生命周期
+
+对于使用 WAL 的常规写入，Mito 按以下顺序处理：
+
+```text
+写请求
+ |
+ v
+追加 WAL -> mutable memtable -> 发布 committed sequence
+ |
+ freeze
+ v
+ immutable memtable -> 写入 SST -> manifest edit
+```
+
+Region worker 先分配 sequence number 和 WAL entry ID，再将 mutation 追加到[预写日志](wal.md)。如果追加失败，Mito 不会更新 memtable。Memtable 更新成功后，Mito 发布 committed sequence，新的读取随后可以看到这些数据。配置了 `skip_wal` 的 Region 会跳过 WAL，但 memtable 更新和可见性顺序不变。
+
+Flush 在启动后台 SST 写入前，先冻结 mutable memtable 并安装一组新的 mutable memtable。后续写入可以继续进行，也不会修改已经冻结的数据。Flush 将 immutable memtable 写为 SST 文件，再持久化包含新文件、flushed WAL checkpoint 和 sequence checkpoint 的 manifest edit。只有 manifest edit 持久化成功后，Mito 才会从当前 Region version 中移除已 flush 的 memtable。Flush 失败时，immutable memtable 会保留，供后续任务重试。
+
+## Region version 和时间分区
+
+每个 Region 包含一个 mutable `TimePartitions` 容器，其中可以有多个 memtable：
+
+```text
+Region version
+├─ mutable TimePartitions
+│ ├─ [t0, t1) -> memtable
+│ └─ [t1, t2) -> memtable
+├─ immutable memtables
+└─ SST files
+```
+
+Mito 根据 time index 的值把每行数据路由到对应分区。分区使用左闭右开的时间范围，并按照固定时长对齐。该时长跟随 Region 的 compaction time window；在取得 compaction time window 前，Mito 使用一天作为初始值。乱序写入可能在最新分区之外创建更早的分区。
+
+冻结 Region 时，Mito 会同时冻结所有 mutable 时间分区，把其中的 memtable 移入 immutable 列表，再创建新的 `TimePartitions` 容器。Flush 失败可能留下多代 immutable memtable，因此读取和后续 flush 不能假定列表中只有一个对象。
+
+## Memtable 实现
+
+Mito 根据 Region 的 SST format、primary key encoding 和 memtable 选项选择实现：
+
+```text
+flat SST format（默认）或 sparse primary-key encoding -> BulkMemtable
+memtable.type=bulk -> BulkMemtable，并强制 flat SST
+primary_key SST + dense encoding（遗留） -> 遗留实现
+```
+
+使用默认 engine 配置时，没有显式指定 SST format 的 Region 会使用 `flat`，因此通常走 `BulkMemtable` 路径，本页其余内容也以它为准。这些规则用于排除不兼容的组合：flat format 或 sparse primary key encoding 必须使用 `BulkMemtable`；显式选择 bulk 实现则会强制使用 flat format。
+
+### BulkMemtable
+
+`BulkMemtable` 使用 flat Arrow 布局把写入保存为 part，而不是将数据行插入按时间序列组织的缓冲区：
+
+```text
+BulkMemtable
+├─ unordered_part
+│ └─ 小批量 BulkPart
+└─ parts
+ ├─ BulkPart (Arrow RecordBatch)
+ ├─ MultiBulkPart (未编码的 RecordBatch)
+ └─ EncodedBulkPart (内存中的 Parquet 数据)
+```
+
+小 part 先积累在 `unordered_part` 中，较大的 part 则直接进入 `parts`。后台 memtable compaction 对符合条件的 part 执行 merge sort，生成 `MultiBulkPart` 或编码为 `EncodedBulkPart`。Scan 利用 part 的统计信息裁剪 range；flush 可以将已编码的 range 写入 SST，无需再次解码和编码数据行。设计动机和性能数据见 [Scaling Time Series to Millions of Cardinalities: GreptimeDB's Flat Format](https://greptime.cn/blogs/2025-12-22-flat-format)。
+
+### 遗留实现
+
+使用遗留 `primary_key` SST format 且为 dense primary key encoding 的 Region 仍然使用 `TimeSeriesMemtable`，它按编码后的 primary key 对数据行分组，而不是保存 flat part。如果 Region 没有 primary key 列，同一个 builder 会创建 `SimpleBulkMemtable`。两者都是为已有表保留的兼容代码，`primary_key` format 退役后可能一并移除；新的工作应面向 bulk 和 flat 路径。
+
+已经删除的 `partition_tree` memtable 不是第三种实现。Option parser 仍接受 `memtable.type=partition_tree` 以兼容旧配置，但不会恢复该实现。Region 最终使用 bulk 和 flat 路径。
+
+## 读取快照
+
+Scan 通过一次 `VersionControl` 快照同时取得 Region version 和 committed sequence，并先确定 version，再应用 sequence 上限。如果单独读取 sequence 后再获取 version，flush 或 compaction 可能在两次读取之间移除旧输入，使 scan 得到不完整的快照。
+
+选定的 version 提供 mutable memtable、immutable memtable 和 SST 文件。Mito 先按照时间范围裁剪数据源，再使用 scan 的 projection、predicate 和 sequence range 从各 memtable 获取 range。Scan 将这些 range 与 SST range 合并，并在所有数据源上应用相同的排序、删除和 merge 语义。即使新的 Region version 已经移除某个 memtable，scan 持有的引用也会让该 memtable 存活到本次读取结束。
+
+## 内存压力
+
+每个 memtable 通过 engine 的 write-buffer manager 记录估算的堆内存分配量。冻结 memtable 后，这部分内存不再计入 mutable memory，但在所有引用释放前仍计入总用量。因此，mutable memory 只反映仍可接收写入的数据，总用量仍包含活跃 scan 保留的内存。
+
+全局 write-buffer 达到限制后，worker 会选择 Region 执行 flush。如果内存用量持续超过配置限制，Mito 会阻塞写入，并在达到更高阈值后拒绝写入。可选的 Region 级限制会单独约束热点 Region，避免其阻塞无关 Region。定期任务、手动请求和 Region 生命周期操作也可以触发 flush。
+
+## 修改约束
+
+修改 memtable 代码时必须保持以下性质：
+
+- 对于使用 WAL 的 Region，先追加 WAL，再把数据安装到 memtable；只有安装成功后才能发布 committed sequence。
+- SST 文件和 manifest edit 持久化前，冻结的 memtable 必须保持可读，并能在 flush 失败后重试。
+- 从同一个 `VersionControl` 快照取得 Region version 和 committed sequence，不得先单独读取 sequence 再获取 version。
+- 保留 scan 和 flush 在 memtable range 与 SST range 之间执行统一删除、去重和 merge 所需的排序及元数据。
+- 通过 write-buffer manager 记录内存分配，并且只在底层内存不再可能被引用时释放计数。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/metric-engine.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/metric-engine.md
index c0cbf681b9..57baecf400 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/metric-engine.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/metric-engine.md
@@ -7,9 +7,9 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述
## 概述
-`Metric` 引擎是 GreptimeDB 的一个组件，属于存储引擎的一种实现，主要针对可观测 metrics 等存在大量小表的场景。
+`Metric` 引擎用于存储包含大量小型指标表的负载。
-它的主要特点是利用合成的物理宽表来存储大量的小表数据，实现相同列复用和元数据复用等效果，从而达到减少小表的存储开销以及提高列式压缩效率等目标。表这一概念在 `Metric` 引擎下变得更更加轻量。
+它将这些逻辑表映射到共享的物理宽表，使其复用列和元数据，从而降低每张表的存储开销并改善列式压缩。
## 概念
@@ -18,7 +18,7 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述
### 逻辑表
逻辑表，即用户定义的表。与普通的表都完全一样，逻辑表的定义包括表的名称、列的定义、索引的定义等。用户的查询、写入等操作都是基于逻辑表进行的。用户在使用过程中不需要关心逻辑表和普通表的区别。
-从实现层面来说，逻辑表是一个虚拟的表，它并不直接读写物理的数据，而是通过将读写请求映射成对应物理表的请求来实现数据的存储与查询。
+逻辑表是虚拟表，本身不直接存储数据。Metric 引擎将其读写请求映射为对应物理表的请求。
### 物理表
物理表是真实存储数据的表，它拥有若干个由分区规则定义的物理 Region。
@@ -27,16 +27,14 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述
`Metric` 引擎的主要设计架构如下：
-
+
-在目前版本的实现中，`Metric` 引擎复用了 `Mito` 引擎来实现物理数据的存储及查询能力，并在此之上同时提供物理表与逻辑表的访问能力。
+`Metric` 引擎将物理存储和查询交给 `Mito` 引擎。每个物理 Region 组包含一个数据 Region 和一个元数据 Region：数据 Region 保存映射到该 Region 组的逻辑表数据，元数据 Region 保存逻辑表及逻辑列的映射。
-在分区方面，逻辑表拥有与物理表完全一致的分区规则及 Region 分布。这是非常自然的，因为逻辑表的数据直接存储在物理表中，所以分区规则也是一致的。
+关联到同一物理表的逻辑表使用相同的分区布局。写入时，Metric 引擎为每行数据记录逻辑表身份；读取时，它在扫描物理 Region 前增加逻辑表过滤条件。
-在路由元数据方面，逻辑表的路由地址为逻辑地址，即该逻辑表所对应的物理表是什么，而后通过该物理表进行二次路由取得真正的物理地址。这一间接路由方式能够显著减少 `Metric` 引擎的 Region 发生迁移调度时所需要修改的元数据数量。
+逻辑表的路由只保存所属物理表的 ID，再由物理表路由解析出持有 Region 的 Datanode。逻辑路由本身不记录 peer，因此迁移物理 Region 只需改写一条物理路由，而不必改写映射到它的每一条逻辑路由。
-在操作方面，`Metric` 引擎支持对逻辑表进行标准的 DML 操作（INSERT、DELETE、SELECT）。然而，对物理表的操作进行了有限的支持以防止误操作，例如禁止直接写入物理表等操作防止影响用户逻辑表的数据。总体上可以认为物理表是对用户只读的。
+逻辑表支持普通的 INSERT、DELETE 和 SELECT 操作。直接写入物理 Region 会绕过逻辑表映射，因此会被拒绝；物理表仍然可以查询。
-为了提升对大量表同时进行 DDL（Data Definition Language，数据操作语言）操作时性能，如 Prometheus Remote Write 冷启动时大量 metrics 带来的自动建表请求，以及前面提到的迁移物理 Region 时大量路由表的修改请求等，`Metric` 引擎引入了一些批量 DDL 操作。这些批量 DDL 操作能够将大量的 DDL 操作合并成一个请求，从而减少了元数据的查询及修改次数，提升了性能。
-
-除了物理表的物理数据 Region 之外，`Metric` 引擎还额外为每一个物理数据 Region 创建了一个物理的元数据 Region，用于存储 `Metric` 引擎自身为了维护映射等状态所需要的一些元数据。这些元数据包括逻辑表与物理表的映射关系，逻辑列与物理列的映射关系等等。
+批量 DDL 用于减少大量逻辑表同时创建或更新时的元数据操作，例如 Prometheus Remote Write 自动建表或物理 Region 迁移。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/overview.md
index 3f44a4543b..c7684eb165 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/overview.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/overview.md
@@ -7,22 +7,26 @@ description: 介绍了 Datanode 的主要职责和组件，包括 gRPC 服务、
## Introduction
-`Datanode` 主要的职责是为 GreptimeDB 存储数据，我们知道在 GreptimeDB 中一个 `table` 可以有一个或者多个 `Region`,
-而 `Datanode` 的职责便是管理这些 `Region` 的读写。`Datanode` 不感知 `table`，可以认为它是一个 `region server`。
-所以 `Frontend` 和 `Metasrv` 按照 `Region` 粒度来操作 `Datanode`。
+Datanode 存储并处理 Region 数据。一张表可以包含多个 Region，但 Datanode 不负责表级路由。Frontend 按 Region 发送数据请求，Metasrv 则控制 Region 的放置和生命周期。
-
+这个边界使同一个 Region server 可以承载不同的存储引擎，而不向 Frontend 或 Metasrv 暴露引擎实现。
+
+
## Components
-一个 datanode 包含了 region server 所需的全部组件。这里列出了比较重要的部分：
-
-- 一个 gRPC 服务来提供对 `Region` 数据的读写，`Frontend` 便是使用这个服务来从 `Datanode` 读写数据。
-- 一个 HTTP 服务，可以通过它来获得当前节点的 metrics、配置信息等
-- `Heartbeat Task` 用来向 `Metasrv` 发送心跳，心跳在 GreptimeDB 的分布式架构中发挥着至关重要的作用，
- 是分布式协调和调度的基础通信通道，心跳的上行消息中包含了重要信息比如 `Region` 的负载，如果 `Metasrv` 做出了调度
- 决定（比如 Region 转移），它会通过心跳的下行消息发送指令到 `Datanode`
-- `Datanode` 不负责解析用户 SQL 或进行分布式规划，用户对一个或多个 `Table` 的查询请求会在 `Frontend` 中被转换为
- `Region` 查询请求，`Datanode` 负责用本地 query engine 执行这些 `Region` 查询计划
-- 一个 `Region Manager` 用来管理 `Datanode` 上的所有 `Region`s
-- GreptimeDB 支持可插拔的多引擎架构，目前已有的 engine 包括 `File Engine` 和 `Mito Engine`
+Datanode 包含以下主要组件：
+
+- Region server 记录已打开的 Region，并把读写和生命周期请求分发给该 Region 注册的 engine。
+- `Mito` 是主要的时序 Region engine。`Metric` 将多个逻辑指标 Region 映射到共享的 Mito Region，`File` 通过 Region 接口访问外部文件。
+- 本地 query engine 执行 Region 查询计划。它不解析客户端 SQL，也不进行集群级规划。
+- Heartbeat task 向 Metasrv 上报节点和 Region 状态，并接收 open、close、upgrade、downgrade 和迁移步骤等指令。
+- gRPC 承载发往 Datanode 的 Region 请求；HTTP 提供 metrics 和配置等节点诊断信息。
+
+## Region 请求生命周期
+
+Mito 写入到达 Region server 后，Region server 根据 Region 元数据选择 Mito。Mito 将 mutation 追加到 WAL，写入 memtable，并在之后把 memtable flush 为 SST 文件。Metric 写入会先补充逻辑表标识，再委托给对应的物理 Mito Region。
+
+读取时，本地 query engine 在 Region engine 提供的 table provider 上执行 Region 计划。Mito scan 获取不可变的 Region version，读取相关 memtable 和 SST 文件，合并并去重数据，最后返回 Arrow record batch 流。
+
+Region 所有权可以在不重启 Datanode 的情况下改变。Metasrv 通过心跳流下发生命周期指令；Region server 将指令应用到对应 engine，并在后续心跳中上报新的 Region role 和统计信息。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/python-scripts.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/python-scripts.md
deleted file mode 100644
index 831278e80d..0000000000
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/python-scripts.md
+++ /dev/null
@@ -1,30 +0,0 @@
----
-keywords: [Python 脚本, 数据分析, CPython, RustPython]
-description: 介绍了在 GreptimeDB 中使用 Python 脚本进行数据分析的两种后端实现：CPython 和嵌入式 RustPython 解释器。
----
-
-# Python 脚本
-
-## 简介
-
-Python 脚本是分析本地数据库中的数据的便捷方式，
-通过将脚本直接在数据库内运行而不是从数据库拉取数据的方式，可以节省大量的数据传输时间。
-下图描述了 Python 脚本的工作原理。
-`RecordBatch`（基本上是表中的一列，带有类型和元数据）可以来自数据库中的任何地方，
-而返回的 `RecordBatch` 可以用 Python 语法注释以指示其元数据，例如类型或空。
-脚本将尽其所能将返回的对象转换为 `RecordBatch`，无论它是 Python 列表、从参数计算出的 `RecordBatch` 还是常量（它被扩展到与输入参数相同的长度）。
-
-
-
-## 两种可选的后端
-
-### CPython 后端
-
-该后端由 [PyO3](https://pyo3.rs/v0.18.1/) 提供支持，可以使用您最喜欢的 Python 库（如 NumPy、Pandas 等），并允许 Conda 管理您的 Python 环境。
-
-但是使用它也涉及一些复杂性。您必须设置正确的 Python 共享库，这可能有点棘手。一般来说，您只需要安装 `python-dev` 包。但是，如果您使用 Homebrew 在 macOS 上安装 Python，则必须创建一个适当的软链接到 `Library/Frameworks/Python.framework`。有关使用 PyO3 crate 与不同 Python 版本的详细说明，请参见 [这里](https://pyo3.rs/v0.18.1/building_and_distribution#configuring-the-python-version)
-
-### 嵌入式 RustPython 解释器
-
-可以运行脚本的实验性 [python 解释器](https://github.com/RustPython/RustPython)，它支持 Python 3.10 语法。您可以使用所有的 Python 语法，更多信息请参见 [Python 脚本的用户指南](/user-guide/python-scripts/overview.md).
-
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/query-engine.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/query-engine.md
index 62d7563bb2..84445759d4 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/query-engine.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/query-engine.md
@@ -7,33 +7,33 @@ description: 介绍了 GreptimeDB 的查询引擎架构，基于 Apache DataFusi
## 介绍
-GreptimeDB 的查询引擎是基于[Apache DataFusion][1]（属于[Apache Arrow][2]的子项目）构建的，它是一个用 Rust 编写的出色的查询引擎。它提供了一整套功能齐全的组件，从逻辑计划、物理计划到执行运行时。下面将解释每个组件如何被整合在一起，以及在执行过程中它们的位置。
+GreptimeDB 的查询引擎基于 [Apache DataFusion][1]。DataFusion 提供逻辑计划、物理计划、优化器框架和执行运行时；GreptimeDB 在此基础上增加各查询语言的 planner、存储相关优化规则、自定义计划节点和分布式执行。
-
+DDL 和其他控制面操作由 statement executor 分发。Query engine 接收数据处理计划，包括 `INSERT ... SELECT` 等操作中读取输入数据的部分。
-入口点是逻辑计划，它被用作查询或执行逻辑等的通用中间表示。逻辑计划的两个主要来源是：1. 用户查询，例如通过 SQL 解析器和规划器的 SQL；2. Frontend 的分布式查询，这将在下一节中详细解释。
+## 查询生命周期
-接下来是物理计划，或称执行计划。与包含所有逻辑计划变体（除特殊扩展计划节点外）的大型枚举的逻辑计划不同，物理计划实际上是一个定义了在执行过程中调用的一组方法的特性。所有数据处理逻辑都包装在实现该特性的相应结构中。它们是对数据执行的实际操作，如聚合器 `MIN` 或 `AVG` ，以及表扫描 `SELECT ... FROM`。
+1. SQL、PromQL 或日志查询 planner 通过 catalog 解析表，并生成 DataFusion logical plan。DataFusion 不直接支持的操作由 GreptimeDB plan extension 表示。
+2. DataFusion 的 analyzer 和 optimizer rule 与 GreptimeDB rule 共同运行。这些规则规范化表达式和类型、改写时间范围操作、将 projection 和 filter 下推到 scan，并在需要时引入分布式计划节点。
+3. Physical planner 将优化后的 logical plan 转换为流式 operator。GreptimeDB 随后应用 scan 并行度、排序和分布式执行相关的 physical rule。
+4. 执行阶段通过 physical plan 拉取 Arrow record batch。存储 scan 接收 projection 和 predicate，下游 operator 消费数据流，无需先物化完整结果。
-优化阶段通过转换逻辑计划和物理计划来提高执行性能，现在全部基于规则。它也被称为“基于规则的优化”。一些规则是 DataFusion 原生的，其他一些是在 GreptimeDB 中自定义的。在未来，我们计划添加更多规则，并利用数据统计进行基于成本的优化 (CBO)。
-
-最后一个阶段"执行"是一个动词，代表从存储读取数据、进行计算并生成预期结果的过程。虽然它比之前提到的概念更抽象，但你可以简单地将它想象为执行一个 Rust 异步函数，并且它确实是一个异步流。
-
-当你想知道 SQL 是如何通过逻辑计划或物理计划中表示时，`EXPLAIN [VERBOSE] ` 是非常有用的。
+使用 [`EXPLAIN`](/reference/sql/explain.md) 查看逻辑和物理计划。`EXPLAIN ANALYZE` 还会执行计划并报告运行时指标。
## 数据表示
-GreptimeDB 使用 [Apache Arrow][2]作为内存中的数据表示格式。它是面向列的，以跨平台格式，也包含许多高性能的基础操作。这些特性使得在许多不同的环境中共享数据和实现计算逻辑变得容易。
+GreptimeDB 使用 [Apache Arrow][2] record batch 作为内存数据表示。一个 record batch 包含等长的列数组和 schema。查询 operator 交换这些 batch 组成的数据流，使 Region scan 到结果编码的执行路径保持列式处理。
## 索引
-在时序数据中，有两个重要的维度：时间戳和标签列（或者类似于关系数据库中的主键）。GreptimeDB 将数据分组到时间桶中，因此能在非常低的成本下定位和提取预期时间范围内的数据。GreptimeDB 中主要使用的持久文件格式 [Apache Parquet][3] 提供了多级索引和过滤器，使得在查询过程中很容易修剪数据。在未来，我们将更多地利用这个特性，并开发我们的分离索引来处理更复杂的用例。
+索引构建和持久化格式属于存储引擎。查询层向 scan 提供 predicate 和 projection，Mito 再利用时间范围、Parquet 统计信息和索引跳过不可能匹配的数据。参见[数据持久化与索引](./data-persistence-indexing.md)。
+
+
-## 分布式查询
+## 分布式执行
-参考 [Distributed Querying][6].
+分布式模式下，Frontend 规划集群级查询，Datanode 执行 Region 本地子计划。[`MergeScan`][6] 是两个阶段之间的边界。
-[1]: https://github.com/apache/arrow-datafusion
+[1]: https://datafusion.apache.org/
[2]: https://arrow.apache.org/
-[3]: https://parquet.apache.org
[6]: ../frontend/distributed-querying.md
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/storage-engine.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/storage-engine.md
index 67f98216fe..d3abb58cfa 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/storage-engine.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/storage-engine.md
@@ -7,7 +7,7 @@ description: 详细介绍了 GreptimeDB 的存储引擎架构、数据模型和
## 概述
-`存储引擎` 负责存储数据库的数据。Mito 是我们默认使用的存储引擎，基于 [LSMT][1]（Log-structured Merge-tree）。我们针对处理时间序列数据的场景做了很多优化，因此 mito 这个存储引擎并不适用于通用用途。
+Mito 是 GreptimeDB 的默认存储引擎，基于 [LSM tree][1]，面向时间序列负载设计，而不是通用的嵌入式存储引擎。
## 架构
下图展示了存储引擎的架构和处理数据的流程。
@@ -20,9 +20,9 @@ description: 详细介绍了 GreptimeDB 的存储引擎架构、数据模型和
- 为尚未刷盘的数据提供高持久性保证。
- 基于 `LogStore` API 实现，不关心底层存储介质。
- WAL 的日志记录可以存储在本地磁盘上，也可以存储在实现了 `LogStore` API 的远程日志服务中，例如 Kafka（remote WAL）。
-- Memtable
- - 数据首先写入 `active memtable`，又称 `mutable memtable`。
- - 当 `mutable memtable` 已满时，它将变为只读的 `immutable memtable`。
+- [Memtable](memtable.md)
+ - Mito 根据 time index 将数据行写入 mutable memtable。
+ - Flush 冻结 mutable memtable，安装一组新的 mutable memtable 以接收写入，再将冻结的 memtable 写为 SST 文件。
- SST
- SST 的全名为有序字符串表（`Sorted String Table`）。
- `immutable memtable` 刷到持久存储后形成一个 SST 文件。
@@ -100,7 +100,9 @@ Mito 会按 primary key 对行分组，并按时间排序，因此 SST 中的数
Mito 支持两种 SST 格式：`flat` 和 `primary_key`。`flat` 是新表的默认格式，适用于各种 primary key 基数，包括高基数 key。`primary_key` 是为了兼容旧表而保留的遗留格式。更多详情请参考 [SST format](/reference/sql/create.md#创建指定-sst-格式的表) 和[表设计指南](/user-guide/deployments-administration/performance-tuning/design-table.md#sst-格式)。
-
+
+
+一个 SST 可能跨越多个 compaction time window。
## 扫描裁剪
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/wal.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/wal.md
index 529adcdf50..36e0689430 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/wal.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/wal.md
@@ -9,20 +9,26 @@ description: 介绍了 GreptimeDB 的预写日志（WAL）机制，包括其命
## 介绍
-我们的存储引擎受到了日志结构合并树（Log-structured Merge Tree，LSMT）的启发。对数据的变更操作直接应用于 MemTable 而不是持久化到磁盘上的数据页，这显著提高了性能，但也带来了持久化相关的问题，特别是在 Datanode 意外崩溃时。与所有类似 LSMT 的存储引擎一样，GreptimeDB 使用预写日志（Write-Ahead Log，WAL）来确保数据被可靠地持久化，并且保证崩溃时的数据完整性。
+Mito 在将数据 flush 为 SST 文件前，先在 [memtable](memtable.md) 中缓冲写入。每个 Region 的 mutation 会先追加到预写日志（WAL），从而恢复尚未进入 SST 的数据。
-预写日志是一个仅提供追加写的文件组。所有的 INSERT 和 DELETE 操作都被转换为操作日志，然后追加到 WAL。一旦操作日志被持久化到底层文件，该操作才可以进一步应用到 MemTable。
+WAL 通过统一的 log-store 抽象访问，可以使用本地 raft-engine 或远端 Kafka。
-当数据节点重新启动时，WAL 中的操作条目将被重放，以重建正确的 MemTable 状态。
+## 写入与恢复流程
-
+正常写入遵循以下顺序：
+
+1. Region worker 分配 sequence number 和 WAL entry ID。
+2. 将 mutation 追加到 WAL。追加失败时，不会把 mutation 写入 memtable。
+3. WAL 追加成功后，Mito 将 mutation 写入 memtable，并发布新的 committed sequence。
+4. Flush 将不可变 memtable 写为 SST 文件，并持久化包含新文件和 `flushed_entry_id` 的 manifest edit。
+5. Manifest edit 持久化后，`flushed_entry_id` 及以前的 WAL entry 被标记为 obsolete；log store 可以稍后再回收物理空间。
+
+Manifest 是恢复边界。正常重新打开 Region 时，Mito 根据 manifest 重建 Region，并从 `flushed_entry_id + 1` 开始重放 WAL。Region 状态切换可以指定更晚的 replay checkpoint，但不会重放早于已持久化 flush 边界的 entry。
## 命名空间
-WAL 的命名空间用于区分来自不同 region 的条目。追加和读取操作必须提供一个命名空间。目前，region ID 被用作命名空间，因为每个 region 都有一个在数据节点重新启动时需要重构的 MemTable。
+WAL entry 按 Region 隔离，而不是按表隔离。追加和读取都需要指定 Region namespace，使单个 Region 可以独立重放或截断。本地 raft-engine 使用 Region ID 作为 namespace ID；Kafka provider 则在基于 topic 的日志中保留 Region 标识。
## 同步/异步刷盘
-默认情况下，WAL 的追加写是异步的，这意味着写入方不会等待操作日志被刷入到磁盘并持久化。这个默认设置提供了更高的性能，但在服务器意外关闭时可能会丢失数据。另一方面，同步刷新提供了更高的可靠性，但其代价是性能更低。
-
-在 v0.4 版本中，新的 region worker 架构可以使用批处理来减轻同步刷盘的开销。
+对于本地 raft-engine，`sync_write` 控制追加写是否等待日志同步到持久化存储，默认值为 `false`。异步写入延迟较低，但主机在缓冲数据同步前故障时，可能丢失最近确认的 entry。Kafka WAL 的持久性由 producer 和集群配置决定，不受这个本地选项控制。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/arrangement.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/arrangement.md
index dd3b6de090..7b35b50259 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/arrangement.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/arrangement.md
@@ -5,6 +5,8 @@ description: 描述了 Arrangement 在数据流进程中的状态存储功能，
# Arrangement
+本页介绍 Flownode 旧 streaming 模式使用的状态结构；batching 模式不使用 Arrangement。
+
Arrangement 存储数据流进程中的状态，存储 flow 的更新流（stream）以供进一步查询和更新。
Arrangement 本质上存储的是带有时间戳的键值对。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/batching_mode.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/batching_mode.md
index bec092b0af..8e2ecdb80b 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/batching_mode.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/batching_mode.md
@@ -9,13 +9,13 @@ description: Flownode 批处理模式概述，这是持续数据聚合当前使
## 概述
-`flownode` 中的批处理模式专为持续数据聚合而设计。它在离散的、微小的时间窗口上周期性地执行用户定义的 SQL 查询。这与原始的流处理模式形成对比；流处理模式现在已经废弃，在该模式下数据会在到达时即被处理。
+`flownode` 中的批处理模式专为持续数据聚合而设计。它在离散的小时间窗口上周期性执行用户定义的 SQL 查询。旧 streaming 路径则在数据到达时进行处理，目前仅为兼容已有 workload 而保留，不推荐新 workload 使用。
其核心思想是：
1. 定义一个带有 SQL 查询的 `flow`，该查询将数据从源表聚合到目标表。
2. 查询通常在时间戳列上包含一个时间窗口函数（例如 `date_bin`）。
3. 当新数据插入源表时，系统会将相应的时间窗口标记为“脏”(dirty)。
-4. 一个后台任务会周期性地唤醒，识别这些脏窗口，并为那些特定的时间范围重新运行聚合查询。
+4. 一个后台任务按自身的节奏运行，在下一次求值时取出待处理的脏窗口，并对这些时间范围重新运行聚合查询。
5. 然后将结果插入到目标表中，从而有效地更新聚合视图。
## 架构
@@ -39,15 +39,15 @@ description: Flownode 批处理模式概述，这是持续数据聚合当前使
- **状态 (`TaskState`)**: 包含任务的动态、可变状态，最重要的是 `DirtyTimeWindows`。
- **执行循环**: 任务运行一个无限循环 (`start_executing_loop`)，该循环：
1. 检查关闭信号。
- 2. 等待一个预定的时间间隔或直到被唤醒。
+ 2. 睡眠到下一次求值时间。设置了求值调度的任务睡眠到下一个调度时间点；自适应任务则按时间窗口大小和最小刷新间隔计算出的轮询间隔睡眠。
3. 基于当前的脏时间窗口集合生成一个新的查询计划 (`gen_insert_plan`)。
4. 对数据库执行查询 (`execute_logical_plan`)。
5. 清理已处理的脏窗口。
### `TaskState` 和 `DirtyTimeWindows`
-- **`TaskState`**: 此结构体跟踪 `BatchingTask` 的运行时状态。它包括 `dirty_time_windows`，这对于确定需要完成哪些操作至关重要。
-- **`DirtyTimeWindows`**: 这是一个关键的数据结构，用于跟踪自上次查询执行以来哪些时间窗口接收到了新数据。它存储一组不重叠的时间范围。当任务的执行循环运行时，它会参考此结构来构建一个 `WHERE` 子句，该子句仅过滤源表中的脏时间窗口。
+- **`TaskState`**: 此结构体跟踪 `BatchingTask` 的运行时状态，包括用于确定待处理工作的 `dirty_time_windows`。
+- **`DirtyTimeWindows`**: 此数据结构跟踪上次查询执行后接收到新数据的时间窗口，并保存一组不重叠的时间范围。执行循环根据它构造 `WHERE` 子句，只从源表选择脏窗口。
### `TimeWindowExpr`
@@ -56,15 +56,15 @@ description: Flownode 批处理模式概述，这是持续数据聚合当前使
- **求值**: 它可以接受一个时间戳并对时间窗口表达式求值，以确定该时间戳所属窗口的开始和结束。
- **窗口大小**: 它还可以从表达式中确定时间窗口的大小（持续时间）。
-这对于标记窗口为脏以及在查询源表时生成正确的过滤条件都至关重要。
+标记脏窗口和生成源表过滤条件使用同一套计算。
## 查询执行流程
以下是批处理模式下查询执行的简化分步演练：
1. **数据摄取**: 新数据被写入源表。
-2. **标记为脏**: `BatchingEngine` 收到有关新数据的通知。它使用与每个相关 flow 关联的 `TimeWindowExpr` 来确定哪些时间窗口受到新数据点的影响。然后将这些窗口添加到相应 `TaskState` 中的 `DirtyTimeWindows` 集合中。
-3. **任务唤醒**: `BatchingTask` 的执行循环被唤醒，原因可能是其周期性调度，也可能是因为它被通知有大量积压的脏窗口。
+2. **标记为脏**: `BatchingEngine` 收到有关新数据的通知。它使用与每个相关 flow 关联的 `TimeWindowExpr` 来确定哪些时间窗口受到新数据点的影响。然后将这些窗口添加到相应 `TaskState` 中的 `DirtyTimeWindows` 集合中。标记脏窗口不会唤醒任务。
+3. **下一次求值**: `BatchingTask` 的执行循环在调度时间点或自适应轮询间隔结束后进入下一次求值，取出待处理的脏窗口。
4. **计划生成**: 任务调用 `gen_insert_plan`。此方法：
- 检查 `DirtyTimeWindows`。
- 生成一系列 `OR` 连接的 `WHERE` 子句（例如 `(ts >= 't1' AND ts < 't2') OR (ts >= 't3' AND ts < 't4') ...`），覆盖所有脏窗口。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/dataflow.md
index 9d07922542..95def5e844 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/dataflow.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/dataflow.md
@@ -1,18 +1,39 @@
---
-keywords: [Dataflow, SQL 查询, 执行计划, 数据流, map, reduce]
-description: 解释了 Dataflow 模块的核心计算功能，包括 SQL 查询转换、内部执行计划、数据流的触发运行和支持的操作。
+keywords: [Flownode, batching mode, streaming mode, Dataflow, 脏时间窗口]
+description: 介绍 Flownode 如何选择并运行 batching 和旧 streaming 两条执行路径。
---
# 数据流
+Flownode 内部有两条执行路径：
+
+- **Batching mode** 是聚合和 TQL workload 的主要执行路径。它查询已经持久化的 source 数据，并将物化结果写入 sink table。
+- **Streaming mode** 是为兼容已有 workload 而保留的旧执行路径，不推荐新 workload 使用。Frontend 会把新到达的行同步给它进行增量处理。
+
+用户不能直接选择执行模式。创建 Flow 时，GreptimeDB 根据查询和 source table 的属性选择执行路径。聚合、`DISTINCT` 和 TQL 查询使用 batching mode；简单的非聚合查询，以及任何 source table 使用 `ttl = 'instant'` 的 Flow，目前仍使用 streaming mode。如果 source table 尚不存在并选择延迟创建，Flow 会先成为 pending batching Flow。
+
+## Batching mode
+
+Batching mode 复用 GreptimeDB 的查询引擎，不需要为每一行输入维护一张算子图。对于基于时间窗口的 Flow，主循环如下：
+
+1. Source table 收到写入后，把受影响的时间窗口标记为 dirty。
+2. `BatchingTask` 按求值调度或自适应轮询节奏运行，并在该次求值时收集待处理的 dirty window。标记 dirty window 不会唤醒任务。
+3. 任务把这些窗口转换成时间谓词，加入 Flow 查询，再请求 Frontend 查询 source table。
+4. 查询结果写入 sink table，更新已重新计算窗口对应的物化结果。
+5. 成功处理的窗口从 dirty set 中移除；执行失败的工作仍可在后续调度中处理。
+
+设置了 evaluation interval、但查询中没有时间窗口表达式的 Flow，会在每次调度时执行完整查询。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见 [Flownode 批处理模式开发者指南](./batching_mode.md)。
+
+## Streaming mode
+
Dataflow 模块（参见 `flow::compute` 模块）是 `flow` 的核心计算模块。
它接收 SQL 查询并将其转换为 `flow` 的内部执行计划。
然后，该执行计划被转化为实际的数据流，而数据流本质上是一个由带有输入和输出端口的函数组成的有向无环图（DAG）。
-数据流会在需要时被触发运行。
+新到达的行变更会增量驱动这张图执行。
-目前该数据流只支持 `map`和 `reduce` 操作，未来将添加对 `join` 等操作的支持。
+Renderer 支持 map/filter/project 和 reduce 操作。执行计划中已经有 join 和 union 节点，但 streaming renderer 尚未实现它们。
在内部，数据流使用 `tuple(row, time, diff)` 以行格式处理数据。
这里 `row` 表示实际传递的数据，可能包含多个 `value` 对象。
`time` 是系统时间，用于跟踪数据流的进度，`diff` 通常表示行的插入或删除（+1 或 -1）。
-因此，`tuple` 表示给定系统时间的 `row` 的插入/删除操作。
+因此，`tuple` 表示给定系统时间的 `row` 的插入/删除操作。有状态算子通过 [Arrangement](./arrangement.md) 保存这些变更的索引 trace。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/distributed-querying.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/distributed-querying.md
index 612174662b..e6d0c35160 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/distributed-querying.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/distributed-querying.md
@@ -1,41 +1,20 @@
---
-keywords: [分布式查询, 查询拆分, 查询合并, TableScan, 物理计划]
-description: 介绍 GreptimeDB 中的分布式查询方法，包括查询的拆分和合并过程，以及 TableScan 节点的作用。
+keywords: [分布式查询, 逻辑计划, MergeScan, Substrait, Region 裁剪]
+description: 介绍 GreptimeDB 如何把逻辑查询计划划分为 Frontend 和 Datanode 上的执行任务。
---
# 分布式查询
-我们知道在 GreptimeDB 中数据是如何分布的（参见“[表分片][1]”），那么如何查询呢？在 GreptimeDB 中，分布式查询非常简单。简单来说，我们只需将查询拆分为子查询，每个子查询负责查询表数据的一个部分，然后将所有结果合并为最终结果。这是一种典型的“拆分 - 合并”方法。具体来说，让我们从查询到达 `frontend` 开始。
+Frontend 和 Datanode 使用同一套基于 DataFusion 的查询引擎。在分布式模式下，Frontend 会增加一个规划步骤，将 Datanode 上执行的工作与 Frontend 上完成的工作分开。
-当查询到达 `frontend` 时，它首先被解析为 SQL 抽象语法树（AST）。我们遍历 AST，并从中生成逻辑计划。顾名思义，逻辑计划只是如何“逻辑地”执行查询的“提示”，它不能被直接运行，因此我们进一步从中生成可执行的物理计划。物理计划是一种类似树形的数据结构，每个节点实际上表示查询的执行方法。一旦我们从上到下运行物理计划树，结果数据将从叶子到根流动，被合并或计算。最终，我们在根节点的输出处得到了查询的结果。
+
-到目前为止，这只是一个典型的“volcano”查询执行模型，你可以在几乎每个 SQL 数据库中看到这种模型。那么“分布式”是在哪里发生的呢？这全部发生在一个名为“TableScan”的物理计划节点中。TableScan 是物理计划树中的一个叶子节点，它负责扫描表的数据（就像它的名称所暗示的）。当 `frontend` 即将扫描表时，它首先需要根据每个 `region` 的数据范围将表扫描拆分为较小的扫描。
+## 分布式规划
-[1]: ./table-sharding.md
+分布式规划器重写逻辑计划，把可以下推的算子移向表扫描，并用 `MergeScan` 节点包装远端子计划。分区列上的谓词还会在任务调度前用于裁剪 Region。
-表的所有 `region` 都有它们存储数据的范围。以下表为例：
+算子能否下推取决于计划形态和算子本身的性质。不支持的部分会保留在 Frontend。初始设计及交换律规则参见[分布式规划器 RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md)。
-```sql
-CREATE TABLE my_table (
- a INT,
- others STRING,
- ts TIMESTAMP TIME INDEX,
-)
-PARTITION ON COLUMNS (a) (
- a < 10,
- a >= 10 AND a < 20,
- a >= 20
-);
-```
+## 分布式计划
-`my_table` 表创建时被设定了 3 个分区。在 GreptimeDB 的当前实现中，将为该表创建 3 个 `region`（分区与 `region` 的比例为 1:1）。这 3 个区域将分别包含以下范围："[-∞, 10)", "[10, 20)" 和 "[20, +∞)"。例如，如果提供了值 "42"，我们将搜索这些范围，并找到包含该值的相应的 `region`（在此示例中为第 3 个 `region`）。
-
-对于查询，我们使用“过滤器”来查找 `region`。 "过滤器"是 "WHERE" 子句中的条件。例如，查询 `SELECT * FROM my_table WHERE a < 10 AND others = 'x'`，其“过滤器”为“a < 10 AND others = 'x'”。然后我们检查这些范围，找出包含满足过滤器条件的值的所有 `region`。
-
-> 如果某个查询没有任何过滤器，则将其视为全表扫描。
-
-找到所需的区域后，我们只需在其中组装子扫描。通过这种方式，我们将查询拆分为子查询，每个子查询都获取表数据的一部分。子查询在 `datanode` 中执行，并在 `frontend` 中等待完成。它们的结果将合并为表扫描请求的最终返回。
-
-下面这张图片总结了分布式查询执行的过程：
-
-
+远端输入是完整的逻辑子计划，并不局限于表扫描。Frontend 使用 [Substrait](https://substrait.io) 序列化子计划，再向持有相应数据的 Datanode 发送 Region 级请求。Datanode 在本地规划并执行子计划，将结果流返回 Frontend。Frontend 合并远端数据流，并执行没有下推的算子。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/overview.md
index ee48da3ce3..6befdc55a9 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/overview.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/overview.md
@@ -5,31 +5,51 @@ description: GreptimeDB Frontend 组件概述 - 为客户端请求提供服务
# Frontend
-**Frontend** 是一个无状态服务，作为 GreptimeDB 中客户端请求的入口点。它为多种数据库协议提供统一接口，并充当代理，将读写请求转发到分布式系统中的相应 Datanode。
+Frontend 是 GreptimeDB 中负责请求编排的无状态服务。Server 层负责终止协议并转换线上消息；Frontend 为这些协议处理器提供数据库行为，包括权限检查、语句执行、路由和分布式查询规划。
+
+Frontend 不存储表数据。它缓存从 Metasrv 获取的 catalog 和路由元数据；元数据发生变化时，Metasrv 通过心跳响应通知 Frontend 失效相应缓存。
## 核心功能
-- **协议支持**：支持多种数据库协议，包括 SQL、PromQL、MySQL 和 PostgreSQL。详见[协议][1]
-- **请求路由**：基于元数据将请求路由到相应的 Datanode
-- **查询分发**：将分布式查询拆分到多个节点
-- **响应聚合**：合并来自多个 Datanode 的结果
-- **认证授权**：安全和访问控制验证
+- 为支持的[协议][1]提供查询和写入行为。
+- 解析 catalog、schema、table 和 Region 路由。
+- 在执行请求前完成权限检查。
+- 规划分布式查询并合并 Datanode 返回的结果。
+- 将表级写入和删除转换为 Region 请求。
## 架构
### 关键组件
-- **协议处理器**：处理不同的数据库协议
-- **目录管理器**：缓存来自 Metasrv 的元数据以实现高效的请求路由和 Schema 校验
-- **分布式规划器**：将逻辑计划转换为分布式执行计划
-- **请求路由器**：为每个请求确定目标 Datanodes
+
+- 协议处理器将 SQL、PromQL、gRPC 写入和可观测性协议转换为 Frontend 的内部请求接口。
+- Catalog manager 和 partition manager 提供表元数据、分区规则和 Region 路由。
+- Statement executor 将查询、DML 和 DDL 分发到各自的执行路径。
+- 分布式规划器把表扫描替换为可跨 Datanode 执行的 `MergeScan` 计划。
### 请求流程
-
+不同操作会走不同的请求路径。
+
+#### 查询
+
+1. 协议处理器创建查询上下文，并完成认证和权限检查。
+2. 对应查询语言的 planner 生成逻辑计划。分布式模式下，planner 根据分区元数据选择 Region 并生成分布式计划。
+3. Frontend 将 Region 子计划发送到对应 Datanode。Datanode 在本地 Region engine 上执行，并返回 Arrow record batch 流。
+4. Frontend 执行剩余算子、合并数据流，再按客户端协议编码结果。
+
+#### 写入和删除
+
+1. Frontend 根据表 schema 校验请求。支持 schema-on-write 的协议可以先创建缺失的表或新增列，再重试写入。
+2. 分区规则把每一行分配给 Region。Frontend 为各目标 Region 构造请求，并路由到当前 Region leader。
+3. Datanode 的 Region server 将请求分发到对应的 Region engine。单机模式下，请求直接发送给内嵌的 Region server。
+
+#### DDL
+
+Statement executor 将 DDL 转换为 task。分布式模式下，Metasrv 以持久化 procedure 执行 task、更新元数据，并协调 Datanode 上的 Region 操作。单机模式复用相同的语句边界，但使用本地元数据和 procedure 实现。
### 部署
-下图是 GreptimeDB 在云上的一个典型的部署。`Frontend` 实例组成了一个集群处理来自客户端的请求：
+下图展示了 GreptimeDB 的一种云上部署。多个 Frontend 实例共同处理客户端请求：
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/table-sharding.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/table-sharding.md
index 63a8ac10a6..cf39afe27b 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/table-sharding.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/table-sharding.md
@@ -5,21 +5,17 @@ description: 介绍 GreptimeDB 中表数据的分片方法，包括分区和 Reg
# 表分片
-对于任何分布式数据库来说，数据的分片都是必不可少的。本文将描述 GreptimeDB 中的表数据如何进行分片。
+GreptimeDB 将一张表分为多个 Region。分区表达式定义每行数据属于哪个 Region，Region 路由则定义当前由哪个 Datanode 持有该 Region。
## 分区
-有关创建分区表的语法，请参阅用户指南中的[表分片](/user-guide/deployments-administration/manage-data/table-sharding.md)部分。
+分区是由一个或多个列上的表达式描述的逻辑行集合。分区布局需要覆盖表的输入域，使每一行都能找到唯一的目标 Region。SQL 语法和支持的表达式参见[表分片](/user-guide/deployments-administration/manage-data/table-sharding.md)。
## Region
-在创建分区后，表中的数据被逻辑上分割。你可能会问："在 GreptimeDB 中，被逻辑上分区的数据是如何存储的？" 答案是保存在 `Region` 当中。
-
-每个 `Region` 对应一个分区，并保存分区的数据。所有的 `Region` 分布在各个 `Datanode` 之中。
-`Metasrv` 管理 `Region` 到 `Datanode` 的路由信息。如果建表后需要调整分区布局，
-GreptimeDB 支持通过显式的 [repartition](/user-guide/deployments-administration/manage-data/repartition.md) 操作拆分或合并分区。
+每个分区对应一个 Region。Region ID 是 Frontend、Datanode 和 Metasrv 用于存储和路由的标识。同一张表的多个 Region 可以放在同一个 Datanode 上。
分区和 Region 的关系参见下图：
@@ -54,3 +50,13 @@ GreptimeDB 支持通过显式的 [repartition](/user-guide/deployments-administr
└──────────────────────────────────┘
可以放在同一个 Datanode 之中
```
+
+## 路由与剪枝
+
+写入时，Frontend 对每行数据计算分区规则，按 Region 分组，再根据路由表把 Region 请求发送到当前 leader。
+
+查询时，分布式 planner 将查询谓词与分区表达式比较，只扫描可能满足谓词的 Region。如果分区元数据缺失或无法安全解释，planner 会退化为扫描所有 Region，避免漏掉数据。
+
+## 调整分区布局
+
+[Repartition](/user-guide/deployments-administration/manage-data/repartition.md) 通过显式的 split 或 merge 调整已有布局。Metasrv 以持久化 procedure 执行变更，更新 Region 路由和分区表达式，并使旧的表路由缓存失效。Frontend 刷新到新元数据后，后续请求使用新的布局。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/getting-started.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/getting-started.md
index e5aa2ca855..999f696ed0 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/getting-started.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/getting-started.md
@@ -15,14 +15,13 @@ description: 介绍如何在本地环境中从源代码编译和运行 GreptimeD
### 构建依赖项
-- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line)（可选）
+- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line)（可选；克隆仓库需要，构建本身不需要）
- C/C++ 工具链：提供编译和链接的基本工具。在 Ubuntu 上，这可用作 `build-essential`。在其他平台上，也有类似的命令。
-- Rust nightly 工具链（[指南][1]）
- - 编译源代码
+- [Rustup][1]。仓库通过 `rust-toolchain.toml` 指定所需的 nightly 工具链。
- Protobuf（[指南][2]）
- 编译 proto 文件
- 请注意，版本需要 >= 3.15。你可以使用 `protoc --version` 检查它。
-- 机器：建议内存在 16GB 以上 或者 使用[mold](https://github.com/rui314/mold)工具以降低链接时的内存使用。
+- 机器：建议 16GB 以上内存。内存较小时，可使用 [mold](https://github.com/rui314/mold) 降低链接阶段的内存占用。
[1]:
[2]:
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/how-to/how-to-write-sdk.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/how-to/how-to-write-sdk.md
index 28ff757763..31e2d45881 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/how-to/how-to-write-sdk.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/how-to/how-to-write-sdk.md
@@ -1,21 +1,17 @@
---
keywords: [gRPC SDK, GreptimeDatabase, GreptimeRequest, GreptimeResponse, 插入请求]
-description: 介绍如何为 GreptimeDB 开发一个 gRPC SDK，包括 GreptimeDatabase 服务的定义、GreptimeRequest 和 GreptimeResponse 的结构。
+description: 介绍 GreptimeDB gRPC 写入 SDK 需要遵守的协议契约和错误处理要求。
---
# 如何为 GreptimeDB 开发一个 gRPC SDK
-GreptimeDB 的 gRPC SDK 只需要处理写请求即可。读请求是标准 SQL 或 PromQL，可以由任何 JDBC 客户端或 Prometheus
-客户端处理。这也是为什么所有的 GreptimeDB SDK 都命名为 "`greptimedb-ingester-`"。请确保你的 GreptimeDB SDK
-遵循相同的命名约定。
+GreptimeDB 的公开 gRPC SDK 是写入客户端。查询通常通过标准 SQL 或 PromQL 客户端完成。除非有单独需求，新 SDK 应聚焦写入和删除，并遵循 `greptimedb-ingester-` 命名约定。面向用户的 API 参见 [gRPC SDK 概述](/user-guide/ingest-data/for-iot/grpc-sdks/overview.md)。
## `GreptimeDatabase` 服务
-GreptimeDB 自定义了一个 gRPC 服务：`GreptimeDatabase`
-。你只需要实现这个服务即可。你可以在[这里](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto)
-找到它的 Protobuf 定义。
+从 [`GreptimeDatabase` Protobuf 定义](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto)生成客户端 stub，不要在 SDK 中手写一份 message 或 service 定义。
-`GreptimeDatabase` 有 2 个 RPC 方法：
+该 service 提供一个 unary method 和一个 client-streaming method：
```protobuf
service GreptimeDatabase {
@@ -25,13 +21,9 @@ service GreptimeDatabase {
}
```
-`Handle` 方法是一个 unary 调用：当 GreptimeDB 服务接收到一个 `GreptimeRequest` 请求后，它立刻处理该请求并返回一个相应的
-`GreptimeResponse`。
+`Handle` 对一个请求返回一个响应，是 SDK insert 和 delete API 通常使用的方法。
-`HandleRequests` 方法则是一个 "[Client Streaming RPC][3]" 方式的调用。
-它可以接受一个连续的 `GreptimeRequest` 请求流，持续地发给 GreptimeDB 服务。
-GreptimeDB 服务会在收到流中的每个请求时立刻进行处理，并最终（流结束时）返回一个总结性的 `GreptimeResponse`。
-通过 `HandleRequests`，我们可以获得一个非常高的请求吞吐量。
+`HandleRequests` 是 [client-streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc)。客户端关闭请求流后，服务端才返回累计响应。SDK 如果暴露 streaming API，需要明确这个确认边界，并将一个 stream 绑定到一个 endpoint。
### `GreptimeRequest`
@@ -51,11 +43,13 @@ message GreptimeRequest {
}
```
-`RequestHeader` 是必需，它包含了一些上下文，鉴权和其他信息。"oneof" 的字段包含了发往 GreptimeDB 服务的请求。
+客户端需要在 `RequestHeader` 中填写服务端要求的 database context 和认证信息，并且只能设置一个 request variant。
-注意我们有两种类型的插入请求，一种是以 "列" 的形式（`InsertRequests`），另一种是以 "行" 的形式（`RowInsertRequests`
-）。通常我们建议使用 "行" 的形式，因为它对于表的插入更自然，更容易使用。但是，如果需要一次插入大量列，或者有大量的 "null"
-值需要插入，那么最好使用 "列" 的形式。
+该 message 还包含供内部调用者使用的 query 和 DDL variant。公开 ingester API 不应暴露它们，因为 `GreptimeDatabase` 不返回 query result stream。
+
+GreptimeDB 同时接受行式 `RowInsertRequests` 和列式 `InsertRequests`。公开写入 API 默认使用行式请求。面向列的客户端可以使用列式请求，但转换过程中必须保持列长度一致，并保留 null、时间戳精度、数据类型和列 semantic type。
+
+删除同样区分行式和列式。SDK 只应暴露能够在不丢失类型信息的前提下完成映射的形式。
### `GreptimeResponse`
@@ -68,6 +62,18 @@ message GreptimeResponse {
}
```
-`ResponseHeader` 包含了返回值的状态码，以及错误信息（如果有的话）。"oneof" 的字段目前只有 "affected rows"。
+成功响应包含 success header 和 `affected_rows`。该值表示服务端确认的行数；关闭请求流时返回的是累计值。
+
+请求失败通过 gRPC status 返回。Trailing metadata 中的 `x-greptime-err-code` 在存在时提供 GreptimeDB error code，错误文本则由 status message 携带。SDK 应保留 gRPC status 并暴露 GreptimeDB error code，不能用一个通用 SDK error 将其覆盖。
+
+## 重试与交付语义
+
+重试次数必须有上限，并且对调用者可见。只有错误被标记为 retryable 且 deadline 仍允许时，才能重试 unary request。Cancellation 和 deadline expiration 不应重试。
+
+响应丢失不代表服务端拒绝了写入。除非调用者的数据模型保证操作幂等，重试这类请求可能插入重复行。SDK 需要说明这一点，并在交付结果不确定时返回最终错误。
+
+不要自动重试只发送了一部分的 `HandleRequests` stream。即使客户端尚未收到累计响应，服务端也可能已经接受了部分请求。此时应关闭失败的 stream，并将不确定状态返回给调用者。
+
+Arrow Flight bulk ingestion 与 `GreptimeDatabase` RPC 应使用不同的 API。它的 batching 和 partial acceptance 需要独立的契约。
-GreptimeDB 现在有很多 SDK，你可以参考[这里](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=)获取一些示例。
+可以参考现有 [GreptimeDB ingester 仓库](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=)的公开 API 约定，但线上行为应以当前 Protobuf 定义和服务端契约为准。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/admin-api.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/admin-api.md
index 3ac3d44f3b..bd28206c6f 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/admin-api.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/admin-api.md
@@ -1,20 +1,25 @@
---
-keywords: [Admin API, 健康检查, leader 查询, 心跳检测, 维护模式]
-description: 介绍 Metasrv 的 Admin API，包括健康检查、leader 查询、心跳检测、维护模式和 Procedure Manager 控制等功能。
+keywords: [Admin API, 健康检查, leader 查询, 心跳检测, 维护模式, 恢复模式, table id sequence]
+description: 介绍 Metasrv 用于状态检查、集群控制和元数据恢复的 Admin API。
---
# Admin API
-Admin 提供了一种简单的方法来查看和管理集群信息，包括 metasrv 健康检测、metasrv leader 查询、数据节点心跳检测、维护模式和 Procedure Manager 控制。
+:::tip
+本页所有 Admin API 都监听 Metasrv 的 `HTTP_PORT`，默认值为 `4000`。
+:::
-Admin API 是一个 HTTP 服务，提供一组可以通过 HTTP 请求调用的 RESTful API。Admin API 简单、用户友好且安全。
+Admin API 通过 HTTP 提供 Metasrv 状态、集群控制和元数据恢复操作。该 API 不提供认证，且部分端点会改变集群行为或元数据分配，部署时必须通过网络策略保护 HTTP 端口。
本页介绍以下 API：
- /health
- /leader
- /heartbeat
+- /node-lease
- /maintenance
- /procedure-manager
+- /recovery
+- /sequence/table
所有这些 API 都在父资源 `/admin` 下。
@@ -22,7 +27,7 @@ Admin API 是一个 HTTP 服务，提供一组可以通过 HTTP 请求调用的
## /health HTTP 端点
-`/health` 端点接受 GET HTTP 请求，你可以使用此端点检查你的 metasrv 实例的健康状况。
+`/health` 端点接受 GET 请求。HTTP 服务运行时返回 `OK`，但不会检查当前 Metasrv 是否为 leader，也不会检查外部依赖是否可用。
### 定义
@@ -116,9 +121,17 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100'
]
```
+## /node-lease HTTP 端点
+
+`/node-lease` 返回 Metasrv 当前记录的 Datanode lease，可用于判断 Metasrv 是否仍将某个 Datanode 视为存活。
+
+```bash
+curl -X GET http://localhost:4000/admin/node-lease
+```
+
## /maintenance HTTP 端点
-集群维护模式是 GreptimeDB 中的一项安全功能，它可以临时禁用自动集群管理操作。此模式在集群升级、计划停机以及任何可能暂时影响集群稳定性的操作期间特别有用。有关更多详细信息，请参阅[集群维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。
+维护模式在升级、计划停机等操作期间临时禁用自动集群管理。它对集群的具体影响参见[集群维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。
`/maintenance` 端点支持以下 HTTP 请求：
@@ -151,3 +164,39 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100'
"status": "running"
}
```
+
+## /recovery HTTP 端点
+
+Recovery mode 控制手动修改 table ID sequence 等元数据修复端点。它只用于恢复工作，不用于常规维护。
+
+- `GET /admin/recovery/status`：查询 recovery mode 是否开启。
+- `POST /admin/recovery/enable`：开启 recovery mode。
+- `POST /admin/recovery/disable`：关闭 recovery mode。
+
+响应体格式如下：
+
+```json
+{
+ "enabled": true
+}
+```
+
+修复完成后应关闭 recovery mode。如果只是计划暂停自动集群操作，应使用[维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。
+
+## /sequence/table HTTP 端点
+
+这些端点用于检查或修复 table ID sequence：
+
+- `GET /admin/sequence/table/next-id`：返回下一个 table ID，但不执行分配。
+- `POST /admin/sequence/table/set-next-id`：推进下一个 table ID。
+
+设置 sequence 前必须开启 recovery mode。新值必须大于当前值，不能通过该 API 回退 sequence。Recovery mode 只是该 API 的前置条件，不能阻止 DDL。执行该操作时，必须遵循[管理 Table ID Sequence](/user-guide/deployments-administration/maintenance/sequence-management.md)中的完整集群操作流程。
+
+```bash
+curl -X POST \
+ -H 'Content-Type: application/json' \
+ -d '{"next_table_id": 2048}' \
+ http://localhost:4000/admin/sequence/table/set-next-id
+```
+
+该操作会影响后续新表分配到的 ID。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/overview.md
index 2e3c525baa..e9f6144f26 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/overview.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/overview.md
@@ -1,162 +1,101 @@
---
-keywords: [Metasrv, 元数据存储, 请求路由, 负载均衡, 高可用性]
-description: 介绍 Metasrv 的功能、架构和与前端的交互方式。
+keywords: [Metasrv, 元数据, 路由, Leader 选举, Procedure, 心跳]
+description: 介绍 Metasrv 提供的元数据及集群协调机制。
---
# Metasrv
-
-
## Metasrv 包含什么
-- 存储元数据（Catalog, Schema, Table, Region 等）
-- 请求路由器。它告诉前端在哪里写入和读取数据。
-- 数据节点的负载均衡，决定谁应该处理新的表创建请求，更准确地说，它做出资源分配决策。
-- 选举与高可用性，GreptimeDB 设计为 Leader-Follower 架构，只有 leader 节点可以写入，而 follower 节点可以读取，follower 节点的数量通常 >= 1，当 leader 不可用时，follower 节点需要能够快速切换为 leader。
-- 统计数据收集（通过每个节点上的心跳报告），如 CPU、负载、节点上的表数量、平均/峰值数据读写大小等，可用作分布式调度的基础。
+Metasrv 是 GreptimeDB 分布式集群中的元数据和协调服务，不参与数据读写链路。它主要负责：
+
+- 存储 Catalog、Schema、Table、Region、路由和节点元数据；
+- 为新 Region 选择 Datanode，并维护表路由；
+- 选举一个 Metasrv leader 负责协调元数据变更；
+- 通过可恢复的 Procedure 执行 DDL、Region 迁移、故障转移和重分区；
+- 通过心跳维护节点租约和 Region 统计信息；
+- 元数据变更时向 Frontend、Datanode 和 Flownode 广播缓存失效；
+- 向 Datanode 下发 Region 生命周期指令。
## 前端如何与 Metasrv 交互
-首先，请求路由器中的路由表结构如下（注意这只是逻辑结构，实际存储结构可能不同，例如端点可能有字典压缩）。
-
-```txt
- table_A
- table_name
- table_schema // 用于物理计划
- regions
- region_1
- mutate_endpoint
- select_endpoint_1, select_endpoint_2
- region_2
- mutate_endpoint
- select_endpoint_1, select_endpoint_2, select_endpoint_3
- region_xxx
- table_B
- ...
-```
+Frontend 从 Metasrv 获取表元数据和 Region 路由，并缓存在本地。修改元数据的语句会发送给 Metasrv leader；普通读写则使用缓存的路由直接访问 Datanode。
-### 创建表
+控制链路和数据链路相互分离：
-1. 前端发送 `CREATE TABLE` 请求到 Metasrv。
-2. 根据请求中包含的分区规则规划 Region 数量。
-3. 检查数据节点可用资源的全局视图（通过心跳收集）并为每个 Region 分配一个节点。
-4. 前端创建表并在成功创建后将 `Schema` 存储到 Metasrv。
+```text
+Frontend
+ |-- 元数据查询和 DDL -------------------> Metasrv leader
+ `-- Region 读写 ------------------------> Datanode
-### `Insert`
+Metasrv leader
+ |-- Region 生命周期指令 ----------------> Datanode
+ `-- 缓存失效 --------------------------> Frontend / Datanode / Flownode
-1. 前端从 Metasrv 获取指定表的路由。注意，最小的路由单元是表的路由（多个 Region），即包含该表所有 Region 的地址。
-2. 最佳实践是前端首先从本地缓存中获取路由并将请求转发到数据节点。如果路由不再有效，则数据节点有义务返回 `Invalid Route` 错误，前端重新从 Metasrv 获取最新数据并更新其缓存。路由信息不经常变化，因此，前端使用惰性策略维护缓存是足够的。
-3. 前端处理可能包含多个表和多个 Region 的一批写入，因此前端需要根据“路由表”拆分用户请求。
+Datanode
+ `-- 心跳、租约续期和 Region 统计信息 ----> Metasrv leader
+```
-### `Select`
+在稳定状态下，表路由为每个 Region 记录一个 leader peer 和零个或多个 follower peer。Leader 是写入目标；支持只读副本的部署可以把读取路由到 follower：
-1. 与 `Insert` 类似，前端首先从本地缓存中获取路由表。
-2. 与 `Insert` 不同，对于 `Select`，前端需要从路由表中提取只读节点（follower），然后根据优先级将请求分发到 leader 或 follower 节点。
-3. 前端的分布式查询引擎根据路由信息分发多个子查询任务并聚合查询结果。
+```text
+Table route
+ |-- Region 0
+ | |-- leader -> Datanode A
+ | `-- followers -> Datanode B, Datanode C
+ `-- Region 1
+ `-- leader -> Datanode D
+```
-## Metasrv 架构
+Region 迁移或故障转移会改变 peer 角色，并可能使 Region 暂时没有 leader。Frontend 刷新缓存路由后，再把后续读写发送给当前 peer。
-
+### 创建表
-## 分布式共识
+1. Frontend 向 Metasrv leader 提交 DDL 请求。
+2. Metasrv 根据分区规则确定 Region，并[为每个 Region 选择 Datanode](/contributor-guide/metasrv/selector.md)。
+3. 持久化的 Procedure 创建 Region，并写入表元数据和路由。发生 leader 切换后，Procedure 可以从已保存的状态继续执行。
+4. 元数据提交后，Metasrv 通知 Frontend 刷新相关缓存。
-如你所见，Metasrv 依赖于分布式共识，因为：
+### `Insert`
-1. 首先，Metasrv 必须选举一个 leader，数据节点只向 leader 发送心跳，我们只使用单个 Metasrv 节点接收心跳，这使得基于全局信息进行一些计算或调度变得容易且快速。至于数据节点如何连接到 leader，这由 MetaClient 决定（使用重定向，心跳请求变为 gRPC 流，使用重定向比转发更不容易出错），这对数据节点是透明的。
-2. 其次，Metasrv 必须为数据节点提供选举 API，以选举“写入”和“只读”节点，并帮助数据节点实现高可用性。
-3. 最后，`Metadata`、`Schema` 和其他数据必须在 Metasrv 上可靠且一致地存储。因此，基于共识的算法是存储它们的理想方法。
+Frontend 解析表路由，按照分区规则拆分数据行，再把各 Region 的写入发送到对应 Datanode。路由发生变化时，相关缓存会失效，Frontend 随后从 Metasrv 重新获取元数据。
-对于 Metasrv 的第一个版本，我们选择 Etcd 作为共识算法组件（Metasrv 设计时考虑适应不同的实现，甚至创建一个新的轮子），原因如下：
+### `Select`
-1. Etcd 提供了我们需要的 API，例如 `Watch`、`Election`、`KV` 等。
-2. 我们只执行两个分布式共识任务：选举（使用 `Watch` 机制）和存储（少量元数据），这两者都不需要我们定制自己的状态机，也不需要基于 raft 定制自己的状态机；少量数据也不需要多 raft 组支持。
-3. Metasrv 的初始版本使用 Etcd，使我们能够专注于 Metasrv 的功能，而不需要在分布式共识算法上花费太多精力，这提高了系统设计（避免与共识算法耦合）并有助于初期的快速开发，同时通过良好的架构设计，未来可以轻松接入优秀的共识算法实现。
+Frontend 在查询规划期间使用表和 Region 元数据。分区列上的谓词用于裁剪 Region，分布式查询引擎再把任务发送给持有这些 Region 的 Datanode。参见[分布式查询](../frontend/distributed-querying.md)。
-## 心跳管理
+## Metasrv 架构
-数据节点与 Metasrv 之间的主要通信方式是心跳请求/响应流，我们希望这是唯一的通信方式。这个想法受到 [TiKV PD](https://github.com/tikv/pd) 设计的启发，我们在 [RheaKV](https://github.com/sofastack/sofa-jraft/tree/master/jraft-rheakv/rheakv-pd) 中有实际经验。请求发送其状态，而 Metasrv 通过心跳响应发送不同的调度指令。
+主要协调路径如下：
+
+```text
+Leader election
+ |
+ v
+Metasrv leader
+├─ DDL manager -> Procedure manager
+├─ Selector -> 新 Region 的放置
+├─ Heartbeat handler chain -> 租约和 Region 统计信息
+├─ Region supervisor -> Region 迁移 Procedure
+├─ Mailbox -> 缓存失效和 Region 指令
+└─ Metadata managers -> KV backend
+```
-心跳可能携带以下数据，但这不是最终设计，我们仍在讨论和探索究竟应该收集哪些数据。
+这些机制共享元数据，但故障边界不同。进程重启可以丢弃缓存和 leader 本地状态；恢复所需的元数据和 Procedure 状态必须持久化。
-```
-service Heartbeat {
- // 心跳，心跳可能有很多内容，例如：
- // 1. 要注册到 Metasrv 并可被其他节点发现的元数据。
- // 2. 一些性能指标，例如负载、CPU 使用率等。
- // 3. 正在执行的计算任务数量。
- rpc Heartbeat(stream HeartbeatRequest) returns (stream HeartbeatResponse) {}
-}
-
-message HeartbeatRequest {
- RequestHeader header = 1;
-
- // 自身节点
- Peer peer = 2;
- // leader 节点
- bool is_leader = 3;
- // 实际报告时间间隔
- TimeInterval report_interval = 4;
- // 节点状态
- NodeStat node_stat = 5;
- // 此节点中的 Region 状态
- repeated RegionStat region_stats = 6;
- // follower 节点和状态，在 follower 节点上为空
- repeated ReplicaStat replica_stats = 7;
-}
-
-message NodeStat {
- // 此期间的读取容量单位
- uint64 rcus = 1;
- // 此期间的写入容量单位
- uint64 wcus = 2;
- // 此节点中的表数量
- uint64 table_num = 3;
- // 此节点中的 Region 数量
- uint64 region_num = 4;
-
- double cpu_usage = 5;
- double load = 6;
- // 节点中的读取磁盘 I/O
- double read_io_rate = 7;
- // 节点中的写入磁盘 I/O
- double write_io_rate = 8;
-
- // 其他
- map attrs = 100;
-}
-
-message RegionStat {
- uint64 region_id = 1;
- TableName table_name = 2;
- // 此期间的读取容量单位
- uint64 rcus = 3;
- // 此期间的写入容量单位
- uint64 wcus = 4;
- // 近似 Region 大小
- uint64 approximate_size = 5;
- // 近似行数
- uint64 approximate_rows = 6;
-
- // 其他
- map attrs = 100;
-}
-
-message ReplicaStat {
- Peer peer = 1;
- bool in_sync = 2;
- bool is_learner = 3;
-}
-```
+## 分布式共识
-## Central Nervous System (CNS)
+Metasrv 将 leader 选举与元数据存储分开。只有选出的 Metasrv leader 执行协调和元数据变更操作，其他 Metasrv 节点会把 client 引导到当前 leader。
-我们要构建一个算法系统，该系统依赖于每个节点的实时和历史心跳数据，应该做出一些更智能的调度决策并将其发送到 Metasrv 的 Autoadmin 单元，该单元分发调度决策，由数据节点本身或更可能由 PaaS 平台执行。
+Key-value backend 保存表元数据、路由、Procedure 状态以及其他必须跨 leader 切换保留的信息。Metasrv 不使用这套选举为 Datanode Region 创建读写副本；Region 可用性由心跳、Region 故障检测和故障转移 Procedure 管理。
-## 工作负载抽象
+## 心跳管理
-工作负载抽象的级别决定了 Metasrv 生成的调度策略（如资源分配）的效率和质量。
+Datanode 与 Metasrv leader 保持心跳流。心跳请求报告节点身份、租约、Region 统计信息以及放置和监控所需的其他状态；响应则携带 Region 生命周期指令、缓存失效等控制消息。
-DynamoDB 定义了 RCUs 和 WCUs（读取容量单位/写入容量单位），解释说 RCU 是一个 4KB 数据的读取请求，WCU 是一个 1KB 数据的写入请求。当使用 RCU 和 WCU 描述工作负载时，更容易实现性能可测量性并获得更有信息量的资源预分配，因为我们可以将不同的硬件能力抽象为 RCU 和 WCU 的组合。
+心跳驱动两套相互独立的机制，调整心跳周期会同时影响两者：
-然而，GreptimeDB 面临比 DynamoDB 更复杂的情况，特别是 RCU 不适合描述需要大量计算的 GreptimeDB 读取工作负载。我们正在努力解决这个问题。
+- **节点租约**：keep-lease handler 为发送心跳的 Datanode 续期。Selector 和 `/node-lease` 端点据此判断 Datanode 是否仍然存活。
+- **Region 故障检测**：Region supervisor 为每个 Region 维护一个基于心跳到达间隔的 Phi Accrual 检测器，其判定与租约是否过期无关。
+只有开启 Region 故障转移时，故障判定才会提交故障转移迁移。该功能默认关闭，并且要求使用 remote WAL，除非显式允许在本地 WAL 上执行。维护模式同样会抑制故障转移。前置条件和开启方式参见 [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md)。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/selector.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/selector.md
index 9048e0daa4..29b521212b 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/selector.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/selector.md
@@ -7,11 +7,7 @@ description: 介绍 Metasrv 中的 Selector，包括其类型和配置方法。
## 介绍
-什么是 `Selector`？顾名思义，它允许用户从给定的 `namespace` 和 `context` 中选择 `Item`s。有一个相关的 `trait`，也叫做 `Selector`，其定义可以在[这里][0]找到。
-
-[0]: https://github.com/GreptimeTeam/greptimedb/blob/main/src/meta-srv/src/selector.rs
-
-在 `Metasrv` 中存在一个特定的场景。当 `Frontend` 向 `Metasrv` 发送建表请求时，`Metasrv` 会创建一个路由表（表的创建细节不在这里赘述）。在创建路由表时，`Metasrv` 需要选择适当的 `Datanode`s，这时候就需要用到 `Selector`。
+建表时，Metasrv 使用 `Selector` 为各 Region 选择 Datanode。Selector 根据当前节点租约进行选择；部分实现还会使用 Region 统计信息。
@@ -19,22 +15,22 @@ description: 介绍 Metasrv 中的 Selector，包括其类型和配置方法。
`Metasrv` 目前提供以下几种类型的 `Selectors`:
-### LeasebasedSelector
+### LeaseBasedSelector
-`LeasebasedSelector` 从所有可用的（也就是在租约期间内）`Datanode` 中随机选择，其特点是简单和快速。
+`LeaseBasedSelector` 从租约有效的 Datanode 中随机选择。
### LoadBasedSelector
`LoadBasedSelector` 按照负载来选择，负载值则由每个 `Datanode` 上的 region 数量决定，较少的 region 表示较低的负载，`LoadBasedSelector` 优先选择低负载的 `Datanode`。
### RoundRobinSelector [默认选项]
-`RoundRobinSelector` 以轮询的方式选择 `Datanode`。在大多数情况下，这是默认的且推荐的选项。如果你不确定选择哪个，通常它就是正确的选择。
+`RoundRobinSelector` 以轮询方式选择 Datanode，是默认选项，也适用于大多数部署。
## 配置
您可以在启动 `Metasrv` 服务时通过名称配置 `Selector`。
-- LeasebasedSelector: `lease_based` 或 `LeaseBased`
+- LeaseBasedSelector: `lease_based` 或 `LeaseBased`
- LoadBasedSelector: `load_based` 或 `LoadBased`
- RoundRobinSelector: `round_robin` 或 `RoundRobin`
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/overview.md
index 9f82a72171..badd6ad62d 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/overview.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/overview.md
@@ -5,9 +5,7 @@ description: 介绍 GreptimeDB 的架构、关键概念和工作原理，包括
# 贡献者指南
-DeepWiki 对 GreptimeDB 的架构和实现进行了详细且清晰的描述，强烈推荐阅读：
-
-[https://deepwiki.com/GreptimeTeam/greptimedb](https://deepwiki.com/GreptimeTeam/greptimedb)
+本指南面向 GreptimeDB 贡献者，介绍理解内部实现所需的设计机制。从源码构建和运行参见[快速开始](/contributor-guide/getting-started.md)。提交要求（CLA、license header、代码格式，以及 PR 必须通过的检查）以源码仓库的 [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md) 为准。
## 架构
@@ -18,7 +16,13 @@ DeepWiki 对 GreptimeDB 的架构和实现进行了详细且清晰的描述，
- [frontend][1]
- [datanode][2]
- [metasrv][3]
+- [flownode][4]
[1]: /contributor-guide/frontend/overview.md
[2]: /contributor-guide/datanode/overview.md
[3]: /contributor-guide/metasrv/overview.md
+[4]: /contributor-guide/flownode/overview.md
+
+## 补充参考
+
+[DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb) 提供了自动生成的 GreptimeDB 源码导读，可用于了解不熟悉的模块。它属于辅助资料；涉及具体版本的行为时，仍应以对应源码为准。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/integration-test.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/integration-test.md
index 63ffa56bc0..767109f773 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/integration-test.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/integration-test.md
@@ -7,8 +7,14 @@ description: 介绍 GreptimeDB 的集成测试，包括测试范围和如何运
## 介绍
-集成测试使用 Rust 测试工具（`#[test]`）编写，与单元测试不同，它们被单独放置在
-[这里](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration)。
-它涵盖了涉及多个组件的场景，其中一个典型案例是与 HTTP/gRPC 相关的功能。你可以查看
-其[文档](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md)以获取更多信息。
+集成测试覆盖跨 crate 或服务边界的行为，例如 HTTP 和 gRPC 处理、分布式组件或外部存储。测试使用 Rust test harness，位于 [`tests-integration`](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration) package。
+运行命令如下：
+
+```shell
+cargo nextest run -p tests-integration
+```
+
+部分 case 依赖外部服务的环境变量或 fixture。运行前按照 package 的[准备说明](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md)配置环境。
+
+只有 crate 级测试或 Sqlness case 无法覆盖所需边界时才使用集成测试。隔离的逻辑仍放在单元测试中，便于快速复现失败。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/overview.md
index 81b6defa45..f38b625082 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/overview.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/overview.md
@@ -5,5 +5,12 @@ description: GreptimeDB 的测试
# 测试
-我们的团队进行了大量测试，以确保 GreptimeDB 的行为。本章将介绍几种用于测试 GreptimeDB 的重要方法，以及如何使用它们。
+选择能够覆盖本次改动的最小测试范围：
+| 测试类型 | 适用场景 | 常用命令 |
+| --- | --- | --- |
+| [单元测试](unit-test.md) | 单个 crate 或组件内的逻辑 | `cargo nextest run -p ` |
+| [Sqlness 测试](sqlness-test.md) | SQL、协议、planner、执行和端到端回归 | `cargo sqlness bare -t ` |
+| [集成测试](integration-test.md) | 跨组件或依赖外部服务的行为 | `cargo nextest run -p tests-integration` |
+
+需要运行完整 Rust workspace 测试时使用 `make test`。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/sqlness-test.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/sqlness-test.md
index d29b69e835..06503a434c 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/sqlness-test.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/sqlness-test.md
@@ -7,36 +7,34 @@ description: 介绍 GreptimeDB 的 Sqlness 测试，包括测试文件类型、
## 介绍
-SQL 是 `GreptimeDB` 的一个重要用户接口。我们为它提供了一个单独的测试套件（名为 `sqlness`）。
+Sqlness 是 GreptimeDB 针对 SQL 和协议行为的端到端回归测试。每个 case 向运行中的 GreptimeDB 发送语句，并将输出与仓库中的结果文件比较。
## Sqlness 手册
### 测试文件
-Sqlness 有两种类型的文件
+每个 case 使用两类文件：
- `.sql`：测试输入，仅包含 SQL
- `.result`：预期的测试输出，包含 SQL 和其结果
-`.result` 文件是预期的执行输出。如果 `.result` 文件发生变化，意味着测试结果不同，测试可能失败。你应该检查变更日志来解决问题。
-
-你只需要在 `.sql` 文件中编写测试 SQL，然后运行测试。
+在 `.sql` 文件中编写输入，运行测试后生成或更新 `.result`。必须检查每一处结果差异，只有行为变化符合预期时才能接受。
### 组织测试案例
-输入案例的根目录是 `tests/cases`。它包含几个子目录，代表不同的测试模式。例如，`standalone/` 包含所有在 `greptimedb standalone start` 模式下运行的测试。
+输入 case 位于 `tests/cases`。第一级目录选择运行环境，例如 `standalone/` 表示使用单机 GreptimeDB。
-在第一级子目录下（例如 `cases/standalone`），你可以随意组织你的测试案例。Sqlness 会递归地遍历每个文件并运行它们。
+在环境目录内，新 case 应与它覆盖的功能放在一起。Sqlness 会递归发现 case 文件。
## 运行测试
-与其他测试不同，这个测试工具是以二进制目标形式存在的。你可以用以下命令运行它
+运行命令如下：
```shell
-cargo run --bin sqlness-runner bare
+cargo sqlness bare
```
-它会自动完成以下步骤：编译 `GreptimeDB`，启动它，抓取测试并将其发送到服务器，然后收集和比较结果。你只需要检查是否有 `.result` 文件发生变化。如果没有，恭喜你，测试通过了 🥳！
+该命令会构建并启动 GreptimeDB、执行选中的 case，再比较输出。`.result` 发生变化只是待审查的结果，不代表新输出一定正确。
### 运行特定测试
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/unit-test.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/unit-test.md
index 79c73775b4..b63b29df94 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/unit-test.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/unit-test.md
@@ -7,22 +7,26 @@ description: 介绍 GreptimeDB 的单元测试，包括如何编写、运行和
## 介绍
-单元测试嵌入在代码库中，通常放置在被测试逻辑的旁边。它们使用 Rust 的 `#[test]` 属性编写，并可以使用 `cargo nextest run` 运行。
+单元测试通常放在被测逻辑旁边，使用 Rust 的 `#[test]` 属性编写。GreptimeDB 主要使用 [`cargo-nextest`](https://nexte.st/) 运行 Rust 测试。
-GreptimeDB 代码库不支持默认的 `cargo` 测试运行器。推荐使用 [`nextest`](https://nexte.st/)。你可以通过以下命令安装它：
+安装命令如下：
```shell
cargo install cargo-nextest --locked
```
-然后运行测试（这里 `--workspace` 不是必须的）
+开发时先运行本次修改的 package：
```shell
-cargo nextest run
+cargo nextest run -p
```
-注意，如果你的 Rust 是通过 `rustup` 安装的，请确保使用 `cargo` 安装 `nextest`，而不是像 `homebrew` 这样的包管理器，否则会弄乱你的本地环境。
+可以继续使用测试名称或 nextest filter 缩小范围。影响范围较广的改动在提交前运行完整 workspace 测试：
+
+```shell
+make test
+```
## 覆盖率
-我们的持续集成（CI）作业有一个“覆盖率检查”步骤。它会报告有多少代码被单元测试覆盖。请在你的补丁中添加必要的单元测试。
+CI 会报告单元测试覆盖率。测试应覆盖本次改变的行为和可能回归的失败路径，而不是只追求覆盖率数字。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/reference/sql/create.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/reference/sql/create.md
index f7fb21170d..cb2d9bdb30 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/reference/sql/create.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/reference/sql/create.md
@@ -26,7 +26,7 @@ CREATE DATABASE [IF NOT EXISTS] db_name [WITH ]
数据库也可以通过使用 `WITH` 关键字配置与 `CREATE TABLE` 语句类似的选项。数据库支持以下选项：
- `ttl` - 数据库中所有表的数据存活时间（不能设置为 `instant`）
-- `memtable.type` - 内存表类型（`time_series`、`partition_tree`）
+- `memtable.type` - memtable 类型（`bulk`、`time_series`）
- `append_mode` - 数据库中的表是否为仅追加模式（`true`/`false`）
- `merge_mode` - 合并重复行的策略（`last_row`、`last_non_null`）
- `skip_wal` - 是否为数据库中的表禁用预写日志（`'true'`/`'false'`）
@@ -74,7 +74,7 @@ CREATE DATABASE test WITH (ttl='7d');
```sql
CREATE DATABASE test WITH (
ttl='30d',
- 'memtable.type'='partition_tree',
+ 'memtable.type'='bulk',
'append_mode'='true'
);
```
@@ -156,7 +156,7 @@ GreptimeDB 提供了丰富的索引实现来加速查询，请在[索引](/user-
| `compaction.twcs.trigger_file_num` | 某个窗口内触发 compaction 的最小文件数量阈值 | 字符串值，如 '8'。只在 `compaction.type` 为 `twcs` 时可用 |
| `compaction.twcs.time_window` | Compaction 时间窗口 | 字符串值，如 '1d' 表示 1 天。该表会根据时间戳将数据分区到不同的时间窗口中。只在 `compaction.type` 为 `twcs` 时可用 |
| `compaction.twcs.max_output_file_size` | TWCS compaction 的最大输出文件大小 | 字符串值，如 '1GB'、'512MB'。设置 TWCS compaction 产生的文件的最大大小。只在 `compaction.type` 为 `twcs` 时可用 |
-| `memtable.type` | memtable 的类型 | 字符串值，支持 `time_series`，`partition_tree` |
+| `memtable.type` | memtable 类型 | 字符串值：`bulk` 或 `time_series`。未设置时，Mito 根据 SST format 选择实现；默认的 flat format 使用 `bulk`。设置 `bulk` 会强制使用 `sst_format=flat`；使用 flat SST 时，即使设置了 `time_series`，Mito 也会选择 bulk 实现。旧值 `partition_tree` 仅为兼容保留，并映射到 bulk 和 flat 路径。 |
| `append_mode` | 该表是否时 append-only 的 | 字符串值。默认值为 'false'，根据 'merge_mode' 按主键和时间戳删除重复行。设置为 'true' 可以开启 append 模式和创建 append-only 表，保留所有重复的行 |
| `merge_mode` | 合并重复行的策略 | 字符串值。只有当 `append_mode` 为 'false' 时可用。默认值为 `last_row`，保留相同主键和时间戳的最后一行。设置为 `last_non_null` 则保留相同主键和时间戳的最后一个非空字段。 |
| `sst_format` | SST 文件的格式 | 字符串值，支持 `primary_key`，`flat`。默认为 `flat`。`flat` 格式建议用于具有高基数主键的表。 |
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/user-guide/deployments-administration/configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/user-guide/deployments-administration/configuration.md
index faaf932d65..c6b52a42e9 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/user-guide/deployments-administration/configuration.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/user-guide/deployments-administration/configuration.md
@@ -481,20 +481,9 @@ create_on_compaction = "auto"
apply_on_query = "auto"
mem_threshold_on_create = "64M"
intermediate_path = ""
-
-[region_engine.mito.memtable]
-type = "time_series"
```
-此外，`mito` 也提供了一个实验性质的 memtable。该 memtable 主要优化大量时间序列下的写入性能和内存占用。其查询性能可能会不如默认的 `time_series` memtable。
-
-```toml
-[region_engine.mito.memtable]
-type = "partition_tree"
-index_max_keys_per_shard = 8192
-data_freeze_threshold = 32768
-fork_dictionary_bytes = "1GiB"
-```
+Mito 根据表选项和 SST format 为每个 Region 选择 memtable 实现。`default_flat_format` 为 `true` 时，没有显式设置 `sst_format` 的 Region 使用 flat SST 和 bulk memtable。`memtable.type` 是数据库或表选项，不是 `[region_engine.mito.memtable]` 引擎配置。详见[表选项](/reference/sql/create.md#表选项)。
以下是可供使用的选项
@@ -528,7 +517,7 @@ fork_dictionary_bytes = "1GiB"
| `scan_memory_on_exhausted` | 字符串 | `fail` | 扫描内存耗尽时的行为。选项：`fail`（快速失败），`wait` 或 `wait()`（等待内存）。 |
| `min_compaction_interval` | 字符串 | `0m` | 两次 compaction 之间的最小时间间隔。设为 "0m"（默认）允许 compactions 立即运行，无限制。 |
| `schedule_compaction_after_edit` | 布尔值 | `true` | 是否允许在成功的 region edit 之后调度 compaction。 设为 `true` 是在 region edit 后调度 compaction 的必要但不充分条件，`min_compaction_interval` 等其他约束仍可能阻止 compaction 被调度。 设为 `false` 则保证 region edit 后不会调度 compaction。 |
-| `default_flat_format` | 布尔值 | `true` | 是否启用 Flat 格式作为默认 SST 格式。 |
+| `default_flat_format` | 布尔值 | `true` | 没有显式设置 `sst_format` 的 Region 是否使用 flat SST。Flat SST 使用 bulk memtable。 |
| `scan_parallelism` | 整数 | `0` | （已弃用，请使用 `max_concurrent_scan_files`）旧版扫描并发度选项。 |
| `index` | -- | -- | Mito 引擎中索引的选项。 |
| `index.aux_path` | 字符串 | `""` | 文件系统中索引的辅助目录路径，用于存储创建索引的中间文件和搜索索引的暂存文件，默认为 `{data_home}/index_intermediate`。为了向后兼容，该目录的默认名称为 `index_intermediate`。此路径包含两个子目录：- `__intm`: 用于存储创建索引时使用的中间文件。- `staging`: 用于存储搜索索引时使用的暂存文件。 |
@@ -543,10 +532,6 @@ fork_dictionary_bytes = "1GiB"
| `inverted_index.apply_on_query` | 字符串 | `auto` | 是否在查询时使用索引 - `auto`: 自动 - `disable`: 从不 |
| `inverted_index.mem_threshold_on_create` | 字符串 | `64M` | 创建索引时如果超过该内存阈值则改为使用外部排序 设置为空会关闭外排，在内存中完成所有排序 |
| `inverted_index.intermediate_path` | 字符串 | `""` | 存放外排临时文件的路径 (默认 `{data_home}/index_intermediate`). |
-| `memtable.type` | 字符串 | `time_series` | Memtable type. - `time_series`: time-series memtable - `partition_tree`: partition tree memtable (实验性功能) |
-| `memtable.index_max_keys_per_shard` | 整数 | `8192` | 一个 shard 内的主键数 只对 `partition_tree` memtable 生效 |
-| `memtable.data_freeze_threshold` | 整数 | `32768` | 一个 shard 内写缓存可容纳的最大行数 只对 `partition_tree` memtable 生效 |
-| `memtable.fork_dictionary_bytes` | 字符串 | `1GiB` | 主键字典的大小 只对 `partition_tree` memtable 生效 |
`metric` 引擎针对包含大量小表的 metrics 数据进行了优化：
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/data-persistence-indexing.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/data-persistence-indexing.md
index e0ef9e8e54..e0ce16721d 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/data-persistence-indexing.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/data-persistence-indexing.md
@@ -5,19 +5,23 @@ description: 介绍了 GreptimeDB 的数据持久化和索引机制，包括 SST
# 数据持久化与索引
-与所有类似 LSMT 的存储引擎一样，MemTables 中的数据被持久化到耐久性存储，例如本地磁盘文件系统或对象存储服务。GreptimeDB 采用 [Apache Parquet][1] 作为其持久文件格式。
+与其他 LSM-tree 存储引擎类似，GreptimeDB 将 memtable 中的数据持久化到本地文件系统或对象存储，并使用 [Apache Parquet][1] 作为持久化文件格式。
## SST 文件格式
Parquet 是一种提供快速数据查询的开源列式存储格式，已经被许多项目采用，例如 Delta Lake。
-Parquet 具有层次结构，类似于“行组 - 列-数据页”。Parquet 文件中的数据被水平分区为行组（row group），在其中相同列的所有值一起存储以形成数据页（data pages）。数据页是最小的存储单元。这种结构极大地提高了性能。
+Parquet 按 row group、column chunk 和 page 组织数据。每个 row group 为每一列保存一个 column chunk，每个 column chunk 再包含一个或多个 page。Page 是编码和压缩单元，读取指定列时则以 column chunk 为 I/O 单元。
首先，数据按列聚集，这使得文件扫描更加高效，特别是当查询只涉及少数列时，这在分析系统中非常常见。
-其次，相同列的数据往往是同质的（比如具备近似的值），这有助于在采用字典和 Run-Length Encoding（RLE）等技术进行压缩。
+其次，同一列中的值通常比较相似，有利于字典编码和 Run-Length Encoding（RLE）等压缩技术发挥作用。
-
+下面这张来自 Apache Parquet 规范的图进一步展示了物理文件布局：column chunk 按 row group 写入，文件元数据及其长度则保存在 footer 中。
+
+
+
+*来源：Apache Parquet [FileLayout.gif](https://github.com/apache/parquet-format/blob/master/doc/images/FileLayout.gif)。Copyright 2014 The Apache Software Foundation，依据 [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) 使用。*
## 数据持久化
@@ -26,18 +30,18 @@ GreptimeDB 提供了 `region_engine.mito.global_write_buffer_size` 的配置项
## SST 文件中的索引数据
-Apache Parquet 文件格式在列块和数据页的头部提供了内置的统计信息，用于剪枝和跳过。
+Parquet 在每个 column chunk 的元数据中保存 row group 级列统计信息，例如最小值、最大值和 null 数量。Page 元数据和可选的 column index 可以提供粒度更细的统计信息。
-
+
-例如，在上述 Parquet 文件中，如果你想要过滤 `name` 等于 `Emily` 的行，你可以轻松跳过行组 0，因为 `name` 字段的最大值是 `Charlie`。这些统计信息减少了 IO 操作。
+例如，查询 `name` 等于 `Emily` 的行时，可以跳过 row group 0，因为其中 `name` 的最大值是 `Charlie`，无需读取该 row group。
## 索引文件
-对于每个 SST 文件，GreptimeDB 不但维护 SST 文件内部索引，还会单独生成一个文件用于存储针对该 SST 文件的索引结构。
+当一个 SST 存在已配置且适用的索引输出时，GreptimeDB 将这些索引写入与该 SST 关联的 Puffin 文件。没有适用索引的 SST 不需要生成 Puffin 文件。
-索引文件采用 [Puffin][3] 格式，这种格式具有较大的灵活性，能够存储更多的元数据，并支持更多的索引结构。
+Puffin 是索引 Blob 及其元数据的容器，使不同索引结构可以共用一个文件。
@@ -58,13 +62,13 @@ GreptimeDB 会将多种索引结构作为 Blob 存储在 Puffin 文件中，包
-例如，上述查询使用倒排索引来定位数据段，数据段满足条件：`job` 等于 `apiserver`，`handler` 符合正则匹配 `.*users` 及 `status` 符合正则匹配 `4..`，然后扫描这些数据段以产生满足所有条件的最终结果，从而显着减少 IO 操作的次数。
+上述查询使用倒排索引定位 `job` 等于 `apiserver`、`handler` 匹配 `.*users` 且 `status` 匹配 `4..` 的数据段。Mito 只扫描这些数据段，再应用剩余过滤条件。
### 倒排索引格式
-
+
-GreptimeDB 按列构建倒排索引，每个倒排索引包含一个 FST 和多个 Bitmap。
+GreptimeDB 按列构建倒排索引。每个列索引包含一个 null bitmap、多个 posting bitmap 和一个 FST。Blob footer 记录定位和解码各列索引所需的 offset、size 和元数据。
FST（Finite State Transducer）允许 GreptimeDB 以紧凑的格式存储列值到 Bitmap 位置的映射，并且提供了优秀的搜索性能和支持复杂搜索（例如正则表达式匹配）；Bitmap 则维护了数据段 ID 列表，每个位表示一个数据段。
@@ -80,7 +84,7 @@ GreptimeDB 把一个 SST 文件分割成多个索引数据段，每个数据段
## 统一数据访问层：OpenDAL
-GreptimeDB 使用 [OpenDAL][2] 提供统一的数据访问层，因此，存储引擎无需与不同的存储 API 交互，数据可以无缝迁移到基于云的存储，如 AWS S3。
+GreptimeDB 使用 [OpenDAL][2] 为本地文件系统和对象存储提供统一访问层。修改配置的存储 backend 不会迁移已有数据。
[1]: https://parquet.apache.org
[2]: https://github.com/datafuselabs/opendal
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/memtable.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/memtable.md
new file mode 100644
index 0000000000..3905a1aecd
--- /dev/null
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/memtable.md
@@ -0,0 +1,100 @@
+---
+keywords: [memtable, Mito engine, 写缓冲, flush, 时间分区, BulkMemtable]
+description: 介绍 Mito 如何在 memtable 中组织 Region 的可变数据，以及如何将这些数据写入 SST 文件。
+---
+
+# Memtable 设计
+
+Memtable 是 Mito 为每个 Region 维护的内存写缓冲。数据 flush 为 SST 文件前，读取可以先从 memtable 获取这些数据。Region version 确定 scan 可以读取的 memtable 和 SST 文件；配合 committed sequence 上限，scan 可以在写入和 flush 推进当前 version 时保持一致性。
+
+## 写入和 flush 生命周期
+
+对于使用 WAL 的常规写入，Mito 按以下顺序处理：
+
+```text
+写请求
+ |
+ v
+追加 WAL -> mutable memtable -> 发布 committed sequence
+ |
+ freeze
+ v
+ immutable memtable -> 写入 SST -> manifest edit
+```
+
+Region worker 先分配 sequence number 和 WAL entry ID，再将 mutation 追加到[预写日志](wal.md)。如果追加失败，Mito 不会更新 memtable。Memtable 更新成功后，Mito 发布 committed sequence，新的读取随后可以看到这些数据。配置了 `skip_wal` 的 Region 会跳过 WAL，但 memtable 更新和可见性顺序不变。
+
+Flush 在启动后台 SST 写入前，先冻结 mutable memtable 并安装一组新的 mutable memtable。后续写入可以继续进行，也不会修改已经冻结的数据。Flush 将 immutable memtable 写为 SST 文件，再持久化包含新文件、flushed WAL checkpoint 和 sequence checkpoint 的 manifest edit。只有 manifest edit 持久化成功后，Mito 才会从当前 Region version 中移除已 flush 的 memtable。Flush 失败时，immutable memtable 会保留，供后续任务重试。
+
+## Region version 和时间分区
+
+每个 Region 包含一个 mutable `TimePartitions` 容器，其中可以有多个 memtable：
+
+```text
+Region version
+├─ mutable TimePartitions
+│ ├─ [t0, t1) -> memtable
+│ └─ [t1, t2) -> memtable
+├─ immutable memtables
+└─ SST files
+```
+
+Mito 根据 time index 的值把每行数据路由到对应分区。分区使用左闭右开的时间范围，并按照固定时长对齐。该时长跟随 Region 的 compaction time window；在取得 compaction time window 前，Mito 使用一天作为初始值。乱序写入可能在最新分区之外创建更早的分区。
+
+冻结 Region 时，Mito 会同时冻结所有 mutable 时间分区，把其中的 memtable 移入 immutable 列表，再创建新的 `TimePartitions` 容器。Flush 失败可能留下多代 immutable memtable，因此读取和后续 flush 不能假定列表中只有一个对象。
+
+## Memtable 实现
+
+Mito 根据 Region 的 SST format、primary key encoding 和 memtable 选项选择实现：
+
+```text
+flat SST format（默认）或 sparse primary-key encoding -> BulkMemtable
+memtable.type=bulk -> BulkMemtable，并强制 flat SST
+primary_key SST + dense encoding（遗留） -> 遗留实现
+```
+
+使用默认 engine 配置时，没有显式指定 SST format 的 Region 会使用 `flat`，因此通常走 `BulkMemtable` 路径，本页其余内容也以它为准。这些规则用于排除不兼容的组合：flat format 或 sparse primary key encoding 必须使用 `BulkMemtable`；显式选择 bulk 实现则会强制使用 flat format。
+
+### BulkMemtable
+
+`BulkMemtable` 使用 flat Arrow 布局把写入保存为 part，而不是将数据行插入按时间序列组织的缓冲区：
+
+```text
+BulkMemtable
+├─ unordered_part
+│ └─ 小批量 BulkPart
+└─ parts
+ ├─ BulkPart (Arrow RecordBatch)
+ ├─ MultiBulkPart (未编码的 RecordBatch)
+ └─ EncodedBulkPart (内存中的 Parquet 数据)
+```
+
+小 part 先积累在 `unordered_part` 中，较大的 part 则直接进入 `parts`。后台 memtable compaction 对符合条件的 part 执行 merge sort，生成 `MultiBulkPart` 或编码为 `EncodedBulkPart`。Scan 利用 part 的统计信息裁剪 range；flush 可以将已编码的 range 写入 SST，无需再次解码和编码数据行。设计动机和性能数据见 [Scaling Time Series to Millions of Cardinalities: GreptimeDB's Flat Format](https://greptime.cn/blogs/2025-12-22-flat-format)。
+
+### 遗留实现
+
+使用遗留 `primary_key` SST format 且为 dense primary key encoding 的 Region 仍然使用 `TimeSeriesMemtable`，它按编码后的 primary key 对数据行分组，而不是保存 flat part。如果 Region 没有 primary key 列，同一个 builder 会创建 `SimpleBulkMemtable`。两者都是为已有表保留的兼容代码，`primary_key` format 退役后可能一并移除；新的工作应面向 bulk 和 flat 路径。
+
+已经删除的 `partition_tree` memtable 不是第三种实现。Option parser 仍接受 `memtable.type=partition_tree` 以兼容旧配置，但不会恢复该实现。Region 最终使用 bulk 和 flat 路径。
+
+## 读取快照
+
+Scan 通过一次 `VersionControl` 快照同时取得 Region version 和 committed sequence，并先确定 version，再应用 sequence 上限。如果单独读取 sequence 后再获取 version，flush 或 compaction 可能在两次读取之间移除旧输入，使 scan 得到不完整的快照。
+
+选定的 version 提供 mutable memtable、immutable memtable 和 SST 文件。Mito 先按照时间范围裁剪数据源，再使用 scan 的 projection、predicate 和 sequence range 从各 memtable 获取 range。Scan 将这些 range 与 SST range 合并，并在所有数据源上应用相同的排序、删除和 merge 语义。即使新的 Region version 已经移除某个 memtable，scan 持有的引用也会让该 memtable 存活到本次读取结束。
+
+## 内存压力
+
+每个 memtable 通过 engine 的 write-buffer manager 记录估算的堆内存分配量。冻结 memtable 后，这部分内存不再计入 mutable memory，但在所有引用释放前仍计入总用量。因此，mutable memory 只反映仍可接收写入的数据，总用量仍包含活跃 scan 保留的内存。
+
+全局 write-buffer 达到限制后，worker 会选择 Region 执行 flush。如果内存用量持续超过配置限制，Mito 会阻塞写入，并在达到更高阈值后拒绝写入。可选的 Region 级限制会单独约束热点 Region，避免其阻塞无关 Region。定期任务、手动请求和 Region 生命周期操作也可以触发 flush。
+
+## 修改约束
+
+修改 memtable 代码时必须保持以下性质：
+
+- 对于使用 WAL 的 Region，先追加 WAL，再把数据安装到 memtable；只有安装成功后才能发布 committed sequence。
+- SST 文件和 manifest edit 持久化前，冻结的 memtable 必须保持可读，并能在 flush 失败后重试。
+- 从同一个 `VersionControl` 快照取得 Region version 和 committed sequence，不得先单独读取 sequence 再获取 version。
+- 保留 scan 和 flush 在 memtable range 与 SST range 之间执行统一删除、去重和 merge 所需的排序及元数据。
+- 通过 write-buffer manager 记录内存分配，并且只在底层内存不再可能被引用时释放计数。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/metric-engine.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/metric-engine.md
index c0cbf681b9..57baecf400 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/metric-engine.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/metric-engine.md
@@ -7,9 +7,9 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述
## 概述
-`Metric` 引擎是 GreptimeDB 的一个组件，属于存储引擎的一种实现，主要针对可观测 metrics 等存在大量小表的场景。
+`Metric` 引擎用于存储包含大量小型指标表的负载。
-它的主要特点是利用合成的物理宽表来存储大量的小表数据，实现相同列复用和元数据复用等效果，从而达到减少小表的存储开销以及提高列式压缩效率等目标。表这一概念在 `Metric` 引擎下变得更更加轻量。
+它将这些逻辑表映射到共享的物理宽表，使其复用列和元数据，从而降低每张表的存储开销并改善列式压缩。
## 概念
@@ -18,7 +18,7 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述
### 逻辑表
逻辑表，即用户定义的表。与普通的表都完全一样，逻辑表的定义包括表的名称、列的定义、索引的定义等。用户的查询、写入等操作都是基于逻辑表进行的。用户在使用过程中不需要关心逻辑表和普通表的区别。
-从实现层面来说，逻辑表是一个虚拟的表，它并不直接读写物理的数据，而是通过将读写请求映射成对应物理表的请求来实现数据的存储与查询。
+逻辑表是虚拟表，本身不直接存储数据。Metric 引擎将其读写请求映射为对应物理表的请求。
### 物理表
物理表是真实存储数据的表，它拥有若干个由分区规则定义的物理 Region。
@@ -27,16 +27,14 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述
`Metric` 引擎的主要设计架构如下：
-
+
-在目前版本的实现中，`Metric` 引擎复用了 `Mito` 引擎来实现物理数据的存储及查询能力，并在此之上同时提供物理表与逻辑表的访问能力。
+`Metric` 引擎将物理存储和查询交给 `Mito` 引擎。每个物理 Region 组包含一个数据 Region 和一个元数据 Region：数据 Region 保存映射到该 Region 组的逻辑表数据，元数据 Region 保存逻辑表及逻辑列的映射。
-在分区方面，逻辑表拥有与物理表完全一致的分区规则及 Region 分布。这是非常自然的，因为逻辑表的数据直接存储在物理表中，所以分区规则也是一致的。
+关联到同一物理表的逻辑表使用相同的分区布局。写入时，Metric 引擎为每行数据记录逻辑表身份；读取时，它在扫描物理 Region 前增加逻辑表过滤条件。
-在路由元数据方面，逻辑表的路由地址为逻辑地址，即该逻辑表所对应的物理表是什么，而后通过该物理表进行二次路由取得真正的物理地址。这一间接路由方式能够显著减少 `Metric` 引擎的 Region 发生迁移调度时所需要修改的元数据数量。
+逻辑表的路由只保存所属物理表的 ID，再由物理表路由解析出持有 Region 的 Datanode。逻辑路由本身不记录 peer，因此迁移物理 Region 只需改写一条物理路由，而不必改写映射到它的每一条逻辑路由。
-在操作方面，`Metric` 引擎支持对逻辑表进行标准的 DML 操作（INSERT、DELETE、SELECT）。然而，对物理表的操作进行了有限的支持以防止误操作，例如禁止直接写入物理表等操作防止影响用户逻辑表的数据。总体上可以认为物理表是对用户只读的。
+逻辑表支持普通的 INSERT、DELETE 和 SELECT 操作。直接写入物理 Region 会绕过逻辑表映射，因此会被拒绝；物理表仍然可以查询。
-为了提升对大量表同时进行 DDL（Data Definition Language，数据操作语言）操作时性能，如 Prometheus Remote Write 冷启动时大量 metrics 带来的自动建表请求，以及前面提到的迁移物理 Region 时大量路由表的修改请求等，`Metric` 引擎引入了一些批量 DDL 操作。这些批量 DDL 操作能够将大量的 DDL 操作合并成一个请求，从而减少了元数据的查询及修改次数，提升了性能。
-
-除了物理表的物理数据 Region 之外，`Metric` 引擎还额外为每一个物理数据 Region 创建了一个物理的元数据 Region，用于存储 `Metric` 引擎自身为了维护映射等状态所需要的一些元数据。这些元数据包括逻辑表与物理表的映射关系，逻辑列与物理列的映射关系等等。
+批量 DDL 用于减少大量逻辑表同时创建或更新时的元数据操作，例如 Prometheus Remote Write 自动建表或物理 Region 迁移。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/overview.md
index 3f44a4543b..c7684eb165 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/overview.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/overview.md
@@ -7,22 +7,26 @@ description: 介绍了 Datanode 的主要职责和组件，包括 gRPC 服务、
## Introduction
-`Datanode` 主要的职责是为 GreptimeDB 存储数据，我们知道在 GreptimeDB 中一个 `table` 可以有一个或者多个 `Region`,
-而 `Datanode` 的职责便是管理这些 `Region` 的读写。`Datanode` 不感知 `table`，可以认为它是一个 `region server`。
-所以 `Frontend` 和 `Metasrv` 按照 `Region` 粒度来操作 `Datanode`。
+Datanode 存储并处理 Region 数据。一张表可以包含多个 Region，但 Datanode 不负责表级路由。Frontend 按 Region 发送数据请求，Metasrv 则控制 Region 的放置和生命周期。
-
+这个边界使同一个 Region server 可以承载不同的存储引擎，而不向 Frontend 或 Metasrv 暴露引擎实现。
+
+
## Components
-一个 datanode 包含了 region server 所需的全部组件。这里列出了比较重要的部分：
-
-- 一个 gRPC 服务来提供对 `Region` 数据的读写，`Frontend` 便是使用这个服务来从 `Datanode` 读写数据。
-- 一个 HTTP 服务，可以通过它来获得当前节点的 metrics、配置信息等
-- `Heartbeat Task` 用来向 `Metasrv` 发送心跳，心跳在 GreptimeDB 的分布式架构中发挥着至关重要的作用，
- 是分布式协调和调度的基础通信通道，心跳的上行消息中包含了重要信息比如 `Region` 的负载，如果 `Metasrv` 做出了调度
- 决定（比如 Region 转移），它会通过心跳的下行消息发送指令到 `Datanode`
-- `Datanode` 不负责解析用户 SQL 或进行分布式规划，用户对一个或多个 `Table` 的查询请求会在 `Frontend` 中被转换为
- `Region` 查询请求，`Datanode` 负责用本地 query engine 执行这些 `Region` 查询计划
-- 一个 `Region Manager` 用来管理 `Datanode` 上的所有 `Region`s
-- GreptimeDB 支持可插拔的多引擎架构，目前已有的 engine 包括 `File Engine` 和 `Mito Engine`
+Datanode 包含以下主要组件：
+
+- Region server 记录已打开的 Region，并把读写和生命周期请求分发给该 Region 注册的 engine。
+- `Mito` 是主要的时序 Region engine。`Metric` 将多个逻辑指标 Region 映射到共享的 Mito Region，`File` 通过 Region 接口访问外部文件。
+- 本地 query engine 执行 Region 查询计划。它不解析客户端 SQL，也不进行集群级规划。
+- Heartbeat task 向 Metasrv 上报节点和 Region 状态，并接收 open、close、upgrade、downgrade 和迁移步骤等指令。
+- gRPC 承载发往 Datanode 的 Region 请求；HTTP 提供 metrics 和配置等节点诊断信息。
+
+## Region 请求生命周期
+
+Mito 写入到达 Region server 后，Region server 根据 Region 元数据选择 Mito。Mito 将 mutation 追加到 WAL，写入 memtable，并在之后把 memtable flush 为 SST 文件。Metric 写入会先补充逻辑表标识，再委托给对应的物理 Mito Region。
+
+读取时，本地 query engine 在 Region engine 提供的 table provider 上执行 Region 计划。Mito scan 获取不可变的 Region version，读取相关 memtable 和 SST 文件，合并并去重数据，最后返回 Arrow record batch 流。
+
+Region 所有权可以在不重启 Datanode 的情况下改变。Metasrv 通过心跳流下发生命周期指令；Region server 将指令应用到对应 engine，并在后续心跳中上报新的 Region role 和统计信息。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/python-scripts.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/python-scripts.md
deleted file mode 100644
index 831278e80d..0000000000
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/python-scripts.md
+++ /dev/null
@@ -1,30 +0,0 @@
----
-keywords: [Python 脚本, 数据分析, CPython, RustPython]
-description: 介绍了在 GreptimeDB 中使用 Python 脚本进行数据分析的两种后端实现：CPython 和嵌入式 RustPython 解释器。
----
-
-# Python 脚本
-
-## 简介
-
-Python 脚本是分析本地数据库中的数据的便捷方式，
-通过将脚本直接在数据库内运行而不是从数据库拉取数据的方式，可以节省大量的数据传输时间。
-下图描述了 Python 脚本的工作原理。
-`RecordBatch`（基本上是表中的一列，带有类型和元数据）可以来自数据库中的任何地方，
-而返回的 `RecordBatch` 可以用 Python 语法注释以指示其元数据，例如类型或空。
-脚本将尽其所能将返回的对象转换为 `RecordBatch`，无论它是 Python 列表、从参数计算出的 `RecordBatch` 还是常量（它被扩展到与输入参数相同的长度）。
-
-
-
-## 两种可选的后端
-
-### CPython 后端
-
-该后端由 [PyO3](https://pyo3.rs/v0.18.1/) 提供支持，可以使用您最喜欢的 Python 库（如 NumPy、Pandas 等），并允许 Conda 管理您的 Python 环境。
-
-但是使用它也涉及一些复杂性。您必须设置正确的 Python 共享库，这可能有点棘手。一般来说，您只需要安装 `python-dev` 包。但是，如果您使用 Homebrew 在 macOS 上安装 Python，则必须创建一个适当的软链接到 `Library/Frameworks/Python.framework`。有关使用 PyO3 crate 与不同 Python 版本的详细说明，请参见 [这里](https://pyo3.rs/v0.18.1/building_and_distribution#configuring-the-python-version)
-
-### 嵌入式 RustPython 解释器
-
-可以运行脚本的实验性 [python 解释器](https://github.com/RustPython/RustPython)，它支持 Python 3.10 语法。您可以使用所有的 Python 语法，更多信息请参见 [Python 脚本的用户指南](/user-guide/python-scripts/overview.md).
-
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/query-engine.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/query-engine.md
index 62d7563bb2..84445759d4 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/query-engine.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/query-engine.md
@@ -7,33 +7,33 @@ description: 介绍了 GreptimeDB 的查询引擎架构，基于 Apache DataFusi
## 介绍
-GreptimeDB 的查询引擎是基于[Apache DataFusion][1]（属于[Apache Arrow][2]的子项目）构建的，它是一个用 Rust 编写的出色的查询引擎。它提供了一整套功能齐全的组件，从逻辑计划、物理计划到执行运行时。下面将解释每个组件如何被整合在一起，以及在执行过程中它们的位置。
+GreptimeDB 的查询引擎基于 [Apache DataFusion][1]。DataFusion 提供逻辑计划、物理计划、优化器框架和执行运行时；GreptimeDB 在此基础上增加各查询语言的 planner、存储相关优化规则、自定义计划节点和分布式执行。
-
+DDL 和其他控制面操作由 statement executor 分发。Query engine 接收数据处理计划，包括 `INSERT ... SELECT` 等操作中读取输入数据的部分。
-入口点是逻辑计划，它被用作查询或执行逻辑等的通用中间表示。逻辑计划的两个主要来源是：1. 用户查询，例如通过 SQL 解析器和规划器的 SQL；2. Frontend 的分布式查询，这将在下一节中详细解释。
+## 查询生命周期
-接下来是物理计划，或称执行计划。与包含所有逻辑计划变体（除特殊扩展计划节点外）的大型枚举的逻辑计划不同，物理计划实际上是一个定义了在执行过程中调用的一组方法的特性。所有数据处理逻辑都包装在实现该特性的相应结构中。它们是对数据执行的实际操作，如聚合器 `MIN` 或 `AVG` ，以及表扫描 `SELECT ... FROM`。
+1. SQL、PromQL 或日志查询 planner 通过 catalog 解析表，并生成 DataFusion logical plan。DataFusion 不直接支持的操作由 GreptimeDB plan extension 表示。
+2. DataFusion 的 analyzer 和 optimizer rule 与 GreptimeDB rule 共同运行。这些规则规范化表达式和类型、改写时间范围操作、将 projection 和 filter 下推到 scan，并在需要时引入分布式计划节点。
+3. Physical planner 将优化后的 logical plan 转换为流式 operator。GreptimeDB 随后应用 scan 并行度、排序和分布式执行相关的 physical rule。
+4. 执行阶段通过 physical plan 拉取 Arrow record batch。存储 scan 接收 projection 和 predicate，下游 operator 消费数据流，无需先物化完整结果。
-优化阶段通过转换逻辑计划和物理计划来提高执行性能，现在全部基于规则。它也被称为“基于规则的优化”。一些规则是 DataFusion 原生的，其他一些是在 GreptimeDB 中自定义的。在未来，我们计划添加更多规则，并利用数据统计进行基于成本的优化 (CBO)。
-
-最后一个阶段"执行"是一个动词，代表从存储读取数据、进行计算并生成预期结果的过程。虽然它比之前提到的概念更抽象，但你可以简单地将它想象为执行一个 Rust 异步函数，并且它确实是一个异步流。
-
-当你想知道 SQL 是如何通过逻辑计划或物理计划中表示时，`EXPLAIN [VERBOSE] ` 是非常有用的。
+使用 [`EXPLAIN`](/reference/sql/explain.md) 查看逻辑和物理计划。`EXPLAIN ANALYZE` 还会执行计划并报告运行时指标。
## 数据表示
-GreptimeDB 使用 [Apache Arrow][2]作为内存中的数据表示格式。它是面向列的，以跨平台格式，也包含许多高性能的基础操作。这些特性使得在许多不同的环境中共享数据和实现计算逻辑变得容易。
+GreptimeDB 使用 [Apache Arrow][2] record batch 作为内存数据表示。一个 record batch 包含等长的列数组和 schema。查询 operator 交换这些 batch 组成的数据流，使 Region scan 到结果编码的执行路径保持列式处理。
## 索引
-在时序数据中，有两个重要的维度：时间戳和标签列（或者类似于关系数据库中的主键）。GreptimeDB 将数据分组到时间桶中，因此能在非常低的成本下定位和提取预期时间范围内的数据。GreptimeDB 中主要使用的持久文件格式 [Apache Parquet][3] 提供了多级索引和过滤器，使得在查询过程中很容易修剪数据。在未来，我们将更多地利用这个特性，并开发我们的分离索引来处理更复杂的用例。
+索引构建和持久化格式属于存储引擎。查询层向 scan 提供 predicate 和 projection，Mito 再利用时间范围、Parquet 统计信息和索引跳过不可能匹配的数据。参见[数据持久化与索引](./data-persistence-indexing.md)。
+
+
-## 分布式查询
+## 分布式执行
-参考 [Distributed Querying][6].
+分布式模式下，Frontend 规划集群级查询，Datanode 执行 Region 本地子计划。[`MergeScan`][6] 是两个阶段之间的边界。
-[1]: https://github.com/apache/arrow-datafusion
+[1]: https://datafusion.apache.org/
[2]: https://arrow.apache.org/
-[3]: https://parquet.apache.org
[6]: ../frontend/distributed-querying.md
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/storage-engine.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/storage-engine.md
index 67f98216fe..d3abb58cfa 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/storage-engine.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/storage-engine.md
@@ -7,7 +7,7 @@ description: 详细介绍了 GreptimeDB 的存储引擎架构、数据模型和
## 概述
-`存储引擎` 负责存储数据库的数据。Mito 是我们默认使用的存储引擎，基于 [LSMT][1]（Log-structured Merge-tree）。我们针对处理时间序列数据的场景做了很多优化，因此 mito 这个存储引擎并不适用于通用用途。
+Mito 是 GreptimeDB 的默认存储引擎，基于 [LSM tree][1]，面向时间序列负载设计，而不是通用的嵌入式存储引擎。
## 架构
下图展示了存储引擎的架构和处理数据的流程。
@@ -20,9 +20,9 @@ description: 详细介绍了 GreptimeDB 的存储引擎架构、数据模型和
- 为尚未刷盘的数据提供高持久性保证。
- 基于 `LogStore` API 实现，不关心底层存储介质。
- WAL 的日志记录可以存储在本地磁盘上，也可以存储在实现了 `LogStore` API 的远程日志服务中，例如 Kafka（remote WAL）。
-- Memtable
- - 数据首先写入 `active memtable`，又称 `mutable memtable`。
- - 当 `mutable memtable` 已满时，它将变为只读的 `immutable memtable`。
+- [Memtable](memtable.md)
+ - Mito 根据 time index 将数据行写入 mutable memtable。
+ - Flush 冻结 mutable memtable，安装一组新的 mutable memtable 以接收写入，再将冻结的 memtable 写为 SST 文件。
- SST
- SST 的全名为有序字符串表（`Sorted String Table`）。
- `immutable memtable` 刷到持久存储后形成一个 SST 文件。
@@ -100,7 +100,9 @@ Mito 会按 primary key 对行分组，并按时间排序，因此 SST 中的数
Mito 支持两种 SST 格式：`flat` 和 `primary_key`。`flat` 是新表的默认格式，适用于各种 primary key 基数，包括高基数 key。`primary_key` 是为了兼容旧表而保留的遗留格式。更多详情请参考 [SST format](/reference/sql/create.md#创建指定-sst-格式的表) 和[表设计指南](/user-guide/deployments-administration/performance-tuning/design-table.md#sst-格式)。
-
+
+
+一个 SST 可能跨越多个 compaction time window。
## 扫描裁剪
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/wal.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/wal.md
index 529adcdf50..36e0689430 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/wal.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/wal.md
@@ -9,20 +9,26 @@ description: 介绍了 GreptimeDB 的预写日志（WAL）机制，包括其命
## 介绍
-我们的存储引擎受到了日志结构合并树（Log-structured Merge Tree，LSMT）的启发。对数据的变更操作直接应用于 MemTable 而不是持久化到磁盘上的数据页，这显著提高了性能，但也带来了持久化相关的问题，特别是在 Datanode 意外崩溃时。与所有类似 LSMT 的存储引擎一样，GreptimeDB 使用预写日志（Write-Ahead Log，WAL）来确保数据被可靠地持久化，并且保证崩溃时的数据完整性。
+Mito 在将数据 flush 为 SST 文件前，先在 [memtable](memtable.md) 中缓冲写入。每个 Region 的 mutation 会先追加到预写日志（WAL），从而恢复尚未进入 SST 的数据。
-预写日志是一个仅提供追加写的文件组。所有的 INSERT 和 DELETE 操作都被转换为操作日志，然后追加到 WAL。一旦操作日志被持久化到底层文件，该操作才可以进一步应用到 MemTable。
+WAL 通过统一的 log-store 抽象访问，可以使用本地 raft-engine 或远端 Kafka。
-当数据节点重新启动时，WAL 中的操作条目将被重放，以重建正确的 MemTable 状态。
+## 写入与恢复流程
-
+正常写入遵循以下顺序：
+
+1. Region worker 分配 sequence number 和 WAL entry ID。
+2. 将 mutation 追加到 WAL。追加失败时，不会把 mutation 写入 memtable。
+3. WAL 追加成功后，Mito 将 mutation 写入 memtable，并发布新的 committed sequence。
+4. Flush 将不可变 memtable 写为 SST 文件，并持久化包含新文件和 `flushed_entry_id` 的 manifest edit。
+5. Manifest edit 持久化后，`flushed_entry_id` 及以前的 WAL entry 被标记为 obsolete；log store 可以稍后再回收物理空间。
+
+Manifest 是恢复边界。正常重新打开 Region 时，Mito 根据 manifest 重建 Region，并从 `flushed_entry_id + 1` 开始重放 WAL。Region 状态切换可以指定更晚的 replay checkpoint，但不会重放早于已持久化 flush 边界的 entry。
## 命名空间
-WAL 的命名空间用于区分来自不同 region 的条目。追加和读取操作必须提供一个命名空间。目前，region ID 被用作命名空间，因为每个 region 都有一个在数据节点重新启动时需要重构的 MemTable。
+WAL entry 按 Region 隔离，而不是按表隔离。追加和读取都需要指定 Region namespace，使单个 Region 可以独立重放或截断。本地 raft-engine 使用 Region ID 作为 namespace ID；Kafka provider 则在基于 topic 的日志中保留 Region 标识。
## 同步/异步刷盘
-默认情况下，WAL 的追加写是异步的，这意味着写入方不会等待操作日志被刷入到磁盘并持久化。这个默认设置提供了更高的性能，但在服务器意外关闭时可能会丢失数据。另一方面，同步刷新提供了更高的可靠性，但其代价是性能更低。
-
-在 v0.4 版本中，新的 region worker 架构可以使用批处理来减轻同步刷盘的开销。
+对于本地 raft-engine，`sync_write` 控制追加写是否等待日志同步到持久化存储，默认值为 `false`。异步写入延迟较低，但主机在缓冲数据同步前故障时，可能丢失最近确认的 entry。Kafka WAL 的持久性由 producer 和集群配置决定，不受这个本地选项控制。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/arrangement.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/arrangement.md
index dd3b6de090..7b35b50259 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/arrangement.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/arrangement.md
@@ -5,6 +5,8 @@ description: 描述了 Arrangement 在数据流进程中的状态存储功能，
# Arrangement
+本页介绍 Flownode 旧 streaming 模式使用的状态结构；batching 模式不使用 Arrangement。
+
Arrangement 存储数据流进程中的状态，存储 flow 的更新流（stream）以供进一步查询和更新。
Arrangement 本质上存储的是带有时间戳的键值对。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/batching_mode.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/batching_mode.md
index bec092b0af..8e2ecdb80b 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/batching_mode.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/batching_mode.md
@@ -9,13 +9,13 @@ description: Flownode 批处理模式概述，这是持续数据聚合当前使
## 概述
-`flownode` 中的批处理模式专为持续数据聚合而设计。它在离散的、微小的时间窗口上周期性地执行用户定义的 SQL 查询。这与原始的流处理模式形成对比；流处理模式现在已经废弃，在该模式下数据会在到达时即被处理。
+`flownode` 中的批处理模式专为持续数据聚合而设计。它在离散的小时间窗口上周期性执行用户定义的 SQL 查询。旧 streaming 路径则在数据到达时进行处理，目前仅为兼容已有 workload 而保留，不推荐新 workload 使用。
其核心思想是：
1. 定义一个带有 SQL 查询的 `flow`，该查询将数据从源表聚合到目标表。
2. 查询通常在时间戳列上包含一个时间窗口函数（例如 `date_bin`）。
3. 当新数据插入源表时，系统会将相应的时间窗口标记为“脏”(dirty)。
-4. 一个后台任务会周期性地唤醒，识别这些脏窗口，并为那些特定的时间范围重新运行聚合查询。
+4. 一个后台任务按自身的节奏运行，在下一次求值时取出待处理的脏窗口，并对这些时间范围重新运行聚合查询。
5. 然后将结果插入到目标表中，从而有效地更新聚合视图。
## 架构
@@ -39,15 +39,15 @@ description: Flownode 批处理模式概述，这是持续数据聚合当前使
- **状态 (`TaskState`)**: 包含任务的动态、可变状态，最重要的是 `DirtyTimeWindows`。
- **执行循环**: 任务运行一个无限循环 (`start_executing_loop`)，该循环：
1. 检查关闭信号。
- 2. 等待一个预定的时间间隔或直到被唤醒。
+ 2. 睡眠到下一次求值时间。设置了求值调度的任务睡眠到下一个调度时间点；自适应任务则按时间窗口大小和最小刷新间隔计算出的轮询间隔睡眠。
3. 基于当前的脏时间窗口集合生成一个新的查询计划 (`gen_insert_plan`)。
4. 对数据库执行查询 (`execute_logical_plan`)。
5. 清理已处理的脏窗口。
### `TaskState` 和 `DirtyTimeWindows`
-- **`TaskState`**: 此结构体跟踪 `BatchingTask` 的运行时状态。它包括 `dirty_time_windows`，这对于确定需要完成哪些操作至关重要。
-- **`DirtyTimeWindows`**: 这是一个关键的数据结构，用于跟踪自上次查询执行以来哪些时间窗口接收到了新数据。它存储一组不重叠的时间范围。当任务的执行循环运行时，它会参考此结构来构建一个 `WHERE` 子句，该子句仅过滤源表中的脏时间窗口。
+- **`TaskState`**: 此结构体跟踪 `BatchingTask` 的运行时状态，包括用于确定待处理工作的 `dirty_time_windows`。
+- **`DirtyTimeWindows`**: 此数据结构跟踪上次查询执行后接收到新数据的时间窗口，并保存一组不重叠的时间范围。执行循环根据它构造 `WHERE` 子句，只从源表选择脏窗口。
### `TimeWindowExpr`
@@ -56,15 +56,15 @@ description: Flownode 批处理模式概述，这是持续数据聚合当前使
- **求值**: 它可以接受一个时间戳并对时间窗口表达式求值，以确定该时间戳所属窗口的开始和结束。
- **窗口大小**: 它还可以从表达式中确定时间窗口的大小（持续时间）。
-这对于标记窗口为脏以及在查询源表时生成正确的过滤条件都至关重要。
+标记脏窗口和生成源表过滤条件使用同一套计算。
## 查询执行流程
以下是批处理模式下查询执行的简化分步演练：
1. **数据摄取**: 新数据被写入源表。
-2. **标记为脏**: `BatchingEngine` 收到有关新数据的通知。它使用与每个相关 flow 关联的 `TimeWindowExpr` 来确定哪些时间窗口受到新数据点的影响。然后将这些窗口添加到相应 `TaskState` 中的 `DirtyTimeWindows` 集合中。
-3. **任务唤醒**: `BatchingTask` 的执行循环被唤醒，原因可能是其周期性调度，也可能是因为它被通知有大量积压的脏窗口。
+2. **标记为脏**: `BatchingEngine` 收到有关新数据的通知。它使用与每个相关 flow 关联的 `TimeWindowExpr` 来确定哪些时间窗口受到新数据点的影响。然后将这些窗口添加到相应 `TaskState` 中的 `DirtyTimeWindows` 集合中。标记脏窗口不会唤醒任务。
+3. **下一次求值**: `BatchingTask` 的执行循环在调度时间点或自适应轮询间隔结束后进入下一次求值，取出待处理的脏窗口。
4. **计划生成**: 任务调用 `gen_insert_plan`。此方法：
- 检查 `DirtyTimeWindows`。
- 生成一系列 `OR` 连接的 `WHERE` 子句（例如 `(ts >= 't1' AND ts < 't2') OR (ts >= 't3' AND ts < 't4') ...`），覆盖所有脏窗口。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/dataflow.md
index 9d07922542..95def5e844 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/dataflow.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/dataflow.md
@@ -1,18 +1,39 @@
---
-keywords: [Dataflow, SQL 查询, 执行计划, 数据流, map, reduce]
-description: 解释了 Dataflow 模块的核心计算功能，包括 SQL 查询转换、内部执行计划、数据流的触发运行和支持的操作。
+keywords: [Flownode, batching mode, streaming mode, Dataflow, 脏时间窗口]
+description: 介绍 Flownode 如何选择并运行 batching 和旧 streaming 两条执行路径。
---
# 数据流
+Flownode 内部有两条执行路径：
+
+- **Batching mode** 是聚合和 TQL workload 的主要执行路径。它查询已经持久化的 source 数据，并将物化结果写入 sink table。
+- **Streaming mode** 是为兼容已有 workload 而保留的旧执行路径，不推荐新 workload 使用。Frontend 会把新到达的行同步给它进行增量处理。
+
+用户不能直接选择执行模式。创建 Flow 时，GreptimeDB 根据查询和 source table 的属性选择执行路径。聚合、`DISTINCT` 和 TQL 查询使用 batching mode；简单的非聚合查询，以及任何 source table 使用 `ttl = 'instant'` 的 Flow，目前仍使用 streaming mode。如果 source table 尚不存在并选择延迟创建，Flow 会先成为 pending batching Flow。
+
+## Batching mode
+
+Batching mode 复用 GreptimeDB 的查询引擎，不需要为每一行输入维护一张算子图。对于基于时间窗口的 Flow，主循环如下：
+
+1. Source table 收到写入后，把受影响的时间窗口标记为 dirty。
+2. `BatchingTask` 按求值调度或自适应轮询节奏运行，并在该次求值时收集待处理的 dirty window。标记 dirty window 不会唤醒任务。
+3. 任务把这些窗口转换成时间谓词，加入 Flow 查询，再请求 Frontend 查询 source table。
+4. 查询结果写入 sink table，更新已重新计算窗口对应的物化结果。
+5. 成功处理的窗口从 dirty set 中移除；执行失败的工作仍可在后续调度中处理。
+
+设置了 evaluation interval、但查询中没有时间窗口表达式的 Flow，会在每次调度时执行完整查询。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见 [Flownode 批处理模式开发者指南](./batching_mode.md)。
+
+## Streaming mode
+
Dataflow 模块（参见 `flow::compute` 模块）是 `flow` 的核心计算模块。
它接收 SQL 查询并将其转换为 `flow` 的内部执行计划。
然后，该执行计划被转化为实际的数据流，而数据流本质上是一个由带有输入和输出端口的函数组成的有向无环图（DAG）。
-数据流会在需要时被触发运行。
+新到达的行变更会增量驱动这张图执行。
-目前该数据流只支持 `map`和 `reduce` 操作，未来将添加对 `join` 等操作的支持。
+Renderer 支持 map/filter/project 和 reduce 操作。执行计划中已经有 join 和 union 节点，但 streaming renderer 尚未实现它们。
在内部，数据流使用 `tuple(row, time, diff)` 以行格式处理数据。
这里 `row` 表示实际传递的数据，可能包含多个 `value` 对象。
`time` 是系统时间，用于跟踪数据流的进度，`diff` 通常表示行的插入或删除（+1 或 -1）。
-因此，`tuple` 表示给定系统时间的 `row` 的插入/删除操作。
+因此，`tuple` 表示给定系统时间的 `row` 的插入/删除操作。有状态算子通过 [Arrangement](./arrangement.md) 保存这些变更的索引 trace。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/distributed-querying.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/distributed-querying.md
index 612174662b..e6d0c35160 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/distributed-querying.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/distributed-querying.md
@@ -1,41 +1,20 @@
---
-keywords: [分布式查询, 查询拆分, 查询合并, TableScan, 物理计划]
-description: 介绍 GreptimeDB 中的分布式查询方法，包括查询的拆分和合并过程，以及 TableScan 节点的作用。
+keywords: [分布式查询, 逻辑计划, MergeScan, Substrait, Region 裁剪]
+description: 介绍 GreptimeDB 如何把逻辑查询计划划分为 Frontend 和 Datanode 上的执行任务。
---
# 分布式查询
-我们知道在 GreptimeDB 中数据是如何分布的（参见“[表分片][1]”），那么如何查询呢？在 GreptimeDB 中，分布式查询非常简单。简单来说，我们只需将查询拆分为子查询，每个子查询负责查询表数据的一个部分，然后将所有结果合并为最终结果。这是一种典型的“拆分 - 合并”方法。具体来说，让我们从查询到达 `frontend` 开始。
+Frontend 和 Datanode 使用同一套基于 DataFusion 的查询引擎。在分布式模式下，Frontend 会增加一个规划步骤，将 Datanode 上执行的工作与 Frontend 上完成的工作分开。
-当查询到达 `frontend` 时，它首先被解析为 SQL 抽象语法树（AST）。我们遍历 AST，并从中生成逻辑计划。顾名思义，逻辑计划只是如何“逻辑地”执行查询的“提示”，它不能被直接运行，因此我们进一步从中生成可执行的物理计划。物理计划是一种类似树形的数据结构，每个节点实际上表示查询的执行方法。一旦我们从上到下运行物理计划树，结果数据将从叶子到根流动，被合并或计算。最终，我们在根节点的输出处得到了查询的结果。
+
-到目前为止，这只是一个典型的“volcano”查询执行模型，你可以在几乎每个 SQL 数据库中看到这种模型。那么“分布式”是在哪里发生的呢？这全部发生在一个名为“TableScan”的物理计划节点中。TableScan 是物理计划树中的一个叶子节点，它负责扫描表的数据（就像它的名称所暗示的）。当 `frontend` 即将扫描表时，它首先需要根据每个 `region` 的数据范围将表扫描拆分为较小的扫描。
+## 分布式规划
-[1]: ./table-sharding.md
+分布式规划器重写逻辑计划，把可以下推的算子移向表扫描，并用 `MergeScan` 节点包装远端子计划。分区列上的谓词还会在任务调度前用于裁剪 Region。
-表的所有 `region` 都有它们存储数据的范围。以下表为例：
+算子能否下推取决于计划形态和算子本身的性质。不支持的部分会保留在 Frontend。初始设计及交换律规则参见[分布式规划器 RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md)。
-```sql
-CREATE TABLE my_table (
- a INT,
- others STRING,
- ts TIMESTAMP TIME INDEX,
-)
-PARTITION ON COLUMNS (a) (
- a < 10,
- a >= 10 AND a < 20,
- a >= 20
-);
-```
+## 分布式计划
-`my_table` 表创建时被设定了 3 个分区。在 GreptimeDB 的当前实现中，将为该表创建 3 个 `region`（分区与 `region` 的比例为 1:1）。这 3 个区域将分别包含以下范围："[-∞, 10)", "[10, 20)" 和 "[20, +∞)"。例如，如果提供了值 "42"，我们将搜索这些范围，并找到包含该值的相应的 `region`（在此示例中为第 3 个 `region`）。
-
-对于查询，我们使用“过滤器”来查找 `region`。 "过滤器"是 "WHERE" 子句中的条件。例如，查询 `SELECT * FROM my_table WHERE a < 10 AND others = 'x'`，其“过滤器”为“a < 10 AND others = 'x'”。然后我们检查这些范围，找出包含满足过滤器条件的值的所有 `region`。
-
-> 如果某个查询没有任何过滤器，则将其视为全表扫描。
-
-找到所需的区域后，我们只需在其中组装子扫描。通过这种方式，我们将查询拆分为子查询，每个子查询都获取表数据的一部分。子查询在 `datanode` 中执行，并在 `frontend` 中等待完成。它们的结果将合并为表扫描请求的最终返回。
-
-下面这张图片总结了分布式查询执行的过程：
-
-
+远端输入是完整的逻辑子计划，并不局限于表扫描。Frontend 使用 [Substrait](https://substrait.io) 序列化子计划，再向持有相应数据的 Datanode 发送 Region 级请求。Datanode 在本地规划并执行子计划，将结果流返回 Frontend。Frontend 合并远端数据流，并执行没有下推的算子。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/overview.md
index ee48da3ce3..6befdc55a9 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/overview.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/overview.md
@@ -5,31 +5,51 @@ description: GreptimeDB Frontend 组件概述 - 为客户端请求提供服务
# Frontend
-**Frontend** 是一个无状态服务，作为 GreptimeDB 中客户端请求的入口点。它为多种数据库协议提供统一接口，并充当代理，将读写请求转发到分布式系统中的相应 Datanode。
+Frontend 是 GreptimeDB 中负责请求编排的无状态服务。Server 层负责终止协议并转换线上消息；Frontend 为这些协议处理器提供数据库行为，包括权限检查、语句执行、路由和分布式查询规划。
+
+Frontend 不存储表数据。它缓存从 Metasrv 获取的 catalog 和路由元数据；元数据发生变化时，Metasrv 通过心跳响应通知 Frontend 失效相应缓存。
## 核心功能
-- **协议支持**：支持多种数据库协议，包括 SQL、PromQL、MySQL 和 PostgreSQL。详见[协议][1]
-- **请求路由**：基于元数据将请求路由到相应的 Datanode
-- **查询分发**：将分布式查询拆分到多个节点
-- **响应聚合**：合并来自多个 Datanode 的结果
-- **认证授权**：安全和访问控制验证
+- 为支持的[协议][1]提供查询和写入行为。
+- 解析 catalog、schema、table 和 Region 路由。
+- 在执行请求前完成权限检查。
+- 规划分布式查询并合并 Datanode 返回的结果。
+- 将表级写入和删除转换为 Region 请求。
## 架构
### 关键组件
-- **协议处理器**：处理不同的数据库协议
-- **目录管理器**：缓存来自 Metasrv 的元数据以实现高效的请求路由和 Schema 校验
-- **分布式规划器**：将逻辑计划转换为分布式执行计划
-- **请求路由器**：为每个请求确定目标 Datanodes
+
+- 协议处理器将 SQL、PromQL、gRPC 写入和可观测性协议转换为 Frontend 的内部请求接口。
+- Catalog manager 和 partition manager 提供表元数据、分区规则和 Region 路由。
+- Statement executor 将查询、DML 和 DDL 分发到各自的执行路径。
+- 分布式规划器把表扫描替换为可跨 Datanode 执行的 `MergeScan` 计划。
### 请求流程
-
+不同操作会走不同的请求路径。
+
+#### 查询
+
+1. 协议处理器创建查询上下文，并完成认证和权限检查。
+2. 对应查询语言的 planner 生成逻辑计划。分布式模式下，planner 根据分区元数据选择 Region 并生成分布式计划。
+3. Frontend 将 Region 子计划发送到对应 Datanode。Datanode 在本地 Region engine 上执行，并返回 Arrow record batch 流。
+4. Frontend 执行剩余算子、合并数据流，再按客户端协议编码结果。
+
+#### 写入和删除
+
+1. Frontend 根据表 schema 校验请求。支持 schema-on-write 的协议可以先创建缺失的表或新增列，再重试写入。
+2. 分区规则把每一行分配给 Region。Frontend 为各目标 Region 构造请求，并路由到当前 Region leader。
+3. Datanode 的 Region server 将请求分发到对应的 Region engine。单机模式下，请求直接发送给内嵌的 Region server。
+
+#### DDL
+
+Statement executor 将 DDL 转换为 task。分布式模式下，Metasrv 以持久化 procedure 执行 task、更新元数据，并协调 Datanode 上的 Region 操作。单机模式复用相同的语句边界，但使用本地元数据和 procedure 实现。
### 部署
-下图是 GreptimeDB 在云上的一个典型的部署。`Frontend` 实例组成了一个集群处理来自客户端的请求：
+下图展示了 GreptimeDB 的一种云上部署。多个 Frontend 实例共同处理客户端请求：
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/table-sharding.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/table-sharding.md
index 63a8ac10a6..cf39afe27b 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/table-sharding.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/table-sharding.md
@@ -5,21 +5,17 @@ description: 介绍 GreptimeDB 中表数据的分片方法，包括分区和 Reg
# 表分片
-对于任何分布式数据库来说，数据的分片都是必不可少的。本文将描述 GreptimeDB 中的表数据如何进行分片。
+GreptimeDB 将一张表分为多个 Region。分区表达式定义每行数据属于哪个 Region，Region 路由则定义当前由哪个 Datanode 持有该 Region。
## 分区
-有关创建分区表的语法，请参阅用户指南中的[表分片](/user-guide/deployments-administration/manage-data/table-sharding.md)部分。
+分区是由一个或多个列上的表达式描述的逻辑行集合。分区布局需要覆盖表的输入域，使每一行都能找到唯一的目标 Region。SQL 语法和支持的表达式参见[表分片](/user-guide/deployments-administration/manage-data/table-sharding.md)。
## Region
-在创建分区后，表中的数据被逻辑上分割。你可能会问："在 GreptimeDB 中，被逻辑上分区的数据是如何存储的？" 答案是保存在 `Region` 当中。
-
-每个 `Region` 对应一个分区，并保存分区的数据。所有的 `Region` 分布在各个 `Datanode` 之中。
-`Metasrv` 管理 `Region` 到 `Datanode` 的路由信息。如果建表后需要调整分区布局，
-GreptimeDB 支持通过显式的 [repartition](/user-guide/deployments-administration/manage-data/repartition.md) 操作拆分或合并分区。
+每个分区对应一个 Region。Region ID 是 Frontend、Datanode 和 Metasrv 用于存储和路由的标识。同一张表的多个 Region 可以放在同一个 Datanode 上。
分区和 Region 的关系参见下图：
@@ -54,3 +50,13 @@ GreptimeDB 支持通过显式的 [repartition](/user-guide/deployments-administr
└──────────────────────────────────┘
可以放在同一个 Datanode 之中
```
+
+## 路由与剪枝
+
+写入时，Frontend 对每行数据计算分区规则，按 Region 分组，再根据路由表把 Region 请求发送到当前 leader。
+
+查询时，分布式 planner 将查询谓词与分区表达式比较，只扫描可能满足谓词的 Region。如果分区元数据缺失或无法安全解释，planner 会退化为扫描所有 Region，避免漏掉数据。
+
+## 调整分区布局
+
+[Repartition](/user-guide/deployments-administration/manage-data/repartition.md) 通过显式的 split 或 merge 调整已有布局。Metasrv 以持久化 procedure 执行变更，更新 Region 路由和分区表达式，并使旧的表路由缓存失效。Frontend 刷新到新元数据后，后续请求使用新的布局。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/getting-started.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/getting-started.md
index e5aa2ca855..999f696ed0 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/getting-started.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/getting-started.md
@@ -15,14 +15,13 @@ description: 介绍如何在本地环境中从源代码编译和运行 GreptimeD
### 构建依赖项
-- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line)（可选）
+- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line)（可选；克隆仓库需要，构建本身不需要）
- C/C++ 工具链：提供编译和链接的基本工具。在 Ubuntu 上，这可用作 `build-essential`。在其他平台上，也有类似的命令。
-- Rust nightly 工具链（[指南][1]）
- - 编译源代码
+- [Rustup][1]。仓库通过 `rust-toolchain.toml` 指定所需的 nightly 工具链。
- Protobuf（[指南][2]）
- 编译 proto 文件
- 请注意，版本需要 >= 3.15。你可以使用 `protoc --version` 检查它。
-- 机器：建议内存在 16GB 以上 或者 使用[mold](https://github.com/rui314/mold)工具以降低链接时的内存使用。
+- 机器：建议 16GB 以上内存。内存较小时，可使用 [mold](https://github.com/rui314/mold) 降低链接阶段的内存占用。
[1]:
[2]:
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/how-to/how-to-write-sdk.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/how-to/how-to-write-sdk.md
index 28ff757763..916aec36f3 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/how-to/how-to-write-sdk.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/how-to/how-to-write-sdk.md
@@ -1,21 +1,17 @@
---
keywords: [gRPC SDK, GreptimeDatabase, GreptimeRequest, GreptimeResponse, 插入请求]
-description: 介绍如何为 GreptimeDB 开发一个 gRPC SDK，包括 GreptimeDatabase 服务的定义、GreptimeRequest 和 GreptimeResponse 的结构。
+description: 介绍 GreptimeDB gRPC 写入 SDK 需要遵守的协议契约和错误处理要求。
---
# 如何为 GreptimeDB 开发一个 gRPC SDK
-GreptimeDB 的 gRPC SDK 只需要处理写请求即可。读请求是标准 SQL 或 PromQL，可以由任何 JDBC 客户端或 Prometheus
-客户端处理。这也是为什么所有的 GreptimeDB SDK 都命名为 "`greptimedb-ingester-`"。请确保你的 GreptimeDB SDK
-遵循相同的命名约定。
+GreptimeDB 的公开 gRPC SDK 是写入客户端。查询通常通过标准 SQL 或 PromQL 客户端完成。除非有单独需求，新 SDK 应聚焦写入和删除，并遵循 `greptimedb-ingester-` 命名约定。面向用户的 API 参见 [gRPC SDK 概述](/user-guide/ingest-data/for-iot/grpc-sdks/overview.md)。
## `GreptimeDatabase` 服务
-GreptimeDB 自定义了一个 gRPC 服务：`GreptimeDatabase`
-。你只需要实现这个服务即可。你可以在[这里](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto)
-找到它的 Protobuf 定义。
+从 [`GreptimeDatabase` Protobuf 定义](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto)生成客户端 stub，不要在 SDK 中手写一份 message 或 service 定义。
-`GreptimeDatabase` 有 2 个 RPC 方法：
+该 service 提供一个 unary method 和一个 client-streaming method：
```protobuf
service GreptimeDatabase {
@@ -25,13 +21,9 @@ service GreptimeDatabase {
}
```
-`Handle` 方法是一个 unary 调用：当 GreptimeDB 服务接收到一个 `GreptimeRequest` 请求后，它立刻处理该请求并返回一个相应的
-`GreptimeResponse`。
+`Handle` 对一个请求返回一个响应，是 SDK insert 和 delete API 通常使用的方法。
-`HandleRequests` 方法则是一个 "[Client Streaming RPC][3]" 方式的调用。
-它可以接受一个连续的 `GreptimeRequest` 请求流，持续地发给 GreptimeDB 服务。
-GreptimeDB 服务会在收到流中的每个请求时立刻进行处理，并最终（流结束时）返回一个总结性的 `GreptimeResponse`。
-通过 `HandleRequests`，我们可以获得一个非常高的请求吞吐量。
+`HandleRequests` 是 [client-streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc)。客户端关闭请求流后，服务端才返回累计响应。SDK 如果暴露 streaming API，需要明确这个确认边界，并将一个 stream 绑定到一个 endpoint。
### `GreptimeRequest`
@@ -51,11 +43,13 @@ message GreptimeRequest {
}
```
-`RequestHeader` 是必需，它包含了一些上下文，鉴权和其他信息。"oneof" 的字段包含了发往 GreptimeDB 服务的请求。
+客户端需要在 `RequestHeader` 中填写服务端要求的 database context 和认证信息，并且只能设置一个 request variant。
-注意我们有两种类型的插入请求，一种是以 "列" 的形式（`InsertRequests`），另一种是以 "行" 的形式（`RowInsertRequests`
-）。通常我们建议使用 "行" 的形式，因为它对于表的插入更自然，更容易使用。但是，如果需要一次插入大量列，或者有大量的 "null"
-值需要插入，那么最好使用 "列" 的形式。
+该 message 还包含供内部调用者使用的 query 和 DDL variant。公开 ingester API 不应暴露它们，因为 `GreptimeDatabase` 不返回 query result stream。
+
+GreptimeDB 同时接受行式 `RowInsertRequests` 和列式 `InsertRequests`。公开写入 API 默认使用行式请求。面向列的客户端可以使用列式请求，但转换过程中必须保持列长度一致，并保留 null、时间戳精度、数据类型和列 semantic type。
+
+删除同样区分行式和列式。SDK 只应暴露能够在不丢失类型信息的前提下完成映射的形式。
### `GreptimeResponse`
@@ -68,6 +62,18 @@ message GreptimeResponse {
}
```
-`ResponseHeader` 包含了返回值的状态码，以及错误信息（如果有的话）。"oneof" 的字段目前只有 "affected rows"。
+成功响应包含 success header 和 `affected_rows`。该值表示服务端确认的行数；关闭请求流时返回的是累计值。
+
+请求失败通过 gRPC status 返回。Trailing metadata 中的 `x-greptime-err-code` 和 `x-greptime-err-retry-hint` 在存在时分别提供 GreptimeDB error code 和 retry classification。SDK 应保留 gRPC status 并暴露这些 GreptimeDB metadata，不能用一个通用 SDK error 将其覆盖。
+
+## 重试与交付语义
+
+重试次数必须有上限，并且对调用者可见。只有错误被标记为 retryable 且 deadline 仍允许时，才能重试 unary request。Cancellation 和 deadline expiration 不应重试。
+
+响应丢失不代表服务端拒绝了写入。除非调用者的数据模型保证操作幂等，重试这类请求可能插入重复行。SDK 需要说明这一点，并在交付结果不确定时返回最终错误。
+
+不要自动重试只发送了一部分的 `HandleRequests` stream。即使客户端尚未收到累计响应，服务端也可能已经接受了部分请求。此时应关闭失败的 stream，并将不确定状态返回给调用者。
+
+Arrow Flight bulk ingestion 与 `GreptimeDatabase` RPC 应使用不同的 API。它的 batching 和 partial acceptance 需要独立的契约。
-GreptimeDB 现在有很多 SDK，你可以参考[这里](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=)获取一些示例。
+可以参考现有 [GreptimeDB ingester 仓库](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=)的公开 API 约定，但线上行为应以当前 Protobuf 定义和服务端契约为准。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/admin-api.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/admin-api.md
index 3ac3d44f3b..bd28206c6f 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/admin-api.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/admin-api.md
@@ -1,20 +1,25 @@
---
-keywords: [Admin API, 健康检查, leader 查询, 心跳检测, 维护模式]
-description: 介绍 Metasrv 的 Admin API，包括健康检查、leader 查询、心跳检测、维护模式和 Procedure Manager 控制等功能。
+keywords: [Admin API, 健康检查, leader 查询, 心跳检测, 维护模式, 恢复模式, table id sequence]
+description: 介绍 Metasrv 用于状态检查、集群控制和元数据恢复的 Admin API。
---
# Admin API
-Admin 提供了一种简单的方法来查看和管理集群信息，包括 metasrv 健康检测、metasrv leader 查询、数据节点心跳检测、维护模式和 Procedure Manager 控制。
+:::tip
+本页所有 Admin API 都监听 Metasrv 的 `HTTP_PORT`，默认值为 `4000`。
+:::
-Admin API 是一个 HTTP 服务，提供一组可以通过 HTTP 请求调用的 RESTful API。Admin API 简单、用户友好且安全。
+Admin API 通过 HTTP 提供 Metasrv 状态、集群控制和元数据恢复操作。该 API 不提供认证，且部分端点会改变集群行为或元数据分配，部署时必须通过网络策略保护 HTTP 端口。
本页介绍以下 API：
- /health
- /leader
- /heartbeat
+- /node-lease
- /maintenance
- /procedure-manager
+- /recovery
+- /sequence/table
所有这些 API 都在父资源 `/admin` 下。
@@ -22,7 +27,7 @@ Admin API 是一个 HTTP 服务，提供一组可以通过 HTTP 请求调用的
## /health HTTP 端点
-`/health` 端点接受 GET HTTP 请求，你可以使用此端点检查你的 metasrv 实例的健康状况。
+`/health` 端点接受 GET 请求。HTTP 服务运行时返回 `OK`，但不会检查当前 Metasrv 是否为 leader，也不会检查外部依赖是否可用。
### 定义
@@ -116,9 +121,17 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100'
]
```
+## /node-lease HTTP 端点
+
+`/node-lease` 返回 Metasrv 当前记录的 Datanode lease，可用于判断 Metasrv 是否仍将某个 Datanode 视为存活。
+
+```bash
+curl -X GET http://localhost:4000/admin/node-lease
+```
+
## /maintenance HTTP 端点
-集群维护模式是 GreptimeDB 中的一项安全功能，它可以临时禁用自动集群管理操作。此模式在集群升级、计划停机以及任何可能暂时影响集群稳定性的操作期间特别有用。有关更多详细信息，请参阅[集群维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。
+维护模式在升级、计划停机等操作期间临时禁用自动集群管理。它对集群的具体影响参见[集群维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。
`/maintenance` 端点支持以下 HTTP 请求：
@@ -151,3 +164,39 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100'
"status": "running"
}
```
+
+## /recovery HTTP 端点
+
+Recovery mode 控制手动修改 table ID sequence 等元数据修复端点。它只用于恢复工作，不用于常规维护。
+
+- `GET /admin/recovery/status`：查询 recovery mode 是否开启。
+- `POST /admin/recovery/enable`：开启 recovery mode。
+- `POST /admin/recovery/disable`：关闭 recovery mode。
+
+响应体格式如下：
+
+```json
+{
+ "enabled": true
+}
+```
+
+修复完成后应关闭 recovery mode。如果只是计划暂停自动集群操作，应使用[维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。
+
+## /sequence/table HTTP 端点
+
+这些端点用于检查或修复 table ID sequence：
+
+- `GET /admin/sequence/table/next-id`：返回下一个 table ID，但不执行分配。
+- `POST /admin/sequence/table/set-next-id`：推进下一个 table ID。
+
+设置 sequence 前必须开启 recovery mode。新值必须大于当前值，不能通过该 API 回退 sequence。Recovery mode 只是该 API 的前置条件，不能阻止 DDL。执行该操作时，必须遵循[管理 Table ID Sequence](/user-guide/deployments-administration/maintenance/sequence-management.md)中的完整集群操作流程。
+
+```bash
+curl -X POST \
+ -H 'Content-Type: application/json' \
+ -d '{"next_table_id": 2048}' \
+ http://localhost:4000/admin/sequence/table/set-next-id
+```
+
+该操作会影响后续新表分配到的 ID。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/overview.md
index 2e3c525baa..e9f6144f26 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/overview.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/overview.md
@@ -1,162 +1,101 @@
---
-keywords: [Metasrv, 元数据存储, 请求路由, 负载均衡, 高可用性]
-description: 介绍 Metasrv 的功能、架构和与前端的交互方式。
+keywords: [Metasrv, 元数据, 路由, Leader 选举, Procedure, 心跳]
+description: 介绍 Metasrv 提供的元数据及集群协调机制。
---
# Metasrv
-
-
## Metasrv 包含什么
-- 存储元数据（Catalog, Schema, Table, Region 等）
-- 请求路由器。它告诉前端在哪里写入和读取数据。
-- 数据节点的负载均衡，决定谁应该处理新的表创建请求，更准确地说，它做出资源分配决策。
-- 选举与高可用性，GreptimeDB 设计为 Leader-Follower 架构，只有 leader 节点可以写入，而 follower 节点可以读取，follower 节点的数量通常 >= 1，当 leader 不可用时，follower 节点需要能够快速切换为 leader。
-- 统计数据收集（通过每个节点上的心跳报告），如 CPU、负载、节点上的表数量、平均/峰值数据读写大小等，可用作分布式调度的基础。
+Metasrv 是 GreptimeDB 分布式集群中的元数据和协调服务，不参与数据读写链路。它主要负责：
+
+- 存储 Catalog、Schema、Table、Region、路由和节点元数据；
+- 为新 Region 选择 Datanode，并维护表路由；
+- 选举一个 Metasrv leader 负责协调元数据变更；
+- 通过可恢复的 Procedure 执行 DDL、Region 迁移、故障转移和重分区；
+- 通过心跳维护节点租约和 Region 统计信息；
+- 元数据变更时向 Frontend、Datanode 和 Flownode 广播缓存失效；
+- 向 Datanode 下发 Region 生命周期指令。
## 前端如何与 Metasrv 交互
-首先，请求路由器中的路由表结构如下（注意这只是逻辑结构，实际存储结构可能不同，例如端点可能有字典压缩）。
-
-```txt
- table_A
- table_name
- table_schema // 用于物理计划
- regions
- region_1
- mutate_endpoint
- select_endpoint_1, select_endpoint_2
- region_2
- mutate_endpoint
- select_endpoint_1, select_endpoint_2, select_endpoint_3
- region_xxx
- table_B
- ...
-```
+Frontend 从 Metasrv 获取表元数据和 Region 路由，并缓存在本地。修改元数据的语句会发送给 Metasrv leader；普通读写则使用缓存的路由直接访问 Datanode。
-### 创建表
+控制链路和数据链路相互分离：
-1. 前端发送 `CREATE TABLE` 请求到 Metasrv。
-2. 根据请求中包含的分区规则规划 Region 数量。
-3. 检查数据节点可用资源的全局视图（通过心跳收集）并为每个 Region 分配一个节点。
-4. 前端创建表并在成功创建后将 `Schema` 存储到 Metasrv。
+```text
+Frontend
+ |-- 元数据查询和 DDL -------------------> Metasrv leader
+ `-- Region 读写 ------------------------> Datanode
-### `Insert`
+Metasrv leader
+ |-- Region 生命周期指令 ----------------> Datanode
+ `-- 缓存失效 --------------------------> Frontend / Datanode / Flownode
-1. 前端从 Metasrv 获取指定表的路由。注意，最小的路由单元是表的路由（多个 Region），即包含该表所有 Region 的地址。
-2. 最佳实践是前端首先从本地缓存中获取路由并将请求转发到数据节点。如果路由不再有效，则数据节点有义务返回 `Invalid Route` 错误，前端重新从 Metasrv 获取最新数据并更新其缓存。路由信息不经常变化，因此，前端使用惰性策略维护缓存是足够的。
-3. 前端处理可能包含多个表和多个 Region 的一批写入，因此前端需要根据“路由表”拆分用户请求。
+Datanode
+ `-- 心跳、租约续期和 Region 统计信息 ----> Metasrv leader
+```
-### `Select`
+在稳定状态下，表路由为每个 Region 记录一个 leader peer 和零个或多个 follower peer。Leader 是写入目标；支持只读副本的部署可以把读取路由到 follower：
-1. 与 `Insert` 类似，前端首先从本地缓存中获取路由表。
-2. 与 `Insert` 不同，对于 `Select`，前端需要从路由表中提取只读节点（follower），然后根据优先级将请求分发到 leader 或 follower 节点。
-3. 前端的分布式查询引擎根据路由信息分发多个子查询任务并聚合查询结果。
+```text
+Table route
+ |-- Region 0
+ | |-- leader -> Datanode A
+ | `-- followers -> Datanode B, Datanode C
+ `-- Region 1
+ `-- leader -> Datanode D
+```
-## Metasrv 架构
+Region 迁移或故障转移会改变 peer 角色，并可能使 Region 暂时没有 leader。Frontend 刷新缓存路由后，再把后续读写发送给当前 peer。
-
+### 创建表
-## 分布式共识
+1. Frontend 向 Metasrv leader 提交 DDL 请求。
+2. Metasrv 根据分区规则确定 Region，并[为每个 Region 选择 Datanode](/contributor-guide/metasrv/selector.md)。
+3. 持久化的 Procedure 创建 Region，并写入表元数据和路由。发生 leader 切换后，Procedure 可以从已保存的状态继续执行。
+4. 元数据提交后，Metasrv 通知 Frontend 刷新相关缓存。
-如你所见，Metasrv 依赖于分布式共识，因为：
+### `Insert`
-1. 首先，Metasrv 必须选举一个 leader，数据节点只向 leader 发送心跳，我们只使用单个 Metasrv 节点接收心跳，这使得基于全局信息进行一些计算或调度变得容易且快速。至于数据节点如何连接到 leader，这由 MetaClient 决定（使用重定向，心跳请求变为 gRPC 流，使用重定向比转发更不容易出错），这对数据节点是透明的。
-2. 其次，Metasrv 必须为数据节点提供选举 API，以选举“写入”和“只读”节点，并帮助数据节点实现高可用性。
-3. 最后，`Metadata`、`Schema` 和其他数据必须在 Metasrv 上可靠且一致地存储。因此，基于共识的算法是存储它们的理想方法。
+Frontend 解析表路由，按照分区规则拆分数据行，再把各 Region 的写入发送到对应 Datanode。路由发生变化时，相关缓存会失效，Frontend 随后从 Metasrv 重新获取元数据。
-对于 Metasrv 的第一个版本，我们选择 Etcd 作为共识算法组件（Metasrv 设计时考虑适应不同的实现，甚至创建一个新的轮子），原因如下：
+### `Select`
-1. Etcd 提供了我们需要的 API，例如 `Watch`、`Election`、`KV` 等。
-2. 我们只执行两个分布式共识任务：选举（使用 `Watch` 机制）和存储（少量元数据），这两者都不需要我们定制自己的状态机，也不需要基于 raft 定制自己的状态机；少量数据也不需要多 raft 组支持。
-3. Metasrv 的初始版本使用 Etcd，使我们能够专注于 Metasrv 的功能，而不需要在分布式共识算法上花费太多精力，这提高了系统设计（避免与共识算法耦合）并有助于初期的快速开发，同时通过良好的架构设计，未来可以轻松接入优秀的共识算法实现。
+Frontend 在查询规划期间使用表和 Region 元数据。分区列上的谓词用于裁剪 Region，分布式查询引擎再把任务发送给持有这些 Region 的 Datanode。参见[分布式查询](../frontend/distributed-querying.md)。
-## 心跳管理
+## Metasrv 架构
-数据节点与 Metasrv 之间的主要通信方式是心跳请求/响应流，我们希望这是唯一的通信方式。这个想法受到 [TiKV PD](https://github.com/tikv/pd) 设计的启发，我们在 [RheaKV](https://github.com/sofastack/sofa-jraft/tree/master/jraft-rheakv/rheakv-pd) 中有实际经验。请求发送其状态，而 Metasrv 通过心跳响应发送不同的调度指令。
+主要协调路径如下：
+
+```text
+Leader election
+ |
+ v
+Metasrv leader
+├─ DDL manager -> Procedure manager
+├─ Selector -> 新 Region 的放置
+├─ Heartbeat handler chain -> 租约和 Region 统计信息
+├─ Region supervisor -> Region 迁移 Procedure
+├─ Mailbox -> 缓存失效和 Region 指令
+└─ Metadata managers -> KV backend
+```
-心跳可能携带以下数据，但这不是最终设计，我们仍在讨论和探索究竟应该收集哪些数据。
+这些机制共享元数据，但故障边界不同。进程重启可以丢弃缓存和 leader 本地状态；恢复所需的元数据和 Procedure 状态必须持久化。
-```
-service Heartbeat {
- // 心跳，心跳可能有很多内容，例如：
- // 1. 要注册到 Metasrv 并可被其他节点发现的元数据。
- // 2. 一些性能指标，例如负载、CPU 使用率等。
- // 3. 正在执行的计算任务数量。
- rpc Heartbeat(stream HeartbeatRequest) returns (stream HeartbeatResponse) {}
-}
-
-message HeartbeatRequest {
- RequestHeader header = 1;
-
- // 自身节点
- Peer peer = 2;
- // leader 节点
- bool is_leader = 3;
- // 实际报告时间间隔
- TimeInterval report_interval = 4;
- // 节点状态
- NodeStat node_stat = 5;
- // 此节点中的 Region 状态
- repeated RegionStat region_stats = 6;
- // follower 节点和状态，在 follower 节点上为空
- repeated ReplicaStat replica_stats = 7;
-}
-
-message NodeStat {
- // 此期间的读取容量单位
- uint64 rcus = 1;
- // 此期间的写入容量单位
- uint64 wcus = 2;
- // 此节点中的表数量
- uint64 table_num = 3;
- // 此节点中的 Region 数量
- uint64 region_num = 4;
-
- double cpu_usage = 5;
- double load = 6;
- // 节点中的读取磁盘 I/O
- double read_io_rate = 7;
- // 节点中的写入磁盘 I/O
- double write_io_rate = 8;
-
- // 其他
- map attrs = 100;
-}
-
-message RegionStat {
- uint64 region_id = 1;
- TableName table_name = 2;
- // 此期间的读取容量单位
- uint64 rcus = 3;
- // 此期间的写入容量单位
- uint64 wcus = 4;
- // 近似 Region 大小
- uint64 approximate_size = 5;
- // 近似行数
- uint64 approximate_rows = 6;
-
- // 其他
- map attrs = 100;
-}
-
-message ReplicaStat {
- Peer peer = 1;
- bool in_sync = 2;
- bool is_learner = 3;
-}
-```
+## 分布式共识
-## Central Nervous System (CNS)
+Metasrv 将 leader 选举与元数据存储分开。只有选出的 Metasrv leader 执行协调和元数据变更操作，其他 Metasrv 节点会把 client 引导到当前 leader。
-我们要构建一个算法系统，该系统依赖于每个节点的实时和历史心跳数据，应该做出一些更智能的调度决策并将其发送到 Metasrv 的 Autoadmin 单元，该单元分发调度决策，由数据节点本身或更可能由 PaaS 平台执行。
+Key-value backend 保存表元数据、路由、Procedure 状态以及其他必须跨 leader 切换保留的信息。Metasrv 不使用这套选举为 Datanode Region 创建读写副本；Region 可用性由心跳、Region 故障检测和故障转移 Procedure 管理。
-## 工作负载抽象
+## 心跳管理
-工作负载抽象的级别决定了 Metasrv 生成的调度策略（如资源分配）的效率和质量。
+Datanode 与 Metasrv leader 保持心跳流。心跳请求报告节点身份、租约、Region 统计信息以及放置和监控所需的其他状态；响应则携带 Region 生命周期指令、缓存失效等控制消息。
-DynamoDB 定义了 RCUs 和 WCUs（读取容量单位/写入容量单位），解释说 RCU 是一个 4KB 数据的读取请求，WCU 是一个 1KB 数据的写入请求。当使用 RCU 和 WCU 描述工作负载时，更容易实现性能可测量性并获得更有信息量的资源预分配，因为我们可以将不同的硬件能力抽象为 RCU 和 WCU 的组合。
+心跳驱动两套相互独立的机制，调整心跳周期会同时影响两者：
-然而，GreptimeDB 面临比 DynamoDB 更复杂的情况，特别是 RCU 不适合描述需要大量计算的 GreptimeDB 读取工作负载。我们正在努力解决这个问题。
+- **节点租约**：keep-lease handler 为发送心跳的 Datanode 续期。Selector 和 `/node-lease` 端点据此判断 Datanode 是否仍然存活。
+- **Region 故障检测**：Region supervisor 为每个 Region 维护一个基于心跳到达间隔的 Phi Accrual 检测器，其判定与租约是否过期无关。
+只有开启 Region 故障转移时，故障判定才会提交故障转移迁移。该功能默认关闭，并且要求使用 remote WAL，除非显式允许在本地 WAL 上执行。维护模式同样会抑制故障转移。前置条件和开启方式参见 [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md)。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/selector.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/selector.md
index 9048e0daa4..29b521212b 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/selector.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/selector.md
@@ -7,11 +7,7 @@ description: 介绍 Metasrv 中的 Selector，包括其类型和配置方法。
## 介绍
-什么是 `Selector`？顾名思义，它允许用户从给定的 `namespace` 和 `context` 中选择 `Item`s。有一个相关的 `trait`，也叫做 `Selector`，其定义可以在[这里][0]找到。
-
-[0]: https://github.com/GreptimeTeam/greptimedb/blob/main/src/meta-srv/src/selector.rs
-
-在 `Metasrv` 中存在一个特定的场景。当 `Frontend` 向 `Metasrv` 发送建表请求时，`Metasrv` 会创建一个路由表（表的创建细节不在这里赘述）。在创建路由表时，`Metasrv` 需要选择适当的 `Datanode`s，这时候就需要用到 `Selector`。
+建表时，Metasrv 使用 `Selector` 为各 Region 选择 Datanode。Selector 根据当前节点租约进行选择；部分实现还会使用 Region 统计信息。
@@ -19,22 +15,22 @@ description: 介绍 Metasrv 中的 Selector，包括其类型和配置方法。
`Metasrv` 目前提供以下几种类型的 `Selectors`:
-### LeasebasedSelector
+### LeaseBasedSelector
-`LeasebasedSelector` 从所有可用的（也就是在租约期间内）`Datanode` 中随机选择，其特点是简单和快速。
+`LeaseBasedSelector` 从租约有效的 Datanode 中随机选择。
### LoadBasedSelector
`LoadBasedSelector` 按照负载来选择，负载值则由每个 `Datanode` 上的 region 数量决定，较少的 region 表示较低的负载，`LoadBasedSelector` 优先选择低负载的 `Datanode`。
### RoundRobinSelector [默认选项]
-`RoundRobinSelector` 以轮询的方式选择 `Datanode`。在大多数情况下，这是默认的且推荐的选项。如果你不确定选择哪个，通常它就是正确的选择。
+`RoundRobinSelector` 以轮询方式选择 Datanode，是默认选项，也适用于大多数部署。
## 配置
您可以在启动 `Metasrv` 服务时通过名称配置 `Selector`。
-- LeasebasedSelector: `lease_based` 或 `LeaseBased`
+- LeaseBasedSelector: `lease_based` 或 `LeaseBased`
- LoadBasedSelector: `load_based` 或 `LoadBased`
- RoundRobinSelector: `round_robin` 或 `RoundRobin`
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/overview.md
index 9f82a72171..badd6ad62d 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/overview.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/overview.md
@@ -5,9 +5,7 @@ description: 介绍 GreptimeDB 的架构、关键概念和工作原理，包括
# 贡献者指南
-DeepWiki 对 GreptimeDB 的架构和实现进行了详细且清晰的描述，强烈推荐阅读：
-
-[https://deepwiki.com/GreptimeTeam/greptimedb](https://deepwiki.com/GreptimeTeam/greptimedb)
+本指南面向 GreptimeDB 贡献者，介绍理解内部实现所需的设计机制。从源码构建和运行参见[快速开始](/contributor-guide/getting-started.md)。提交要求（CLA、license header、代码格式，以及 PR 必须通过的检查）以源码仓库的 [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md) 为准。
## 架构
@@ -18,7 +16,13 @@ DeepWiki 对 GreptimeDB 的架构和实现进行了详细且清晰的描述，
- [frontend][1]
- [datanode][2]
- [metasrv][3]
+- [flownode][4]
[1]: /contributor-guide/frontend/overview.md
[2]: /contributor-guide/datanode/overview.md
[3]: /contributor-guide/metasrv/overview.md
+[4]: /contributor-guide/flownode/overview.md
+
+## 补充参考
+
+[DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb) 提供了自动生成的 GreptimeDB 源码导读，可用于了解不熟悉的模块。它属于辅助资料；涉及具体版本的行为时，仍应以对应源码为准。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/integration-test.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/integration-test.md
index 63ffa56bc0..767109f773 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/integration-test.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/integration-test.md
@@ -7,8 +7,14 @@ description: 介绍 GreptimeDB 的集成测试，包括测试范围和如何运
## 介绍
-集成测试使用 Rust 测试工具（`#[test]`）编写，与单元测试不同，它们被单独放置在
-[这里](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration)。
-它涵盖了涉及多个组件的场景，其中一个典型案例是与 HTTP/gRPC 相关的功能。你可以查看
-其[文档](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md)以获取更多信息。
+集成测试覆盖跨 crate 或服务边界的行为，例如 HTTP 和 gRPC 处理、分布式组件或外部存储。测试使用 Rust test harness，位于 [`tests-integration`](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration) package。
+运行命令如下：
+
+```shell
+cargo nextest run -p tests-integration
+```
+
+部分 case 依赖外部服务的环境变量或 fixture。运行前按照 package 的[准备说明](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md)配置环境。
+
+只有 crate 级测试或 Sqlness case 无法覆盖所需边界时才使用集成测试。隔离的逻辑仍放在单元测试中，便于快速复现失败。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/overview.md
index 81b6defa45..9ac8eb2ec1 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/overview.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/overview.md
@@ -5,5 +5,13 @@ description: GreptimeDB 的测试
# 测试
-我们的团队进行了大量测试，以确保 GreptimeDB 的行为。本章将介绍几种用于测试 GreptimeDB 的重要方法，以及如何使用它们。
+选择能够覆盖本次改动的最小测试范围：
+| 测试类型 | 适用场景 | 常用命令 |
+| --- | --- | --- |
+| [单元测试](unit-test.md) | 单个 crate 或组件内的逻辑 | `cargo nextest run -p ` |
+| [Sqlness 测试](sqlness-test.md) | SQL、协议、planner、执行和端到端回归 | `cargo sqlness bare -t ` |
+| [集成测试](integration-test.md) | 跨组件或依赖外部服务的行为 | `cargo nextest run -p tests-integration` |
+| [兼容性测试](https://github.com/GreptimeTeam/greptimedb/blob/main/tests/compatibility/README.md) | 读取旧版本写入的数据或元数据 | `cargo sqlness compat --from-version ` |
+
+需要运行完整 Rust workspace 测试时使用 `make test`。如果改动涉及持久化元数据、WAL record、SST 文件或线上协议，并且输入可能由旧版本产生，还需要补充兼容性测试。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/sqlness-test.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/sqlness-test.md
index d29b69e835..06503a434c 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/sqlness-test.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/sqlness-test.md
@@ -7,36 +7,34 @@ description: 介绍 GreptimeDB 的 Sqlness 测试，包括测试文件类型、
## 介绍
-SQL 是 `GreptimeDB` 的一个重要用户接口。我们为它提供了一个单独的测试套件（名为 `sqlness`）。
+Sqlness 是 GreptimeDB 针对 SQL 和协议行为的端到端回归测试。每个 case 向运行中的 GreptimeDB 发送语句，并将输出与仓库中的结果文件比较。
## Sqlness 手册
### 测试文件
-Sqlness 有两种类型的文件
+每个 case 使用两类文件：
- `.sql`：测试输入，仅包含 SQL
- `.result`：预期的测试输出，包含 SQL 和其结果
-`.result` 文件是预期的执行输出。如果 `.result` 文件发生变化，意味着测试结果不同，测试可能失败。你应该检查变更日志来解决问题。
-
-你只需要在 `.sql` 文件中编写测试 SQL，然后运行测试。
+在 `.sql` 文件中编写输入，运行测试后生成或更新 `.result`。必须检查每一处结果差异，只有行为变化符合预期时才能接受。
### 组织测试案例
-输入案例的根目录是 `tests/cases`。它包含几个子目录，代表不同的测试模式。例如，`standalone/` 包含所有在 `greptimedb standalone start` 模式下运行的测试。
+输入 case 位于 `tests/cases`。第一级目录选择运行环境，例如 `standalone/` 表示使用单机 GreptimeDB。
-在第一级子目录下（例如 `cases/standalone`），你可以随意组织你的测试案例。Sqlness 会递归地遍历每个文件并运行它们。
+在环境目录内，新 case 应与它覆盖的功能放在一起。Sqlness 会递归发现 case 文件。
## 运行测试
-与其他测试不同，这个测试工具是以二进制目标形式存在的。你可以用以下命令运行它
+运行命令如下：
```shell
-cargo run --bin sqlness-runner bare
+cargo sqlness bare
```
-它会自动完成以下步骤：编译 `GreptimeDB`，启动它，抓取测试并将其发送到服务器，然后收集和比较结果。你只需要检查是否有 `.result` 文件发生变化。如果没有，恭喜你，测试通过了 🥳！
+该命令会构建并启动 GreptimeDB、执行选中的 case，再比较输出。`.result` 发生变化只是待审查的结果，不代表新输出一定正确。
### 运行特定测试
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/unit-test.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/unit-test.md
index 79c73775b4..b63b29df94 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/unit-test.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/unit-test.md
@@ -7,22 +7,26 @@ description: 介绍 GreptimeDB 的单元测试，包括如何编写、运行和
## 介绍
-单元测试嵌入在代码库中，通常放置在被测试逻辑的旁边。它们使用 Rust 的 `#[test]` 属性编写，并可以使用 `cargo nextest run` 运行。
+单元测试通常放在被测逻辑旁边，使用 Rust 的 `#[test]` 属性编写。GreptimeDB 主要使用 [`cargo-nextest`](https://nexte.st/) 运行 Rust 测试。
-GreptimeDB 代码库不支持默认的 `cargo` 测试运行器。推荐使用 [`nextest`](https://nexte.st/)。你可以通过以下命令安装它：
+安装命令如下：
```shell
cargo install cargo-nextest --locked
```
-然后运行测试（这里 `--workspace` 不是必须的）
+开发时先运行本次修改的 package：
```shell
-cargo nextest run
+cargo nextest run -p
```
-注意，如果你的 Rust 是通过 `rustup` 安装的，请确保使用 `cargo` 安装 `nextest`，而不是像 `homebrew` 这样的包管理器，否则会弄乱你的本地环境。
+可以继续使用测试名称或 nextest filter 缩小范围。影响范围较广的改动在提交前运行完整 workspace 测试：
+
+```shell
+make test
+```
## 覆盖率
-我们的持续集成（CI）作业有一个“覆盖率检查”步骤。它会报告有多少代码被单元测试覆盖。请在你的补丁中添加必要的单元测试。
+CI 会报告单元测试覆盖率。测试应覆盖本次改变的行为和可能回归的失败路径，而不是只追求覆盖率数字。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/reference/sql/create.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/reference/sql/create.md
index 5e1d660bf0..660ca1751e 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/reference/sql/create.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/reference/sql/create.md
@@ -26,7 +26,7 @@ CREATE DATABASE [IF NOT EXISTS] db_name [WITH ]
数据库也可以通过使用 `WITH` 关键字配置与 `CREATE TABLE` 语句类似的选项。数据库支持以下选项：
- `ttl` - 数据库中所有表的数据存活时间（不能设置为 `instant`）
-- `memtable.type` - 内存表类型（`time_series`、`partition_tree`）
+- `memtable.type` - memtable 类型（`bulk`、`time_series`）
- `append_mode` - 数据库中的表是否为仅追加模式（`true`/`false`）
- `merge_mode` - 合并重复行的策略（`last_row`、`last_non_null`）
- `skip_wal` - 是否为数据库中的表禁用预写日志（`'true'`/`'false'`）
@@ -74,7 +74,7 @@ CREATE DATABASE test WITH (ttl='7d');
```sql
CREATE DATABASE test WITH (
ttl='30d',
- 'memtable.type'='partition_tree',
+ 'memtable.type'='bulk',
'append_mode'='true'
);
```
@@ -156,7 +156,7 @@ GreptimeDB 提供了丰富的索引实现来加速查询，请在[索引](/user-
| `compaction.twcs.trigger_file_num` | 某个窗口内触发 compaction 的最小文件数量阈值 | 字符串值，如 '8'。只在 `compaction.type` 为 `twcs` 时可用 |
| `compaction.twcs.time_window` | Compaction 时间窗口 | 字符串值，如 '1d' 表示 1 天。该表会根据时间戳将数据分区到不同的时间窗口中。只在 `compaction.type` 为 `twcs` 时可用 |
| `compaction.twcs.max_output_file_size` | TWCS compaction 的最大输出文件大小 | 字符串值，如 '1GB'、'512MB'。设置 TWCS compaction 产生的文件的最大大小。只在 `compaction.type` 为 `twcs` 时可用 |
-| `memtable.type` | memtable 的类型 | 字符串值，支持 `time_series`，`partition_tree` |
+| `memtable.type` | memtable 类型 | 字符串值：`bulk` 或 `time_series`。未设置时，Mito 根据 SST format 选择实现；默认的 flat format 使用 `bulk`。设置 `bulk` 会强制使用 `sst_format=flat`；使用 flat SST 时，即使设置了 `time_series`，Mito 也会选择 bulk 实现。旧值 `partition_tree` 仅为兼容保留，并映射到 bulk 和 flat 路径。 |
| `append_mode` | 该表是否时 append-only 的 | 字符串值。默认值为 'false'，根据 'merge_mode' 按主键和时间戳删除重复行。设置为 'true' 可以开启 append 模式和创建 append-only 表，保留所有重复的行 |
| `merge_mode` | 合并重复行的策略 | 字符串值。只有当 `append_mode` 为 'false' 时可用。默认值为 `last_row`，保留相同主键和时间戳的最后一行。设置为 `last_non_null` 则保留相同主键和时间戳的最后一个非空字段。 |
| `sst_format` | SST 文件的格式 | 字符串值，支持 `primary_key`，`flat`。默认为 `flat`。`flat` 格式建议用于具有高基数主键的表。 |
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/user-guide/deployments-administration/configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/user-guide/deployments-administration/configuration.md
index 3324915c65..b7c37a192d 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/user-guide/deployments-administration/configuration.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/user-guide/deployments-administration/configuration.md
@@ -600,20 +600,9 @@ create_on_compaction = "auto"
apply_on_query = "auto"
mem_threshold_on_create = "64M"
intermediate_path = ""
-
-[region_engine.mito.memtable]
-type = "time_series"
```
-此外，`mito` 也提供了一个实验性质的 memtable。该 memtable 主要优化大量时间序列下的写入性能和内存占用。其查询性能可能会不如默认的 `time_series` memtable。
-
-```toml
-[region_engine.mito.memtable]
-type = "partition_tree"
-index_max_keys_per_shard = 8192
-data_freeze_threshold = 32768
-fork_dictionary_bytes = "1GiB"
-```
+Mito 根据表选项和 SST format 为每个 Region 选择 memtable 实现。`default_flat_format` 为 `true` 时，没有显式设置 `sst_format` 的 Region 使用 flat SST 和 bulk memtable。`memtable.type` 是数据库或表选项，不是 `[region_engine.mito.memtable]` 引擎配置。详见[表选项](/reference/sql/create.md#表选项)。
以下是可供使用的选项
@@ -648,7 +637,7 @@ fork_dictionary_bytes = "1GiB"
| `scan_memory_on_exhausted` | 字符串 | `fail` | 扫描内存耗尽时的行为。选项：`fail`（快速失败），`wait` 或 `wait()`（等待内存）。 |
| `min_compaction_interval` | 字符串 | `0m` | 两次 compaction 之间的最小时间间隔。设为 "0m"（默认）允许 compactions 立即运行，无限制。 |
| `schedule_compaction_after_edit` | 布尔值 | `true` | 是否允许在成功的 region edit 之后调度 compaction。 设为 `true` 是在 region edit 后调度 compaction 的必要但不充分条件，`min_compaction_interval` 等其他约束仍可能阻止 compaction 被调度。 设为 `false` 则保证 region edit 后不会调度 compaction。 |
-| `default_flat_format` | 布尔值 | `true` | 是否启用 Flat 格式作为默认 SST 格式。 |
+| `default_flat_format` | 布尔值 | `true` | 没有显式设置 `sst_format` 的 Region 是否使用 flat SST。Flat SST 使用 bulk memtable。 |
| `scan_parallelism` | 整数 | `0` | （已弃用，请使用 `max_concurrent_scan_files`）旧版扫描并发度选项。 |
| `index` | -- | -- | Mito 引擎中索引的选项。 |
| `index.aux_path` | 字符串 | `""` | 文件系统中索引的辅助目录路径，用于存储创建索引的中间文件和搜索索引的暂存文件，默认为 `{data_home}/index_intermediate`。为了向后兼容，该目录的默认名称为 `index_intermediate`。此路径包含两个子目录：- `__intm`: 用于存储创建索引时使用的中间文件。- `staging`: 用于存储搜索索引时使用的暂存文件。 |
@@ -663,10 +652,6 @@ fork_dictionary_bytes = "1GiB"
| `inverted_index.apply_on_query` | 字符串 | `auto` | 是否在查询时使用索引 - `auto`: 自动 - `disable`: 从不 |
| `inverted_index.mem_threshold_on_create` | 字符串 | `64M` | 创建索引时如果超过该内存阈值则改为使用外部排序 设置为空会关闭外排，在内存中完成所有排序 |
| `inverted_index.intermediate_path` | 字符串 | `""` | 存放外排临时文件的路径 (默认 `{data_home}/index_intermediate`). |
-| `memtable.type` | 字符串 | `time_series` | Memtable type. - `time_series`: time-series memtable - `partition_tree`: partition tree memtable (实验性功能) |
-| `memtable.index_max_keys_per_shard` | 整数 | `8192` | 一个 shard 内的主键数 只对 `partition_tree` memtable 生效 |
-| `memtable.data_freeze_threshold` | 整数 | `32768` | 一个 shard 内写缓存可容纳的最大行数 只对 `partition_tree` memtable 生效 |
-| `memtable.fork_dictionary_bytes` | 字符串 | `1GiB` | 主键字典的大小 只对 `partition_tree` memtable 生效 |
`metric` 引擎针对包含大量小表的 metrics 数据进行了优化。
diff --git a/sidebars.ts b/sidebars.ts
index 6955ab0039..0a82e3abc3 100644
--- a/sidebars.ts
+++ b/sidebars.ts
@@ -762,6 +762,7 @@ const sidebars: SidebarsConfig = {
label: 'Overview',
},
'contributor-guide/datanode/storage-engine',
+ 'contributor-guide/datanode/memtable',
'contributor-guide/datanode/query-engine',
'contributor-guide/datanode/data-persistence-indexing',
'contributor-guide/datanode/wal',
diff --git a/static/datanode-architecture.svg b/static/datanode-architecture.svg
new file mode 100644
index 0000000000..ef62bc07d1
--- /dev/null
+++ b/static/datanode-architecture.svg
@@ -0,0 +1,61 @@
+
diff --git a/static/datanode-architecture.zh.svg b/static/datanode-architecture.zh.svg
new file mode 100644
index 0000000000..07419628ea
--- /dev/null
+++ b/static/datanode-architecture.zh.svg
@@ -0,0 +1,61 @@
+
diff --git a/static/inverted-index-blob-layout.svg b/static/inverted-index-blob-layout.svg
new file mode 100644
index 0000000000..282ccb8bb8
--- /dev/null
+++ b/static/inverted-index-blob-layout.svg
@@ -0,0 +1,53 @@
+
diff --git a/static/inverted-index-blob-layout.zh.svg b/static/inverted-index-blob-layout.zh.svg
new file mode 100644
index 0000000000..55b461a778
--- /dev/null
+++ b/static/inverted-index-blob-layout.zh.svg
@@ -0,0 +1,53 @@
+
diff --git a/static/metric-engine-architecture.svg b/static/metric-engine-architecture.svg
new file mode 100644
index 0000000000..1025b1f26b
--- /dev/null
+++ b/static/metric-engine-architecture.svg
@@ -0,0 +1,51 @@
+
diff --git a/static/metric-engine-architecture.zh.svg b/static/metric-engine-architecture.zh.svg
new file mode 100644
index 0000000000..1b3e60ee63
--- /dev/null
+++ b/static/metric-engine-architecture.zh.svg
@@ -0,0 +1,51 @@
+
diff --git a/static/mito-sst-layout.svg b/static/mito-sst-layout.svg
new file mode 100644
index 0000000000..1b88d85c21
--- /dev/null
+++ b/static/mito-sst-layout.svg
@@ -0,0 +1,52 @@
+
diff --git a/static/mito-sst-layout.zh.svg b/static/mito-sst-layout.zh.svg
new file mode 100644
index 0000000000..e01d5f2911
--- /dev/null
+++ b/static/mito-sst-layout.zh.svg
@@ -0,0 +1,52 @@
+
diff --git a/static/parquet-file-layout.gif b/static/parquet-file-layout.gif
new file mode 100644
index 0000000000..b54641d175
Binary files /dev/null and b/static/parquet-file-layout.gif differ
diff --git a/static/parquet-row-group-statistics.svg b/static/parquet-row-group-statistics.svg
new file mode 100644
index 0000000000..e3850cef9d
--- /dev/null
+++ b/static/parquet-row-group-statistics.svg
@@ -0,0 +1,45 @@
+
diff --git a/static/parquet-row-group-statistics.zh.svg b/static/parquet-row-group-statistics.zh.svg
new file mode 100644
index 0000000000..780d2c8e95
--- /dev/null
+++ b/static/parquet-row-group-statistics.zh.svg
@@ -0,0 +1,45 @@
+
diff --git a/versioned_docs/version-1.0/contributor-guide/datanode/data-persistence-indexing.md b/versioned_docs/version-1.0/contributor-guide/datanode/data-persistence-indexing.md
index 9f657c5edf..13f4d85e66 100644
--- a/versioned_docs/version-1.0/contributor-guide/datanode/data-persistence-indexing.md
+++ b/versioned_docs/version-1.0/contributor-guide/datanode/data-persistence-indexing.md
@@ -5,19 +5,23 @@ description: Explanation of data persistence and indexing in GreptimeDB, includi
# Data Persistence and Indexing
-Similar to all LSMT-like storage engines, data in MemTables is persisted to durable storage, for example, the local disk file system or object storage service. GreptimeDB adopts [Apache Parquet][1] as its persistent file format.
+Like other LSM-tree storage engines, GreptimeDB persists data from memtables to durable storage such as a local filesystem or object storage. It uses [Apache Parquet][1] as the persistent file format.
## SST File Format
Parquet is an open source columnar format that provides fast data querying and has already been adopted by many projects, such as Delta Lake.
-Parquet has a hierarchical structure like "row groups-columns-data pages". Data in a Parquet file is horizontally partitioned into row groups, in which all values of the same column are stored together to form a data page. Data page is the minimal storage unit. This structure greatly improves performance.
+Parquet organizes data as row groups, column chunks, and pages. A row group contains one column chunk for each column, and each column chunk contains one or more pages. Pages are the units of encoding and compression; column chunks are the I/O units for reading selected columns.
First, clustering data by column makes file scanning more efficient, especially when only a few columns are queried, which is very common in analytical systems.
-Second, data of the same column tends to be homogeneous which helps with compression when apply techniques like dictionary and Run-Length Encoding (RLE).
+Second, values within a column tend to be similar, which improves compression with techniques such as dictionary encoding and run-length encoding (RLE).
-
+The following diagram from the Apache Parquet specification also shows the physical file layout: column chunks are stored by row group, while file metadata and its length are written in the footer.
+
+
+
+*Source: Apache Parquet [FileLayout.gif](https://github.com/apache/parquet-format/blob/master/doc/images/FileLayout.gif). Copyright 2014 The Apache Software Foundation, licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).*
## Data Persistence
@@ -27,17 +31,17 @@ When the size of data buffered in MemTables reaches that threshold, GreptimeDB w
## Indexing Data in SST Files
-Apache Parquet file format provides inherent statistics in headers of column chunks and data pages, which are used for pruning and skipping.
+Parquet stores row-group column statistics such as minimum, maximum, and null count in each column chunk's metadata. Page metadata and optional column indexes can provide finer-grained statistics.
-
+
-For example, in the above Parquet file, if you want to filter rows where `name` = `Emily`, you can easily skip row group 0 because the max value for `name` field is `Charlie`. This statistical information reduces IO operations.
+For example, a query filtering for `name` = `Emily` can skip row group 0 because the maximum `name` value is `Charlie`. This avoids reading that row group.
## Index Files
-For each SST file, GreptimeDB not only maintains an internal index but also generates a separate file to store the index structures specific to that SST file.
+When an SST has one or more configured index outputs, GreptimeDB writes them to a Puffin file associated with that SST. An SST with no applicable index does not need a Puffin file.
-The index files utilize the [Puffin][3] format, which offers significant flexibility, allowing for the storage of additional metadata and supporting a broader range of index structures.
+Puffin provides a container for index blobs and their metadata, allowing different index structures to share one file.
@@ -57,13 +61,13 @@ The inverted index enables GreptimeDB to skip data segments that do not meet que
-For instance, the query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches the regex `.*users`, and `status` matches the regex `4...`. It then scans these data segments to produce the final results that meet all conditions, significantly reducing the number of IO operations.
+The query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches `.*users`, and `status` matches `4..`. It scans only those segments before applying the remaining filters.
### Inverted Index Format
-
+
-GreptimeDB builds inverted indexes by column, with each inverted index consisting of an FST and multiple Bitmaps.
+GreptimeDB builds inverted indexes by column. Each column index contains a null bitmap, multiple posting bitmaps, and an FST. The blob footer records the offsets, sizes, and metadata needed to locate and decode the column indexes.
The FST (Finite State Transducer) enables GreptimeDB to store mappings from column values to Bitmap positions in a compact format and provides excellent search performance and supports complex search capabilities (such as regular expression matching). The Bitmaps maintain a list of data segment IDs, with each bit representing a data segment.
@@ -77,7 +81,7 @@ The number of rows in a data segment is controlled by the engine option `index.i
## Unified Data Access Layer: OpenDAL
-GreptimeDB uses [OpenDAL][2] to provide a unified data access layer, thus, the storage engine does not need to interact with different storage APIs, and data can be migrated to cloud-based storage like AWS S3 seamlessly.
+GreptimeDB uses [OpenDAL][2] to provide a common access layer for local filesystems and object stores. Changing the configured storage backend does not migrate existing data.
[1]: https://parquet.apache.org
[2]: https://github.com/datafuselabs/opendal
diff --git a/versioned_docs/version-1.0/contributor-guide/datanode/metric-engine.md b/versioned_docs/version-1.0/contributor-guide/datanode/metric-engine.md
index 064872ce14..fde3c178a3 100644
--- a/versioned_docs/version-1.0/contributor-guide/datanode/metric-engine.md
+++ b/versioned_docs/version-1.0/contributor-guide/datanode/metric-engine.md
@@ -7,9 +7,9 @@ description: Overview of the Metric engine in GreptimeDB, its concepts, architec
## Overview
-The `Metric` engine is a component of GreptimeDB, and it's an implementation of the storage engine. It mainly targets scenarios with a large number of small tables for observable metrics.
+The `Metric` engine stores workloads with many small metric tables.
-Its main feature is to use synthetic physical wide tables to store a large amount of small table data, achieving effects such as reuse of the same column and metadata. This reduces storage overhead for small tables and improves columnar compression efficiency. The concept of a table becomes even more lightweight under the `Metric` engine.
+It maps those logical tables onto shared physical wide tables so they can reuse columns and metadata. This reduces per-table storage overhead and improves columnar compression.
## Concepts
@@ -19,7 +19,7 @@ The `Metric` engine introduces two new concepts: "logical table" and "physical t
A logical table refers to user-defined tables. Just like any other ordinary table, its definition includes the name of the table, column definitions, index definitions etc. All operations such as queries or write-ins by users are based on these logical tables. Users don't need to worry about differences between logical and ordinary tables during usage.
-From an implementation standpoint, a logical table is virtual; it doesn't directly read or write physical data but maps read/write requests into corresponding requests for physical tables in order to implement data storage and querying.
+A logical table is virtual. The engine maps its read and write requests to the corresponding physical table instead of storing data for it directly.
### Physical Table
@@ -29,16 +29,14 @@ A physical table is a table that actually stores data, possessing several physic
The main design architecture of the `Metric` engine is as follows:
-
+
-In the current version implementation, the `Metric` engine reuses the `Mito` engine to achieve storage and query capabilities for physical data. It also provides access to both physical tables and logical tables simultaneously.
+The `Metric` engine delegates physical storage and queries to the `Mito` engine. Each physical Region group contains a data Region, which stores rows from its mapped logical tables, and a metadata Region, which stores the logical-table and logical-column mappings.
-Regarding partitioning, logical tables have identical partition rules and Region distribution as physical tables. This makes sense because the data of logical tables are directly stored in physical tables, so their partition rules are consistent.
+Logical tables associated with the same physical table share its partition layout. During writes, the engine records the logical table identity with each row. During reads, it adds a logical-table filter before scanning the physical Region.
-Concerning routing metadata, the routing address of a logical table is a logical address - what its corresponding physical table is - then through this physical table for secondary routing to obtain the real physical address. This indirect routing method can significantly reduce the number of metadata modifications required when Region migration scheduling occurs in Metric engines.
+A logical table's route stores only the ID of its physical table; the physical table route resolves that to the Datanodes holding the Regions. Because logical routes do not name peers, migrating a physical Region rewrites one physical route instead of every logical route that maps to it.
-Operationally speaking, The `Metric` engine supports standard DML operations (INSERT, DELETE, SELECT) on logical tables. However, it only supports limited operations on physical tables to prevent misoperations - for example, writing directly to a physical table is prohibited as it could affect user's logical table data. Generally speaking, users can consider that they have read-only access to these physical tables.
+Logical tables support normal INSERT, DELETE, and SELECT operations. Direct writes to a physical Region are rejected because they would bypass the logical-table mapping; querying a physical table remains supported.
-To improve performance during simultaneous DDL (Data Definition Language) operations on many tables, the 'Metric' engine has introduced some batch DDL operations. These batch DDL operations can merge lots of DDL actions into one request thereby reducing queries and modifications times for metadata thus enhancing performance. This feature is particularly beneficial in scenarios such as the automatic creation requests brought about by large amounts of metrics during Prometheus Remote Write cold start-up, as well as the modification requests for numerous route-tables mentioned earlier during migration of many physical regions.
-
-Apart from physical data regions belonging to physical tables, the 'Metric' engine creates an additional metadata region physically for each individual physical data region used in storing some metadata needed by itself while maintaining mapping and other states. This metadata includes the mapping relationship between logical tables and physical tables, the mapping relationship between logical columns and physical columns etc.
+Batch DDL operations reduce metadata work when many logical tables are created or updated together, such as during Prometheus Remote Write auto-creation or physical Region migration.
diff --git a/versioned_docs/version-1.0/contributor-guide/datanode/overview.md b/versioned_docs/version-1.0/contributor-guide/datanode/overview.md
index d0afe21b34..a21c112faa 100644
--- a/versioned_docs/version-1.0/contributor-guide/datanode/overview.md
+++ b/versioned_docs/version-1.0/contributor-guide/datanode/overview.md
@@ -7,28 +7,26 @@ description: Overview of Datanode in GreptimeDB, its responsibilities, component
## Introduction
-`Datanode` is mainly responsible for storing the actual data for GreptimeDB. As we know, in GreptimeDB,
-a `table` can have one or more `Region`s, and `Datanode` is responsible for managing the reading and writing
-of these `Region`s. `Datanode` is not aware of `table` and can be considered as a `region server`. Therefore,
-`Frontend` and `Metasrv` operate `Datanode` at the granularity of `Region`.
+A Datanode stores and processes Region data. A table can contain multiple Regions, but the Datanode does not own table-level routing. Frontend sends data requests by Region, while Metasrv controls Region placement and lifecycle.
-
+This boundary lets the same Region server host different storage engines without exposing their implementation to Frontend or Metasrv.
+
+
## Components
-A `Datanode` contains all the components needed for a `region server`. Here we list some of the vital parts:
-
-- A gRPC service is provided for reading and writing region data, and `Frontend` uses this service
- to read and write data from `Datanode`s.
-- An HTTP service, through which you can obtain metrics, configuration information, etc., of the current node.
-- `Heartbeat Task` is used to send heartbeat to the `Metasrv`. The heartbeat plays a crucial role in the
- distributed architecture of GreptimeDB and serves as a basic communication channel for distributed coordination.
- The upstream heartbeat messages contain important information such as the workload of a `Region`. If the
- `Metasrv `has made scheduling(such as `Region` migration) decisions, it will send instructions to the
- `Datanode` via downstream heartbeat messages.
-- The `Datanode` does not parse user SQL or perform distributed planning. The user's query requests for one or
- more `Table`s will be transformed into `Region` query requests in the `Frontend`. The `Datanode` is responsible
- for executing these `Region` query plans with its local query engine.
-- A `Region Manager` is used to manage all `Region`s on a `Datanode`.
-- GreptimeDB supports a pluggable multi-engine architecture, with existing engines including `File Engine` and
- `Mito Engine`.
+The main components are:
+
+- The Region server tracks open Regions and dispatches reads, writes, and lifecycle requests to the engine registered for each Region.
+- `Mito` is the primary time-series Region engine. `Metric` maps many logical metric Regions onto shared Mito Regions, and `File` exposes external files through the Region interface.
+- The local query engine executes Region query plans. It does not parse client SQL or perform cluster-wide planning.
+- The heartbeat task reports node and Region state to Metasrv and receives instructions such as open, close, upgrade, downgrade, and migration steps.
+- gRPC carries Region requests to the Datanode. HTTP exposes node diagnostics such as metrics and configuration.
+
+## Region Request Lifecycle
+
+For a Mito write, the Region server selects Mito from the Region metadata. Mito appends the mutation to the WAL, applies it to a memtable, and later flushes the memtable to SST files. A Metric write is first rewritten with the logical-table identity and then delegated to its physical Mito Region.
+
+For a read, the local query engine executes the Region plan against a table provider backed by the Region engine. A Mito scan takes an immutable Region version, reads the relevant memtables and SST files, merges and deduplicates rows, and returns a stream of Arrow record batches.
+
+Region ownership can change without restarting the Datanode. Metasrv sends lifecycle instructions over the heartbeat stream; the Region server applies them to the engine and reports the new Region role and statistics in subsequent heartbeats.
diff --git a/versioned_docs/version-1.0/contributor-guide/datanode/python-scripts.md b/versioned_docs/version-1.0/contributor-guide/datanode/python-scripts.md
deleted file mode 100644
index 98909142a6..0000000000
--- a/versioned_docs/version-1.0/contributor-guide/datanode/python-scripts.md
+++ /dev/null
@@ -1,35 +0,0 @@
----
-keywords: [Python scripts, data analysis, CPython backend, RustPython interpreter, RecordBatch]
-description: Guide on using Python scripts for data analysis in GreptimeDB, including backend options and setup instructions.
----
-
-# Python Scripts
-
-## Introduction
-
-Python scripts are methods for analyzing data in GreptimeDB,
-by running it in the database directly instead of fetching all the data from the database and running it locally.
-This approach saves a lot of data transfer costs.
-The image below depicts how the script works.
-The `RecordBatch` (which is basically a column in a table with type and nullability metadata)
-can come from anywhere in the database,
-and the returned `RecordBatch` can be annotated in Python grammar to indicate its metadata,
-such as type or nullability.
-The script will do its best to convert the returned object to a `RecordBatch`,
-whether it is a Python list, a `RecordBatch` computed from parameters,
-or a constant (which is extended to the same length as the input arguments).
-
-
-
-## Two optional backends
-
-### CPython Backend powered by PyO3
-
-This backend is powered by [PyO3](https://pyo3.rs/v0.18.1/), enabling the use of your favourite Python libraries (such as NumPy, Pandas, etc.) and allowing Conda to manage your Python environment.
-
-But using it also involves some complications. You must set up the correct Python shared library, which can be a bit challenging. In general, you just need to install the `python-dev` package. However, if you are using Homebrew to install Python on macOS, you must create a proper soft link to `Library/Frameworks/Python.framework`. Detailed instructions on using PyO3 crate with different Python Version can be found [here](https://pyo3.rs/v0.18.1/building_and_distribution#configuring-the-python-version)
-
-### Embedded RustPython Interpreter
-
-An experiment [python interpreter](https://github.com/RustPython/RustPython) to run
-the coprocessor script, it supports Python 3.10 grammar. You can use all the very Python syntax, see [User Guide/Python Coprocessor](/user-guide/python-scripts/overview.md) for more!
diff --git a/versioned_docs/version-1.0/contributor-guide/datanode/query-engine.md b/versioned_docs/version-1.0/contributor-guide/datanode/query-engine.md
index 5b83a30613..91f307aa59 100644
--- a/versioned_docs/version-1.0/contributor-guide/datanode/query-engine.md
+++ b/versioned_docs/version-1.0/contributor-guide/datanode/query-engine.md
@@ -7,51 +7,30 @@ description: Overview of GreptimeDB's query engine, its architecture, data repre
## Introduction
-GreptimeDB's query engine is built on [Apache DataFusion][1] (subproject under [Apache
-Arrow][2]), a brilliant query engine written in Rust. It provides a set of well functional components from
-logical plan, physical plan and the execution runtime. Below explains how each component is orchestrated and their positions during execution.
+GreptimeDB's query engine is built on [Apache DataFusion][1]. DataFusion supplies the logical and physical plan interfaces, optimizer framework, and execution runtime. GreptimeDB adds planners for its query languages, storage-aware optimizer rules, custom plan nodes, and distributed execution.
-
+DDL and other control-plane operations are dispatched by the statement executor. The query engine receives plans for data processing, including the input side of operations such as `INSERT ... SELECT`.
-The entry point is the logical plan, which is used as the general intermediate representation of a
-query or execution logic etc. Two noticeable sources of logical plan are from: 1. the user query, like
-SQL through SQL parser and planner; 2. the Frontend's distributed query, which is explained in details in the following section.
+## Query Lifecycle
-Next is the physical plan, or the execution plan. Unlike the logical plan which is a big
-enumeration containing all the logical plan variants (except the special extension plan node), the
-physical plan is in fact a trait that defines a group of methods invoked during
-execution. All data processing logics are packed in corresponding structures that
-implement the trait. They are the actual operations performed on the data, like
-aggregator `MIN` or `AVG`, and table scan `SELECT ... FROM`.
+1. The SQL, PromQL, or log-query planner resolves tables through the catalog and produces a DataFusion logical plan. GreptimeDB plan extensions represent operations that DataFusion does not provide directly.
+2. DataFusion analyzer and optimizer rules run together with GreptimeDB rules. These rules normalize expressions and types, rewrite time-range operations, push projections and filters toward scans, and introduce distributed plan nodes when required.
+3. The physical planner converts the optimized logical plan into streaming operators. GreptimeDB then applies physical rules for scan parallelism, ordering, and distributed execution.
+4. Execution pulls Arrow record batches through the physical plan. Storage scans receive the projection and predicates, and downstream operators consume the resulting stream without materializing the complete result first.
-The optimization phase which improves execution performance by transforming both logical and physical plans, is now all based on rules. It is also called, "Rule Based Optimization". Some of the rules are DataFusion native and others are customized in Greptime DB. In the future, we plan to add more
-rules and leverage the data statistics for Cost Based Optimization/CBO.
-
-The last phase "execute" is a verb, stands for the procedure that reads data from storage, performs
-calculations and generates the expected results. Although it's more abstract than previously mentioned concepts, you can just
-simply imagine it as executing a Rust async function. And it's indeed a future (stream).
-
-`EXPLAIN [VERBOSE] ` is very useful if you want to see how your SQL is represented in the logical or physical plan.
+Use [`EXPLAIN`](/reference/sql/explain.md) to inspect the logical and physical plans. `EXPLAIN ANALYZE` also executes the plan and reports runtime metrics.
## Data Representation
-GreptimeDB uses [Apache Arrow][2] as the in-memory data representation. It's column-oriented, in
-cross-platform format, and also contains many high-performance data operators. These features
-make it easy to share data in many different environments and implement calculation logic.
+GreptimeDB uses [Apache Arrow][2] record batches as its in-memory data representation. A record batch contains equal-length column arrays and a schema. Query operators exchange streams of these batches, which keeps the execution path columnar from Region scans through result encoding.
## Indexing
-In time series data, there are two important dimensions: timestamp and tag columns (or like
-primary key in a general relational database). GreptimeDB groups data in time buckets, so it's efficient
-to locate and extract data within the expected time range at a very low cost. The mainly used persistent file format [Apache Parquet][3] in GreptimeDB helps a lot -- it
-provides multi-level indices and filters that make it easy to prune data during querying. In the future, we
-will make more use of this feature, and develop our separated index to handle more complex use cases.
+Index construction and persistent index formats belong to the storage engine. The query layer supplies predicates and projections to a scan; Mito then uses time ranges, Parquet statistics, and indexes to avoid reading data that cannot match. See [Data Persistence and Indexing](./data-persistence-indexing.md).
## Distributed Execution
-Covered in [Distributed Querying][6].
+In distributed mode, the Frontend plans the cluster-wide query and Datanodes execute Region-local subplans. [`MergeScan`](../frontend/distributed-querying.md) is the boundary between those stages.
-[1]: https://github.com/apache/arrow-datafusion
+[1]: https://datafusion.apache.org/
[2]: https://arrow.apache.org/
-[3]: https://parquet.apache.org
-[6]: ../frontend/distributed-querying.md
diff --git a/versioned_docs/version-1.0/contributor-guide/datanode/storage-engine.md b/versioned_docs/version-1.0/contributor-guide/datanode/storage-engine.md
index c220c72208..b6963815b3 100644
--- a/versioned_docs/version-1.0/contributor-guide/datanode/storage-engine.md
+++ b/versioned_docs/version-1.0/contributor-guide/datanode/storage-engine.md
@@ -7,7 +7,7 @@ description: Overview of the storage engine in GreptimeDB, its architecture, com
## Introduction
-The `storage engine` is responsible for storing the data of the database. Mito, based on [LSMT][1] (Log-structured Merge-tree), is the storage engine we use by default. We have made significant optimizations for handling time-series data scenarios, so mito engine is not suitable for general purposes.
+Mito is GreptimeDB's default storage engine. It uses an [LSM tree][1] and is designed for time-series workloads rather than as a general-purpose embedded storage engine.
## Architecture
@@ -24,8 +24,8 @@ The architecture is the same as a traditional LSMT engine:
- Log records of the WAL can be stored on the local disk, or in a remote log service such as
Kafka (remote WAL) that implements the `Log Store` API.
- Memtables:
- - Data is written into the `active memtable`, aka `mutable memtable` first.
- - When a `mutable memtable` is full, it will be changed to a `read-only memtable`, aka `immutable memtable`.
+ - Mito routes rows by time index into mutable memtables.
+ - A flush freezes the mutable memtables, installs a new mutable set for writes, and writes the frozen memtables to SST files.
- SST
- The full name of SST, aka SSTable is `Sorted String Table`.
- `Immutable memtable` is flushed to persistent storage and produces an SST file.
@@ -103,7 +103,9 @@ Each Parquet SST is split into row groups, the unit that Parquet can read or ski
Mito supports two SST formats: `flat` and `primary_key`. `flat` is the default for new tables and works well across primary-key cardinalities, including high-cardinality keys. `primary_key` is the legacy format kept for compatibility with older tables. See [SST format](/reference/sql/create.md#create-a-table-with-sst-format) and the [table design guide](/user-guide/deployments-administration/performance-tuning/design-table.md#sst-format) for more details.
-
+
+
+An SST may span more than one compaction time window.
## Scan Pruning
diff --git a/versioned_docs/version-1.0/contributor-guide/datanode/wal.md b/versioned_docs/version-1.0/contributor-guide/datanode/wal.md
index 4ecb19ef02..bb6e733bae 100644
--- a/versioned_docs/version-1.0/contributor-guide/datanode/wal.md
+++ b/versioned_docs/version-1.0/contributor-guide/datanode/wal.md
@@ -7,30 +7,26 @@ description: Introduction to Write-Ahead Logging (WAL) in GreptimeDB, its purpos
## Introduction
-Our storage engine is inspired by the Log-structured Merge Tree (LSMT). Mutating operations are
-applied to a MemTable instead of persisting to disk, which significantly improves performance but
-also brings durability-related issues, especially when the Datanode crashes unexpectedly. Similar
-to all LSMT-like storage engines, GreptimeDB uses a write-ahead log (WAL) to ensure data durability
-and is safe from crashing.
+Mito buffers writes in memtables before flushing them to SST files. It first appends each Region's mutations to the write-ahead log (WAL), so data that has not reached an SST can be recovered.
-WAL is an append-only file group. All `INSERT` and `DELETE` operations are transformed into
-operation entries and then appended to WAL. Once operation entries are persisted to the underlying
-file, the operation can be further applied to MemTable.
+The WAL uses a common log-store abstraction with local raft-engine and remote Kafka providers.
-When the Datanode restarts, operation entries in WAL are replayed to reconstruct the correct
-in-memory state.
+## Write and Recovery Cycle
-
+The order of a normal write is:
+
+1. The Region worker assigns sequence numbers and a WAL entry ID.
+2. It appends the mutations to the WAL. If the append fails, the mutations are not applied to the memtable.
+3. After the append succeeds, Mito writes the mutations to the memtable and publishes the new committed sequence.
+4. A flush writes immutable SST files and persists a manifest edit containing the new files and `flushed_entry_id`.
+5. After the manifest edit is durable, WAL entries through `flushed_entry_id` are marked obsolete. The log store may reclaim them later.
+
+The manifest is the recovery boundary. On a normal reopen, Mito rebuilds the Region from the manifest and replays WAL entries starting at `flushed_entry_id + 1`. Region transitions may supply a later replay checkpoint, but they never replay entries before the persisted flush boundary.
## Namespace
-Namespace of WAL is used to separate entries from different tables (different regions). Append and
-read operations must provide a Namespace. Currently, region ID is used as the Namespace, because
-each region has a MemTable that needs to be reconstructed when Datanode restarts.
+WAL entries are isolated by Region, not by table. Each append and read identifies a Region namespace so one Region can be replayed or truncated independently. The local raft-engine provider uses the Region ID as its namespace ID. Kafka keeps Region identity within the provider's topic-backed log.
## Synchronous/Asynchronous flush
-By default, appending to WAL is asynchronous, which means the writer will not wait until entries are
-flushed to disk. This setting provides higher performance, but may lose data when running host shutdown unexpectedly. In the other hand, synchronous flush provides higher durability at the cost of performance.
-
-In v0.4 version, the new region worker architecture can use batching to alleviate the overhead of sync flush.
+For the local raft-engine provider, `sync_write` controls whether an append waits for the log to be synced to durable storage. It defaults to `false`. Asynchronous writes reduce latency but can lose recently acknowledged entries if the host fails before buffered data is synced. Kafka WAL durability is controlled by its producer and cluster settings instead of this local option.
diff --git a/versioned_docs/version-1.0/contributor-guide/flownode/arrangement.md b/versioned_docs/version-1.0/contributor-guide/flownode/arrangement.md
index aed75af777..8b472ea316 100644
--- a/versioned_docs/version-1.0/contributor-guide/flownode/arrangement.md
+++ b/versioned_docs/version-1.0/contributor-guide/flownode/arrangement.md
@@ -5,6 +5,8 @@ description: Details on the arrangement component in Flownode, which stores stat
# Arrangement
+This page describes state used by Flownode's legacy streaming mode. Batching mode does not use an Arrangement.
+
Arrangement stores the state in the dataflow's process. It stores the streams of update flows for further querying and updating.
The arrangement essentially stores key-value pairs with timestamps to mark their change time.
diff --git a/versioned_docs/version-1.0/contributor-guide/flownode/batching_mode.md b/versioned_docs/version-1.0/contributor-guide/flownode/batching_mode.md
index df6cd15cf8..d90dc87bfd 100644
--- a/versioned_docs/version-1.0/contributor-guide/flownode/batching_mode.md
+++ b/versioned_docs/version-1.0/contributor-guide/flownode/batching_mode.md
@@ -9,13 +9,13 @@ This guide provides a brief overview of the batching mode in `flownode`. It's in
## Overview
-The batching mode in `flownode` is designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. This is in contrast to a streaming mode where data is processed as it arrives.
+The batching mode in `flownode` is designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. This is in contrast to the legacy streaming path, which processes data as it arrives and is retained for compatibility but deprecated for new workloads.
The core idea is to:
1. Define a `flow` with a SQL query that aggregates data from a source table into a sink table.
2. The query typically includes a time window function (e.g., `date_bin`) on a timestamp column.
3. When new data is inserted into the source table, the system marks the corresponding time windows as "dirty."
-4. A background task periodically wakes up, identifies these dirty windows, and re-runs the aggregation query for those specific time ranges.
+4. A background task runs on its own cadence, consumes the pending dirty windows at its next evaluation, and re-runs the aggregation query for those time ranges.
5. The results are then inserted into the sink table, effectively updating the aggregated view.
## Architecture
@@ -39,15 +39,15 @@ A `BatchingTask` represents a single, independent data flow. Each task is associ
- **State (`TaskState`)**: This contains the dynamic, mutable state of the task, most importantly the `DirtyTimeWindows`.
- **Execution Loop**: The task runs an infinite loop (`start_executing_loop`) that:
1. Checks for a shutdown signal.
- 2. Waits for a scheduled interval or until it's woken up.
+ 2. Sleeps until its next evaluation time. A task with an evaluation schedule sleeps until the next scheduled time; an adaptive task sleeps for a polling interval derived from the time window size and the minimum refresh duration.
3. Generates a new query plan (`gen_insert_plan`) based on the current set of dirty time windows.
4. Executes the query (`execute_logical_plan`) against the database.
5. Cleans up the processed dirty windows.
### `TaskState` and `DirtyTimeWindows`
-- **`TaskState`**: This struct tracks the runtime state of a `BatchingTask`. It includes `dirty_time_windows`, which is crucial for determining what work needs to be done.
-- **`DirtyTimeWindows`**: This is a key data structure that keeps track of which time windows have received new data since the last query execution. It stores a set of non-overlapping time ranges. When a task's execution loop runs, it consults this structure to build a `WHERE` clause that filters the source table for only the dirty time windows.
+- **`TaskState`**: This struct tracks the runtime state of a `BatchingTask`, including the `dirty_time_windows` that determine its pending work.
+- **`DirtyTimeWindows`**: This data structure tracks which time windows have received new data since the last query execution. It stores a set of non-overlapping time ranges. The execution loop uses it to build a `WHERE` clause that selects only the dirty windows from the source table.
### `TimeWindowExpr`
@@ -56,15 +56,15 @@ The `TimeWindowExpr` is a helper utility for dealing with time window expression
- **Evaluation**: It can take a timestamp and evaluate the time window expression to determine the start and end of the window that the timestamp falls into.
- **Window Size**: It can also determine the size (duration) of the time window from the expression.
-This is essential for both marking windows as dirty and for generating the correct filter conditions when querying the source table.
+The same calculation is used to mark dirty windows and generate the source-table filters.
## Query Execution Flow
Here's a simplified step-by-step walkthrough of how a query is executed in batch mode:
1. **Data Ingestion**: New data is written to a source table.
-2. **Marking Dirty**: The `BatchingEngine` receives a notification about the new data. It uses the `TimeWindowExpr` associated with each relevant flow to determine which time windows are affected by the new data points. These windows are then added to the `DirtyTimeWindows` set in the corresponding `TaskState`.
-3. **Task Wake-up**: The `BatchingTask`'s execution loop wakes up, either due to its periodic schedule or because it was notified of a large backlog of dirty windows.
+2. **Marking Dirty**: The `BatchingEngine` receives a notification about the new data. It uses the `TimeWindowExpr` associated with each relevant flow to determine which time windows are affected by the new data points. These windows are then added to the `DirtyTimeWindows` set in the corresponding `TaskState`. Marking a window dirty does not wake the task.
+3. **Next Evaluation**: The `BatchingTask`'s execution loop reaches its next evaluation, either at a scheduled time or after its adaptive polling interval, and consumes the pending dirty windows.
4. **Plan Generation**: The task calls `gen_insert_plan`. This method:
- Inspects the `DirtyTimeWindows`.
- Generates a series of `OR`'d `WHERE` clauses (e.g., `(ts >= 't1' AND ts < 't2') OR (ts >= 't3' AND ts < 't4') ...`) that cover the dirty windows.
diff --git a/versioned_docs/version-1.0/contributor-guide/flownode/dataflow.md b/versioned_docs/version-1.0/contributor-guide/flownode/dataflow.md
index 000a65edb3..8000f14166 100644
--- a/versioned_docs/version-1.0/contributor-guide/flownode/dataflow.md
+++ b/versioned_docs/version-1.0/contributor-guide/flownode/dataflow.md
@@ -1,17 +1,38 @@
---
-keywords: [dataflow module, SQL query transformation, execution plan, DAG, map and reduce operations]
-description: Explanation of the dataflow module in Flownode, its operations, internal data handling, and future enhancements.
+keywords: [Flownode, batching mode, streaming mode, dataflow, dirty time windows]
+description: How Flownode selects and runs its batching and legacy streaming execution paths.
---
# Dataflow
+Flownode has two internal execution paths:
+
+- **Batching mode** is the primary path for aggregation and TQL workloads. It evaluates queries over persisted source data and writes materialized results to a sink table.
+- **Streaming mode** is the legacy path retained for compatibility and deprecated for new workloads. It incrementally processes rows mirrored from Frontend as they arrive.
+
+Users do not select the mode directly. When a Flow is created, GreptimeDB chooses the path from the query and source-table properties. Aggregation, `DISTINCT`, and TQL queries use batching mode. Simple non-aggregation queries, and any Flow whose source table has `ttl = 'instant'`, currently use streaming mode.
+
+## Batching mode
+
+Batching mode reuses GreptimeDB's query engine instead of maintaining an operator graph for every incoming row. For a time-windowed Flow, its main loop is:
+
+1. A source-table write marks the affected time windows as dirty.
+2. A `BatchingTask` runs on its evaluation schedule or adaptive polling cadence and collects the pending dirty windows at that evaluation. Marking a window dirty does not wake the task.
+3. The task adds time predicates for those windows to the Flow query and asks Frontend to execute it against the source tables.
+4. The query result is inserted into the sink table, updating the materialized result for windows that were evaluated.
+5. Successfully processed windows are removed from the dirty set. Failed work remains available for a later evaluation.
+
+Flows with an evaluation interval but without a time-window expression run the complete query on each scheduled evaluation. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Flownode Batching Mode Developer Guide](./batching_mode.md) for the task and dirty-window components.
+
+## Streaming mode
+
The `dataflow` module (see `flow::compute` module) is the core computing module of `flow`.
It takes a SQL query and transforms it into flow's internal execution plan.
This execution plan is then rendered into an actual dataflow, which is essentially a directed acyclic graph (DAG) of functions with input and output ports.
-The dataflow is triggered to run when needed.
+New row changes drive the graph incrementally.
-Currently, this dataflow only supports `map` and `reduce` operations. Support for `join` operations will be added in the future.
+The renderer supports map/filter/project and reduce operations. Join and union plan nodes exist, but their streaming renderers are not implemented.
Internally, the dataflow handles data in row format, using a tuple `(row, time, diff)`. Here, `row` represents the actual data being passed, which may contain multiple `Value` objects.
`time` is the system time which tracks the progress of the dataflow, and `diff` typically represents the insertion or deletion of the row (+1 or -1).
-Therefore, the tuple represents the insert/delete operation of the `row` at a given system `time`.
\ No newline at end of file
+Therefore, the tuple represents the insert/delete operation of the `row` at a given system `time`. Stateful operators keep indexed traces of these changes in an [Arrangement](./arrangement.md).
diff --git a/versioned_docs/version-1.0/contributor-guide/frontend/distributed-querying.md b/versioned_docs/version-1.0/contributor-guide/frontend/distributed-querying.md
index 21ee07d7e8..ca3822e113 100644
--- a/versioned_docs/version-1.0/contributor-guide/frontend/distributed-querying.md
+++ b/versioned_docs/version-1.0/contributor-guide/frontend/distributed-querying.md
@@ -5,29 +5,16 @@ description: Describes the process of distributed querying in GreptimeDB, focusi
# Distributed Querying
-Most steps of querying in frontend and datanode are identical. The only difference is that
-Frontend have a "special" step in planning phase to make the logical query plan distributed.
-Let's reference it as "dist planner" in the following text.
-
-The modified, distributed logical plan has multiple stages, each of them is executed in different
-server node.
+Frontend and Datanode use the same DataFusion-based query engine. In distributed mode, Frontend adds a planning step that separates work performed by Datanodes from work completed by Frontend.
## Dist Planner
-Planner will traverse the input logical plan, and split it into multiple stages by the "[commutativity
-rule](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md)".
+The distributed planner rewrites the logical plan. It pushes compatible operators toward table scans and wraps remote subplans in `MergeScan` nodes. Partition predicates are also used to prune Regions before the remote work is scheduled.
-This rule is under heavy development. At present it will consider things like:
-- whether the operator itself is commutative
-- how the partition rule is configured
-- etc...
+Whether an operator can be pushed down depends on the plan shape and the operator's properties. Unsupported parts remain on Frontend. The original design and its commutativity rules are described in the [distributed planner RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md).
## Dist Plan
-Except the first stage, which have to read data from files in storage. All other stages' leaf node
-are actually a gRPC call to its previous stage.
-
-Sub-plan in a stage is itself a complete logical plan, and can be executed independently without
-the follow up stages. The plan is encoded in [substrait format](https://substrait.io).
+A remote input is a complete logical subplan, not just a table scan. Frontend serializes the subplan in [Substrait](https://substrait.io) format and sends a Region-specific request to the Datanode that owns the data. The Datanode plans and executes it locally, then streams the result back. Frontend merges the remote streams and executes any operators that were not pushed down.
diff --git a/versioned_docs/version-1.0/contributor-guide/frontend/overview.md b/versioned_docs/version-1.0/contributor-guide/frontend/overview.md
index 3a2f63e7ae..18cabfc2b8 100644
--- a/versioned_docs/version-1.0/contributor-guide/frontend/overview.md
+++ b/versioned_docs/version-1.0/contributor-guide/frontend/overview.md
@@ -5,27 +5,47 @@ description: Overview of GreptimeDB's Frontend component - a stateless proxy ser
# Frontend
-The **Frontend** is a stateless service that serves as the entry point for client requests in GreptimeDB. It provides a unified interface for multiple database protocols and acts as a proxy that forwards read/write requests to appropriate Datanodes in the distributed system.
+Frontend is GreptimeDB's stateless request-orchestration service. The server layer terminates protocols and converts wire messages; Frontend supplies the database behavior behind those handlers, including permission checks, statement execution, routing, and distributed query planning.
+
+Frontend does not store table data. It caches catalog and route metadata obtained from Metasrv, and Metasrv invalidates those caches through heartbeat responses when metadata changes.
## Core Functions
-- **Protocol Support**: Multiple database protocols including SQL, PromQL, MySQL, and PostgreSQL. See [Protocols][1] for details
-- **Request Routing**: Routes requests to appropriate Datanodes based on metadata
-- **Query Distribution**: Splits distributed queries across multiple nodes
-- **Response Aggregation**: Combines results from multiple Datanodes
-- **Authorization**: Security and access control validation
+- Provide query and ingestion behavior for the supported [protocols][1].
+- Resolve catalogs, schemas, tables, and Region routes.
+- Validate permissions before executing a request.
+- Plan distributed queries and merge results from Datanodes.
+- Convert table-level writes and deletes into Region requests.
## Architecture
### Key Components
-- **Protocol Handlers**: Handle different database protocols
-- **Catalog Manager**: Caches metadata from Metasrv to enable efficient request routing and schema validation
-- **Dist Planner**: Converts logical plans to distributed execution plans
-- **Request Router**: Determines target Datanodes for each request
+
+- Protocol handlers adapt SQL, PromQL, gRPC ingestion, and observability protocols to Frontend's internal request interfaces.
+- The catalog and partition managers provide table metadata, partition rules, and Region routes.
+- The statement executor dispatches queries, DML, and DDL to their respective execution paths.
+- The distributed planner replaces table scans with `MergeScan` plans that can run across Datanodes.
### Request Flow
-
+The request path depends on the operation.
+
+#### Queries
+
+1. A protocol handler creates the query context and performs authentication and permission checks.
+2. The language-specific planner produces a logical plan. In distributed mode, the planner uses partition metadata to select Regions and constructs a distributed plan.
+3. Frontend sends Region subplans to the owning Datanodes. Datanodes execute them against local Region engines and return streams of Arrow record batches.
+4. Frontend runs the remaining operators, merges the streams, and formats the result for the client protocol.
+
+#### Writes and deletes
+
+1. Frontend validates the request against the table schema. Protocols that support schema-on-write may create a missing table or add columns before retrying the write.
+2. The partition rule assigns rows to Regions. Frontend builds one Region request per target and routes it to the current Region leader.
+3. The Datanode's Region server dispatches each request to the Region engine. In standalone mode, the same request is sent to an embedded Region server instead of over RPC.
+
+#### DDL
+
+The statement executor converts DDL into a task. In distributed mode, Metasrv runs that task as a persisted procedure, updates metadata, and coordinates Region operations on Datanodes. Standalone mode uses the same statement boundary with local implementations of the metadata and procedure services.
### Deployment
diff --git a/versioned_docs/version-1.0/contributor-guide/frontend/table-sharding.md b/versioned_docs/version-1.0/contributor-guide/frontend/table-sharding.md
index a60276d14e..beaece40c0 100644
--- a/versioned_docs/version-1.0/contributor-guide/frontend/table-sharding.md
+++ b/versioned_docs/version-1.0/contributor-guide/frontend/table-sharding.md
@@ -5,21 +5,15 @@ description: Explains how table data in GreptimeDB is sharded and distributed, i
# Table Sharding
-The sharding of stored data is essential to any distributed database. This document will describe how table's data in GreptimeDB is being sharded, and distributed.
+GreptimeDB shards a table into Regions. Partition expressions define which rows belong to each Region, while Region routes define which Datanode currently owns each Region.
## Partition
-For the syntax of creating a partitioned table, please refer to the [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) section in the User Guide.
+A partition is a logical row set described by an expression over one or more columns. The partition layout must cover the table's input domain so each row has one target Region. See [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) for the SQL syntax and supported expressions.
## Region
-The data within a table is logically split after creating partitions. You may ask the question "
-how are the data, after being logically partitioned, stored in the GreptimeDB? The answer is in "`Region`"s.
-
-Each region is corresponding to a partition, and stores the data in the partition. The regions are distributed among
-`Datanode`s. `Metasrv` manages the route information that maps regions to Datanodes.
-If the partition layout needs to change after table creation, GreptimeDB supports explicit
-[repartitioning](/user-guide/deployments-administration/manage-data/repartition.md) through split and merge operations.
+Each partition maps to one Region. Region IDs remain the storage and routing identity used by Frontend, Datanode, and Metasrv. Multiple Regions from the same table may be placed on one Datanode.
The relationship between partition and region can be viewed as the following diagram:
@@ -53,3 +47,14 @@ The relationship between partition and region can be viewed as the following dia
│ │
└──────────────────────────────────┘
Could be placed in one Datanode
+```
+
+## Routing and Pruning
+
+For writes, Frontend evaluates the partition rule for each row, groups rows by Region, and sends Region requests to the current leaders from the route table.
+
+For queries, the distributed planner compares query predicates with the partition expressions. It scans only Regions that can satisfy the predicates. If partition metadata is missing or cannot be interpreted safely, the planner falls back to all Regions rather than risk omitting data.
+
+## Changing the Partition Layout
+
+[Repartitioning](/user-guide/deployments-administration/manage-data/repartition.md) changes an existing layout through explicit split and merge operations. Metasrv runs the change as a persisted procedure, updates the Region routes and partition expressions, and invalidates stale table-route caches. New requests use the published layout after their Frontend refreshes that metadata.
diff --git a/versioned_docs/version-1.0/contributor-guide/getting-started.md b/versioned_docs/version-1.0/contributor-guide/getting-started.md
index b17184dfa8..7e8cca3ad8 100644
--- a/versioned_docs/version-1.0/contributor-guide/getting-started.md
+++ b/versioned_docs/version-1.0/contributor-guide/getting-started.md
@@ -15,14 +15,13 @@ At the moment, GreptimeDB supports Linux (both amd64 and arm64), macOS (both amd
### Build Dependencies
-- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) (optional)
+- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) (optional; needed to clone the repository, not to build it)
- C/C++ Toolchain: provides essential tools for compiling and linking. This is available either as `build-essential` on ubuntu or a similar name on other platforms.
-- Rust nightly toolchain ([guide][1])
- - Compile the source code
+- [Rustup][1]. The repository pins the required nightly toolchain in `rust-toolchain.toml`.
- Protobuf ([guide][2])
- Compile the proto file
- Note that the version needs to be >= 3.15. You can check it with `protoc --version`
-- Machine: Recommended memory is 16GB or more, or use the [mold](https://github.com/rui314/mold) tool to reduce memory usage during linking.
+- Machine: 16GB of memory or more is recommended. On a smaller machine, use [mold](https://github.com/rui314/mold) to reduce memory usage during linking.
[1]:
[2]:
diff --git a/versioned_docs/version-1.0/contributor-guide/how-to/how-to-write-sdk.md b/versioned_docs/version-1.0/contributor-guide/how-to/how-to-write-sdk.md
index 40d0bc3713..e306448642 100644
--- a/versioned_docs/version-1.0/contributor-guide/how-to/how-to-write-sdk.md
+++ b/versioned_docs/version-1.0/contributor-guide/how-to/how-to-write-sdk.md
@@ -1,21 +1,17 @@
---
keywords: [gRPC SDK, GreptimeDatabase, Handle, HandleRequests, GreptimeRequest, GreptimeResponse]
-description: Explains how to write a gRPC SDK for GreptimeDB, focusing on the GreptimeDatabase service, its methods, and the structure of requests and responses.
+description: Protocol contracts and error-handling requirements for a GreptimeDB gRPC ingestion SDK.
---
# How to write a gRPC SDK for GreptimeDB
-A GreptimeDB gRPC SDK only needs to handle the writes. The reads are standard SQL and PromQL, can be handled by any JDBC
-client or Prometheus client. This is also why GreptimeDB gRPC SDKs are all named
-like "`greptimedb-ingester-`". Please make sure your GreptimeDB SDK follow the same naming convention.
+GreptimeDB's public gRPC SDKs are ingestion clients. Queries normally use SQL or PromQL through their standard clients. A new SDK should therefore focus on writes and deletes unless it has a separate requirement, and follow the `greptimedb-ingester-` naming convention. See the [gRPC SDK overview](/user-guide/ingest-data/for-iot/grpc-sdks/overview.md) for the user-facing API.
## `GreptimeDatabase` Service
-GreptimeDB defines a custom gRPC service called `GreptimeDatabase`. All you need to do in your SDK are implement it. You
-can find its Protobuf
-definitions [here](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto).
+Generate client stubs from the [`GreptimeDatabase` Protobuf definition](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto). Do not maintain a handwritten copy of the messages or service definition.
-The service contains two RPC methods:
+The service provides a unary method and a client-streaming method:
```protobuf
service GreptimeDatabase {
@@ -25,13 +21,9 @@ service GreptimeDatabase {
}
```
-The `Handle` method is for unary call: when a `GreptimeRequest` is received and processed by a GreptimeDB
-server, it responds with a `GreptimeResponse` immediately.
+`Handle` returns one response for one request. It is the usual choice for an SDK's insert and delete APIs.
-The `HandleRequests` acts in
-a "[Client streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc)" style. It ingests a
-stream of `GreptimeRequest`, and handles them on the fly. After all the requests have been handled, it returns a
-summarized `GreptimeResponse`. Through `HandleRequests`, we can achieve a very high throughput of requests handling.
+`HandleRequests` is a [client-streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc). The server returns a cumulative response only after the client closes the request stream. An SDK that exposes streaming must document this acknowledgement boundary and bind the stream to one endpoint.
### `GreptimeRequest`
@@ -51,13 +43,13 @@ message GreptimeRequest {
}
```
-A `RequestHeader` is needed, it includes some context, authentication and others. The "oneof" field contains the request
-to the GreptimeDB server.
+A client must populate `RequestHeader` with the database context and authentication expected by the server. Set exactly one request variant.
-Note that we have two types of insertions, one is in the form of "column" (the `InsertRequests`), and the other is "
-row" (`RowInsertRequests`). It's generally recommended to use the "row" form, since it's more natural for insertions on
-a table, and easier to use. However, if there's a need to insert a large number of columns at once, or there're plenty
-of "null" values to insert, the "column" form is better to be used.
+The message also contains query and DDL variants used by internal callers. The public ingester API should not expose them: `GreptimeDatabase` does not return query result streams.
+
+GreptimeDB accepts row-oriented `RowInsertRequests` and column-oriented `InsertRequests`. Row-oriented requests are the default for public ingestion APIs. A column-native client may use the column form, but it must keep column lengths consistent and preserve null values, timestamp precision, data types, and column semantic types during conversion.
+
+Deletes have the same row-oriented and column-oriented distinction. Expose only the forms that the SDK can map without losing type information.
### `GreptimeResponse`
@@ -70,8 +62,18 @@ message GreptimeResponse {
}
```
-The `ResponseHeader` contains the response's status code, and error message (if there's any). The "oneof" response only
-contains the affected rows for now.
+On success, the response contains a successful header and `affected_rows`. Treat that value as the number acknowledged by the server, including the cumulative value returned when a request stream closes.
+
+Request failures are returned as a gRPC status. When present, the trailing metadata key `x-greptime-err-code` carries GreptimeDB's error code, and the status message carries the error text. Preserve the gRPC status and expose the GreptimeDB error code rather than replacing them with a generic SDK error.
+
+## Retry and Delivery Semantics
+
+Retries must be bounded and observable. A unary request may be retried only when the failure is classified as retryable and the deadline still permits it. Do not retry cancellation or deadline-expiration errors.
+
+A lost response does not prove that the server rejected a write. Retrying such a request can insert duplicate rows unless the caller's data model makes the operation idempotent. Document this possibility and return the final error when delivery is ambiguous.
+
+Do not transparently retry a partially sent `HandleRequests` stream. The server may already have accepted some requests even though the client has not received the cumulative response. Close the failed stream and report the ambiguity to the caller.
+
+Keep Arrow Flight bulk ingestion separate from the `GreptimeDatabase` RPCs. Its batching and partial-acceptance behavior needs its own API contract.
-GreptimeDB has a lot of SDKs now, you can refer to
-them [here](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=) for some examples.
+Use the existing [GreptimeDB ingester repositories](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=) to compare public API conventions, but derive wire behavior from the current Protobuf definition and server contract.
diff --git a/versioned_docs/version-1.0/contributor-guide/metasrv/admin-api.md b/versioned_docs/version-1.0/contributor-guide/metasrv/admin-api.md
index b091b48b70..d1f2ee6dbe 100644
--- a/versioned_docs/version-1.0/contributor-guide/metasrv/admin-api.md
+++ b/versioned_docs/version-1.0/contributor-guide/metasrv/admin-api.md
@@ -1,6 +1,6 @@
---
-keywords: [admin api, health check, leader query, heartbeat, maintenance mode, RESTful API]
-description: Details the Admin API for Metasrv, including endpoints for health checks, leader queries, heartbeat data, maintenance mode, and Procedure Manager controls.
+keywords: [admin api, health check, leader query, heartbeat, maintenance mode, recovery mode, table id sequence]
+description: Details the Metasrv Admin API for status inspection, cluster controls, and metadata recovery.
---
# Admin API
@@ -9,16 +9,17 @@ description: Details the Admin API for Metasrv, including endpoints for health c
Note that all Admin API endpoints in this document listen on Metasrv's `HTTP_PORT`, which defaults to `4000`.
:::
-The Admin API provides a simple way to view and manage cluster information, including metasrv health detection, metasrv leader query, datanode heartbeat detection, maintenance mode, and Procedure Manager controls.
-
-The Admin API is an HTTP service that provides a set of RESTful APIs that can be called through HTTP requests. The Admin API is simple, user-friendly and safe.
+The Admin API exposes Metasrv status, cluster controls, and metadata recovery operations over HTTP. It does not provide authentication, and some endpoints change cluster behavior or metadata allocation. Deployments must protect the HTTP port with network-level controls.
This page covers the following APIs:
- /health
- /leader
- /heartbeat
+- /node-lease
- /maintenance
- /procedure-manager
+- /recovery
+- /sequence/table
All these APIs are under the parent resource `/admin`.
@@ -26,7 +27,7 @@ In the following sections, we assume that your metasrv instance is running on lo
## /health HTTP endpoint
-The `/health` endpoint accepts GET HTTP requests and you can use this endpoint to check the health of your metasrv instance.
+The `/health` endpoint accepts GET requests and returns `OK` when the HTTP service is running. It does not check whether this Metasrv is the leader or whether external dependencies are available.
### Definition
@@ -120,9 +121,17 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100'
]
```
+## /node-lease HTTP endpoint
+
+The `/node-lease` endpoint returns the current leases recorded for Datanodes. Use it when diagnosing whether Metasrv still considers a Datanode active.
+
+```bash
+curl -X GET http://localhost:4000/admin/node-lease
+```
+
## /maintenance HTTP endpoint
-Cluster Maintenance Mode is a safety feature in GreptimeDB that temporarily disables automatic cluster management operations. This mode is particularly useful during cluster upgrades, planned downtime, and any operation that might temporarily affect cluster stability. For more details, please refer to [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md).
+Maintenance mode temporarily disables automatic cluster management operations during upgrades, planned downtime, or similar work. See [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md) for its effect on the cluster.
The `/maintenance` endpoint supports the following HTTP requests:
@@ -155,3 +164,39 @@ The response body uses the following format:
"status": "running"
}
```
+
+## /recovery HTTP endpoints
+
+Recovery mode gates metadata repair endpoints such as manual table ID sequence changes. It is intended for recovery work, not routine maintenance.
+
+- `GET /admin/recovery/status`: query whether recovery mode is enabled.
+- `POST /admin/recovery/enable`: enable recovery mode.
+- `POST /admin/recovery/disable`: disable recovery mode.
+
+The response body uses the following format:
+
+```json
+{
+ "enabled": true
+}
+```
+
+Disable recovery mode after the repair is complete. Use [maintenance mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md) instead when the goal is to suspend automatic cluster operations during planned maintenance.
+
+## /sequence/table HTTP endpoints
+
+These endpoints inspect or repair the table ID sequence:
+
+- `GET /admin/sequence/table/next-id`: return the next table ID without allocating it.
+- `POST /admin/sequence/table/set-next-id`: advance the next table ID.
+
+Setting the sequence requires recovery mode. The new value must be greater than the current value; the API cannot move the sequence backwards. Recovery mode is an API precondition, not a DDL barrier. Follow [Manage table ID sequences](/user-guide/deployments-administration/maintenance/sequence-management.md) for the required cluster-wide procedure.
+
+```bash
+curl -X POST \
+ -H 'Content-Type: application/json' \
+ -d '{"next_table_id": 2048}' \
+ http://localhost:4000/admin/sequence/table/set-next-id
+```
+
+Changing this value affects IDs allocated to future tables.
diff --git a/versioned_docs/version-1.0/contributor-guide/metasrv/overview.md b/versioned_docs/version-1.0/contributor-guide/metasrv/overview.md
index 7aa052dd7e..6458b6380d 100644
--- a/versioned_docs/version-1.0/contributor-guide/metasrv/overview.md
+++ b/versioned_docs/version-1.0/contributor-guide/metasrv/overview.md
@@ -1,161 +1,101 @@
---
-keywords: [metasrv, metadata, request-router, load balancing, election, high availability, heartbeat]
-description: Provides an overview of the Metasrv service, its components, interactions with the Frontend, architecture, and key functionalities like distributed consensus and heartbeat management.
+keywords: [metasrv, metadata, routing, leader election, procedure, heartbeat]
+description: Overview of the metadata and coordination mechanisms provided by Metasrv.
---
# Metasrv
-
-
## What's in Metasrv
-- Store metadata (Catalog, Schema, Table, Region, etc.)
-- Request-Router. It tells the Frontend where to write and read data.
-- Load balancing for Datanode, determines who should handle new table creation requests, more precisely, it makes resource allocation decisions.
-- Election & High Availability, GreptimeDB is designed in a Leader-Follower architecture, only Leader nodes can write while Follower nodes can read, the number of Follower nodes is usually >= 1, and Follower nodes need to be able to switch to Leader quickly when Leader is not available.
-- Statistical data collection (reported via Heartbeats on each node), such as CPU, Load, number of Tables on the node, average/peak data read/write size, etc., can be used as the basis for distributed scheduling.
+Metasrv is the metadata and coordination service in a distributed GreptimeDB cluster. It does not sit on the data path. Its main responsibilities are:
+
+- storing Catalog, Schema, Table, Region, route, and node metadata;
+- choosing Datanodes for new Regions and maintaining table routes;
+- electing one Metasrv leader to coordinate metadata changes;
+- running recoverable procedures for DDL, Region migration, failover, and repartitioning;
+- tracking node leases and Region statistics through heartbeats;
+- broadcasting cache invalidations to Frontends, Datanodes, and Flownodes when metadata changes;
+- sending Region lifecycle instructions to Datanodes.
## How the Frontend interacts with Metasrv
-First, the routing table in Request-Router is in the following structure (note that this is only the logical structure, the actual storage structure varies, for example, endpoints may have dictionary compression).
+Frontend obtains table metadata and Region routes from Metasrv and caches them locally. Metadata-changing statements are sent to the Metasrv leader, while reads and writes use the cached routes to reach Datanodes directly.
+
+The control and data paths are separate:
+
+```text
+Frontend
+ |-- metadata lookup and DDL ------------> Metasrv leader
+ `-- Region reads and writes ------------> Datanode
+
+Metasrv leader
+ |-- Region lifecycle instructions ------> Datanode
+ `-- cache invalidations ----------------> Frontend / Datanode / Flownode
+Datanode
+ `-- heartbeat, lease renewal, Region stats -> Metasrv leader
```
- table_A
- table_name
- table_schema // for physical plan
- regions
- region_1
- mutate_endpoint
- select_endpoint_1, select_endpoint_2
- region_2
- mutate_endpoint
- select_endpoint_1, select_endpoint_2, select_endpoint_3
- region_xxx
- table_B
- ...
+
+In steady state, a table route records one leader peer and zero or more follower peers for each Region. The leader is the write target. Deployments with read-replica support can route reads to followers:
+
+```text
+Table route
+ |-- Region 0
+ | |-- leader -> Datanode A
+ | `-- followers -> Datanode B, Datanode C
+ `-- Region 1
+ `-- leader -> Datanode D
```
+Region migration or failover changes peer roles and can temporarily leave a Region without a leader. Frontend refreshes its cached route before sending subsequent reads or writes to the current peers.
+
### Create Table
-1. The Frontend sends `CREATE TABLE` requests to Metasrv.
-2. Plan the number of Regions according to the partition rules contained in the request.
-3. Check the global view of resources available to Datanodes (collected by Heartbeats) and assign one node to each region.
-4. The Frontend creates the table and stores the `Schema` to Metasrv after successful creation.
+1. Frontend submits the DDL request to the Metasrv leader.
+2. Metasrv derives Regions from the partition rules and [selects a Datanode for each Region](/contributor-guide/metasrv/selector.md).
+3. A persisted procedure creates the Regions and records the table and route metadata. If leadership changes, the procedure can resume from its persisted state.
+4. Metasrv notifies Frontends after the metadata change is committed so their caches can be refreshed.
### Insert
-1. The Frontend fetches the routes of the specified table from Metasrv. Note that the smallest routing unit is the route of the table (several regions), i.e., it contains the addresses of all regions of this table.
-2. The best practice is that the Frontend first fetches the routes from its local cache and forwards the request to the Datanode. If the route is no longer valid, then Datanode is obliged to return an `Invalid Route` error, and the Frontend re-fetches the latest data from Metasrv and updates its cache. Route information does not change frequently, thus, it's sufficient for Frontend uses the Lazy policy to maintain the cache.
-3. The Frontend processes a batch of writes that may contain multiple tables and multiple regions, so the Frontend needs to split user requests based on the 'route table'.
+Frontend resolves the table route, splits rows according to the partition rules, and sends each Region write to the corresponding Datanode. Route changes cause the cached metadata to be invalidated and fetched again from Metasrv.
### Select
-1. As with `Insert`, the Frontend first fetches the route table from the local cache.
-2. Unlike `Insert`, for `Select`, the Frontend needs to extract the read-only node (follower) from the route table, then dispatch the request to the leader or follower node depending on the priority.
-3. The distributed query engine in the Frontend distributes multiple sub-query tasks based on the routing information and aggregates the query results.
+Frontend uses table and Region metadata while planning the query. Predicates on partition columns prune Regions, and the distributed query engine sends work to the Datanodes that own the selected Regions. See [Distributed Querying](../frontend/distributed-querying.md).
## Metasrv Architecture
-
-
-## Distributed Consensus
+The main coordination paths are:
+
+```text
+Leader election
+ |
+ v
+Metasrv leader
+├─ DDL manager -> Procedure manager
+├─ Selector -> new Region placement
+├─ Heartbeat handler chain -> leases and Region statistics
+├─ Region supervisor -> Region migration procedures
+├─ Mailbox -> cache invalidations and Region instructions
+└─ Metadata managers -> KV backend
+```
-As you can see, Metasrv has a dependency on distributed consensus because:
+These mechanisms share metadata, but they have different failure boundaries. A process restart may discard caches and leader-local state; metadata and procedure state required for recovery must be durable.
-1. First, Metasrv has to elect a leader, Datanode only sends heartbeats to the leader, and we only use a single metasrv node to receive heartbeats, which makes it easy to do some calculations or scheduling accurately and quickly based on global information. As for how the Datanode connects to the leader, this is for MetaClient to decide (using a redirect, Heartbeat requests becomes a gRPC stream, and using redirect will be less error-prone than forwarding), and it is transparent to the Datanode.
-2. Second, Metasrv must provide an election API for Datanode to elect "write" and "read-only" nodes and help Datanode achieve high availability.
-3. Finally, `Metadata`, `Schema` and other data must be reliably and consistently stored on Metasrv. Therefore, consensus-based algorithms are the ideal approach for storing them.
+## Distributed Consensus
-For the first version of Metasrv, we choose Etcd as the consensus algorithm component (Metasrv is designed to consider adapting different implementations and even creating a new wheel) for the following reasons:
+Metasrv separates leader election from metadata storage. Only the elected Metasrv leader performs coordination and metadata-changing operations. Other Metasrv nodes direct clients to the current leader.
-1. Etcd provides exactly the API we need, such as `Watch`, `Election`, `KV`, etc.
-2. We only perform two tasks with distributed consensus: elections (using the `Watch` mechanism) and storing (a small amount of metadata), and neither of them requires us to customize our own state machine, nor do we need to customize our own state machine based on raft; the small amount of data also does not require multi-raft-group support.
-3. The initial version of Metasrv uses Etcd, which allows us to focus on the capabilities of Metasrv and not spend too much effort on distributed consensus algorithms, which improves the design of the system (avoiding coupling with consensus algorithms) and helps with rapid development at the beginning, as well as allows easy access to good consensus algorithm implementations in the future through good architectural designs.
+The key-value backend stores table metadata, routes, procedure state, and other information that must survive a leader change. Metasrv does not use this election to create leader and follower replicas for Datanode Regions; Region availability is managed through heartbeats, Region failure detection, and failover procedures.
## Heartbeat Management
-The primary means of communication between Datanode and Metasrv is the Heartbeat Request/Response Stream, and we want this to be the only way to communicate. This idea is inspired by the design of [TiKV PD](https://github.com/tikv/pd), and we have practical experience in [RheaKV](https://github.com/sofastack/sofa-jraft/tree/master/jraft-rheakv/rheakv-pd). The request sends its state, while Metasrv sends different scheduling instructions via Heartbeat Response.
-
-A heartbeat will probably carry the data listed below, but this is not the final design, and we are still discussing and exploring exactly which data should be mostly collected.
-
-```
-service Heartbeat {
- // Heartbeat, there may be many contents of the heartbeat, such as:
- // 1. Metadata to be registered to metasrv and discoverable by other nodes.
- // 2. Some performance metrics, such as Load, CPU usage, etc.
- // 3. The number of computing tasks being executed.
- rpc Heartbeat(stream HeartbeatRequest) returns (stream HeartbeatResponse) {}
-}
-
-message HeartbeatRequest {
- RequestHeader header = 1;
-
- // Self peer
- Peer peer = 2;
- // Leader node
- bool is_leader = 3;
- // Actually reported time interval
- TimeInterval report_interval = 4;
- // Node stat
- NodeStat node_stat = 5;
- // Region stats in this node
- repeated RegionStat region_stats = 6;
- // Follower nodes and stats, empty on follower nodes
- repeated ReplicaStat replica_stats = 7;
-}
-
-message NodeStat {
- // The read capacity units during this period
- uint64 rcus = 1;
- // The write capacity units during this period
- uint64 wcus = 2;
- // Table number in this node
- uint64 table_num = 3;
- // Region number in this node
- uint64 region_num = 4;
-
- double cpu_usage = 5;
- double load = 6;
- // Read disk I/O in the node
- double read_io_rate = 7;
- // Write disk I/O in the node
- double write_io_rate = 8;
-
- // Others
- map attrs = 100;
-}
-
-message RegionStat {
- uint64 region_id = 1;
- TableName table_name = 2;
- // The read capacity units during this period
- uint64 rcus = 3;
- // The write capacity units during this period
- uint64 wcus = 4;
- // Approximate region size
- uint64 approximate_size = 5;
- // Approximate number of rows
- uint64 approximate_rows = 6;
-
- // Others
- map attrs = 100;
-}
-
-message ReplicaStat {
- Peer peer = 1;
- bool in_sync = 2;
- bool is_learner = 3;
-}
-```
-
-## Central Nervous System (CNS)
-
-We are to build an algorithmic system, which relies on real-time and historical heartbeat data from each node, should make some smarter scheduling decisions and send them to Metasrv's Autoadmin unit, which distributes the scheduling decisions, either by the Datanode itself or more likely by the PaaS platform.
-
-## Abstraction of Workloads
+Datanodes maintain heartbeat streams to the Metasrv leader. Heartbeat requests report node identity, lease information, Region statistics, and other state used for placement and supervision. Responses carry control messages such as Region lifecycle instructions and cache invalidations.
-The level of workload abstraction determines the efficiency and quality of the scheduling strategy generated by Metasrv such as resource allocation.
+A heartbeat drives two independent mechanisms, and a change to heartbeat timing affects both:
-DynamoDB defines RCUs & WCUs (Read Capacity Units / Write Capacity Units), explaining that a RCU is a read request of 4KB data, and a WCU is a write request of 1KB data. When using RCU and WCU to describe workloads, it's easier to achieve performance measurability and get more informative resource preallocation because we can abstract different hardware capabilities as a combination of RCU and WCU.
+- **Node lease.** The keep-lease handler renews the sending Datanode's lease. Selectors and the `/node-lease` endpoint use these leases to decide whether a Datanode is still active.
+- **Region failure detection.** The Region supervisor keeps a per-Region Phi Accrual detector over heartbeat arrival intervals. Its verdict is independent of lease expiry.
-However, GreptimeDB still faces a more complex situation than DynamoDB, in particular, RCU doesn't fit to describe GreptimeDB's read workloads which require a lot of computation. We are working on that.
+A failure verdict submits a failover migration only when Region failover is enabled; it is disabled by default and requires remote WAL unless explicitly allowed on local WAL. Maintenance mode also suppresses failover. See [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md) for the prerequisites and how to enable it.
diff --git a/versioned_docs/version-1.0/contributor-guide/metasrv/selector.md b/versioned_docs/version-1.0/contributor-guide/metasrv/selector.md
index 790ffb849b..23190cd6a9 100644
--- a/versioned_docs/version-1.0/contributor-guide/metasrv/selector.md
+++ b/versioned_docs/version-1.0/contributor-guide/metasrv/selector.md
@@ -7,32 +7,28 @@ description: Describes the different types of selectors in the Metasrv service,
## Introduction
-What is the `Selector`? As its name suggests, it allows users to select specific items from a given `namespace` and `context`. There is a related trait, also named `Selector`, whose definition can be found [below][0].
-
-[0]: https://github.com/GreptimeTeam/greptimedb/blob/main/src/meta-srv/src/selector.rs
-
-There is a specific scenario in `Metasrv` service. When a request to create a table is sent to the `Metasrv` service, it creates a routing table (the details of table creation will not be described here). The `Metasrv` service needs to select the appropriate `Datanode` list when creating a routing table.
+When a table is created, Metasrv uses a `Selector` to choose Datanodes for its Regions. Selection uses the current node leases and, depending on the selector, Region statistics.
## Selector Type
The `Metasrv` service currently offers the following types of `Selectors`:
-### LeasebasedSelector
+### LeaseBasedSelector
-`LeasebasedSelector` randomly selects from all available (in lease) `Datanode`s, its characteristic is simplicity and fast.
+`LeaseBasedSelector` randomly selects from Datanodes with valid leases.
### LoadBasedSelector
The `LoadBasedSelector` load value is determined by the number of regions on each `Datanode`, fewer regions indicate lower load, and `LoadBasedSelector` prioritizes selecting low-load `Datanodes`.
### RoundRobinSelector [default]
-`RoundRobinSelector` selects `Datanode`s in a round-robin fashion. It is recommended and the default option in most cases. If you're unsure which to choose, it's usually the right choice.
+`RoundRobinSelector` selects `Datanode`s in a round-robin fashion. It is the default and recommended choice for most deployments.
## Configuration
You can configure the `Selector` by its name when starting the `Metasrv` service.
-- LeasebasedSelector: `lease_based` or `LeaseBased`
+- LeaseBasedSelector: `lease_based` or `LeaseBased`
- LoadBasedSelector: `load_based` or `LoadBased`
- RoundRobinSelector: `round_robin` or `RoundRobin`
diff --git a/versioned_docs/version-1.0/contributor-guide/overview.md b/versioned_docs/version-1.0/contributor-guide/overview.md
index 875b4b1e16..6e6ec669c9 100644
--- a/versioned_docs/version-1.0/contributor-guide/overview.md
+++ b/versioned_docs/version-1.0/contributor-guide/overview.md
@@ -5,9 +5,7 @@ description: Overview of GreptimeDB's architecture, key components, and how they
# Contributor Guide
-DeepWiki provides a detailed and clear explanation of GreptimeDB's architecture and implementation. Highly recommended:
-
-[https://deepwiki.com/GreptimeTeam/greptimedb](https://deepwiki.com/GreptimeTeam/greptimedb)
+This guide explains the internal design of GreptimeDB for contributors. Start with [Getting Started](/contributor-guide/getting-started.md) to build and run it from source. Submission requirements, including the CLA, license headers, formatting, and the checks a pull request must pass, are maintained in the source repository's [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md).
## Architecture
@@ -18,8 +16,13 @@ For more details on each component, see the following guides:
- [frontend][1]
- [datanode][2]
- [metasrv][3]
+- [flownode][4]
[1]: /contributor-guide/frontend/overview.md
[2]: /contributor-guide/datanode/overview.md
[3]: /contributor-guide/metasrv/overview.md
+[4]: /contributor-guide/flownode/overview.md
+
+## Additional reference
+[DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb) provides an automatically generated walkthrough of the GreptimeDB repository. It can help when exploring an unfamiliar area, but it is a secondary reference: verify version-sensitive behavior against the source code.
diff --git a/versioned_docs/version-1.0/contributor-guide/tests/integration-test.md b/versioned_docs/version-1.0/contributor-guide/tests/integration-test.md
index 5d5f6cb1a5..bc52ee4854 100644
--- a/versioned_docs/version-1.0/contributor-guide/tests/integration-test.md
+++ b/versioned_docs/version-1.0/contributor-guide/tests/integration-test.md
@@ -7,7 +7,14 @@ description: Guide on writing and running integration tests in GreptimeDB, cover
## Introduction
-Integration testing is written with Rust test harness (`#[test]`), unlike unit testing, they are placed separately
-[here](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration).
-It covers scenarios involving multiple components, in which one typical case is HTTP/gRPC-related features. You can check
-its [documentation](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md) for more information.
+Integration tests cover behavior that crosses crate or service boundaries, such as HTTP and gRPC handling, distributed components, or external storage. They use Rust's test harness and live in the [`tests-integration`](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration) package.
+
+Run the package with:
+
+```shell
+cargo nextest run -p tests-integration
+```
+
+Some cases require environment variables or fixtures for external services. Follow the package's [setup instructions](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md) before running those cases.
+
+Use an integration test when a crate-level test or a Sqlness case cannot exercise the required boundary. Keep isolated logic in unit tests so that failures remain fast to reproduce.
diff --git a/versioned_docs/version-1.0/contributor-guide/tests/overview.md b/versioned_docs/version-1.0/contributor-guide/tests/overview.md
index feeefe2b6d..07290b79bd 100644
--- a/versioned_docs/version-1.0/contributor-guide/tests/overview.md
+++ b/versioned_docs/version-1.0/contributor-guide/tests/overview.md
@@ -5,4 +5,12 @@ description: Overview of the testing methods used in GreptimeDB to ensure its be
# Tests
-Our team has conducted lots of tests to ensure the behaviours of `GreptimeDB` . This chapter will introduce several significant methods used to test `GreptimeDB`, and how to work with them.
+Choose the narrowest test that exercises the behavior you changed:
+
+| Test type | Use it for | Typical command |
+| --- | --- | --- |
+| [Unit test](unit-test.md) | Logic contained within one crate or component | `cargo nextest run -p ` |
+| [Sqlness test](sqlness-test.md) | SQL, protocol, planner, execution, and end-to-end regressions | `cargo sqlness bare -t ` |
+| [Integration test](integration-test.md) | Behavior that crosses components or requires external services | `cargo nextest run -p tests-integration` |
+
+Run `make test` when a change needs the full Rust workspace test suite.
diff --git a/versioned_docs/version-1.0/contributor-guide/tests/sqlness-test.md b/versioned_docs/version-1.0/contributor-guide/tests/sqlness-test.md
index b5fdcf2b0b..0262ff6851 100644
--- a/versioned_docs/version-1.0/contributor-guide/tests/sqlness-test.md
+++ b/versioned_docs/version-1.0/contributor-guide/tests/sqlness-test.md
@@ -7,42 +7,34 @@ description: Instructions for running SQL tests in GreptimeDB using the `sqlness
## Introduction
-SQL is an important user interface for `GreptimeDB`. We have a separate test suite for it (named `sqlness`).
+Sqlness is GreptimeDB's end-to-end regression suite for SQL and protocol behavior. A case sends statements to a running GreptimeDB instance and compares the output with a checked-in result file.
## Sqlness manual
### Case file
-Sqlness has two types of file
+Each case uses two files:
- `.sql`: test input, SQL only
- `.result`: expected test output, SQL and its results
-The `.result` file is the expected execution output. If you see `.result` files changed,
-it means the test gets a different result and indicates it may fail. You should
-check the change logs to solve the problem.
-
-You only need to write test SQL in the `.sql` file, and run the test.
+Write the input in the `.sql` file and run the test to generate or update `.result`. Review every result diff: accept it only when the behavior change is intended.
### Case organization
-The root dir of input cases is `tests/cases`. It contains several sub-directories stand for different test
-modes. E.g., `standalone/` contains all the tests to run under `greptimedb standalone start` mode.
+Input cases live under `tests/cases`. The first directory level selects an environment. For example, `standalone/` runs against a standalone GreptimeDB instance.
-Under the first level of sub-directory (e.g. the `cases/standalone`), you can organize your cases as you like.
-Sqlness walks through every file recursively and runs them.
+Within an environment, group a new case with the feature it exercises. Sqlness discovers case files recursively.
## Run the test
-Unlike other tests, this harness is in a binary target form. You can run it with
+Run the suite with:
```shell
-cargo run --bin sqlness-runner bare
+cargo sqlness bare
```
-It automatically finishes the following procedures: compile `GreptimeDB`, start it, grab tests and feed it to
-the server, then collect and compare the results. You only need to check whether any `.result` files changed.
-If not, congratulations, the test is passed 🥳!
+The command builds and starts GreptimeDB, runs the selected cases, and compares their output. A changed `.result` file is part of the review, not proof that the new output is correct.
### Run a specific test
diff --git a/versioned_docs/version-1.0/contributor-guide/tests/unit-test.md b/versioned_docs/version-1.0/contributor-guide/tests/unit-test.md
index 82e30bf5fa..183a72aab4 100644
--- a/versioned_docs/version-1.0/contributor-guide/tests/unit-test.md
+++ b/versioned_docs/version-1.0/contributor-guide/tests/unit-test.md
@@ -8,25 +8,26 @@ description: Guide on writing and running unit tests in GreptimeDB using Rust's
## Introduction
Unit tests are embedded into the codebase, usually placed next to the logic being tested.
-They are written using Rust's `#[test]` attribute and can run with `cargo nextest run`.
+They are written using Rust's `#[test]` attribute. GreptimeDB uses [`cargo-nextest`](https://nexte.st/) as its primary Rust test runner.
-The default test runner ships with `cargo` is not supported in GreptimeDB codebase. It's recommended
-to use [`nextest`](https://nexte.st/) instead. You can install it with
+Install it with:
```shell
cargo install cargo-nextest --locked
```
-And run the tests (here the `--workspace` is not necessary)
+Run the package you changed first:
```shell
-cargo nextest run
+cargo nextest run -p
```
-Notes if your Rust is installed via `rustup`, be sure to install `nextest` with `cargo` rather
-than the package manager like `homebrew`. Otherwise it will mess up your local environment.
+Use a test name or nextest filter to narrow the run further while developing. Before submitting a change with broad effects, run the full workspace suite:
+
+```shell
+make test
+```
## Coverage
-Our continuous integration (CI) jobs have a "coverage checking" step. It will report how many
-codes are covered by unit tests. Please add the necessary unit test to your patch.
+CI reports unit-test coverage. Add tests for changed behavior and failure cases that could otherwise regress; coverage percentage alone is not the goal.
diff --git a/versioned_docs/version-1.1/contributor-guide/datanode/data-persistence-indexing.md b/versioned_docs/version-1.1/contributor-guide/datanode/data-persistence-indexing.md
index 6403ffc2df..1fa5d6d258 100644
--- a/versioned_docs/version-1.1/contributor-guide/datanode/data-persistence-indexing.md
+++ b/versioned_docs/version-1.1/contributor-guide/datanode/data-persistence-indexing.md
@@ -5,19 +5,23 @@ description: Explanation of data persistence and indexing in GreptimeDB, includi
# Data Persistence and Indexing
-Similar to all LSMT-like storage engines, data in MemTables is persisted to durable storage, for example, the local disk file system or object storage service. GreptimeDB adopts [Apache Parquet][1] as its persistent file format.
+Like other LSM-tree storage engines, GreptimeDB persists data from memtables to durable storage such as a local filesystem or object storage. It uses [Apache Parquet][1] as the persistent file format.
## SST File Format
Parquet is an open source columnar format that provides fast data querying and has already been adopted by many projects, such as Delta Lake.
-Parquet has a hierarchical structure like "row groups-columns-data pages". Data in a Parquet file is horizontally partitioned into row groups, in which all values of the same column are stored together to form a data page. Data page is the minimal storage unit. This structure greatly improves performance.
+Parquet organizes data as row groups, column chunks, and pages. A row group contains one column chunk for each column, and each column chunk contains one or more pages. Pages are the units of encoding and compression; column chunks are the I/O units for reading selected columns.
First, clustering data by column makes file scanning more efficient, especially when only a few columns are queried, which is very common in analytical systems.
-Second, data of the same column tends to be homogeneous which helps with compression when apply techniques like dictionary and Run-Length Encoding (RLE).
+Second, values within a column tend to be similar, which improves compression with techniques such as dictionary encoding and run-length encoding (RLE).
-
+The following diagram from the Apache Parquet specification also shows the physical file layout: column chunks are stored by row group, while file metadata and its length are written in the footer.
+
+
+
+*Source: Apache Parquet [FileLayout.gif](https://github.com/apache/parquet-format/blob/master/doc/images/FileLayout.gif). Copyright 2014 The Apache Software Foundation, licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).*
## Data Persistence
@@ -27,17 +31,17 @@ When the size of data buffered in MemTables reaches that threshold, GreptimeDB w
## Indexing Data in SST Files
-Apache Parquet file format provides inherent statistics in headers of column chunks and data pages, which are used for pruning and skipping.
+Parquet stores row-group column statistics such as minimum, maximum, and null count in each column chunk's metadata. Page metadata and optional column indexes can provide finer-grained statistics.
-
+
-For example, in the above Parquet file, if you want to filter rows where `name` = `Emily`, you can easily skip row group 0 because the max value for `name` field is `Charlie`. This statistical information reduces IO operations.
+For example, a query filtering for `name` = `Emily` can skip row group 0 because the maximum `name` value is `Charlie`. This avoids reading that row group.
## Index Files
-For each SST file, GreptimeDB not only maintains an internal index but also generates a separate file to store the index structures specific to that SST file.
+When an SST has one or more configured index outputs, GreptimeDB writes them to a Puffin file associated with that SST. An SST with no applicable index does not need a Puffin file.
-The index files utilize the [Puffin][3] format, which offers significant flexibility, allowing for the storage of additional metadata and supporting a broader range of index structures.
+Puffin provides a container for index blobs and their metadata, allowing different index structures to share one file.
@@ -57,13 +61,13 @@ The inverted index enables GreptimeDB to skip data segments that do not meet que
-For instance, the query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches the regex `.*users`, and `status` matches the regex `4...`. It then scans these data segments to produce the final results that meet all conditions, significantly reducing the number of IO operations.
+The query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches `.*users`, and `status` matches `4..`. It scans only those segments before applying the remaining filters.
### Inverted Index Format
-
+
-GreptimeDB builds inverted indexes by column, with each inverted index consisting of an FST and multiple Bitmaps.
+GreptimeDB builds inverted indexes by column. Each column index contains a null bitmap, multiple posting bitmaps, and an FST. The blob footer records the offsets, sizes, and metadata needed to locate and decode the column indexes.
The FST (Finite State Transducer) enables GreptimeDB to store mappings from column values to Bitmap positions in a compact format and provides excellent search performance and supports complex search capabilities (such as regular expression matching). The Bitmaps maintain a list of data segment IDs, with each bit representing a data segment.
@@ -77,7 +81,7 @@ The number of rows in a data segment is controlled by the engine option `index.i
## Unified Data Access Layer: OpenDAL
-GreptimeDB uses [OpenDAL][2] to provide a unified data access layer, thus, the storage engine does not need to interact with different storage APIs, and data can be migrated to cloud-based storage like AWS S3 seamlessly.
+GreptimeDB uses [OpenDAL][2] to provide a common access layer for local filesystems and object stores. Changing the configured storage backend does not migrate existing data.
[1]: https://parquet.apache.org
[2]: https://github.com/datafuselabs/opendal
diff --git a/versioned_docs/version-1.1/contributor-guide/datanode/memtable.md b/versioned_docs/version-1.1/contributor-guide/datanode/memtable.md
new file mode 100644
index 0000000000..92e3eb58de
--- /dev/null
+++ b/versioned_docs/version-1.1/contributor-guide/datanode/memtable.md
@@ -0,0 +1,100 @@
+---
+keywords: [memtable, Mito engine, write buffer, flush, time partition, BulkMemtable]
+description: How Mito organizes mutable Region data in memtables and moves it into SST files.
+---
+
+# Memtable design
+
+A memtable is Mito's in-memory write buffer for a Region. It makes writes available to reads before a flush creates SST files. A Region version identifies the memtables and SST files that a scan may read. Together with a committed-sequence fence, it keeps the scan consistent while writes and flushes advance the current version.
+
+## Write and flush lifecycle
+
+For a normal WAL-backed write, Mito uses this order:
+
+```text
+write request
+ |
+ v
+WAL append -> mutable memtable -> publish committed sequence
+ |
+ freeze
+ v
+ immutable memtable -> SST write -> manifest edit
+```
+
+The Region worker assigns sequence numbers and a WAL entry ID before appending the mutation to the [write-ahead log](wal.md). If the WAL append fails, Mito does not update the memtable. After the memtable update succeeds, Mito publishes the committed sequence and the rows become visible to new reads. A Region configured with `skip_wal` omits the WAL append, but keeps the same memtable and visibility ordering.
+
+A flush freezes the mutable memtables and installs a new mutable set before starting the background SST write. New writes therefore continue without changing the frozen data. The flush writes the immutable memtables to SST files, then persists a manifest edit containing the files and the flushed WAL and sequence checkpoints. Only after that edit is durable does Mito remove the flushed memtables from the current Region version. If the flush fails, the immutable memtables remain available for a later attempt.
+
+## Region versions and time partitions
+
+Each Region has one mutable `TimePartitions` container, which can hold more than one memtable:
+
+```text
+Region version
+├─ mutable TimePartitions
+│ ├─ [t0, t1) -> memtable
+│ └─ [t1, t2) -> memtable
+├─ immutable memtables
+└─ SST files
+```
+
+Mito routes each row to a partition by its time-index value. Partition ranges are half-open and aligned to a fixed duration. The duration follows the Region's compaction time window; Mito uses one day until a compaction window is available. An out-of-order write can create an earlier partition alongside the latest one.
+
+Freezing a Region freezes all mutable time partitions together. Mito moves their memtables to the immutable list and creates a new `TimePartitions` container. A failed flush can leave more than one generation of immutable memtables, so reads and later flushes must not assume that the list contains a single item.
+
+## Memtable implementations
+
+Mito selects a memtable implementation from the Region's SST format, primary-key encoding, and memtable options:
+
+```text
+flat SST format (the default) or sparse primary-key encoding -> BulkMemtable
+memtable.type=bulk -> BulkMemtable, and forces flat SST
+primary_key SST with dense encoding (legacy) -> a legacy implementation
+```
+
+With the default engine configuration, a Region without an explicit SST format uses `flat`, so `BulkMemtable` is the normal path and the rest of this page describes it. The rules exist to prevent incompatible combinations: flat format or sparse primary-key encoding requires `BulkMemtable`, and explicitly selecting the bulk implementation forces flat format.
+
+### BulkMemtable
+
+`BulkMemtable` stores writes as parts in the flat Arrow layout instead of inserting rows into per-series buffers:
+
+```text
+BulkMemtable
+├─ unordered_part
+│ └─ small BulkPart batches
+└─ parts
+ ├─ BulkPart (Arrow RecordBatch)
+ ├─ MultiBulkPart (raw RecordBatches)
+ └─ EncodedBulkPart (in-memory Parquet)
+```
+
+Small parts accumulate in `unordered_part`; larger parts enter `parts` directly. Background memtable compaction merge-sorts eligible parts into a `MultiBulkPart` or encodes them as an `EncodedBulkPart`. Scans use part statistics to prune ranges, and flush can write encoded ranges to SST without decoding and encoding the rows again. For the design rationale and performance results, see [Scaling Time Series to Millions of Cardinalities: GreptimeDB's Flat Format](https://greptime.com/blogs/2025-12-22-flat-format).
+
+### Legacy implementations
+
+Regions on the legacy `primary_key` SST format with dense primary-key encoding still use `TimeSeriesMemtable`, which groups rows by encoded primary key rather than storing flat parts. A Region with no primary-key columns gets `SimpleBulkMemtable` from the same builder. Both are compatibility code for existing tables and may be removed once the `primary_key` format is retired; new work targets the bulk and flat path.
+
+The removed `partition_tree` memtable is not a third implementation. The option parser accepts `memtable.type=partition_tree` for compatibility, but it does not recreate that implementation. The Region uses the bulk and flat path.
+
+## Read snapshots
+
+A scan obtains the Region version and committed sequence together from `VersionControl`. It selects the version before applying the sequence fence. Reading the sequence separately before the version could pair that sequence with a later version after flush or compaction removes an older input, producing an incomplete snapshot.
+
+The selected version supplies mutable memtables, immutable memtables, and SST files. Mito first prunes sources by time range, then asks each memtable for ranges using the scan's projection, predicate, and sequence bounds. The scan merges the resulting ranges with SST ranges and applies the same ordering, deletion, and merge semantics across all sources. References held by the scan keep an older memtable alive even after a newer Region version removes it.
+
+## Memory pressure
+
+Each memtable tracks its estimated heap allocation through the engine's write-buffer manager. Freezing a memtable removes its allocation from the mutable-memory count, but total usage includes the allocation until all references to that memtable are released. The mutable-memory count therefore tracks data that can still accept writes, while total usage continues to include memory retained by active scans.
+
+The global write-buffer limit causes workers to select Regions for flush. If memory remains above the configured limits, Mito stalls writes and can reject them at a higher threshold. An optional per-Region limit applies the same pressure to one hot Region without stalling unrelated Regions. Periodic, manual, and Region lifecycle operations can also request a flush.
+
+## Constraints for changes
+
+Changes to memtable code must preserve these properties:
+
+- For a WAL-backed Region, append to the WAL before installing rows in a memtable. Publish the committed sequence only after installation succeeds.
+- Keep frozen memtables readable and retryable until the SST files and manifest edit are durable.
+- Obtain the Region version and committed sequence from the same `VersionControl` snapshot; never read the sequence separately before the version.
+- Preserve the ordering and metadata that scans and flushes need to apply the same deletion, deduplication, and merge rules across memtable and SST ranges.
+- Charge allocations to the write-buffer manager and release them only when the underlying memory can no longer be referenced.
diff --git a/versioned_docs/version-1.1/contributor-guide/datanode/metric-engine.md b/versioned_docs/version-1.1/contributor-guide/datanode/metric-engine.md
index 064872ce14..fde3c178a3 100644
--- a/versioned_docs/version-1.1/contributor-guide/datanode/metric-engine.md
+++ b/versioned_docs/version-1.1/contributor-guide/datanode/metric-engine.md
@@ -7,9 +7,9 @@ description: Overview of the Metric engine in GreptimeDB, its concepts, architec
## Overview
-The `Metric` engine is a component of GreptimeDB, and it's an implementation of the storage engine. It mainly targets scenarios with a large number of small tables for observable metrics.
+The `Metric` engine stores workloads with many small metric tables.
-Its main feature is to use synthetic physical wide tables to store a large amount of small table data, achieving effects such as reuse of the same column and metadata. This reduces storage overhead for small tables and improves columnar compression efficiency. The concept of a table becomes even more lightweight under the `Metric` engine.
+It maps those logical tables onto shared physical wide tables so they can reuse columns and metadata. This reduces per-table storage overhead and improves columnar compression.
## Concepts
@@ -19,7 +19,7 @@ The `Metric` engine introduces two new concepts: "logical table" and "physical t
A logical table refers to user-defined tables. Just like any other ordinary table, its definition includes the name of the table, column definitions, index definitions etc. All operations such as queries or write-ins by users are based on these logical tables. Users don't need to worry about differences between logical and ordinary tables during usage.
-From an implementation standpoint, a logical table is virtual; it doesn't directly read or write physical data but maps read/write requests into corresponding requests for physical tables in order to implement data storage and querying.
+A logical table is virtual. The engine maps its read and write requests to the corresponding physical table instead of storing data for it directly.
### Physical Table
@@ -29,16 +29,14 @@ A physical table is a table that actually stores data, possessing several physic
The main design architecture of the `Metric` engine is as follows:
-
+
-In the current version implementation, the `Metric` engine reuses the `Mito` engine to achieve storage and query capabilities for physical data. It also provides access to both physical tables and logical tables simultaneously.
+The `Metric` engine delegates physical storage and queries to the `Mito` engine. Each physical Region group contains a data Region, which stores rows from its mapped logical tables, and a metadata Region, which stores the logical-table and logical-column mappings.
-Regarding partitioning, logical tables have identical partition rules and Region distribution as physical tables. This makes sense because the data of logical tables are directly stored in physical tables, so their partition rules are consistent.
+Logical tables associated with the same physical table share its partition layout. During writes, the engine records the logical table identity with each row. During reads, it adds a logical-table filter before scanning the physical Region.
-Concerning routing metadata, the routing address of a logical table is a logical address - what its corresponding physical table is - then through this physical table for secondary routing to obtain the real physical address. This indirect routing method can significantly reduce the number of metadata modifications required when Region migration scheduling occurs in Metric engines.
+A logical table's route stores only the ID of its physical table; the physical table route resolves that to the Datanodes holding the Regions. Because logical routes do not name peers, migrating a physical Region rewrites one physical route instead of every logical route that maps to it.
-Operationally speaking, The `Metric` engine supports standard DML operations (INSERT, DELETE, SELECT) on logical tables. However, it only supports limited operations on physical tables to prevent misoperations - for example, writing directly to a physical table is prohibited as it could affect user's logical table data. Generally speaking, users can consider that they have read-only access to these physical tables.
+Logical tables support normal INSERT, DELETE, and SELECT operations. Direct writes to a physical Region are rejected because they would bypass the logical-table mapping; querying a physical table remains supported.
-To improve performance during simultaneous DDL (Data Definition Language) operations on many tables, the 'Metric' engine has introduced some batch DDL operations. These batch DDL operations can merge lots of DDL actions into one request thereby reducing queries and modifications times for metadata thus enhancing performance. This feature is particularly beneficial in scenarios such as the automatic creation requests brought about by large amounts of metrics during Prometheus Remote Write cold start-up, as well as the modification requests for numerous route-tables mentioned earlier during migration of many physical regions.
-
-Apart from physical data regions belonging to physical tables, the 'Metric' engine creates an additional metadata region physically for each individual physical data region used in storing some metadata needed by itself while maintaining mapping and other states. This metadata includes the mapping relationship between logical tables and physical tables, the mapping relationship between logical columns and physical columns etc.
+Batch DDL operations reduce metadata work when many logical tables are created or updated together, such as during Prometheus Remote Write auto-creation or physical Region migration.
diff --git a/versioned_docs/version-1.1/contributor-guide/datanode/overview.md b/versioned_docs/version-1.1/contributor-guide/datanode/overview.md
index d0afe21b34..a21c112faa 100644
--- a/versioned_docs/version-1.1/contributor-guide/datanode/overview.md
+++ b/versioned_docs/version-1.1/contributor-guide/datanode/overview.md
@@ -7,28 +7,26 @@ description: Overview of Datanode in GreptimeDB, its responsibilities, component
## Introduction
-`Datanode` is mainly responsible for storing the actual data for GreptimeDB. As we know, in GreptimeDB,
-a `table` can have one or more `Region`s, and `Datanode` is responsible for managing the reading and writing
-of these `Region`s. `Datanode` is not aware of `table` and can be considered as a `region server`. Therefore,
-`Frontend` and `Metasrv` operate `Datanode` at the granularity of `Region`.
+A Datanode stores and processes Region data. A table can contain multiple Regions, but the Datanode does not own table-level routing. Frontend sends data requests by Region, while Metasrv controls Region placement and lifecycle.
-
+This boundary lets the same Region server host different storage engines without exposing their implementation to Frontend or Metasrv.
+
+
## Components
-A `Datanode` contains all the components needed for a `region server`. Here we list some of the vital parts:
-
-- A gRPC service is provided for reading and writing region data, and `Frontend` uses this service
- to read and write data from `Datanode`s.
-- An HTTP service, through which you can obtain metrics, configuration information, etc., of the current node.
-- `Heartbeat Task` is used to send heartbeat to the `Metasrv`. The heartbeat plays a crucial role in the
- distributed architecture of GreptimeDB and serves as a basic communication channel for distributed coordination.
- The upstream heartbeat messages contain important information such as the workload of a `Region`. If the
- `Metasrv `has made scheduling(such as `Region` migration) decisions, it will send instructions to the
- `Datanode` via downstream heartbeat messages.
-- The `Datanode` does not parse user SQL or perform distributed planning. The user's query requests for one or
- more `Table`s will be transformed into `Region` query requests in the `Frontend`. The `Datanode` is responsible
- for executing these `Region` query plans with its local query engine.
-- A `Region Manager` is used to manage all `Region`s on a `Datanode`.
-- GreptimeDB supports a pluggable multi-engine architecture, with existing engines including `File Engine` and
- `Mito Engine`.
+The main components are:
+
+- The Region server tracks open Regions and dispatches reads, writes, and lifecycle requests to the engine registered for each Region.
+- `Mito` is the primary time-series Region engine. `Metric` maps many logical metric Regions onto shared Mito Regions, and `File` exposes external files through the Region interface.
+- The local query engine executes Region query plans. It does not parse client SQL or perform cluster-wide planning.
+- The heartbeat task reports node and Region state to Metasrv and receives instructions such as open, close, upgrade, downgrade, and migration steps.
+- gRPC carries Region requests to the Datanode. HTTP exposes node diagnostics such as metrics and configuration.
+
+## Region Request Lifecycle
+
+For a Mito write, the Region server selects Mito from the Region metadata. Mito appends the mutation to the WAL, applies it to a memtable, and later flushes the memtable to SST files. A Metric write is first rewritten with the logical-table identity and then delegated to its physical Mito Region.
+
+For a read, the local query engine executes the Region plan against a table provider backed by the Region engine. A Mito scan takes an immutable Region version, reads the relevant memtables and SST files, merges and deduplicates rows, and returns a stream of Arrow record batches.
+
+Region ownership can change without restarting the Datanode. Metasrv sends lifecycle instructions over the heartbeat stream; the Region server applies them to the engine and reports the new Region role and statistics in subsequent heartbeats.
diff --git a/versioned_docs/version-1.1/contributor-guide/datanode/python-scripts.md b/versioned_docs/version-1.1/contributor-guide/datanode/python-scripts.md
deleted file mode 100644
index 98909142a6..0000000000
--- a/versioned_docs/version-1.1/contributor-guide/datanode/python-scripts.md
+++ /dev/null
@@ -1,35 +0,0 @@
----
-keywords: [Python scripts, data analysis, CPython backend, RustPython interpreter, RecordBatch]
-description: Guide on using Python scripts for data analysis in GreptimeDB, including backend options and setup instructions.
----
-
-# Python Scripts
-
-## Introduction
-
-Python scripts are methods for analyzing data in GreptimeDB,
-by running it in the database directly instead of fetching all the data from the database and running it locally.
-This approach saves a lot of data transfer costs.
-The image below depicts how the script works.
-The `RecordBatch` (which is basically a column in a table with type and nullability metadata)
-can come from anywhere in the database,
-and the returned `RecordBatch` can be annotated in Python grammar to indicate its metadata,
-such as type or nullability.
-The script will do its best to convert the returned object to a `RecordBatch`,
-whether it is a Python list, a `RecordBatch` computed from parameters,
-or a constant (which is extended to the same length as the input arguments).
-
-
-
-## Two optional backends
-
-### CPython Backend powered by PyO3
-
-This backend is powered by [PyO3](https://pyo3.rs/v0.18.1/), enabling the use of your favourite Python libraries (such as NumPy, Pandas, etc.) and allowing Conda to manage your Python environment.
-
-But using it also involves some complications. You must set up the correct Python shared library, which can be a bit challenging. In general, you just need to install the `python-dev` package. However, if you are using Homebrew to install Python on macOS, you must create a proper soft link to `Library/Frameworks/Python.framework`. Detailed instructions on using PyO3 crate with different Python Version can be found [here](https://pyo3.rs/v0.18.1/building_and_distribution#configuring-the-python-version)
-
-### Embedded RustPython Interpreter
-
-An experiment [python interpreter](https://github.com/RustPython/RustPython) to run
-the coprocessor script, it supports Python 3.10 grammar. You can use all the very Python syntax, see [User Guide/Python Coprocessor](/user-guide/python-scripts/overview.md) for more!
diff --git a/versioned_docs/version-1.1/contributor-guide/datanode/query-engine.md b/versioned_docs/version-1.1/contributor-guide/datanode/query-engine.md
index 5b83a30613..91f307aa59 100644
--- a/versioned_docs/version-1.1/contributor-guide/datanode/query-engine.md
+++ b/versioned_docs/version-1.1/contributor-guide/datanode/query-engine.md
@@ -7,51 +7,30 @@ description: Overview of GreptimeDB's query engine, its architecture, data repre
## Introduction
-GreptimeDB's query engine is built on [Apache DataFusion][1] (subproject under [Apache
-Arrow][2]), a brilliant query engine written in Rust. It provides a set of well functional components from
-logical plan, physical plan and the execution runtime. Below explains how each component is orchestrated and their positions during execution.
+GreptimeDB's query engine is built on [Apache DataFusion][1]. DataFusion supplies the logical and physical plan interfaces, optimizer framework, and execution runtime. GreptimeDB adds planners for its query languages, storage-aware optimizer rules, custom plan nodes, and distributed execution.
-
+DDL and other control-plane operations are dispatched by the statement executor. The query engine receives plans for data processing, including the input side of operations such as `INSERT ... SELECT`.
-The entry point is the logical plan, which is used as the general intermediate representation of a
-query or execution logic etc. Two noticeable sources of logical plan are from: 1. the user query, like
-SQL through SQL parser and planner; 2. the Frontend's distributed query, which is explained in details in the following section.
+## Query Lifecycle
-Next is the physical plan, or the execution plan. Unlike the logical plan which is a big
-enumeration containing all the logical plan variants (except the special extension plan node), the
-physical plan is in fact a trait that defines a group of methods invoked during
-execution. All data processing logics are packed in corresponding structures that
-implement the trait. They are the actual operations performed on the data, like
-aggregator `MIN` or `AVG`, and table scan `SELECT ... FROM`.
+1. The SQL, PromQL, or log-query planner resolves tables through the catalog and produces a DataFusion logical plan. GreptimeDB plan extensions represent operations that DataFusion does not provide directly.
+2. DataFusion analyzer and optimizer rules run together with GreptimeDB rules. These rules normalize expressions and types, rewrite time-range operations, push projections and filters toward scans, and introduce distributed plan nodes when required.
+3. The physical planner converts the optimized logical plan into streaming operators. GreptimeDB then applies physical rules for scan parallelism, ordering, and distributed execution.
+4. Execution pulls Arrow record batches through the physical plan. Storage scans receive the projection and predicates, and downstream operators consume the resulting stream without materializing the complete result first.
-The optimization phase which improves execution performance by transforming both logical and physical plans, is now all based on rules. It is also called, "Rule Based Optimization". Some of the rules are DataFusion native and others are customized in Greptime DB. In the future, we plan to add more
-rules and leverage the data statistics for Cost Based Optimization/CBO.
-
-The last phase "execute" is a verb, stands for the procedure that reads data from storage, performs
-calculations and generates the expected results. Although it's more abstract than previously mentioned concepts, you can just
-simply imagine it as executing a Rust async function. And it's indeed a future (stream).
-
-`EXPLAIN [VERBOSE] ` is very useful if you want to see how your SQL is represented in the logical or physical plan.
+Use [`EXPLAIN`](/reference/sql/explain.md) to inspect the logical and physical plans. `EXPLAIN ANALYZE` also executes the plan and reports runtime metrics.
## Data Representation
-GreptimeDB uses [Apache Arrow][2] as the in-memory data representation. It's column-oriented, in
-cross-platform format, and also contains many high-performance data operators. These features
-make it easy to share data in many different environments and implement calculation logic.
+GreptimeDB uses [Apache Arrow][2] record batches as its in-memory data representation. A record batch contains equal-length column arrays and a schema. Query operators exchange streams of these batches, which keeps the execution path columnar from Region scans through result encoding.
## Indexing
-In time series data, there are two important dimensions: timestamp and tag columns (or like
-primary key in a general relational database). GreptimeDB groups data in time buckets, so it's efficient
-to locate and extract data within the expected time range at a very low cost. The mainly used persistent file format [Apache Parquet][3] in GreptimeDB helps a lot -- it
-provides multi-level indices and filters that make it easy to prune data during querying. In the future, we
-will make more use of this feature, and develop our separated index to handle more complex use cases.
+Index construction and persistent index formats belong to the storage engine. The query layer supplies predicates and projections to a scan; Mito then uses time ranges, Parquet statistics, and indexes to avoid reading data that cannot match. See [Data Persistence and Indexing](./data-persistence-indexing.md).
## Distributed Execution
-Covered in [Distributed Querying][6].
+In distributed mode, the Frontend plans the cluster-wide query and Datanodes execute Region-local subplans. [`MergeScan`](../frontend/distributed-querying.md) is the boundary between those stages.
-[1]: https://github.com/apache/arrow-datafusion
+[1]: https://datafusion.apache.org/
[2]: https://arrow.apache.org/
-[3]: https://parquet.apache.org
-[6]: ../frontend/distributed-querying.md
diff --git a/versioned_docs/version-1.1/contributor-guide/datanode/storage-engine.md b/versioned_docs/version-1.1/contributor-guide/datanode/storage-engine.md
index c220c72208..36fef62833 100644
--- a/versioned_docs/version-1.1/contributor-guide/datanode/storage-engine.md
+++ b/versioned_docs/version-1.1/contributor-guide/datanode/storage-engine.md
@@ -7,7 +7,7 @@ description: Overview of the storage engine in GreptimeDB, its architecture, com
## Introduction
-The `storage engine` is responsible for storing the data of the database. Mito, based on [LSMT][1] (Log-structured Merge-tree), is the storage engine we use by default. We have made significant optimizations for handling time-series data scenarios, so mito engine is not suitable for general purposes.
+Mito is GreptimeDB's default storage engine. It uses an [LSM tree][1] and is designed for time-series workloads rather than as a general-purpose embedded storage engine.
## Architecture
@@ -23,9 +23,9 @@ The architecture is the same as a traditional LSMT engine:
media.
- Log records of the WAL can be stored on the local disk, or in a remote log service such as
Kafka (remote WAL) that implements the `Log Store` API.
-- Memtables:
- - Data is written into the `active memtable`, aka `mutable memtable` first.
- - When a `mutable memtable` is full, it will be changed to a `read-only memtable`, aka `immutable memtable`.
+- [Memtables](memtable.md):
+ - Mito routes rows by time index into mutable memtables.
+ - A flush freezes the mutable memtables, installs a new mutable set for writes, and writes the frozen memtables to SST files.
- SST
- The full name of SST, aka SSTable is `Sorted String Table`.
- `Immutable memtable` is flushed to persistent storage and produces an SST file.
@@ -103,7 +103,9 @@ Each Parquet SST is split into row groups, the unit that Parquet can read or ski
Mito supports two SST formats: `flat` and `primary_key`. `flat` is the default for new tables and works well across primary-key cardinalities, including high-cardinality keys. `primary_key` is the legacy format kept for compatibility with older tables. See [SST format](/reference/sql/create.md#create-a-table-with-sst-format) and the [table design guide](/user-guide/deployments-administration/performance-tuning/design-table.md#sst-format) for more details.
-
+
+
+An SST may span more than one compaction time window.
## Scan Pruning
diff --git a/versioned_docs/version-1.1/contributor-guide/datanode/wal.md b/versioned_docs/version-1.1/contributor-guide/datanode/wal.md
index 4ecb19ef02..8f898d91d6 100644
--- a/versioned_docs/version-1.1/contributor-guide/datanode/wal.md
+++ b/versioned_docs/version-1.1/contributor-guide/datanode/wal.md
@@ -7,30 +7,26 @@ description: Introduction to Write-Ahead Logging (WAL) in GreptimeDB, its purpos
## Introduction
-Our storage engine is inspired by the Log-structured Merge Tree (LSMT). Mutating operations are
-applied to a MemTable instead of persisting to disk, which significantly improves performance but
-also brings durability-related issues, especially when the Datanode crashes unexpectedly. Similar
-to all LSMT-like storage engines, GreptimeDB uses a write-ahead log (WAL) to ensure data durability
-and is safe from crashing.
+Mito buffers writes in [memtables](memtable.md) before flushing them to SST files. It first appends each Region's mutations to the write-ahead log (WAL), so data that has not reached an SST can be recovered.
-WAL is an append-only file group. All `INSERT` and `DELETE` operations are transformed into
-operation entries and then appended to WAL. Once operation entries are persisted to the underlying
-file, the operation can be further applied to MemTable.
+The WAL uses a common log-store abstraction with local raft-engine and remote Kafka providers.
-When the Datanode restarts, operation entries in WAL are replayed to reconstruct the correct
-in-memory state.
+## Write and Recovery Cycle
-
+The order of a normal write is:
+
+1. The Region worker assigns sequence numbers and a WAL entry ID.
+2. It appends the mutations to the WAL. If the append fails, the mutations are not applied to the memtable.
+3. After the append succeeds, Mito writes the mutations to the memtable and publishes the new committed sequence.
+4. A flush writes immutable SST files and persists a manifest edit containing the new files and `flushed_entry_id`.
+5. After the manifest edit is durable, WAL entries through `flushed_entry_id` are marked obsolete. The log store may reclaim them later.
+
+The manifest is the recovery boundary. On a normal reopen, Mito rebuilds the Region from the manifest and replays WAL entries starting at `flushed_entry_id + 1`. Region transitions may supply a later replay checkpoint, but they never replay entries before the persisted flush boundary.
## Namespace
-Namespace of WAL is used to separate entries from different tables (different regions). Append and
-read operations must provide a Namespace. Currently, region ID is used as the Namespace, because
-each region has a MemTable that needs to be reconstructed when Datanode restarts.
+WAL entries are isolated by Region, not by table. Each append and read identifies a Region namespace so one Region can be replayed or truncated independently. The local raft-engine provider uses the Region ID as its namespace ID. Kafka keeps Region identity within the provider's topic-backed log.
## Synchronous/Asynchronous flush
-By default, appending to WAL is asynchronous, which means the writer will not wait until entries are
-flushed to disk. This setting provides higher performance, but may lose data when running host shutdown unexpectedly. In the other hand, synchronous flush provides higher durability at the cost of performance.
-
-In v0.4 version, the new region worker architecture can use batching to alleviate the overhead of sync flush.
+For the local raft-engine provider, `sync_write` controls whether an append waits for the log to be synced to durable storage. It defaults to `false`. Asynchronous writes reduce latency but can lose recently acknowledged entries if the host fails before buffered data is synced. Kafka WAL durability is controlled by its producer and cluster settings instead of this local option.
diff --git a/versioned_docs/version-1.1/contributor-guide/flownode/arrangement.md b/versioned_docs/version-1.1/contributor-guide/flownode/arrangement.md
index aed75af777..8b472ea316 100644
--- a/versioned_docs/version-1.1/contributor-guide/flownode/arrangement.md
+++ b/versioned_docs/version-1.1/contributor-guide/flownode/arrangement.md
@@ -5,6 +5,8 @@ description: Details on the arrangement component in Flownode, which stores stat
# Arrangement
+This page describes state used by Flownode's legacy streaming mode. Batching mode does not use an Arrangement.
+
Arrangement stores the state in the dataflow's process. It stores the streams of update flows for further querying and updating.
The arrangement essentially stores key-value pairs with timestamps to mark their change time.
diff --git a/versioned_docs/version-1.1/contributor-guide/flownode/batching_mode.md b/versioned_docs/version-1.1/contributor-guide/flownode/batching_mode.md
index 37a695ce9c..aa8099d1ad 100644
--- a/versioned_docs/version-1.1/contributor-guide/flownode/batching_mode.md
+++ b/versioned_docs/version-1.1/contributor-guide/flownode/batching_mode.md
@@ -9,13 +9,13 @@ This guide provides a brief overview of the batching mode in `flownode`. It's in
## Overview
-The batching mode in `flownode` is designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. This is in contrast to the original streaming mode, now deprecated, where data was processed as it arrived.
+The batching mode in `flownode` is designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. This is in contrast to the legacy streaming path, which processes data as it arrives and is retained for compatibility but deprecated for new workloads.
The core idea is to:
1. Define a `flow` with a SQL query that aggregates data from a source table into a sink table.
2. The query typically includes a time window function (e.g., `date_bin`) on a timestamp column.
3. When new data is inserted into the source table, the system marks the corresponding time windows as "dirty."
-4. A background task periodically wakes up, identifies these dirty windows, and re-runs the aggregation query for those specific time ranges.
+4. A background task runs on its own cadence, consumes the pending dirty windows at its next evaluation, and re-runs the aggregation query for those time ranges.
5. The results are then inserted into the sink table, effectively updating the aggregated view.
## Architecture
@@ -39,15 +39,15 @@ A `BatchingTask` represents a single, independent data flow. Each task is associ
- **State (`TaskState`)**: This contains the dynamic, mutable state of the task, most importantly the `DirtyTimeWindows`.
- **Execution Loop**: The task runs an infinite loop (`start_executing_loop`) that:
1. Checks for a shutdown signal.
- 2. Waits for a scheduled interval or until it's woken up.
+ 2. Sleeps until its next evaluation time. A task with an evaluation schedule sleeps until the next scheduled time; an adaptive task sleeps for a polling interval derived from the time window size and the minimum refresh duration.
3. Generates a new query plan (`gen_insert_plan`) based on the current set of dirty time windows.
4. Executes the query (`execute_logical_plan`) against the database.
5. Cleans up the processed dirty windows.
### `TaskState` and `DirtyTimeWindows`
-- **`TaskState`**: This struct tracks the runtime state of a `BatchingTask`. It includes `dirty_time_windows`, which is crucial for determining what work needs to be done.
-- **`DirtyTimeWindows`**: This is a key data structure that keeps track of which time windows have received new data since the last query execution. It stores a set of non-overlapping time ranges. When a task's execution loop runs, it consults this structure to build a `WHERE` clause that filters the source table for only the dirty time windows.
+- **`TaskState`**: This struct tracks the runtime state of a `BatchingTask`, including the `dirty_time_windows` that determine its pending work.
+- **`DirtyTimeWindows`**: This data structure tracks which time windows have received new data since the last query execution. It stores a set of non-overlapping time ranges. The execution loop uses it to build a `WHERE` clause that selects only the dirty windows from the source table.
### `TimeWindowExpr`
@@ -56,15 +56,15 @@ The `TimeWindowExpr` is a helper utility for dealing with time window expression
- **Evaluation**: It can take a timestamp and evaluate the time window expression to determine the start and end of the window that the timestamp falls into.
- **Window Size**: It can also determine the size (duration) of the time window from the expression.
-This is essential for both marking windows as dirty and for generating the correct filter conditions when querying the source table.
+The same calculation is used to mark dirty windows and generate the source-table filters.
## Query Execution Flow
Here's a simplified step-by-step walkthrough of how a query is executed in batch mode:
1. **Data Ingestion**: New data is written to a source table.
-2. **Marking Dirty**: The `BatchingEngine` receives a notification about the new data. It uses the `TimeWindowExpr` associated with each relevant flow to determine which time windows are affected by the new data points. These windows are then added to the `DirtyTimeWindows` set in the corresponding `TaskState`.
-3. **Task Wake-up**: The `BatchingTask`'s execution loop wakes up, either due to its periodic schedule or because it was notified of a large backlog of dirty windows.
+2. **Marking Dirty**: The `BatchingEngine` receives a notification about the new data. It uses the `TimeWindowExpr` associated with each relevant flow to determine which time windows are affected by the new data points. These windows are then added to the `DirtyTimeWindows` set in the corresponding `TaskState`. Marking a window dirty does not wake the task.
+3. **Next Evaluation**: The `BatchingTask`'s execution loop reaches its next evaluation, either at a scheduled time or after its adaptive polling interval, and consumes the pending dirty windows.
4. **Plan Generation**: The task calls `gen_insert_plan`. This method:
- Inspects the `DirtyTimeWindows`.
- Generates a series of `OR`'d `WHERE` clauses (e.g., `(ts >= 't1' AND ts < 't2') OR (ts >= 't3' AND ts < 't4') ...`) that cover the dirty windows.
diff --git a/versioned_docs/version-1.1/contributor-guide/flownode/dataflow.md b/versioned_docs/version-1.1/contributor-guide/flownode/dataflow.md
index 000a65edb3..c876054d6d 100644
--- a/versioned_docs/version-1.1/contributor-guide/flownode/dataflow.md
+++ b/versioned_docs/version-1.1/contributor-guide/flownode/dataflow.md
@@ -1,17 +1,38 @@
---
-keywords: [dataflow module, SQL query transformation, execution plan, DAG, map and reduce operations]
-description: Explanation of the dataflow module in Flownode, its operations, internal data handling, and future enhancements.
+keywords: [Flownode, batching mode, streaming mode, dataflow, dirty time windows]
+description: How Flownode selects and runs its batching and legacy streaming execution paths.
---
# Dataflow
+Flownode has two internal execution paths:
+
+- **Batching mode** is the primary path for aggregation and TQL workloads. It evaluates queries over persisted source data and writes materialized results to a sink table.
+- **Streaming mode** is the legacy path retained for compatibility and deprecated for new workloads. It incrementally processes rows mirrored from Frontend as they arrive.
+
+Users do not select the mode directly. When a Flow is created, GreptimeDB chooses the path from the query and source-table properties. Aggregation, `DISTINCT`, and TQL queries use batching mode. Simple non-aggregation queries, and any Flow whose source table has `ttl = 'instant'`, currently use streaming mode. A Flow deferred because its source table does not yet exist starts as a pending batching Flow.
+
+## Batching mode
+
+Batching mode reuses GreptimeDB's query engine instead of maintaining an operator graph for every incoming row. For a time-windowed Flow, its main loop is:
+
+1. A source-table write marks the affected time windows as dirty.
+2. A `BatchingTask` runs on its evaluation schedule or adaptive polling cadence and collects the pending dirty windows at that evaluation. Marking a window dirty does not wake the task.
+3. The task adds time predicates for those windows to the Flow query and asks Frontend to execute it against the source tables.
+4. The query result is inserted into the sink table, updating the materialized result for windows that were evaluated.
+5. Successfully processed windows are removed from the dirty set. Failed work remains available for a later evaluation.
+
+Flows with an evaluation interval but without a time-window expression run the complete query on each scheduled evaluation. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Flownode Batching Mode Developer Guide](./batching_mode.md) for the task and dirty-window components.
+
+## Streaming mode
+
The `dataflow` module (see `flow::compute` module) is the core computing module of `flow`.
It takes a SQL query and transforms it into flow's internal execution plan.
This execution plan is then rendered into an actual dataflow, which is essentially a directed acyclic graph (DAG) of functions with input and output ports.
-The dataflow is triggered to run when needed.
+New row changes drive the graph incrementally.
-Currently, this dataflow only supports `map` and `reduce` operations. Support for `join` operations will be added in the future.
+The renderer supports map/filter/project and reduce operations. Join and union plan nodes exist, but their streaming renderers are not implemented.
Internally, the dataflow handles data in row format, using a tuple `(row, time, diff)`. Here, `row` represents the actual data being passed, which may contain multiple `Value` objects.
`time` is the system time which tracks the progress of the dataflow, and `diff` typically represents the insertion or deletion of the row (+1 or -1).
-Therefore, the tuple represents the insert/delete operation of the `row` at a given system `time`.
\ No newline at end of file
+Therefore, the tuple represents the insert/delete operation of the `row` at a given system `time`. Stateful operators keep indexed traces of these changes in an [Arrangement](./arrangement.md).
diff --git a/versioned_docs/version-1.1/contributor-guide/frontend/distributed-querying.md b/versioned_docs/version-1.1/contributor-guide/frontend/distributed-querying.md
index 21ee07d7e8..ca3822e113 100644
--- a/versioned_docs/version-1.1/contributor-guide/frontend/distributed-querying.md
+++ b/versioned_docs/version-1.1/contributor-guide/frontend/distributed-querying.md
@@ -5,29 +5,16 @@ description: Describes the process of distributed querying in GreptimeDB, focusi
# Distributed Querying
-Most steps of querying in frontend and datanode are identical. The only difference is that
-Frontend have a "special" step in planning phase to make the logical query plan distributed.
-Let's reference it as "dist planner" in the following text.
-
-The modified, distributed logical plan has multiple stages, each of them is executed in different
-server node.
+Frontend and Datanode use the same DataFusion-based query engine. In distributed mode, Frontend adds a planning step that separates work performed by Datanodes from work completed by Frontend.
## Dist Planner
-Planner will traverse the input logical plan, and split it into multiple stages by the "[commutativity
-rule](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md)".
+The distributed planner rewrites the logical plan. It pushes compatible operators toward table scans and wraps remote subplans in `MergeScan` nodes. Partition predicates are also used to prune Regions before the remote work is scheduled.
-This rule is under heavy development. At present it will consider things like:
-- whether the operator itself is commutative
-- how the partition rule is configured
-- etc...
+Whether an operator can be pushed down depends on the plan shape and the operator's properties. Unsupported parts remain on Frontend. The original design and its commutativity rules are described in the [distributed planner RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md).
## Dist Plan
-Except the first stage, which have to read data from files in storage. All other stages' leaf node
-are actually a gRPC call to its previous stage.
-
-Sub-plan in a stage is itself a complete logical plan, and can be executed independently without
-the follow up stages. The plan is encoded in [substrait format](https://substrait.io).
+A remote input is a complete logical subplan, not just a table scan. Frontend serializes the subplan in [Substrait](https://substrait.io) format and sends a Region-specific request to the Datanode that owns the data. The Datanode plans and executes it locally, then streams the result back. Frontend merges the remote streams and executes any operators that were not pushed down.
diff --git a/versioned_docs/version-1.1/contributor-guide/frontend/overview.md b/versioned_docs/version-1.1/contributor-guide/frontend/overview.md
index 3a2f63e7ae..18cabfc2b8 100644
--- a/versioned_docs/version-1.1/contributor-guide/frontend/overview.md
+++ b/versioned_docs/version-1.1/contributor-guide/frontend/overview.md
@@ -5,27 +5,47 @@ description: Overview of GreptimeDB's Frontend component - a stateless proxy ser
# Frontend
-The **Frontend** is a stateless service that serves as the entry point for client requests in GreptimeDB. It provides a unified interface for multiple database protocols and acts as a proxy that forwards read/write requests to appropriate Datanodes in the distributed system.
+Frontend is GreptimeDB's stateless request-orchestration service. The server layer terminates protocols and converts wire messages; Frontend supplies the database behavior behind those handlers, including permission checks, statement execution, routing, and distributed query planning.
+
+Frontend does not store table data. It caches catalog and route metadata obtained from Metasrv, and Metasrv invalidates those caches through heartbeat responses when metadata changes.
## Core Functions
-- **Protocol Support**: Multiple database protocols including SQL, PromQL, MySQL, and PostgreSQL. See [Protocols][1] for details
-- **Request Routing**: Routes requests to appropriate Datanodes based on metadata
-- **Query Distribution**: Splits distributed queries across multiple nodes
-- **Response Aggregation**: Combines results from multiple Datanodes
-- **Authorization**: Security and access control validation
+- Provide query and ingestion behavior for the supported [protocols][1].
+- Resolve catalogs, schemas, tables, and Region routes.
+- Validate permissions before executing a request.
+- Plan distributed queries and merge results from Datanodes.
+- Convert table-level writes and deletes into Region requests.
## Architecture
### Key Components
-- **Protocol Handlers**: Handle different database protocols
-- **Catalog Manager**: Caches metadata from Metasrv to enable efficient request routing and schema validation
-- **Dist Planner**: Converts logical plans to distributed execution plans
-- **Request Router**: Determines target Datanodes for each request
+
+- Protocol handlers adapt SQL, PromQL, gRPC ingestion, and observability protocols to Frontend's internal request interfaces.
+- The catalog and partition managers provide table metadata, partition rules, and Region routes.
+- The statement executor dispatches queries, DML, and DDL to their respective execution paths.
+- The distributed planner replaces table scans with `MergeScan` plans that can run across Datanodes.
### Request Flow
-
+The request path depends on the operation.
+
+#### Queries
+
+1. A protocol handler creates the query context and performs authentication and permission checks.
+2. The language-specific planner produces a logical plan. In distributed mode, the planner uses partition metadata to select Regions and constructs a distributed plan.
+3. Frontend sends Region subplans to the owning Datanodes. Datanodes execute them against local Region engines and return streams of Arrow record batches.
+4. Frontend runs the remaining operators, merges the streams, and formats the result for the client protocol.
+
+#### Writes and deletes
+
+1. Frontend validates the request against the table schema. Protocols that support schema-on-write may create a missing table or add columns before retrying the write.
+2. The partition rule assigns rows to Regions. Frontend builds one Region request per target and routes it to the current Region leader.
+3. The Datanode's Region server dispatches each request to the Region engine. In standalone mode, the same request is sent to an embedded Region server instead of over RPC.
+
+#### DDL
+
+The statement executor converts DDL into a task. In distributed mode, Metasrv runs that task as a persisted procedure, updates metadata, and coordinates Region operations on Datanodes. Standalone mode uses the same statement boundary with local implementations of the metadata and procedure services.
### Deployment
diff --git a/versioned_docs/version-1.1/contributor-guide/frontend/table-sharding.md b/versioned_docs/version-1.1/contributor-guide/frontend/table-sharding.md
index a60276d14e..beaece40c0 100644
--- a/versioned_docs/version-1.1/contributor-guide/frontend/table-sharding.md
+++ b/versioned_docs/version-1.1/contributor-guide/frontend/table-sharding.md
@@ -5,21 +5,15 @@ description: Explains how table data in GreptimeDB is sharded and distributed, i
# Table Sharding
-The sharding of stored data is essential to any distributed database. This document will describe how table's data in GreptimeDB is being sharded, and distributed.
+GreptimeDB shards a table into Regions. Partition expressions define which rows belong to each Region, while Region routes define which Datanode currently owns each Region.
## Partition
-For the syntax of creating a partitioned table, please refer to the [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) section in the User Guide.
+A partition is a logical row set described by an expression over one or more columns. The partition layout must cover the table's input domain so each row has one target Region. See [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) for the SQL syntax and supported expressions.
## Region
-The data within a table is logically split after creating partitions. You may ask the question "
-how are the data, after being logically partitioned, stored in the GreptimeDB? The answer is in "`Region`"s.
-
-Each region is corresponding to a partition, and stores the data in the partition. The regions are distributed among
-`Datanode`s. `Metasrv` manages the route information that maps regions to Datanodes.
-If the partition layout needs to change after table creation, GreptimeDB supports explicit
-[repartitioning](/user-guide/deployments-administration/manage-data/repartition.md) through split and merge operations.
+Each partition maps to one Region. Region IDs remain the storage and routing identity used by Frontend, Datanode, and Metasrv. Multiple Regions from the same table may be placed on one Datanode.
The relationship between partition and region can be viewed as the following diagram:
@@ -53,3 +47,14 @@ The relationship between partition and region can be viewed as the following dia
│ │
└──────────────────────────────────┘
Could be placed in one Datanode
+```
+
+## Routing and Pruning
+
+For writes, Frontend evaluates the partition rule for each row, groups rows by Region, and sends Region requests to the current leaders from the route table.
+
+For queries, the distributed planner compares query predicates with the partition expressions. It scans only Regions that can satisfy the predicates. If partition metadata is missing or cannot be interpreted safely, the planner falls back to all Regions rather than risk omitting data.
+
+## Changing the Partition Layout
+
+[Repartitioning](/user-guide/deployments-administration/manage-data/repartition.md) changes an existing layout through explicit split and merge operations. Metasrv runs the change as a persisted procedure, updates the Region routes and partition expressions, and invalidates stale table-route caches. New requests use the published layout after their Frontend refreshes that metadata.
diff --git a/versioned_docs/version-1.1/contributor-guide/getting-started.md b/versioned_docs/version-1.1/contributor-guide/getting-started.md
index b17184dfa8..7e8cca3ad8 100644
--- a/versioned_docs/version-1.1/contributor-guide/getting-started.md
+++ b/versioned_docs/version-1.1/contributor-guide/getting-started.md
@@ -15,14 +15,13 @@ At the moment, GreptimeDB supports Linux (both amd64 and arm64), macOS (both amd
### Build Dependencies
-- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) (optional)
+- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) (optional; needed to clone the repository, not to build it)
- C/C++ Toolchain: provides essential tools for compiling and linking. This is available either as `build-essential` on ubuntu or a similar name on other platforms.
-- Rust nightly toolchain ([guide][1])
- - Compile the source code
+- [Rustup][1]. The repository pins the required nightly toolchain in `rust-toolchain.toml`.
- Protobuf ([guide][2])
- Compile the proto file
- Note that the version needs to be >= 3.15. You can check it with `protoc --version`
-- Machine: Recommended memory is 16GB or more, or use the [mold](https://github.com/rui314/mold) tool to reduce memory usage during linking.
+- Machine: 16GB of memory or more is recommended. On a smaller machine, use [mold](https://github.com/rui314/mold) to reduce memory usage during linking.
[1]:
[2]:
diff --git a/versioned_docs/version-1.1/contributor-guide/how-to/how-to-write-sdk.md b/versioned_docs/version-1.1/contributor-guide/how-to/how-to-write-sdk.md
index 40d0bc3713..e306448642 100644
--- a/versioned_docs/version-1.1/contributor-guide/how-to/how-to-write-sdk.md
+++ b/versioned_docs/version-1.1/contributor-guide/how-to/how-to-write-sdk.md
@@ -1,21 +1,17 @@
---
keywords: [gRPC SDK, GreptimeDatabase, Handle, HandleRequests, GreptimeRequest, GreptimeResponse]
-description: Explains how to write a gRPC SDK for GreptimeDB, focusing on the GreptimeDatabase service, its methods, and the structure of requests and responses.
+description: Protocol contracts and error-handling requirements for a GreptimeDB gRPC ingestion SDK.
---
# How to write a gRPC SDK for GreptimeDB
-A GreptimeDB gRPC SDK only needs to handle the writes. The reads are standard SQL and PromQL, can be handled by any JDBC
-client or Prometheus client. This is also why GreptimeDB gRPC SDKs are all named
-like "`greptimedb-ingester-`". Please make sure your GreptimeDB SDK follow the same naming convention.
+GreptimeDB's public gRPC SDKs are ingestion clients. Queries normally use SQL or PromQL through their standard clients. A new SDK should therefore focus on writes and deletes unless it has a separate requirement, and follow the `greptimedb-ingester-` naming convention. See the [gRPC SDK overview](/user-guide/ingest-data/for-iot/grpc-sdks/overview.md) for the user-facing API.
## `GreptimeDatabase` Service
-GreptimeDB defines a custom gRPC service called `GreptimeDatabase`. All you need to do in your SDK are implement it. You
-can find its Protobuf
-definitions [here](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto).
+Generate client stubs from the [`GreptimeDatabase` Protobuf definition](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto). Do not maintain a handwritten copy of the messages or service definition.
-The service contains two RPC methods:
+The service provides a unary method and a client-streaming method:
```protobuf
service GreptimeDatabase {
@@ -25,13 +21,9 @@ service GreptimeDatabase {
}
```
-The `Handle` method is for unary call: when a `GreptimeRequest` is received and processed by a GreptimeDB
-server, it responds with a `GreptimeResponse` immediately.
+`Handle` returns one response for one request. It is the usual choice for an SDK's insert and delete APIs.
-The `HandleRequests` acts in
-a "[Client streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc)" style. It ingests a
-stream of `GreptimeRequest`, and handles them on the fly. After all the requests have been handled, it returns a
-summarized `GreptimeResponse`. Through `HandleRequests`, we can achieve a very high throughput of requests handling.
+`HandleRequests` is a [client-streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc). The server returns a cumulative response only after the client closes the request stream. An SDK that exposes streaming must document this acknowledgement boundary and bind the stream to one endpoint.
### `GreptimeRequest`
@@ -51,13 +43,13 @@ message GreptimeRequest {
}
```
-A `RequestHeader` is needed, it includes some context, authentication and others. The "oneof" field contains the request
-to the GreptimeDB server.
+A client must populate `RequestHeader` with the database context and authentication expected by the server. Set exactly one request variant.
-Note that we have two types of insertions, one is in the form of "column" (the `InsertRequests`), and the other is "
-row" (`RowInsertRequests`). It's generally recommended to use the "row" form, since it's more natural for insertions on
-a table, and easier to use. However, if there's a need to insert a large number of columns at once, or there're plenty
-of "null" values to insert, the "column" form is better to be used.
+The message also contains query and DDL variants used by internal callers. The public ingester API should not expose them: `GreptimeDatabase` does not return query result streams.
+
+GreptimeDB accepts row-oriented `RowInsertRequests` and column-oriented `InsertRequests`. Row-oriented requests are the default for public ingestion APIs. A column-native client may use the column form, but it must keep column lengths consistent and preserve null values, timestamp precision, data types, and column semantic types during conversion.
+
+Deletes have the same row-oriented and column-oriented distinction. Expose only the forms that the SDK can map without losing type information.
### `GreptimeResponse`
@@ -70,8 +62,18 @@ message GreptimeResponse {
}
```
-The `ResponseHeader` contains the response's status code, and error message (if there's any). The "oneof" response only
-contains the affected rows for now.
+On success, the response contains a successful header and `affected_rows`. Treat that value as the number acknowledged by the server, including the cumulative value returned when a request stream closes.
+
+Request failures are returned as a gRPC status. When present, the trailing metadata key `x-greptime-err-code` carries GreptimeDB's error code, and the status message carries the error text. Preserve the gRPC status and expose the GreptimeDB error code rather than replacing them with a generic SDK error.
+
+## Retry and Delivery Semantics
+
+Retries must be bounded and observable. A unary request may be retried only when the failure is classified as retryable and the deadline still permits it. Do not retry cancellation or deadline-expiration errors.
+
+A lost response does not prove that the server rejected a write. Retrying such a request can insert duplicate rows unless the caller's data model makes the operation idempotent. Document this possibility and return the final error when delivery is ambiguous.
+
+Do not transparently retry a partially sent `HandleRequests` stream. The server may already have accepted some requests even though the client has not received the cumulative response. Close the failed stream and report the ambiguity to the caller.
+
+Keep Arrow Flight bulk ingestion separate from the `GreptimeDatabase` RPCs. Its batching and partial-acceptance behavior needs its own API contract.
-GreptimeDB has a lot of SDKs now, you can refer to
-them [here](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=) for some examples.
+Use the existing [GreptimeDB ingester repositories](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=) to compare public API conventions, but derive wire behavior from the current Protobuf definition and server contract.
diff --git a/versioned_docs/version-1.1/contributor-guide/metasrv/admin-api.md b/versioned_docs/version-1.1/contributor-guide/metasrv/admin-api.md
index b091b48b70..d1f2ee6dbe 100644
--- a/versioned_docs/version-1.1/contributor-guide/metasrv/admin-api.md
+++ b/versioned_docs/version-1.1/contributor-guide/metasrv/admin-api.md
@@ -1,6 +1,6 @@
---
-keywords: [admin api, health check, leader query, heartbeat, maintenance mode, RESTful API]
-description: Details the Admin API for Metasrv, including endpoints for health checks, leader queries, heartbeat data, maintenance mode, and Procedure Manager controls.
+keywords: [admin api, health check, leader query, heartbeat, maintenance mode, recovery mode, table id sequence]
+description: Details the Metasrv Admin API for status inspection, cluster controls, and metadata recovery.
---
# Admin API
@@ -9,16 +9,17 @@ description: Details the Admin API for Metasrv, including endpoints for health c
Note that all Admin API endpoints in this document listen on Metasrv's `HTTP_PORT`, which defaults to `4000`.
:::
-The Admin API provides a simple way to view and manage cluster information, including metasrv health detection, metasrv leader query, datanode heartbeat detection, maintenance mode, and Procedure Manager controls.
-
-The Admin API is an HTTP service that provides a set of RESTful APIs that can be called through HTTP requests. The Admin API is simple, user-friendly and safe.
+The Admin API exposes Metasrv status, cluster controls, and metadata recovery operations over HTTP. It does not provide authentication, and some endpoints change cluster behavior or metadata allocation. Deployments must protect the HTTP port with network-level controls.
This page covers the following APIs:
- /health
- /leader
- /heartbeat
+- /node-lease
- /maintenance
- /procedure-manager
+- /recovery
+- /sequence/table
All these APIs are under the parent resource `/admin`.
@@ -26,7 +27,7 @@ In the following sections, we assume that your metasrv instance is running on lo
## /health HTTP endpoint
-The `/health` endpoint accepts GET HTTP requests and you can use this endpoint to check the health of your metasrv instance.
+The `/health` endpoint accepts GET requests and returns `OK` when the HTTP service is running. It does not check whether this Metasrv is the leader or whether external dependencies are available.
### Definition
@@ -120,9 +121,17 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100'
]
```
+## /node-lease HTTP endpoint
+
+The `/node-lease` endpoint returns the current leases recorded for Datanodes. Use it when diagnosing whether Metasrv still considers a Datanode active.
+
+```bash
+curl -X GET http://localhost:4000/admin/node-lease
+```
+
## /maintenance HTTP endpoint
-Cluster Maintenance Mode is a safety feature in GreptimeDB that temporarily disables automatic cluster management operations. This mode is particularly useful during cluster upgrades, planned downtime, and any operation that might temporarily affect cluster stability. For more details, please refer to [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md).
+Maintenance mode temporarily disables automatic cluster management operations during upgrades, planned downtime, or similar work. See [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md) for its effect on the cluster.
The `/maintenance` endpoint supports the following HTTP requests:
@@ -155,3 +164,39 @@ The response body uses the following format:
"status": "running"
}
```
+
+## /recovery HTTP endpoints
+
+Recovery mode gates metadata repair endpoints such as manual table ID sequence changes. It is intended for recovery work, not routine maintenance.
+
+- `GET /admin/recovery/status`: query whether recovery mode is enabled.
+- `POST /admin/recovery/enable`: enable recovery mode.
+- `POST /admin/recovery/disable`: disable recovery mode.
+
+The response body uses the following format:
+
+```json
+{
+ "enabled": true
+}
+```
+
+Disable recovery mode after the repair is complete. Use [maintenance mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md) instead when the goal is to suspend automatic cluster operations during planned maintenance.
+
+## /sequence/table HTTP endpoints
+
+These endpoints inspect or repair the table ID sequence:
+
+- `GET /admin/sequence/table/next-id`: return the next table ID without allocating it.
+- `POST /admin/sequence/table/set-next-id`: advance the next table ID.
+
+Setting the sequence requires recovery mode. The new value must be greater than the current value; the API cannot move the sequence backwards. Recovery mode is an API precondition, not a DDL barrier. Follow [Manage table ID sequences](/user-guide/deployments-administration/maintenance/sequence-management.md) for the required cluster-wide procedure.
+
+```bash
+curl -X POST \
+ -H 'Content-Type: application/json' \
+ -d '{"next_table_id": 2048}' \
+ http://localhost:4000/admin/sequence/table/set-next-id
+```
+
+Changing this value affects IDs allocated to future tables.
diff --git a/versioned_docs/version-1.1/contributor-guide/metasrv/overview.md b/versioned_docs/version-1.1/contributor-guide/metasrv/overview.md
index 7aa052dd7e..6458b6380d 100644
--- a/versioned_docs/version-1.1/contributor-guide/metasrv/overview.md
+++ b/versioned_docs/version-1.1/contributor-guide/metasrv/overview.md
@@ -1,161 +1,101 @@
---
-keywords: [metasrv, metadata, request-router, load balancing, election, high availability, heartbeat]
-description: Provides an overview of the Metasrv service, its components, interactions with the Frontend, architecture, and key functionalities like distributed consensus and heartbeat management.
+keywords: [metasrv, metadata, routing, leader election, procedure, heartbeat]
+description: Overview of the metadata and coordination mechanisms provided by Metasrv.
---
# Metasrv
-
-
## What's in Metasrv
-- Store metadata (Catalog, Schema, Table, Region, etc.)
-- Request-Router. It tells the Frontend where to write and read data.
-- Load balancing for Datanode, determines who should handle new table creation requests, more precisely, it makes resource allocation decisions.
-- Election & High Availability, GreptimeDB is designed in a Leader-Follower architecture, only Leader nodes can write while Follower nodes can read, the number of Follower nodes is usually >= 1, and Follower nodes need to be able to switch to Leader quickly when Leader is not available.
-- Statistical data collection (reported via Heartbeats on each node), such as CPU, Load, number of Tables on the node, average/peak data read/write size, etc., can be used as the basis for distributed scheduling.
+Metasrv is the metadata and coordination service in a distributed GreptimeDB cluster. It does not sit on the data path. Its main responsibilities are:
+
+- storing Catalog, Schema, Table, Region, route, and node metadata;
+- choosing Datanodes for new Regions and maintaining table routes;
+- electing one Metasrv leader to coordinate metadata changes;
+- running recoverable procedures for DDL, Region migration, failover, and repartitioning;
+- tracking node leases and Region statistics through heartbeats;
+- broadcasting cache invalidations to Frontends, Datanodes, and Flownodes when metadata changes;
+- sending Region lifecycle instructions to Datanodes.
## How the Frontend interacts with Metasrv
-First, the routing table in Request-Router is in the following structure (note that this is only the logical structure, the actual storage structure varies, for example, endpoints may have dictionary compression).
+Frontend obtains table metadata and Region routes from Metasrv and caches them locally. Metadata-changing statements are sent to the Metasrv leader, while reads and writes use the cached routes to reach Datanodes directly.
+
+The control and data paths are separate:
+
+```text
+Frontend
+ |-- metadata lookup and DDL ------------> Metasrv leader
+ `-- Region reads and writes ------------> Datanode
+
+Metasrv leader
+ |-- Region lifecycle instructions ------> Datanode
+ `-- cache invalidations ----------------> Frontend / Datanode / Flownode
+Datanode
+ `-- heartbeat, lease renewal, Region stats -> Metasrv leader
```
- table_A
- table_name
- table_schema // for physical plan
- regions
- region_1
- mutate_endpoint
- select_endpoint_1, select_endpoint_2
- region_2
- mutate_endpoint
- select_endpoint_1, select_endpoint_2, select_endpoint_3
- region_xxx
- table_B
- ...
+
+In steady state, a table route records one leader peer and zero or more follower peers for each Region. The leader is the write target. Deployments with read-replica support can route reads to followers:
+
+```text
+Table route
+ |-- Region 0
+ | |-- leader -> Datanode A
+ | `-- followers -> Datanode B, Datanode C
+ `-- Region 1
+ `-- leader -> Datanode D
```
+Region migration or failover changes peer roles and can temporarily leave a Region without a leader. Frontend refreshes its cached route before sending subsequent reads or writes to the current peers.
+
### Create Table
-1. The Frontend sends `CREATE TABLE` requests to Metasrv.
-2. Plan the number of Regions according to the partition rules contained in the request.
-3. Check the global view of resources available to Datanodes (collected by Heartbeats) and assign one node to each region.
-4. The Frontend creates the table and stores the `Schema` to Metasrv after successful creation.
+1. Frontend submits the DDL request to the Metasrv leader.
+2. Metasrv derives Regions from the partition rules and [selects a Datanode for each Region](/contributor-guide/metasrv/selector.md).
+3. A persisted procedure creates the Regions and records the table and route metadata. If leadership changes, the procedure can resume from its persisted state.
+4. Metasrv notifies Frontends after the metadata change is committed so their caches can be refreshed.
### Insert
-1. The Frontend fetches the routes of the specified table from Metasrv. Note that the smallest routing unit is the route of the table (several regions), i.e., it contains the addresses of all regions of this table.
-2. The best practice is that the Frontend first fetches the routes from its local cache and forwards the request to the Datanode. If the route is no longer valid, then Datanode is obliged to return an `Invalid Route` error, and the Frontend re-fetches the latest data from Metasrv and updates its cache. Route information does not change frequently, thus, it's sufficient for Frontend uses the Lazy policy to maintain the cache.
-3. The Frontend processes a batch of writes that may contain multiple tables and multiple regions, so the Frontend needs to split user requests based on the 'route table'.
+Frontend resolves the table route, splits rows according to the partition rules, and sends each Region write to the corresponding Datanode. Route changes cause the cached metadata to be invalidated and fetched again from Metasrv.
### Select
-1. As with `Insert`, the Frontend first fetches the route table from the local cache.
-2. Unlike `Insert`, for `Select`, the Frontend needs to extract the read-only node (follower) from the route table, then dispatch the request to the leader or follower node depending on the priority.
-3. The distributed query engine in the Frontend distributes multiple sub-query tasks based on the routing information and aggregates the query results.
+Frontend uses table and Region metadata while planning the query. Predicates on partition columns prune Regions, and the distributed query engine sends work to the Datanodes that own the selected Regions. See [Distributed Querying](../frontend/distributed-querying.md).
## Metasrv Architecture
-
-
-## Distributed Consensus
+The main coordination paths are:
+
+```text
+Leader election
+ |
+ v
+Metasrv leader
+├─ DDL manager -> Procedure manager
+├─ Selector -> new Region placement
+├─ Heartbeat handler chain -> leases and Region statistics
+├─ Region supervisor -> Region migration procedures
+├─ Mailbox -> cache invalidations and Region instructions
+└─ Metadata managers -> KV backend
+```
-As you can see, Metasrv has a dependency on distributed consensus because:
+These mechanisms share metadata, but they have different failure boundaries. A process restart may discard caches and leader-local state; metadata and procedure state required for recovery must be durable.
-1. First, Metasrv has to elect a leader, Datanode only sends heartbeats to the leader, and we only use a single metasrv node to receive heartbeats, which makes it easy to do some calculations or scheduling accurately and quickly based on global information. As for how the Datanode connects to the leader, this is for MetaClient to decide (using a redirect, Heartbeat requests becomes a gRPC stream, and using redirect will be less error-prone than forwarding), and it is transparent to the Datanode.
-2. Second, Metasrv must provide an election API for Datanode to elect "write" and "read-only" nodes and help Datanode achieve high availability.
-3. Finally, `Metadata`, `Schema` and other data must be reliably and consistently stored on Metasrv. Therefore, consensus-based algorithms are the ideal approach for storing them.
+## Distributed Consensus
-For the first version of Metasrv, we choose Etcd as the consensus algorithm component (Metasrv is designed to consider adapting different implementations and even creating a new wheel) for the following reasons:
+Metasrv separates leader election from metadata storage. Only the elected Metasrv leader performs coordination and metadata-changing operations. Other Metasrv nodes direct clients to the current leader.
-1. Etcd provides exactly the API we need, such as `Watch`, `Election`, `KV`, etc.
-2. We only perform two tasks with distributed consensus: elections (using the `Watch` mechanism) and storing (a small amount of metadata), and neither of them requires us to customize our own state machine, nor do we need to customize our own state machine based on raft; the small amount of data also does not require multi-raft-group support.
-3. The initial version of Metasrv uses Etcd, which allows us to focus on the capabilities of Metasrv and not spend too much effort on distributed consensus algorithms, which improves the design of the system (avoiding coupling with consensus algorithms) and helps with rapid development at the beginning, as well as allows easy access to good consensus algorithm implementations in the future through good architectural designs.
+The key-value backend stores table metadata, routes, procedure state, and other information that must survive a leader change. Metasrv does not use this election to create leader and follower replicas for Datanode Regions; Region availability is managed through heartbeats, Region failure detection, and failover procedures.
## Heartbeat Management
-The primary means of communication between Datanode and Metasrv is the Heartbeat Request/Response Stream, and we want this to be the only way to communicate. This idea is inspired by the design of [TiKV PD](https://github.com/tikv/pd), and we have practical experience in [RheaKV](https://github.com/sofastack/sofa-jraft/tree/master/jraft-rheakv/rheakv-pd). The request sends its state, while Metasrv sends different scheduling instructions via Heartbeat Response.
-
-A heartbeat will probably carry the data listed below, but this is not the final design, and we are still discussing and exploring exactly which data should be mostly collected.
-
-```
-service Heartbeat {
- // Heartbeat, there may be many contents of the heartbeat, such as:
- // 1. Metadata to be registered to metasrv and discoverable by other nodes.
- // 2. Some performance metrics, such as Load, CPU usage, etc.
- // 3. The number of computing tasks being executed.
- rpc Heartbeat(stream HeartbeatRequest) returns (stream HeartbeatResponse) {}
-}
-
-message HeartbeatRequest {
- RequestHeader header = 1;
-
- // Self peer
- Peer peer = 2;
- // Leader node
- bool is_leader = 3;
- // Actually reported time interval
- TimeInterval report_interval = 4;
- // Node stat
- NodeStat node_stat = 5;
- // Region stats in this node
- repeated RegionStat region_stats = 6;
- // Follower nodes and stats, empty on follower nodes
- repeated ReplicaStat replica_stats = 7;
-}
-
-message NodeStat {
- // The read capacity units during this period
- uint64 rcus = 1;
- // The write capacity units during this period
- uint64 wcus = 2;
- // Table number in this node
- uint64 table_num = 3;
- // Region number in this node
- uint64 region_num = 4;
-
- double cpu_usage = 5;
- double load = 6;
- // Read disk I/O in the node
- double read_io_rate = 7;
- // Write disk I/O in the node
- double write_io_rate = 8;
-
- // Others
- map attrs = 100;
-}
-
-message RegionStat {
- uint64 region_id = 1;
- TableName table_name = 2;
- // The read capacity units during this period
- uint64 rcus = 3;
- // The write capacity units during this period
- uint64 wcus = 4;
- // Approximate region size
- uint64 approximate_size = 5;
- // Approximate number of rows
- uint64 approximate_rows = 6;
-
- // Others
- map attrs = 100;
-}
-
-message ReplicaStat {
- Peer peer = 1;
- bool in_sync = 2;
- bool is_learner = 3;
-}
-```
-
-## Central Nervous System (CNS)
-
-We are to build an algorithmic system, which relies on real-time and historical heartbeat data from each node, should make some smarter scheduling decisions and send them to Metasrv's Autoadmin unit, which distributes the scheduling decisions, either by the Datanode itself or more likely by the PaaS platform.
-
-## Abstraction of Workloads
+Datanodes maintain heartbeat streams to the Metasrv leader. Heartbeat requests report node identity, lease information, Region statistics, and other state used for placement and supervision. Responses carry control messages such as Region lifecycle instructions and cache invalidations.
-The level of workload abstraction determines the efficiency and quality of the scheduling strategy generated by Metasrv such as resource allocation.
+A heartbeat drives two independent mechanisms, and a change to heartbeat timing affects both:
-DynamoDB defines RCUs & WCUs (Read Capacity Units / Write Capacity Units), explaining that a RCU is a read request of 4KB data, and a WCU is a write request of 1KB data. When using RCU and WCU to describe workloads, it's easier to achieve performance measurability and get more informative resource preallocation because we can abstract different hardware capabilities as a combination of RCU and WCU.
+- **Node lease.** The keep-lease handler renews the sending Datanode's lease. Selectors and the `/node-lease` endpoint use these leases to decide whether a Datanode is still active.
+- **Region failure detection.** The Region supervisor keeps a per-Region Phi Accrual detector over heartbeat arrival intervals. Its verdict is independent of lease expiry.
-However, GreptimeDB still faces a more complex situation than DynamoDB, in particular, RCU doesn't fit to describe GreptimeDB's read workloads which require a lot of computation. We are working on that.
+A failure verdict submits a failover migration only when Region failover is enabled; it is disabled by default and requires remote WAL unless explicitly allowed on local WAL. Maintenance mode also suppresses failover. See [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md) for the prerequisites and how to enable it.
diff --git a/versioned_docs/version-1.1/contributor-guide/metasrv/selector.md b/versioned_docs/version-1.1/contributor-guide/metasrv/selector.md
index 790ffb849b..23190cd6a9 100644
--- a/versioned_docs/version-1.1/contributor-guide/metasrv/selector.md
+++ b/versioned_docs/version-1.1/contributor-guide/metasrv/selector.md
@@ -7,32 +7,28 @@ description: Describes the different types of selectors in the Metasrv service,
## Introduction
-What is the `Selector`? As its name suggests, it allows users to select specific items from a given `namespace` and `context`. There is a related trait, also named `Selector`, whose definition can be found [below][0].
-
-[0]: https://github.com/GreptimeTeam/greptimedb/blob/main/src/meta-srv/src/selector.rs
-
-There is a specific scenario in `Metasrv` service. When a request to create a table is sent to the `Metasrv` service, it creates a routing table (the details of table creation will not be described here). The `Metasrv` service needs to select the appropriate `Datanode` list when creating a routing table.
+When a table is created, Metasrv uses a `Selector` to choose Datanodes for its Regions. Selection uses the current node leases and, depending on the selector, Region statistics.
## Selector Type
The `Metasrv` service currently offers the following types of `Selectors`:
-### LeasebasedSelector
+### LeaseBasedSelector
-`LeasebasedSelector` randomly selects from all available (in lease) `Datanode`s, its characteristic is simplicity and fast.
+`LeaseBasedSelector` randomly selects from Datanodes with valid leases.
### LoadBasedSelector
The `LoadBasedSelector` load value is determined by the number of regions on each `Datanode`, fewer regions indicate lower load, and `LoadBasedSelector` prioritizes selecting low-load `Datanodes`.
### RoundRobinSelector [default]
-`RoundRobinSelector` selects `Datanode`s in a round-robin fashion. It is recommended and the default option in most cases. If you're unsure which to choose, it's usually the right choice.
+`RoundRobinSelector` selects `Datanode`s in a round-robin fashion. It is the default and recommended choice for most deployments.
## Configuration
You can configure the `Selector` by its name when starting the `Metasrv` service.
-- LeasebasedSelector: `lease_based` or `LeaseBased`
+- LeaseBasedSelector: `lease_based` or `LeaseBased`
- LoadBasedSelector: `load_based` or `LoadBased`
- RoundRobinSelector: `round_robin` or `RoundRobin`
diff --git a/versioned_docs/version-1.1/contributor-guide/overview.md b/versioned_docs/version-1.1/contributor-guide/overview.md
index 875b4b1e16..6e6ec669c9 100644
--- a/versioned_docs/version-1.1/contributor-guide/overview.md
+++ b/versioned_docs/version-1.1/contributor-guide/overview.md
@@ -5,9 +5,7 @@ description: Overview of GreptimeDB's architecture, key components, and how they
# Contributor Guide
-DeepWiki provides a detailed and clear explanation of GreptimeDB's architecture and implementation. Highly recommended:
-
-[https://deepwiki.com/GreptimeTeam/greptimedb](https://deepwiki.com/GreptimeTeam/greptimedb)
+This guide explains the internal design of GreptimeDB for contributors. Start with [Getting Started](/contributor-guide/getting-started.md) to build and run it from source. Submission requirements, including the CLA, license headers, formatting, and the checks a pull request must pass, are maintained in the source repository's [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md).
## Architecture
@@ -18,8 +16,13 @@ For more details on each component, see the following guides:
- [frontend][1]
- [datanode][2]
- [metasrv][3]
+- [flownode][4]
[1]: /contributor-guide/frontend/overview.md
[2]: /contributor-guide/datanode/overview.md
[3]: /contributor-guide/metasrv/overview.md
+[4]: /contributor-guide/flownode/overview.md
+
+## Additional reference
+[DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb) provides an automatically generated walkthrough of the GreptimeDB repository. It can help when exploring an unfamiliar area, but it is a secondary reference: verify version-sensitive behavior against the source code.
diff --git a/versioned_docs/version-1.1/contributor-guide/tests/integration-test.md b/versioned_docs/version-1.1/contributor-guide/tests/integration-test.md
index 5d5f6cb1a5..bc52ee4854 100644
--- a/versioned_docs/version-1.1/contributor-guide/tests/integration-test.md
+++ b/versioned_docs/version-1.1/contributor-guide/tests/integration-test.md
@@ -7,7 +7,14 @@ description: Guide on writing and running integration tests in GreptimeDB, cover
## Introduction
-Integration testing is written with Rust test harness (`#[test]`), unlike unit testing, they are placed separately
-[here](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration).
-It covers scenarios involving multiple components, in which one typical case is HTTP/gRPC-related features. You can check
-its [documentation](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md) for more information.
+Integration tests cover behavior that crosses crate or service boundaries, such as HTTP and gRPC handling, distributed components, or external storage. They use Rust's test harness and live in the [`tests-integration`](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration) package.
+
+Run the package with:
+
+```shell
+cargo nextest run -p tests-integration
+```
+
+Some cases require environment variables or fixtures for external services. Follow the package's [setup instructions](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md) before running those cases.
+
+Use an integration test when a crate-level test or a Sqlness case cannot exercise the required boundary. Keep isolated logic in unit tests so that failures remain fast to reproduce.
diff --git a/versioned_docs/version-1.1/contributor-guide/tests/overview.md b/versioned_docs/version-1.1/contributor-guide/tests/overview.md
index feeefe2b6d..07290b79bd 100644
--- a/versioned_docs/version-1.1/contributor-guide/tests/overview.md
+++ b/versioned_docs/version-1.1/contributor-guide/tests/overview.md
@@ -5,4 +5,12 @@ description: Overview of the testing methods used in GreptimeDB to ensure its be
# Tests
-Our team has conducted lots of tests to ensure the behaviours of `GreptimeDB` . This chapter will introduce several significant methods used to test `GreptimeDB`, and how to work with them.
+Choose the narrowest test that exercises the behavior you changed:
+
+| Test type | Use it for | Typical command |
+| --- | --- | --- |
+| [Unit test](unit-test.md) | Logic contained within one crate or component | `cargo nextest run -p ` |
+| [Sqlness test](sqlness-test.md) | SQL, protocol, planner, execution, and end-to-end regressions | `cargo sqlness bare -t ` |
+| [Integration test](integration-test.md) | Behavior that crosses components or requires external services | `cargo nextest run -p tests-integration` |
+
+Run `make test` when a change needs the full Rust workspace test suite.
diff --git a/versioned_docs/version-1.1/contributor-guide/tests/sqlness-test.md b/versioned_docs/version-1.1/contributor-guide/tests/sqlness-test.md
index b5fdcf2b0b..0262ff6851 100644
--- a/versioned_docs/version-1.1/contributor-guide/tests/sqlness-test.md
+++ b/versioned_docs/version-1.1/contributor-guide/tests/sqlness-test.md
@@ -7,42 +7,34 @@ description: Instructions for running SQL tests in GreptimeDB using the `sqlness
## Introduction
-SQL is an important user interface for `GreptimeDB`. We have a separate test suite for it (named `sqlness`).
+Sqlness is GreptimeDB's end-to-end regression suite for SQL and protocol behavior. A case sends statements to a running GreptimeDB instance and compares the output with a checked-in result file.
## Sqlness manual
### Case file
-Sqlness has two types of file
+Each case uses two files:
- `.sql`: test input, SQL only
- `.result`: expected test output, SQL and its results
-The `.result` file is the expected execution output. If you see `.result` files changed,
-it means the test gets a different result and indicates it may fail. You should
-check the change logs to solve the problem.
-
-You only need to write test SQL in the `.sql` file, and run the test.
+Write the input in the `.sql` file and run the test to generate or update `.result`. Review every result diff: accept it only when the behavior change is intended.
### Case organization
-The root dir of input cases is `tests/cases`. It contains several sub-directories stand for different test
-modes. E.g., `standalone/` contains all the tests to run under `greptimedb standalone start` mode.
+Input cases live under `tests/cases`. The first directory level selects an environment. For example, `standalone/` runs against a standalone GreptimeDB instance.
-Under the first level of sub-directory (e.g. the `cases/standalone`), you can organize your cases as you like.
-Sqlness walks through every file recursively and runs them.
+Within an environment, group a new case with the feature it exercises. Sqlness discovers case files recursively.
## Run the test
-Unlike other tests, this harness is in a binary target form. You can run it with
+Run the suite with:
```shell
-cargo run --bin sqlness-runner bare
+cargo sqlness bare
```
-It automatically finishes the following procedures: compile `GreptimeDB`, start it, grab tests and feed it to
-the server, then collect and compare the results. You only need to check whether any `.result` files changed.
-If not, congratulations, the test is passed 🥳!
+The command builds and starts GreptimeDB, runs the selected cases, and compares their output. A changed `.result` file is part of the review, not proof that the new output is correct.
### Run a specific test
diff --git a/versioned_docs/version-1.1/contributor-guide/tests/unit-test.md b/versioned_docs/version-1.1/contributor-guide/tests/unit-test.md
index 82e30bf5fa..183a72aab4 100644
--- a/versioned_docs/version-1.1/contributor-guide/tests/unit-test.md
+++ b/versioned_docs/version-1.1/contributor-guide/tests/unit-test.md
@@ -8,25 +8,26 @@ description: Guide on writing and running unit tests in GreptimeDB using Rust's
## Introduction
Unit tests are embedded into the codebase, usually placed next to the logic being tested.
-They are written using Rust's `#[test]` attribute and can run with `cargo nextest run`.
+They are written using Rust's `#[test]` attribute. GreptimeDB uses [`cargo-nextest`](https://nexte.st/) as its primary Rust test runner.
-The default test runner ships with `cargo` is not supported in GreptimeDB codebase. It's recommended
-to use [`nextest`](https://nexte.st/) instead. You can install it with
+Install it with:
```shell
cargo install cargo-nextest --locked
```
-And run the tests (here the `--workspace` is not necessary)
+Run the package you changed first:
```shell
-cargo nextest run
+cargo nextest run -p
```
-Notes if your Rust is installed via `rustup`, be sure to install `nextest` with `cargo` rather
-than the package manager like `homebrew`. Otherwise it will mess up your local environment.
+Use a test name or nextest filter to narrow the run further while developing. Before submitting a change with broad effects, run the full workspace suite:
+
+```shell
+make test
+```
## Coverage
-Our continuous integration (CI) jobs have a "coverage checking" step. It will report how many
-codes are covered by unit tests. Please add the necessary unit test to your patch.
+CI reports unit-test coverage. Add tests for changed behavior and failure cases that could otherwise regress; coverage percentage alone is not the goal.
diff --git a/versioned_docs/version-1.1/reference/sql/create.md b/versioned_docs/version-1.1/reference/sql/create.md
index 87f3f97cb0..9493db1369 100644
--- a/versioned_docs/version-1.1/reference/sql/create.md
+++ b/versioned_docs/version-1.1/reference/sql/create.md
@@ -26,7 +26,7 @@ If the `db_name` database already exists, then GreptimeDB has the following beha
The database can also carry options similar to the `CREATE TABLE` statement by using the `WITH` keyword. The following options are available for databases:
- `ttl` - Time-To-Live for data in all tables within the database (cannot be set to `instant`)
-- `memtable.type` - Type of memtable (`time_series`, `partition_tree`)
+- `memtable.type` - Type of memtable (`bulk`, `time_series`)
- `append_mode` - Whether tables in the database should be append-only (`true`/`false`)
- `merge_mode` - Strategy for merging duplicate rows (`last_row`, `last_non_null`)
- `skip_wal` - Whether to disable Write-Ahead-Log for tables in the database (`'true'`/`'false'`)
@@ -74,7 +74,7 @@ Create a database with multiple options, including append mode and custom memtab
```sql
CREATE DATABASE test WITH (
ttl='30d',
- 'memtable.type'='partition_tree',
+ 'memtable.type'='bulk',
'append_mode'='true'
);
```
@@ -154,7 +154,7 @@ Users can add table options by using `WITH`. The valid options contain the follo
| `compaction.twcs.trigger_file_num` | Number of files in a specific time window to trigger a compaction | String value, such as '8'. Only available when `compaction.type` is `twcs`. You can refer to this [document](https://cassandra.apache.org/doc/latest/cassandra/managing/operating/compaction/twcs.html) to learn more about the `twcs` compaction strategy. |
| `compaction.twcs.time_window` | Compaction time window | String value, such as '1d' for 1 day. The table usually partitions rows into different time windows by their timestamps. Only available when `compaction.type` is `twcs`. |
| `compaction.twcs.max_output_file_size` | Maximum allowed output file size for TWCS compaction | String value, such as '1GB', '512MB'. Sets the maximum size for files produced by TWCS compaction. Only available when `compaction.type` is `twcs`. |
-| `memtable.type` | Type of the memtable. | String value, supports `time_series`, `partition_tree`. |
+| `memtable.type` | Type of the memtable | String value: `bulk` or `time_series`. If omitted, Mito selects the implementation from the SST format; the default flat format uses `bulk`. Setting `bulk` forces `sst_format=flat`, and flat SSTs use the bulk implementation even if `time_series` is specified. The legacy value `partition_tree` is accepted for compatibility and maps to the bulk and flat path. |
| `append_mode` | Whether the table is append-only | String value. Default is 'false', which removes duplicate rows by primary keys and timestamps according to the `merge_mode`. Setting it to 'true' to enable append mode and create an append-only table which keeps duplicate rows. |
| `merge_mode` | The strategy to merge duplicate rows | String value. Only available when `append_mode` is 'false'. Default is `last_row`, which keeps the last row for the same primary key and timestamp. Setting it to `last_non_null` to keep the last non-null field for the same primary key and timestamp. |
| `sst_format` | The format of SST files | String value, supports `primary_key`, `flat`. Default is `flat`. `flat` is recommended for tables which have a large number of unique primary keys. |
diff --git a/versioned_docs/version-1.1/user-guide/deployments-administration/configuration.md b/versioned_docs/version-1.1/user-guide/deployments-administration/configuration.md
index 40588adc2a..31b63bfba2 100644
--- a/versioned_docs/version-1.1/user-guide/deployments-administration/configuration.md
+++ b/versioned_docs/version-1.1/user-guide/deployments-administration/configuration.md
@@ -479,20 +479,9 @@ create_on_compaction = "auto"
apply_on_query = "auto"
mem_threshold_on_create = "64M"
intermediate_path = ""
-
-[region_engine.mito.memtable]
-type = "time_series"
```
-The `mito` engine provides an experimental memtable which optimizes for write performance and memory efficiency under large amounts of time-series. Its read performance might not as fast as the default `time_series` memtable.
-
-```toml
-[region_engine.mito.memtable]
-type = "partition_tree"
-index_max_keys_per_shard = 8192
-data_freeze_threshold = 32768
-fork_dictionary_bytes = "1GiB"
-```
+Mito selects the memtable implementation for each Region according to its table options and SST format. When `default_flat_format` is `true`, Regions without an explicit `sst_format` use flat SSTs and the bulk memtable. Configure `memtable.type` as a database or table option; `[region_engine.mito.memtable]` is not an engine setting. See [table options](/reference/sql/create.md#table-options).
Available options:
@@ -526,7 +515,7 @@ Available options:
| `scan_memory_on_exhausted` | String | `fail` | Behavior when scan memory is exhausted. Options: `fail` (fail fast), `wait` or `wait()` (wait for memory). |
| `min_compaction_interval` | String | `0m` | Minimum time interval between two compactions. Set to "0m" (default) to allow compactions to run immediately without restriction. |
| `schedule_compaction_after_edit` | Bool | `true` | Whether to allow scheduling a compaction after a successful region edit. Setting this to `true` is a necessary but not sufficient condition for scheduling compaction after a region edit. Other constraints, such as `min_compaction_interval`, may still prevent compaction from being scheduled. Setting this to `false` guarantees that compaction will not be scheduled after a region edit. |
-| `default_flat_format` | Bool | `true` | Whether to enable flat format as the default SST format. |
+| `default_flat_format` | Bool | `true` | Whether Regions without an explicit `sst_format` use flat SSTs. Flat SSTs use the bulk memtable. |
| `scan_parallelism` | Integer | `0` | (Deprecated, use `max_concurrent_scan_files` instead) Legacy option for scan parallelism. |
| `index` | -- | -- | The options for index in Mito engine. |
| `index.aux_path` | String | `""` | Auxiliary directory path for the index in the filesystem. This path is used to store intermediate files for creating the index and staging files for searching the index. It defaults to `{data_home}/index_intermediate`. The default name for this directory is `index_intermediate` for backward compatibility. This path contains two subdirectories: `__intm` for storing intermediate files used during index creation, and `staging` for storing staging files used during index searching. |
@@ -542,10 +531,6 @@ Available options:
| `inverted_index.apply_on_query` | String | `auto` | Whether to apply the index on query - `auto`: automatically - `disable`: never |
| `inverted_index.mem_threshold_on_create` | String | `64M` | Memory threshold for performing an external sort during index creation. Setting to empty will disable external sorting, forcing all sorting operations to happen in memory. |
| `inverted_index.intermediate_path` | String | `""` | File system path to store intermediate files for external sorting (default `{data_home}/index_intermediate`). |
-| `memtable.type` | String | `time_series` | Memtable type. - `time_series`: time-series memtable - `partition_tree`: partition tree memtable (experimental) |
-| `memtable.index_max_keys_per_shard` | Integer | `8192` | The max number of keys in one shard. Only available for `partition_tree` memtable. |
-| `memtable.data_freeze_threshold` | Integer | `32768` | The max rows of data inside the actively writing buffer in one shard. Only available for `partition_tree` memtable. |
-| `memtable.fork_dictionary_bytes` | String | `1GiB` | Max dictionary bytes. Only available for `partition_tree` memtable. |
The `metric` engine is optimized for handling metrics data with a large number of small tables:
diff --git a/versioned_docs/version-1.2/contributor-guide/datanode/data-persistence-indexing.md b/versioned_docs/version-1.2/contributor-guide/datanode/data-persistence-indexing.md
index 6403ffc2df..1fa5d6d258 100644
--- a/versioned_docs/version-1.2/contributor-guide/datanode/data-persistence-indexing.md
+++ b/versioned_docs/version-1.2/contributor-guide/datanode/data-persistence-indexing.md
@@ -5,19 +5,23 @@ description: Explanation of data persistence and indexing in GreptimeDB, includi
# Data Persistence and Indexing
-Similar to all LSMT-like storage engines, data in MemTables is persisted to durable storage, for example, the local disk file system or object storage service. GreptimeDB adopts [Apache Parquet][1] as its persistent file format.
+Like other LSM-tree storage engines, GreptimeDB persists data from memtables to durable storage such as a local filesystem or object storage. It uses [Apache Parquet][1] as the persistent file format.
## SST File Format
Parquet is an open source columnar format that provides fast data querying and has already been adopted by many projects, such as Delta Lake.
-Parquet has a hierarchical structure like "row groups-columns-data pages". Data in a Parquet file is horizontally partitioned into row groups, in which all values of the same column are stored together to form a data page. Data page is the minimal storage unit. This structure greatly improves performance.
+Parquet organizes data as row groups, column chunks, and pages. A row group contains one column chunk for each column, and each column chunk contains one or more pages. Pages are the units of encoding and compression; column chunks are the I/O units for reading selected columns.
First, clustering data by column makes file scanning more efficient, especially when only a few columns are queried, which is very common in analytical systems.
-Second, data of the same column tends to be homogeneous which helps with compression when apply techniques like dictionary and Run-Length Encoding (RLE).
+Second, values within a column tend to be similar, which improves compression with techniques such as dictionary encoding and run-length encoding (RLE).
-
+The following diagram from the Apache Parquet specification also shows the physical file layout: column chunks are stored by row group, while file metadata and its length are written in the footer.
+
+
+
+*Source: Apache Parquet [FileLayout.gif](https://github.com/apache/parquet-format/blob/master/doc/images/FileLayout.gif). Copyright 2014 The Apache Software Foundation, licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).*
## Data Persistence
@@ -27,17 +31,17 @@ When the size of data buffered in MemTables reaches that threshold, GreptimeDB w
## Indexing Data in SST Files
-Apache Parquet file format provides inherent statistics in headers of column chunks and data pages, which are used for pruning and skipping.
+Parquet stores row-group column statistics such as minimum, maximum, and null count in each column chunk's metadata. Page metadata and optional column indexes can provide finer-grained statistics.
-
+
-For example, in the above Parquet file, if you want to filter rows where `name` = `Emily`, you can easily skip row group 0 because the max value for `name` field is `Charlie`. This statistical information reduces IO operations.
+For example, a query filtering for `name` = `Emily` can skip row group 0 because the maximum `name` value is `Charlie`. This avoids reading that row group.
## Index Files
-For each SST file, GreptimeDB not only maintains an internal index but also generates a separate file to store the index structures specific to that SST file.
+When an SST has one or more configured index outputs, GreptimeDB writes them to a Puffin file associated with that SST. An SST with no applicable index does not need a Puffin file.
-The index files utilize the [Puffin][3] format, which offers significant flexibility, allowing for the storage of additional metadata and supporting a broader range of index structures.
+Puffin provides a container for index blobs and their metadata, allowing different index structures to share one file.
@@ -57,13 +61,13 @@ The inverted index enables GreptimeDB to skip data segments that do not meet que
-For instance, the query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches the regex `.*users`, and `status` matches the regex `4...`. It then scans these data segments to produce the final results that meet all conditions, significantly reducing the number of IO operations.
+The query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches `.*users`, and `status` matches `4..`. It scans only those segments before applying the remaining filters.
### Inverted Index Format
-
+
-GreptimeDB builds inverted indexes by column, with each inverted index consisting of an FST and multiple Bitmaps.
+GreptimeDB builds inverted indexes by column. Each column index contains a null bitmap, multiple posting bitmaps, and an FST. The blob footer records the offsets, sizes, and metadata needed to locate and decode the column indexes.
The FST (Finite State Transducer) enables GreptimeDB to store mappings from column values to Bitmap positions in a compact format and provides excellent search performance and supports complex search capabilities (such as regular expression matching). The Bitmaps maintain a list of data segment IDs, with each bit representing a data segment.
@@ -77,7 +81,7 @@ The number of rows in a data segment is controlled by the engine option `index.i
## Unified Data Access Layer: OpenDAL
-GreptimeDB uses [OpenDAL][2] to provide a unified data access layer, thus, the storage engine does not need to interact with different storage APIs, and data can be migrated to cloud-based storage like AWS S3 seamlessly.
+GreptimeDB uses [OpenDAL][2] to provide a common access layer for local filesystems and object stores. Changing the configured storage backend does not migrate existing data.
[1]: https://parquet.apache.org
[2]: https://github.com/datafuselabs/opendal
diff --git a/versioned_docs/version-1.2/contributor-guide/datanode/memtable.md b/versioned_docs/version-1.2/contributor-guide/datanode/memtable.md
new file mode 100644
index 0000000000..92e3eb58de
--- /dev/null
+++ b/versioned_docs/version-1.2/contributor-guide/datanode/memtable.md
@@ -0,0 +1,100 @@
+---
+keywords: [memtable, Mito engine, write buffer, flush, time partition, BulkMemtable]
+description: How Mito organizes mutable Region data in memtables and moves it into SST files.
+---
+
+# Memtable design
+
+A memtable is Mito's in-memory write buffer for a Region. It makes writes available to reads before a flush creates SST files. A Region version identifies the memtables and SST files that a scan may read. Together with a committed-sequence fence, it keeps the scan consistent while writes and flushes advance the current version.
+
+## Write and flush lifecycle
+
+For a normal WAL-backed write, Mito uses this order:
+
+```text
+write request
+ |
+ v
+WAL append -> mutable memtable -> publish committed sequence
+ |
+ freeze
+ v
+ immutable memtable -> SST write -> manifest edit
+```
+
+The Region worker assigns sequence numbers and a WAL entry ID before appending the mutation to the [write-ahead log](wal.md). If the WAL append fails, Mito does not update the memtable. After the memtable update succeeds, Mito publishes the committed sequence and the rows become visible to new reads. A Region configured with `skip_wal` omits the WAL append, but keeps the same memtable and visibility ordering.
+
+A flush freezes the mutable memtables and installs a new mutable set before starting the background SST write. New writes therefore continue without changing the frozen data. The flush writes the immutable memtables to SST files, then persists a manifest edit containing the files and the flushed WAL and sequence checkpoints. Only after that edit is durable does Mito remove the flushed memtables from the current Region version. If the flush fails, the immutable memtables remain available for a later attempt.
+
+## Region versions and time partitions
+
+Each Region has one mutable `TimePartitions` container, which can hold more than one memtable:
+
+```text
+Region version
+├─ mutable TimePartitions
+│ ├─ [t0, t1) -> memtable
+│ └─ [t1, t2) -> memtable
+├─ immutable memtables
+└─ SST files
+```
+
+Mito routes each row to a partition by its time-index value. Partition ranges are half-open and aligned to a fixed duration. The duration follows the Region's compaction time window; Mito uses one day until a compaction window is available. An out-of-order write can create an earlier partition alongside the latest one.
+
+Freezing a Region freezes all mutable time partitions together. Mito moves their memtables to the immutable list and creates a new `TimePartitions` container. A failed flush can leave more than one generation of immutable memtables, so reads and later flushes must not assume that the list contains a single item.
+
+## Memtable implementations
+
+Mito selects a memtable implementation from the Region's SST format, primary-key encoding, and memtable options:
+
+```text
+flat SST format (the default) or sparse primary-key encoding -> BulkMemtable
+memtable.type=bulk -> BulkMemtable, and forces flat SST
+primary_key SST with dense encoding (legacy) -> a legacy implementation
+```
+
+With the default engine configuration, a Region without an explicit SST format uses `flat`, so `BulkMemtable` is the normal path and the rest of this page describes it. The rules exist to prevent incompatible combinations: flat format or sparse primary-key encoding requires `BulkMemtable`, and explicitly selecting the bulk implementation forces flat format.
+
+### BulkMemtable
+
+`BulkMemtable` stores writes as parts in the flat Arrow layout instead of inserting rows into per-series buffers:
+
+```text
+BulkMemtable
+├─ unordered_part
+│ └─ small BulkPart batches
+└─ parts
+ ├─ BulkPart (Arrow RecordBatch)
+ ├─ MultiBulkPart (raw RecordBatches)
+ └─ EncodedBulkPart (in-memory Parquet)
+```
+
+Small parts accumulate in `unordered_part`; larger parts enter `parts` directly. Background memtable compaction merge-sorts eligible parts into a `MultiBulkPart` or encodes them as an `EncodedBulkPart`. Scans use part statistics to prune ranges, and flush can write encoded ranges to SST without decoding and encoding the rows again. For the design rationale and performance results, see [Scaling Time Series to Millions of Cardinalities: GreptimeDB's Flat Format](https://greptime.com/blogs/2025-12-22-flat-format).
+
+### Legacy implementations
+
+Regions on the legacy `primary_key` SST format with dense primary-key encoding still use `TimeSeriesMemtable`, which groups rows by encoded primary key rather than storing flat parts. A Region with no primary-key columns gets `SimpleBulkMemtable` from the same builder. Both are compatibility code for existing tables and may be removed once the `primary_key` format is retired; new work targets the bulk and flat path.
+
+The removed `partition_tree` memtable is not a third implementation. The option parser accepts `memtable.type=partition_tree` for compatibility, but it does not recreate that implementation. The Region uses the bulk and flat path.
+
+## Read snapshots
+
+A scan obtains the Region version and committed sequence together from `VersionControl`. It selects the version before applying the sequence fence. Reading the sequence separately before the version could pair that sequence with a later version after flush or compaction removes an older input, producing an incomplete snapshot.
+
+The selected version supplies mutable memtables, immutable memtables, and SST files. Mito first prunes sources by time range, then asks each memtable for ranges using the scan's projection, predicate, and sequence bounds. The scan merges the resulting ranges with SST ranges and applies the same ordering, deletion, and merge semantics across all sources. References held by the scan keep an older memtable alive even after a newer Region version removes it.
+
+## Memory pressure
+
+Each memtable tracks its estimated heap allocation through the engine's write-buffer manager. Freezing a memtable removes its allocation from the mutable-memory count, but total usage includes the allocation until all references to that memtable are released. The mutable-memory count therefore tracks data that can still accept writes, while total usage continues to include memory retained by active scans.
+
+The global write-buffer limit causes workers to select Regions for flush. If memory remains above the configured limits, Mito stalls writes and can reject them at a higher threshold. An optional per-Region limit applies the same pressure to one hot Region without stalling unrelated Regions. Periodic, manual, and Region lifecycle operations can also request a flush.
+
+## Constraints for changes
+
+Changes to memtable code must preserve these properties:
+
+- For a WAL-backed Region, append to the WAL before installing rows in a memtable. Publish the committed sequence only after installation succeeds.
+- Keep frozen memtables readable and retryable until the SST files and manifest edit are durable.
+- Obtain the Region version and committed sequence from the same `VersionControl` snapshot; never read the sequence separately before the version.
+- Preserve the ordering and metadata that scans and flushes need to apply the same deletion, deduplication, and merge rules across memtable and SST ranges.
+- Charge allocations to the write-buffer manager and release them only when the underlying memory can no longer be referenced.
diff --git a/versioned_docs/version-1.2/contributor-guide/datanode/metric-engine.md b/versioned_docs/version-1.2/contributor-guide/datanode/metric-engine.md
index 064872ce14..fde3c178a3 100644
--- a/versioned_docs/version-1.2/contributor-guide/datanode/metric-engine.md
+++ b/versioned_docs/version-1.2/contributor-guide/datanode/metric-engine.md
@@ -7,9 +7,9 @@ description: Overview of the Metric engine in GreptimeDB, its concepts, architec
## Overview
-The `Metric` engine is a component of GreptimeDB, and it's an implementation of the storage engine. It mainly targets scenarios with a large number of small tables for observable metrics.
+The `Metric` engine stores workloads with many small metric tables.
-Its main feature is to use synthetic physical wide tables to store a large amount of small table data, achieving effects such as reuse of the same column and metadata. This reduces storage overhead for small tables and improves columnar compression efficiency. The concept of a table becomes even more lightweight under the `Metric` engine.
+It maps those logical tables onto shared physical wide tables so they can reuse columns and metadata. This reduces per-table storage overhead and improves columnar compression.
## Concepts
@@ -19,7 +19,7 @@ The `Metric` engine introduces two new concepts: "logical table" and "physical t
A logical table refers to user-defined tables. Just like any other ordinary table, its definition includes the name of the table, column definitions, index definitions etc. All operations such as queries or write-ins by users are based on these logical tables. Users don't need to worry about differences between logical and ordinary tables during usage.
-From an implementation standpoint, a logical table is virtual; it doesn't directly read or write physical data but maps read/write requests into corresponding requests for physical tables in order to implement data storage and querying.
+A logical table is virtual. The engine maps its read and write requests to the corresponding physical table instead of storing data for it directly.
### Physical Table
@@ -29,16 +29,14 @@ A physical table is a table that actually stores data, possessing several physic
The main design architecture of the `Metric` engine is as follows:
-
+
-In the current version implementation, the `Metric` engine reuses the `Mito` engine to achieve storage and query capabilities for physical data. It also provides access to both physical tables and logical tables simultaneously.
+The `Metric` engine delegates physical storage and queries to the `Mito` engine. Each physical Region group contains a data Region, which stores rows from its mapped logical tables, and a metadata Region, which stores the logical-table and logical-column mappings.
-Regarding partitioning, logical tables have identical partition rules and Region distribution as physical tables. This makes sense because the data of logical tables are directly stored in physical tables, so their partition rules are consistent.
+Logical tables associated with the same physical table share its partition layout. During writes, the engine records the logical table identity with each row. During reads, it adds a logical-table filter before scanning the physical Region.
-Concerning routing metadata, the routing address of a logical table is a logical address - what its corresponding physical table is - then through this physical table for secondary routing to obtain the real physical address. This indirect routing method can significantly reduce the number of metadata modifications required when Region migration scheduling occurs in Metric engines.
+A logical table's route stores only the ID of its physical table; the physical table route resolves that to the Datanodes holding the Regions. Because logical routes do not name peers, migrating a physical Region rewrites one physical route instead of every logical route that maps to it.
-Operationally speaking, The `Metric` engine supports standard DML operations (INSERT, DELETE, SELECT) on logical tables. However, it only supports limited operations on physical tables to prevent misoperations - for example, writing directly to a physical table is prohibited as it could affect user's logical table data. Generally speaking, users can consider that they have read-only access to these physical tables.
+Logical tables support normal INSERT, DELETE, and SELECT operations. Direct writes to a physical Region are rejected because they would bypass the logical-table mapping; querying a physical table remains supported.
-To improve performance during simultaneous DDL (Data Definition Language) operations on many tables, the 'Metric' engine has introduced some batch DDL operations. These batch DDL operations can merge lots of DDL actions into one request thereby reducing queries and modifications times for metadata thus enhancing performance. This feature is particularly beneficial in scenarios such as the automatic creation requests brought about by large amounts of metrics during Prometheus Remote Write cold start-up, as well as the modification requests for numerous route-tables mentioned earlier during migration of many physical regions.
-
-Apart from physical data regions belonging to physical tables, the 'Metric' engine creates an additional metadata region physically for each individual physical data region used in storing some metadata needed by itself while maintaining mapping and other states. This metadata includes the mapping relationship between logical tables and physical tables, the mapping relationship between logical columns and physical columns etc.
+Batch DDL operations reduce metadata work when many logical tables are created or updated together, such as during Prometheus Remote Write auto-creation or physical Region migration.
diff --git a/versioned_docs/version-1.2/contributor-guide/datanode/overview.md b/versioned_docs/version-1.2/contributor-guide/datanode/overview.md
index d0afe21b34..a21c112faa 100644
--- a/versioned_docs/version-1.2/contributor-guide/datanode/overview.md
+++ b/versioned_docs/version-1.2/contributor-guide/datanode/overview.md
@@ -7,28 +7,26 @@ description: Overview of Datanode in GreptimeDB, its responsibilities, component
## Introduction
-`Datanode` is mainly responsible for storing the actual data for GreptimeDB. As we know, in GreptimeDB,
-a `table` can have one or more `Region`s, and `Datanode` is responsible for managing the reading and writing
-of these `Region`s. `Datanode` is not aware of `table` and can be considered as a `region server`. Therefore,
-`Frontend` and `Metasrv` operate `Datanode` at the granularity of `Region`.
+A Datanode stores and processes Region data. A table can contain multiple Regions, but the Datanode does not own table-level routing. Frontend sends data requests by Region, while Metasrv controls Region placement and lifecycle.
-
+This boundary lets the same Region server host different storage engines without exposing their implementation to Frontend or Metasrv.
+
+
## Components
-A `Datanode` contains all the components needed for a `region server`. Here we list some of the vital parts:
-
-- A gRPC service is provided for reading and writing region data, and `Frontend` uses this service
- to read and write data from `Datanode`s.
-- An HTTP service, through which you can obtain metrics, configuration information, etc., of the current node.
-- `Heartbeat Task` is used to send heartbeat to the `Metasrv`. The heartbeat plays a crucial role in the
- distributed architecture of GreptimeDB and serves as a basic communication channel for distributed coordination.
- The upstream heartbeat messages contain important information such as the workload of a `Region`. If the
- `Metasrv `has made scheduling(such as `Region` migration) decisions, it will send instructions to the
- `Datanode` via downstream heartbeat messages.
-- The `Datanode` does not parse user SQL or perform distributed planning. The user's query requests for one or
- more `Table`s will be transformed into `Region` query requests in the `Frontend`. The `Datanode` is responsible
- for executing these `Region` query plans with its local query engine.
-- A `Region Manager` is used to manage all `Region`s on a `Datanode`.
-- GreptimeDB supports a pluggable multi-engine architecture, with existing engines including `File Engine` and
- `Mito Engine`.
+The main components are:
+
+- The Region server tracks open Regions and dispatches reads, writes, and lifecycle requests to the engine registered for each Region.
+- `Mito` is the primary time-series Region engine. `Metric` maps many logical metric Regions onto shared Mito Regions, and `File` exposes external files through the Region interface.
+- The local query engine executes Region query plans. It does not parse client SQL or perform cluster-wide planning.
+- The heartbeat task reports node and Region state to Metasrv and receives instructions such as open, close, upgrade, downgrade, and migration steps.
+- gRPC carries Region requests to the Datanode. HTTP exposes node diagnostics such as metrics and configuration.
+
+## Region Request Lifecycle
+
+For a Mito write, the Region server selects Mito from the Region metadata. Mito appends the mutation to the WAL, applies it to a memtable, and later flushes the memtable to SST files. A Metric write is first rewritten with the logical-table identity and then delegated to its physical Mito Region.
+
+For a read, the local query engine executes the Region plan against a table provider backed by the Region engine. A Mito scan takes an immutable Region version, reads the relevant memtables and SST files, merges and deduplicates rows, and returns a stream of Arrow record batches.
+
+Region ownership can change without restarting the Datanode. Metasrv sends lifecycle instructions over the heartbeat stream; the Region server applies them to the engine and reports the new Region role and statistics in subsequent heartbeats.
diff --git a/versioned_docs/version-1.2/contributor-guide/datanode/python-scripts.md b/versioned_docs/version-1.2/contributor-guide/datanode/python-scripts.md
deleted file mode 100644
index 98909142a6..0000000000
--- a/versioned_docs/version-1.2/contributor-guide/datanode/python-scripts.md
+++ /dev/null
@@ -1,35 +0,0 @@
----
-keywords: [Python scripts, data analysis, CPython backend, RustPython interpreter, RecordBatch]
-description: Guide on using Python scripts for data analysis in GreptimeDB, including backend options and setup instructions.
----
-
-# Python Scripts
-
-## Introduction
-
-Python scripts are methods for analyzing data in GreptimeDB,
-by running it in the database directly instead of fetching all the data from the database and running it locally.
-This approach saves a lot of data transfer costs.
-The image below depicts how the script works.
-The `RecordBatch` (which is basically a column in a table with type and nullability metadata)
-can come from anywhere in the database,
-and the returned `RecordBatch` can be annotated in Python grammar to indicate its metadata,
-such as type or nullability.
-The script will do its best to convert the returned object to a `RecordBatch`,
-whether it is a Python list, a `RecordBatch` computed from parameters,
-or a constant (which is extended to the same length as the input arguments).
-
-
-
-## Two optional backends
-
-### CPython Backend powered by PyO3
-
-This backend is powered by [PyO3](https://pyo3.rs/v0.18.1/), enabling the use of your favourite Python libraries (such as NumPy, Pandas, etc.) and allowing Conda to manage your Python environment.
-
-But using it also involves some complications. You must set up the correct Python shared library, which can be a bit challenging. In general, you just need to install the `python-dev` package. However, if you are using Homebrew to install Python on macOS, you must create a proper soft link to `Library/Frameworks/Python.framework`. Detailed instructions on using PyO3 crate with different Python Version can be found [here](https://pyo3.rs/v0.18.1/building_and_distribution#configuring-the-python-version)
-
-### Embedded RustPython Interpreter
-
-An experiment [python interpreter](https://github.com/RustPython/RustPython) to run
-the coprocessor script, it supports Python 3.10 grammar. You can use all the very Python syntax, see [User Guide/Python Coprocessor](/user-guide/python-scripts/overview.md) for more!
diff --git a/versioned_docs/version-1.2/contributor-guide/datanode/query-engine.md b/versioned_docs/version-1.2/contributor-guide/datanode/query-engine.md
index 5b83a30613..91f307aa59 100644
--- a/versioned_docs/version-1.2/contributor-guide/datanode/query-engine.md
+++ b/versioned_docs/version-1.2/contributor-guide/datanode/query-engine.md
@@ -7,51 +7,30 @@ description: Overview of GreptimeDB's query engine, its architecture, data repre
## Introduction
-GreptimeDB's query engine is built on [Apache DataFusion][1] (subproject under [Apache
-Arrow][2]), a brilliant query engine written in Rust. It provides a set of well functional components from
-logical plan, physical plan and the execution runtime. Below explains how each component is orchestrated and their positions during execution.
+GreptimeDB's query engine is built on [Apache DataFusion][1]. DataFusion supplies the logical and physical plan interfaces, optimizer framework, and execution runtime. GreptimeDB adds planners for its query languages, storage-aware optimizer rules, custom plan nodes, and distributed execution.
-
+DDL and other control-plane operations are dispatched by the statement executor. The query engine receives plans for data processing, including the input side of operations such as `INSERT ... SELECT`.
-The entry point is the logical plan, which is used as the general intermediate representation of a
-query or execution logic etc. Two noticeable sources of logical plan are from: 1. the user query, like
-SQL through SQL parser and planner; 2. the Frontend's distributed query, which is explained in details in the following section.
+## Query Lifecycle
-Next is the physical plan, or the execution plan. Unlike the logical plan which is a big
-enumeration containing all the logical plan variants (except the special extension plan node), the
-physical plan is in fact a trait that defines a group of methods invoked during
-execution. All data processing logics are packed in corresponding structures that
-implement the trait. They are the actual operations performed on the data, like
-aggregator `MIN` or `AVG`, and table scan `SELECT ... FROM`.
+1. The SQL, PromQL, or log-query planner resolves tables through the catalog and produces a DataFusion logical plan. GreptimeDB plan extensions represent operations that DataFusion does not provide directly.
+2. DataFusion analyzer and optimizer rules run together with GreptimeDB rules. These rules normalize expressions and types, rewrite time-range operations, push projections and filters toward scans, and introduce distributed plan nodes when required.
+3. The physical planner converts the optimized logical plan into streaming operators. GreptimeDB then applies physical rules for scan parallelism, ordering, and distributed execution.
+4. Execution pulls Arrow record batches through the physical plan. Storage scans receive the projection and predicates, and downstream operators consume the resulting stream without materializing the complete result first.
-The optimization phase which improves execution performance by transforming both logical and physical plans, is now all based on rules. It is also called, "Rule Based Optimization". Some of the rules are DataFusion native and others are customized in Greptime DB. In the future, we plan to add more
-rules and leverage the data statistics for Cost Based Optimization/CBO.
-
-The last phase "execute" is a verb, stands for the procedure that reads data from storage, performs
-calculations and generates the expected results. Although it's more abstract than previously mentioned concepts, you can just
-simply imagine it as executing a Rust async function. And it's indeed a future (stream).
-
-`EXPLAIN [VERBOSE] ` is very useful if you want to see how your SQL is represented in the logical or physical plan.
+Use [`EXPLAIN`](/reference/sql/explain.md) to inspect the logical and physical plans. `EXPLAIN ANALYZE` also executes the plan and reports runtime metrics.
## Data Representation
-GreptimeDB uses [Apache Arrow][2] as the in-memory data representation. It's column-oriented, in
-cross-platform format, and also contains many high-performance data operators. These features
-make it easy to share data in many different environments and implement calculation logic.
+GreptimeDB uses [Apache Arrow][2] record batches as its in-memory data representation. A record batch contains equal-length column arrays and a schema. Query operators exchange streams of these batches, which keeps the execution path columnar from Region scans through result encoding.
## Indexing
-In time series data, there are two important dimensions: timestamp and tag columns (or like
-primary key in a general relational database). GreptimeDB groups data in time buckets, so it's efficient
-to locate and extract data within the expected time range at a very low cost. The mainly used persistent file format [Apache Parquet][3] in GreptimeDB helps a lot -- it
-provides multi-level indices and filters that make it easy to prune data during querying. In the future, we
-will make more use of this feature, and develop our separated index to handle more complex use cases.
+Index construction and persistent index formats belong to the storage engine. The query layer supplies predicates and projections to a scan; Mito then uses time ranges, Parquet statistics, and indexes to avoid reading data that cannot match. See [Data Persistence and Indexing](./data-persistence-indexing.md).
## Distributed Execution
-Covered in [Distributed Querying][6].
+In distributed mode, the Frontend plans the cluster-wide query and Datanodes execute Region-local subplans. [`MergeScan`](../frontend/distributed-querying.md) is the boundary between those stages.
-[1]: https://github.com/apache/arrow-datafusion
+[1]: https://datafusion.apache.org/
[2]: https://arrow.apache.org/
-[3]: https://parquet.apache.org
-[6]: ../frontend/distributed-querying.md
diff --git a/versioned_docs/version-1.2/contributor-guide/datanode/storage-engine.md b/versioned_docs/version-1.2/contributor-guide/datanode/storage-engine.md
index c220c72208..36fef62833 100644
--- a/versioned_docs/version-1.2/contributor-guide/datanode/storage-engine.md
+++ b/versioned_docs/version-1.2/contributor-guide/datanode/storage-engine.md
@@ -7,7 +7,7 @@ description: Overview of the storage engine in GreptimeDB, its architecture, com
## Introduction
-The `storage engine` is responsible for storing the data of the database. Mito, based on [LSMT][1] (Log-structured Merge-tree), is the storage engine we use by default. We have made significant optimizations for handling time-series data scenarios, so mito engine is not suitable for general purposes.
+Mito is GreptimeDB's default storage engine. It uses an [LSM tree][1] and is designed for time-series workloads rather than as a general-purpose embedded storage engine.
## Architecture
@@ -23,9 +23,9 @@ The architecture is the same as a traditional LSMT engine:
media.
- Log records of the WAL can be stored on the local disk, or in a remote log service such as
Kafka (remote WAL) that implements the `Log Store` API.
-- Memtables:
- - Data is written into the `active memtable`, aka `mutable memtable` first.
- - When a `mutable memtable` is full, it will be changed to a `read-only memtable`, aka `immutable memtable`.
+- [Memtables](memtable.md):
+ - Mito routes rows by time index into mutable memtables.
+ - A flush freezes the mutable memtables, installs a new mutable set for writes, and writes the frozen memtables to SST files.
- SST
- The full name of SST, aka SSTable is `Sorted String Table`.
- `Immutable memtable` is flushed to persistent storage and produces an SST file.
@@ -103,7 +103,9 @@ Each Parquet SST is split into row groups, the unit that Parquet can read or ski
Mito supports two SST formats: `flat` and `primary_key`. `flat` is the default for new tables and works well across primary-key cardinalities, including high-cardinality keys. `primary_key` is the legacy format kept for compatibility with older tables. See [SST format](/reference/sql/create.md#create-a-table-with-sst-format) and the [table design guide](/user-guide/deployments-administration/performance-tuning/design-table.md#sst-format) for more details.
-
+
+
+An SST may span more than one compaction time window.
## Scan Pruning
diff --git a/versioned_docs/version-1.2/contributor-guide/datanode/wal.md b/versioned_docs/version-1.2/contributor-guide/datanode/wal.md
index 4ecb19ef02..8f898d91d6 100644
--- a/versioned_docs/version-1.2/contributor-guide/datanode/wal.md
+++ b/versioned_docs/version-1.2/contributor-guide/datanode/wal.md
@@ -7,30 +7,26 @@ description: Introduction to Write-Ahead Logging (WAL) in GreptimeDB, its purpos
## Introduction
-Our storage engine is inspired by the Log-structured Merge Tree (LSMT). Mutating operations are
-applied to a MemTable instead of persisting to disk, which significantly improves performance but
-also brings durability-related issues, especially when the Datanode crashes unexpectedly. Similar
-to all LSMT-like storage engines, GreptimeDB uses a write-ahead log (WAL) to ensure data durability
-and is safe from crashing.
+Mito buffers writes in [memtables](memtable.md) before flushing them to SST files. It first appends each Region's mutations to the write-ahead log (WAL), so data that has not reached an SST can be recovered.
-WAL is an append-only file group. All `INSERT` and `DELETE` operations are transformed into
-operation entries and then appended to WAL. Once operation entries are persisted to the underlying
-file, the operation can be further applied to MemTable.
+The WAL uses a common log-store abstraction with local raft-engine and remote Kafka providers.
-When the Datanode restarts, operation entries in WAL are replayed to reconstruct the correct
-in-memory state.
+## Write and Recovery Cycle
-
+The order of a normal write is:
+
+1. The Region worker assigns sequence numbers and a WAL entry ID.
+2. It appends the mutations to the WAL. If the append fails, the mutations are not applied to the memtable.
+3. After the append succeeds, Mito writes the mutations to the memtable and publishes the new committed sequence.
+4. A flush writes immutable SST files and persists a manifest edit containing the new files and `flushed_entry_id`.
+5. After the manifest edit is durable, WAL entries through `flushed_entry_id` are marked obsolete. The log store may reclaim them later.
+
+The manifest is the recovery boundary. On a normal reopen, Mito rebuilds the Region from the manifest and replays WAL entries starting at `flushed_entry_id + 1`. Region transitions may supply a later replay checkpoint, but they never replay entries before the persisted flush boundary.
## Namespace
-Namespace of WAL is used to separate entries from different tables (different regions). Append and
-read operations must provide a Namespace. Currently, region ID is used as the Namespace, because
-each region has a MemTable that needs to be reconstructed when Datanode restarts.
+WAL entries are isolated by Region, not by table. Each append and read identifies a Region namespace so one Region can be replayed or truncated independently. The local raft-engine provider uses the Region ID as its namespace ID. Kafka keeps Region identity within the provider's topic-backed log.
## Synchronous/Asynchronous flush
-By default, appending to WAL is asynchronous, which means the writer will not wait until entries are
-flushed to disk. This setting provides higher performance, but may lose data when running host shutdown unexpectedly. In the other hand, synchronous flush provides higher durability at the cost of performance.
-
-In v0.4 version, the new region worker architecture can use batching to alleviate the overhead of sync flush.
+For the local raft-engine provider, `sync_write` controls whether an append waits for the log to be synced to durable storage. It defaults to `false`. Asynchronous writes reduce latency but can lose recently acknowledged entries if the host fails before buffered data is synced. Kafka WAL durability is controlled by its producer and cluster settings instead of this local option.
diff --git a/versioned_docs/version-1.2/contributor-guide/flownode/arrangement.md b/versioned_docs/version-1.2/contributor-guide/flownode/arrangement.md
index aed75af777..8b472ea316 100644
--- a/versioned_docs/version-1.2/contributor-guide/flownode/arrangement.md
+++ b/versioned_docs/version-1.2/contributor-guide/flownode/arrangement.md
@@ -5,6 +5,8 @@ description: Details on the arrangement component in Flownode, which stores stat
# Arrangement
+This page describes state used by Flownode's legacy streaming mode. Batching mode does not use an Arrangement.
+
Arrangement stores the state in the dataflow's process. It stores the streams of update flows for further querying and updating.
The arrangement essentially stores key-value pairs with timestamps to mark their change time.
diff --git a/versioned_docs/version-1.2/contributor-guide/flownode/batching_mode.md b/versioned_docs/version-1.2/contributor-guide/flownode/batching_mode.md
index 37a695ce9c..aa8099d1ad 100644
--- a/versioned_docs/version-1.2/contributor-guide/flownode/batching_mode.md
+++ b/versioned_docs/version-1.2/contributor-guide/flownode/batching_mode.md
@@ -9,13 +9,13 @@ This guide provides a brief overview of the batching mode in `flownode`. It's in
## Overview
-The batching mode in `flownode` is designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. This is in contrast to the original streaming mode, now deprecated, where data was processed as it arrived.
+The batching mode in `flownode` is designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. This is in contrast to the legacy streaming path, which processes data as it arrives and is retained for compatibility but deprecated for new workloads.
The core idea is to:
1. Define a `flow` with a SQL query that aggregates data from a source table into a sink table.
2. The query typically includes a time window function (e.g., `date_bin`) on a timestamp column.
3. When new data is inserted into the source table, the system marks the corresponding time windows as "dirty."
-4. A background task periodically wakes up, identifies these dirty windows, and re-runs the aggregation query for those specific time ranges.
+4. A background task runs on its own cadence, consumes the pending dirty windows at its next evaluation, and re-runs the aggregation query for those time ranges.
5. The results are then inserted into the sink table, effectively updating the aggregated view.
## Architecture
@@ -39,15 +39,15 @@ A `BatchingTask` represents a single, independent data flow. Each task is associ
- **State (`TaskState`)**: This contains the dynamic, mutable state of the task, most importantly the `DirtyTimeWindows`.
- **Execution Loop**: The task runs an infinite loop (`start_executing_loop`) that:
1. Checks for a shutdown signal.
- 2. Waits for a scheduled interval or until it's woken up.
+ 2. Sleeps until its next evaluation time. A task with an evaluation schedule sleeps until the next scheduled time; an adaptive task sleeps for a polling interval derived from the time window size and the minimum refresh duration.
3. Generates a new query plan (`gen_insert_plan`) based on the current set of dirty time windows.
4. Executes the query (`execute_logical_plan`) against the database.
5. Cleans up the processed dirty windows.
### `TaskState` and `DirtyTimeWindows`
-- **`TaskState`**: This struct tracks the runtime state of a `BatchingTask`. It includes `dirty_time_windows`, which is crucial for determining what work needs to be done.
-- **`DirtyTimeWindows`**: This is a key data structure that keeps track of which time windows have received new data since the last query execution. It stores a set of non-overlapping time ranges. When a task's execution loop runs, it consults this structure to build a `WHERE` clause that filters the source table for only the dirty time windows.
+- **`TaskState`**: This struct tracks the runtime state of a `BatchingTask`, including the `dirty_time_windows` that determine its pending work.
+- **`DirtyTimeWindows`**: This data structure tracks which time windows have received new data since the last query execution. It stores a set of non-overlapping time ranges. The execution loop uses it to build a `WHERE` clause that selects only the dirty windows from the source table.
### `TimeWindowExpr`
@@ -56,15 +56,15 @@ The `TimeWindowExpr` is a helper utility for dealing with time window expression
- **Evaluation**: It can take a timestamp and evaluate the time window expression to determine the start and end of the window that the timestamp falls into.
- **Window Size**: It can also determine the size (duration) of the time window from the expression.
-This is essential for both marking windows as dirty and for generating the correct filter conditions when querying the source table.
+The same calculation is used to mark dirty windows and generate the source-table filters.
## Query Execution Flow
Here's a simplified step-by-step walkthrough of how a query is executed in batch mode:
1. **Data Ingestion**: New data is written to a source table.
-2. **Marking Dirty**: The `BatchingEngine` receives a notification about the new data. It uses the `TimeWindowExpr` associated with each relevant flow to determine which time windows are affected by the new data points. These windows are then added to the `DirtyTimeWindows` set in the corresponding `TaskState`.
-3. **Task Wake-up**: The `BatchingTask`'s execution loop wakes up, either due to its periodic schedule or because it was notified of a large backlog of dirty windows.
+2. **Marking Dirty**: The `BatchingEngine` receives a notification about the new data. It uses the `TimeWindowExpr` associated with each relevant flow to determine which time windows are affected by the new data points. These windows are then added to the `DirtyTimeWindows` set in the corresponding `TaskState`. Marking a window dirty does not wake the task.
+3. **Next Evaluation**: The `BatchingTask`'s execution loop reaches its next evaluation, either at a scheduled time or after its adaptive polling interval, and consumes the pending dirty windows.
4. **Plan Generation**: The task calls `gen_insert_plan`. This method:
- Inspects the `DirtyTimeWindows`.
- Generates a series of `OR`'d `WHERE` clauses (e.g., `(ts >= 't1' AND ts < 't2') OR (ts >= 't3' AND ts < 't4') ...`) that cover the dirty windows.
diff --git a/versioned_docs/version-1.2/contributor-guide/flownode/dataflow.md b/versioned_docs/version-1.2/contributor-guide/flownode/dataflow.md
index 000a65edb3..c876054d6d 100644
--- a/versioned_docs/version-1.2/contributor-guide/flownode/dataflow.md
+++ b/versioned_docs/version-1.2/contributor-guide/flownode/dataflow.md
@@ -1,17 +1,38 @@
---
-keywords: [dataflow module, SQL query transformation, execution plan, DAG, map and reduce operations]
-description: Explanation of the dataflow module in Flownode, its operations, internal data handling, and future enhancements.
+keywords: [Flownode, batching mode, streaming mode, dataflow, dirty time windows]
+description: How Flownode selects and runs its batching and legacy streaming execution paths.
---
# Dataflow
+Flownode has two internal execution paths:
+
+- **Batching mode** is the primary path for aggregation and TQL workloads. It evaluates queries over persisted source data and writes materialized results to a sink table.
+- **Streaming mode** is the legacy path retained for compatibility and deprecated for new workloads. It incrementally processes rows mirrored from Frontend as they arrive.
+
+Users do not select the mode directly. When a Flow is created, GreptimeDB chooses the path from the query and source-table properties. Aggregation, `DISTINCT`, and TQL queries use batching mode. Simple non-aggregation queries, and any Flow whose source table has `ttl = 'instant'`, currently use streaming mode. A Flow deferred because its source table does not yet exist starts as a pending batching Flow.
+
+## Batching mode
+
+Batching mode reuses GreptimeDB's query engine instead of maintaining an operator graph for every incoming row. For a time-windowed Flow, its main loop is:
+
+1. A source-table write marks the affected time windows as dirty.
+2. A `BatchingTask` runs on its evaluation schedule or adaptive polling cadence and collects the pending dirty windows at that evaluation. Marking a window dirty does not wake the task.
+3. The task adds time predicates for those windows to the Flow query and asks Frontend to execute it against the source tables.
+4. The query result is inserted into the sink table, updating the materialized result for windows that were evaluated.
+5. Successfully processed windows are removed from the dirty set. Failed work remains available for a later evaluation.
+
+Flows with an evaluation interval but without a time-window expression run the complete query on each scheduled evaluation. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Flownode Batching Mode Developer Guide](./batching_mode.md) for the task and dirty-window components.
+
+## Streaming mode
+
The `dataflow` module (see `flow::compute` module) is the core computing module of `flow`.
It takes a SQL query and transforms it into flow's internal execution plan.
This execution plan is then rendered into an actual dataflow, which is essentially a directed acyclic graph (DAG) of functions with input and output ports.
-The dataflow is triggered to run when needed.
+New row changes drive the graph incrementally.
-Currently, this dataflow only supports `map` and `reduce` operations. Support for `join` operations will be added in the future.
+The renderer supports map/filter/project and reduce operations. Join and union plan nodes exist, but their streaming renderers are not implemented.
Internally, the dataflow handles data in row format, using a tuple `(row, time, diff)`. Here, `row` represents the actual data being passed, which may contain multiple `Value` objects.
`time` is the system time which tracks the progress of the dataflow, and `diff` typically represents the insertion or deletion of the row (+1 or -1).
-Therefore, the tuple represents the insert/delete operation of the `row` at a given system `time`.
\ No newline at end of file
+Therefore, the tuple represents the insert/delete operation of the `row` at a given system `time`. Stateful operators keep indexed traces of these changes in an [Arrangement](./arrangement.md).
diff --git a/versioned_docs/version-1.2/contributor-guide/frontend/distributed-querying.md b/versioned_docs/version-1.2/contributor-guide/frontend/distributed-querying.md
index 21ee07d7e8..ca3822e113 100644
--- a/versioned_docs/version-1.2/contributor-guide/frontend/distributed-querying.md
+++ b/versioned_docs/version-1.2/contributor-guide/frontend/distributed-querying.md
@@ -5,29 +5,16 @@ description: Describes the process of distributed querying in GreptimeDB, focusi
# Distributed Querying
-Most steps of querying in frontend and datanode are identical. The only difference is that
-Frontend have a "special" step in planning phase to make the logical query plan distributed.
-Let's reference it as "dist planner" in the following text.
-
-The modified, distributed logical plan has multiple stages, each of them is executed in different
-server node.
+Frontend and Datanode use the same DataFusion-based query engine. In distributed mode, Frontend adds a planning step that separates work performed by Datanodes from work completed by Frontend.
## Dist Planner
-Planner will traverse the input logical plan, and split it into multiple stages by the "[commutativity
-rule](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md)".
+The distributed planner rewrites the logical plan. It pushes compatible operators toward table scans and wraps remote subplans in `MergeScan` nodes. Partition predicates are also used to prune Regions before the remote work is scheduled.
-This rule is under heavy development. At present it will consider things like:
-- whether the operator itself is commutative
-- how the partition rule is configured
-- etc...
+Whether an operator can be pushed down depends on the plan shape and the operator's properties. Unsupported parts remain on Frontend. The original design and its commutativity rules are described in the [distributed planner RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md).
## Dist Plan
-Except the first stage, which have to read data from files in storage. All other stages' leaf node
-are actually a gRPC call to its previous stage.
-
-Sub-plan in a stage is itself a complete logical plan, and can be executed independently without
-the follow up stages. The plan is encoded in [substrait format](https://substrait.io).
+A remote input is a complete logical subplan, not just a table scan. Frontend serializes the subplan in [Substrait](https://substrait.io) format and sends a Region-specific request to the Datanode that owns the data. The Datanode plans and executes it locally, then streams the result back. Frontend merges the remote streams and executes any operators that were not pushed down.
diff --git a/versioned_docs/version-1.2/contributor-guide/frontend/overview.md b/versioned_docs/version-1.2/contributor-guide/frontend/overview.md
index 3a2f63e7ae..18cabfc2b8 100644
--- a/versioned_docs/version-1.2/contributor-guide/frontend/overview.md
+++ b/versioned_docs/version-1.2/contributor-guide/frontend/overview.md
@@ -5,27 +5,47 @@ description: Overview of GreptimeDB's Frontend component - a stateless proxy ser
# Frontend
-The **Frontend** is a stateless service that serves as the entry point for client requests in GreptimeDB. It provides a unified interface for multiple database protocols and acts as a proxy that forwards read/write requests to appropriate Datanodes in the distributed system.
+Frontend is GreptimeDB's stateless request-orchestration service. The server layer terminates protocols and converts wire messages; Frontend supplies the database behavior behind those handlers, including permission checks, statement execution, routing, and distributed query planning.
+
+Frontend does not store table data. It caches catalog and route metadata obtained from Metasrv, and Metasrv invalidates those caches through heartbeat responses when metadata changes.
## Core Functions
-- **Protocol Support**: Multiple database protocols including SQL, PromQL, MySQL, and PostgreSQL. See [Protocols][1] for details
-- **Request Routing**: Routes requests to appropriate Datanodes based on metadata
-- **Query Distribution**: Splits distributed queries across multiple nodes
-- **Response Aggregation**: Combines results from multiple Datanodes
-- **Authorization**: Security and access control validation
+- Provide query and ingestion behavior for the supported [protocols][1].
+- Resolve catalogs, schemas, tables, and Region routes.
+- Validate permissions before executing a request.
+- Plan distributed queries and merge results from Datanodes.
+- Convert table-level writes and deletes into Region requests.
## Architecture
### Key Components
-- **Protocol Handlers**: Handle different database protocols
-- **Catalog Manager**: Caches metadata from Metasrv to enable efficient request routing and schema validation
-- **Dist Planner**: Converts logical plans to distributed execution plans
-- **Request Router**: Determines target Datanodes for each request
+
+- Protocol handlers adapt SQL, PromQL, gRPC ingestion, and observability protocols to Frontend's internal request interfaces.
+- The catalog and partition managers provide table metadata, partition rules, and Region routes.
+- The statement executor dispatches queries, DML, and DDL to their respective execution paths.
+- The distributed planner replaces table scans with `MergeScan` plans that can run across Datanodes.
### Request Flow
-
+The request path depends on the operation.
+
+#### Queries
+
+1. A protocol handler creates the query context and performs authentication and permission checks.
+2. The language-specific planner produces a logical plan. In distributed mode, the planner uses partition metadata to select Regions and constructs a distributed plan.
+3. Frontend sends Region subplans to the owning Datanodes. Datanodes execute them against local Region engines and return streams of Arrow record batches.
+4. Frontend runs the remaining operators, merges the streams, and formats the result for the client protocol.
+
+#### Writes and deletes
+
+1. Frontend validates the request against the table schema. Protocols that support schema-on-write may create a missing table or add columns before retrying the write.
+2. The partition rule assigns rows to Regions. Frontend builds one Region request per target and routes it to the current Region leader.
+3. The Datanode's Region server dispatches each request to the Region engine. In standalone mode, the same request is sent to an embedded Region server instead of over RPC.
+
+#### DDL
+
+The statement executor converts DDL into a task. In distributed mode, Metasrv runs that task as a persisted procedure, updates metadata, and coordinates Region operations on Datanodes. Standalone mode uses the same statement boundary with local implementations of the metadata and procedure services.
### Deployment
diff --git a/versioned_docs/version-1.2/contributor-guide/frontend/table-sharding.md b/versioned_docs/version-1.2/contributor-guide/frontend/table-sharding.md
index a60276d14e..beaece40c0 100644
--- a/versioned_docs/version-1.2/contributor-guide/frontend/table-sharding.md
+++ b/versioned_docs/version-1.2/contributor-guide/frontend/table-sharding.md
@@ -5,21 +5,15 @@ description: Explains how table data in GreptimeDB is sharded and distributed, i
# Table Sharding
-The sharding of stored data is essential to any distributed database. This document will describe how table's data in GreptimeDB is being sharded, and distributed.
+GreptimeDB shards a table into Regions. Partition expressions define which rows belong to each Region, while Region routes define which Datanode currently owns each Region.
## Partition
-For the syntax of creating a partitioned table, please refer to the [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) section in the User Guide.
+A partition is a logical row set described by an expression over one or more columns. The partition layout must cover the table's input domain so each row has one target Region. See [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) for the SQL syntax and supported expressions.
## Region
-The data within a table is logically split after creating partitions. You may ask the question "
-how are the data, after being logically partitioned, stored in the GreptimeDB? The answer is in "`Region`"s.
-
-Each region is corresponding to a partition, and stores the data in the partition. The regions are distributed among
-`Datanode`s. `Metasrv` manages the route information that maps regions to Datanodes.
-If the partition layout needs to change after table creation, GreptimeDB supports explicit
-[repartitioning](/user-guide/deployments-administration/manage-data/repartition.md) through split and merge operations.
+Each partition maps to one Region. Region IDs remain the storage and routing identity used by Frontend, Datanode, and Metasrv. Multiple Regions from the same table may be placed on one Datanode.
The relationship between partition and region can be viewed as the following diagram:
@@ -53,3 +47,14 @@ The relationship between partition and region can be viewed as the following dia
│ │
└──────────────────────────────────┘
Could be placed in one Datanode
+```
+
+## Routing and Pruning
+
+For writes, Frontend evaluates the partition rule for each row, groups rows by Region, and sends Region requests to the current leaders from the route table.
+
+For queries, the distributed planner compares query predicates with the partition expressions. It scans only Regions that can satisfy the predicates. If partition metadata is missing or cannot be interpreted safely, the planner falls back to all Regions rather than risk omitting data.
+
+## Changing the Partition Layout
+
+[Repartitioning](/user-guide/deployments-administration/manage-data/repartition.md) changes an existing layout through explicit split and merge operations. Metasrv runs the change as a persisted procedure, updates the Region routes and partition expressions, and invalidates stale table-route caches. New requests use the published layout after their Frontend refreshes that metadata.
diff --git a/versioned_docs/version-1.2/contributor-guide/getting-started.md b/versioned_docs/version-1.2/contributor-guide/getting-started.md
index b17184dfa8..7e8cca3ad8 100644
--- a/versioned_docs/version-1.2/contributor-guide/getting-started.md
+++ b/versioned_docs/version-1.2/contributor-guide/getting-started.md
@@ -15,14 +15,13 @@ At the moment, GreptimeDB supports Linux (both amd64 and arm64), macOS (both amd
### Build Dependencies
-- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) (optional)
+- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) (optional; needed to clone the repository, not to build it)
- C/C++ Toolchain: provides essential tools for compiling and linking. This is available either as `build-essential` on ubuntu or a similar name on other platforms.
-- Rust nightly toolchain ([guide][1])
- - Compile the source code
+- [Rustup][1]. The repository pins the required nightly toolchain in `rust-toolchain.toml`.
- Protobuf ([guide][2])
- Compile the proto file
- Note that the version needs to be >= 3.15. You can check it with `protoc --version`
-- Machine: Recommended memory is 16GB or more, or use the [mold](https://github.com/rui314/mold) tool to reduce memory usage during linking.
+- Machine: 16GB of memory or more is recommended. On a smaller machine, use [mold](https://github.com/rui314/mold) to reduce memory usage during linking.
[1]:
[2]:
diff --git a/versioned_docs/version-1.2/contributor-guide/how-to/how-to-write-sdk.md b/versioned_docs/version-1.2/contributor-guide/how-to/how-to-write-sdk.md
index 40d0bc3713..0fd47f94e4 100644
--- a/versioned_docs/version-1.2/contributor-guide/how-to/how-to-write-sdk.md
+++ b/versioned_docs/version-1.2/contributor-guide/how-to/how-to-write-sdk.md
@@ -1,21 +1,17 @@
---
keywords: [gRPC SDK, GreptimeDatabase, Handle, HandleRequests, GreptimeRequest, GreptimeResponse]
-description: Explains how to write a gRPC SDK for GreptimeDB, focusing on the GreptimeDatabase service, its methods, and the structure of requests and responses.
+description: Protocol contracts and error-handling requirements for a GreptimeDB gRPC ingestion SDK.
---
# How to write a gRPC SDK for GreptimeDB
-A GreptimeDB gRPC SDK only needs to handle the writes. The reads are standard SQL and PromQL, can be handled by any JDBC
-client or Prometheus client. This is also why GreptimeDB gRPC SDKs are all named
-like "`greptimedb-ingester-`". Please make sure your GreptimeDB SDK follow the same naming convention.
+GreptimeDB's public gRPC SDKs are ingestion clients. Queries normally use SQL or PromQL through their standard clients. A new SDK should therefore focus on writes and deletes unless it has a separate requirement, and follow the `greptimedb-ingester-` naming convention. See the [gRPC SDK overview](/user-guide/ingest-data/for-iot/grpc-sdks/overview.md) for the user-facing API.
## `GreptimeDatabase` Service
-GreptimeDB defines a custom gRPC service called `GreptimeDatabase`. All you need to do in your SDK are implement it. You
-can find its Protobuf
-definitions [here](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto).
+Generate client stubs from the [`GreptimeDatabase` Protobuf definition](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto). Do not maintain a handwritten copy of the messages or service definition.
-The service contains two RPC methods:
+The service provides a unary method and a client-streaming method:
```protobuf
service GreptimeDatabase {
@@ -25,13 +21,9 @@ service GreptimeDatabase {
}
```
-The `Handle` method is for unary call: when a `GreptimeRequest` is received and processed by a GreptimeDB
-server, it responds with a `GreptimeResponse` immediately.
+`Handle` returns one response for one request. It is the usual choice for an SDK's insert and delete APIs.
-The `HandleRequests` acts in
-a "[Client streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc)" style. It ingests a
-stream of `GreptimeRequest`, and handles them on the fly. After all the requests have been handled, it returns a
-summarized `GreptimeResponse`. Through `HandleRequests`, we can achieve a very high throughput of requests handling.
+`HandleRequests` is a [client-streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc). The server returns a cumulative response only after the client closes the request stream. An SDK that exposes streaming must document this acknowledgement boundary and bind the stream to one endpoint.
### `GreptimeRequest`
@@ -51,13 +43,13 @@ message GreptimeRequest {
}
```
-A `RequestHeader` is needed, it includes some context, authentication and others. The "oneof" field contains the request
-to the GreptimeDB server.
+A client must populate `RequestHeader` with the database context and authentication expected by the server. Set exactly one request variant.
-Note that we have two types of insertions, one is in the form of "column" (the `InsertRequests`), and the other is "
-row" (`RowInsertRequests`). It's generally recommended to use the "row" form, since it's more natural for insertions on
-a table, and easier to use. However, if there's a need to insert a large number of columns at once, or there're plenty
-of "null" values to insert, the "column" form is better to be used.
+The message also contains query and DDL variants used by internal callers. The public ingester API should not expose them: `GreptimeDatabase` does not return query result streams.
+
+GreptimeDB accepts row-oriented `RowInsertRequests` and column-oriented `InsertRequests`. Row-oriented requests are the default for public ingestion APIs. A column-native client may use the column form, but it must keep column lengths consistent and preserve null values, timestamp precision, data types, and column semantic types during conversion.
+
+Deletes have the same row-oriented and column-oriented distinction. Expose only the forms that the SDK can map without losing type information.
### `GreptimeResponse`
@@ -70,8 +62,18 @@ message GreptimeResponse {
}
```
-The `ResponseHeader` contains the response's status code, and error message (if there's any). The "oneof" response only
-contains the affected rows for now.
+On success, the response contains a successful header and `affected_rows`. Treat that value as the number acknowledged by the server, including the cumulative value returned when a request stream closes.
+
+Request failures are returned as a gRPC status. When present, the trailing metadata keys `x-greptime-err-code` and `x-greptime-err-retry-hint` carry GreptimeDB's error code and retry classification. Preserve the gRPC status and expose the GreptimeDB metadata rather than replacing them with a generic SDK error.
+
+## Retry and Delivery Semantics
+
+Retries must be bounded and observable. A unary request may be retried only when the failure is classified as retryable and the deadline still permits it. Do not retry cancellation or deadline-expiration errors.
+
+A lost response does not prove that the server rejected a write. Retrying such a request can insert duplicate rows unless the caller's data model makes the operation idempotent. Document this possibility and return the final error when delivery is ambiguous.
+
+Do not transparently retry a partially sent `HandleRequests` stream. The server may already have accepted some requests even though the client has not received the cumulative response. Close the failed stream and report the ambiguity to the caller.
+
+Keep Arrow Flight bulk ingestion separate from the `GreptimeDatabase` RPCs. Its batching and partial-acceptance behavior needs its own API contract.
-GreptimeDB has a lot of SDKs now, you can refer to
-them [here](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=) for some examples.
+Use the existing [GreptimeDB ingester repositories](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=) to compare public API conventions, but derive wire behavior from the current Protobuf definition and server contract.
diff --git a/versioned_docs/version-1.2/contributor-guide/metasrv/admin-api.md b/versioned_docs/version-1.2/contributor-guide/metasrv/admin-api.md
index b091b48b70..d1f2ee6dbe 100644
--- a/versioned_docs/version-1.2/contributor-guide/metasrv/admin-api.md
+++ b/versioned_docs/version-1.2/contributor-guide/metasrv/admin-api.md
@@ -1,6 +1,6 @@
---
-keywords: [admin api, health check, leader query, heartbeat, maintenance mode, RESTful API]
-description: Details the Admin API for Metasrv, including endpoints for health checks, leader queries, heartbeat data, maintenance mode, and Procedure Manager controls.
+keywords: [admin api, health check, leader query, heartbeat, maintenance mode, recovery mode, table id sequence]
+description: Details the Metasrv Admin API for status inspection, cluster controls, and metadata recovery.
---
# Admin API
@@ -9,16 +9,17 @@ description: Details the Admin API for Metasrv, including endpoints for health c
Note that all Admin API endpoints in this document listen on Metasrv's `HTTP_PORT`, which defaults to `4000`.
:::
-The Admin API provides a simple way to view and manage cluster information, including metasrv health detection, metasrv leader query, datanode heartbeat detection, maintenance mode, and Procedure Manager controls.
-
-The Admin API is an HTTP service that provides a set of RESTful APIs that can be called through HTTP requests. The Admin API is simple, user-friendly and safe.
+The Admin API exposes Metasrv status, cluster controls, and metadata recovery operations over HTTP. It does not provide authentication, and some endpoints change cluster behavior or metadata allocation. Deployments must protect the HTTP port with network-level controls.
This page covers the following APIs:
- /health
- /leader
- /heartbeat
+- /node-lease
- /maintenance
- /procedure-manager
+- /recovery
+- /sequence/table
All these APIs are under the parent resource `/admin`.
@@ -26,7 +27,7 @@ In the following sections, we assume that your metasrv instance is running on lo
## /health HTTP endpoint
-The `/health` endpoint accepts GET HTTP requests and you can use this endpoint to check the health of your metasrv instance.
+The `/health` endpoint accepts GET requests and returns `OK` when the HTTP service is running. It does not check whether this Metasrv is the leader or whether external dependencies are available.
### Definition
@@ -120,9 +121,17 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100'
]
```
+## /node-lease HTTP endpoint
+
+The `/node-lease` endpoint returns the current leases recorded for Datanodes. Use it when diagnosing whether Metasrv still considers a Datanode active.
+
+```bash
+curl -X GET http://localhost:4000/admin/node-lease
+```
+
## /maintenance HTTP endpoint
-Cluster Maintenance Mode is a safety feature in GreptimeDB that temporarily disables automatic cluster management operations. This mode is particularly useful during cluster upgrades, planned downtime, and any operation that might temporarily affect cluster stability. For more details, please refer to [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md).
+Maintenance mode temporarily disables automatic cluster management operations during upgrades, planned downtime, or similar work. See [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md) for its effect on the cluster.
The `/maintenance` endpoint supports the following HTTP requests:
@@ -155,3 +164,39 @@ The response body uses the following format:
"status": "running"
}
```
+
+## /recovery HTTP endpoints
+
+Recovery mode gates metadata repair endpoints such as manual table ID sequence changes. It is intended for recovery work, not routine maintenance.
+
+- `GET /admin/recovery/status`: query whether recovery mode is enabled.
+- `POST /admin/recovery/enable`: enable recovery mode.
+- `POST /admin/recovery/disable`: disable recovery mode.
+
+The response body uses the following format:
+
+```json
+{
+ "enabled": true
+}
+```
+
+Disable recovery mode after the repair is complete. Use [maintenance mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md) instead when the goal is to suspend automatic cluster operations during planned maintenance.
+
+## /sequence/table HTTP endpoints
+
+These endpoints inspect or repair the table ID sequence:
+
+- `GET /admin/sequence/table/next-id`: return the next table ID without allocating it.
+- `POST /admin/sequence/table/set-next-id`: advance the next table ID.
+
+Setting the sequence requires recovery mode. The new value must be greater than the current value; the API cannot move the sequence backwards. Recovery mode is an API precondition, not a DDL barrier. Follow [Manage table ID sequences](/user-guide/deployments-administration/maintenance/sequence-management.md) for the required cluster-wide procedure.
+
+```bash
+curl -X POST \
+ -H 'Content-Type: application/json' \
+ -d '{"next_table_id": 2048}' \
+ http://localhost:4000/admin/sequence/table/set-next-id
+```
+
+Changing this value affects IDs allocated to future tables.
diff --git a/versioned_docs/version-1.2/contributor-guide/metasrv/overview.md b/versioned_docs/version-1.2/contributor-guide/metasrv/overview.md
index 7aa052dd7e..6458b6380d 100644
--- a/versioned_docs/version-1.2/contributor-guide/metasrv/overview.md
+++ b/versioned_docs/version-1.2/contributor-guide/metasrv/overview.md
@@ -1,161 +1,101 @@
---
-keywords: [metasrv, metadata, request-router, load balancing, election, high availability, heartbeat]
-description: Provides an overview of the Metasrv service, its components, interactions with the Frontend, architecture, and key functionalities like distributed consensus and heartbeat management.
+keywords: [metasrv, metadata, routing, leader election, procedure, heartbeat]
+description: Overview of the metadata and coordination mechanisms provided by Metasrv.
---
# Metasrv
-
-
## What's in Metasrv
-- Store metadata (Catalog, Schema, Table, Region, etc.)
-- Request-Router. It tells the Frontend where to write and read data.
-- Load balancing for Datanode, determines who should handle new table creation requests, more precisely, it makes resource allocation decisions.
-- Election & High Availability, GreptimeDB is designed in a Leader-Follower architecture, only Leader nodes can write while Follower nodes can read, the number of Follower nodes is usually >= 1, and Follower nodes need to be able to switch to Leader quickly when Leader is not available.
-- Statistical data collection (reported via Heartbeats on each node), such as CPU, Load, number of Tables on the node, average/peak data read/write size, etc., can be used as the basis for distributed scheduling.
+Metasrv is the metadata and coordination service in a distributed GreptimeDB cluster. It does not sit on the data path. Its main responsibilities are:
+
+- storing Catalog, Schema, Table, Region, route, and node metadata;
+- choosing Datanodes for new Regions and maintaining table routes;
+- electing one Metasrv leader to coordinate metadata changes;
+- running recoverable procedures for DDL, Region migration, failover, and repartitioning;
+- tracking node leases and Region statistics through heartbeats;
+- broadcasting cache invalidations to Frontends, Datanodes, and Flownodes when metadata changes;
+- sending Region lifecycle instructions to Datanodes.
## How the Frontend interacts with Metasrv
-First, the routing table in Request-Router is in the following structure (note that this is only the logical structure, the actual storage structure varies, for example, endpoints may have dictionary compression).
+Frontend obtains table metadata and Region routes from Metasrv and caches them locally. Metadata-changing statements are sent to the Metasrv leader, while reads and writes use the cached routes to reach Datanodes directly.
+
+The control and data paths are separate:
+
+```text
+Frontend
+ |-- metadata lookup and DDL ------------> Metasrv leader
+ `-- Region reads and writes ------------> Datanode
+
+Metasrv leader
+ |-- Region lifecycle instructions ------> Datanode
+ `-- cache invalidations ----------------> Frontend / Datanode / Flownode
+Datanode
+ `-- heartbeat, lease renewal, Region stats -> Metasrv leader
```
- table_A
- table_name
- table_schema // for physical plan
- regions
- region_1
- mutate_endpoint
- select_endpoint_1, select_endpoint_2
- region_2
- mutate_endpoint
- select_endpoint_1, select_endpoint_2, select_endpoint_3
- region_xxx
- table_B
- ...
+
+In steady state, a table route records one leader peer and zero or more follower peers for each Region. The leader is the write target. Deployments with read-replica support can route reads to followers:
+
+```text
+Table route
+ |-- Region 0
+ | |-- leader -> Datanode A
+ | `-- followers -> Datanode B, Datanode C
+ `-- Region 1
+ `-- leader -> Datanode D
```
+Region migration or failover changes peer roles and can temporarily leave a Region without a leader. Frontend refreshes its cached route before sending subsequent reads or writes to the current peers.
+
### Create Table
-1. The Frontend sends `CREATE TABLE` requests to Metasrv.
-2. Plan the number of Regions according to the partition rules contained in the request.
-3. Check the global view of resources available to Datanodes (collected by Heartbeats) and assign one node to each region.
-4. The Frontend creates the table and stores the `Schema` to Metasrv after successful creation.
+1. Frontend submits the DDL request to the Metasrv leader.
+2. Metasrv derives Regions from the partition rules and [selects a Datanode for each Region](/contributor-guide/metasrv/selector.md).
+3. A persisted procedure creates the Regions and records the table and route metadata. If leadership changes, the procedure can resume from its persisted state.
+4. Metasrv notifies Frontends after the metadata change is committed so their caches can be refreshed.
### Insert
-1. The Frontend fetches the routes of the specified table from Metasrv. Note that the smallest routing unit is the route of the table (several regions), i.e., it contains the addresses of all regions of this table.
-2. The best practice is that the Frontend first fetches the routes from its local cache and forwards the request to the Datanode. If the route is no longer valid, then Datanode is obliged to return an `Invalid Route` error, and the Frontend re-fetches the latest data from Metasrv and updates its cache. Route information does not change frequently, thus, it's sufficient for Frontend uses the Lazy policy to maintain the cache.
-3. The Frontend processes a batch of writes that may contain multiple tables and multiple regions, so the Frontend needs to split user requests based on the 'route table'.
+Frontend resolves the table route, splits rows according to the partition rules, and sends each Region write to the corresponding Datanode. Route changes cause the cached metadata to be invalidated and fetched again from Metasrv.
### Select
-1. As with `Insert`, the Frontend first fetches the route table from the local cache.
-2. Unlike `Insert`, for `Select`, the Frontend needs to extract the read-only node (follower) from the route table, then dispatch the request to the leader or follower node depending on the priority.
-3. The distributed query engine in the Frontend distributes multiple sub-query tasks based on the routing information and aggregates the query results.
+Frontend uses table and Region metadata while planning the query. Predicates on partition columns prune Regions, and the distributed query engine sends work to the Datanodes that own the selected Regions. See [Distributed Querying](../frontend/distributed-querying.md).
## Metasrv Architecture
-
-
-## Distributed Consensus
+The main coordination paths are:
+
+```text
+Leader election
+ |
+ v
+Metasrv leader
+├─ DDL manager -> Procedure manager
+├─ Selector -> new Region placement
+├─ Heartbeat handler chain -> leases and Region statistics
+├─ Region supervisor -> Region migration procedures
+├─ Mailbox -> cache invalidations and Region instructions
+└─ Metadata managers -> KV backend
+```
-As you can see, Metasrv has a dependency on distributed consensus because:
+These mechanisms share metadata, but they have different failure boundaries. A process restart may discard caches and leader-local state; metadata and procedure state required for recovery must be durable.
-1. First, Metasrv has to elect a leader, Datanode only sends heartbeats to the leader, and we only use a single metasrv node to receive heartbeats, which makes it easy to do some calculations or scheduling accurately and quickly based on global information. As for how the Datanode connects to the leader, this is for MetaClient to decide (using a redirect, Heartbeat requests becomes a gRPC stream, and using redirect will be less error-prone than forwarding), and it is transparent to the Datanode.
-2. Second, Metasrv must provide an election API for Datanode to elect "write" and "read-only" nodes and help Datanode achieve high availability.
-3. Finally, `Metadata`, `Schema` and other data must be reliably and consistently stored on Metasrv. Therefore, consensus-based algorithms are the ideal approach for storing them.
+## Distributed Consensus
-For the first version of Metasrv, we choose Etcd as the consensus algorithm component (Metasrv is designed to consider adapting different implementations and even creating a new wheel) for the following reasons:
+Metasrv separates leader election from metadata storage. Only the elected Metasrv leader performs coordination and metadata-changing operations. Other Metasrv nodes direct clients to the current leader.
-1. Etcd provides exactly the API we need, such as `Watch`, `Election`, `KV`, etc.
-2. We only perform two tasks with distributed consensus: elections (using the `Watch` mechanism) and storing (a small amount of metadata), and neither of them requires us to customize our own state machine, nor do we need to customize our own state machine based on raft; the small amount of data also does not require multi-raft-group support.
-3. The initial version of Metasrv uses Etcd, which allows us to focus on the capabilities of Metasrv and not spend too much effort on distributed consensus algorithms, which improves the design of the system (avoiding coupling with consensus algorithms) and helps with rapid development at the beginning, as well as allows easy access to good consensus algorithm implementations in the future through good architectural designs.
+The key-value backend stores table metadata, routes, procedure state, and other information that must survive a leader change. Metasrv does not use this election to create leader and follower replicas for Datanode Regions; Region availability is managed through heartbeats, Region failure detection, and failover procedures.
## Heartbeat Management
-The primary means of communication between Datanode and Metasrv is the Heartbeat Request/Response Stream, and we want this to be the only way to communicate. This idea is inspired by the design of [TiKV PD](https://github.com/tikv/pd), and we have practical experience in [RheaKV](https://github.com/sofastack/sofa-jraft/tree/master/jraft-rheakv/rheakv-pd). The request sends its state, while Metasrv sends different scheduling instructions via Heartbeat Response.
-
-A heartbeat will probably carry the data listed below, but this is not the final design, and we are still discussing and exploring exactly which data should be mostly collected.
-
-```
-service Heartbeat {
- // Heartbeat, there may be many contents of the heartbeat, such as:
- // 1. Metadata to be registered to metasrv and discoverable by other nodes.
- // 2. Some performance metrics, such as Load, CPU usage, etc.
- // 3. The number of computing tasks being executed.
- rpc Heartbeat(stream HeartbeatRequest) returns (stream HeartbeatResponse) {}
-}
-
-message HeartbeatRequest {
- RequestHeader header = 1;
-
- // Self peer
- Peer peer = 2;
- // Leader node
- bool is_leader = 3;
- // Actually reported time interval
- TimeInterval report_interval = 4;
- // Node stat
- NodeStat node_stat = 5;
- // Region stats in this node
- repeated RegionStat region_stats = 6;
- // Follower nodes and stats, empty on follower nodes
- repeated ReplicaStat replica_stats = 7;
-}
-
-message NodeStat {
- // The read capacity units during this period
- uint64 rcus = 1;
- // The write capacity units during this period
- uint64 wcus = 2;
- // Table number in this node
- uint64 table_num = 3;
- // Region number in this node
- uint64 region_num = 4;
-
- double cpu_usage = 5;
- double load = 6;
- // Read disk I/O in the node
- double read_io_rate = 7;
- // Write disk I/O in the node
- double write_io_rate = 8;
-
- // Others
- map attrs = 100;
-}
-
-message RegionStat {
- uint64 region_id = 1;
- TableName table_name = 2;
- // The read capacity units during this period
- uint64 rcus = 3;
- // The write capacity units during this period
- uint64 wcus = 4;
- // Approximate region size
- uint64 approximate_size = 5;
- // Approximate number of rows
- uint64 approximate_rows = 6;
-
- // Others
- map attrs = 100;
-}
-
-message ReplicaStat {
- Peer peer = 1;
- bool in_sync = 2;
- bool is_learner = 3;
-}
-```
-
-## Central Nervous System (CNS)
-
-We are to build an algorithmic system, which relies on real-time and historical heartbeat data from each node, should make some smarter scheduling decisions and send them to Metasrv's Autoadmin unit, which distributes the scheduling decisions, either by the Datanode itself or more likely by the PaaS platform.
-
-## Abstraction of Workloads
+Datanodes maintain heartbeat streams to the Metasrv leader. Heartbeat requests report node identity, lease information, Region statistics, and other state used for placement and supervision. Responses carry control messages such as Region lifecycle instructions and cache invalidations.
-The level of workload abstraction determines the efficiency and quality of the scheduling strategy generated by Metasrv such as resource allocation.
+A heartbeat drives two independent mechanisms, and a change to heartbeat timing affects both:
-DynamoDB defines RCUs & WCUs (Read Capacity Units / Write Capacity Units), explaining that a RCU is a read request of 4KB data, and a WCU is a write request of 1KB data. When using RCU and WCU to describe workloads, it's easier to achieve performance measurability and get more informative resource preallocation because we can abstract different hardware capabilities as a combination of RCU and WCU.
+- **Node lease.** The keep-lease handler renews the sending Datanode's lease. Selectors and the `/node-lease` endpoint use these leases to decide whether a Datanode is still active.
+- **Region failure detection.** The Region supervisor keeps a per-Region Phi Accrual detector over heartbeat arrival intervals. Its verdict is independent of lease expiry.
-However, GreptimeDB still faces a more complex situation than DynamoDB, in particular, RCU doesn't fit to describe GreptimeDB's read workloads which require a lot of computation. We are working on that.
+A failure verdict submits a failover migration only when Region failover is enabled; it is disabled by default and requires remote WAL unless explicitly allowed on local WAL. Maintenance mode also suppresses failover. See [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md) for the prerequisites and how to enable it.
diff --git a/versioned_docs/version-1.2/contributor-guide/metasrv/selector.md b/versioned_docs/version-1.2/contributor-guide/metasrv/selector.md
index 790ffb849b..23190cd6a9 100644
--- a/versioned_docs/version-1.2/contributor-guide/metasrv/selector.md
+++ b/versioned_docs/version-1.2/contributor-guide/metasrv/selector.md
@@ -7,32 +7,28 @@ description: Describes the different types of selectors in the Metasrv service,
## Introduction
-What is the `Selector`? As its name suggests, it allows users to select specific items from a given `namespace` and `context`. There is a related trait, also named `Selector`, whose definition can be found [below][0].
-
-[0]: https://github.com/GreptimeTeam/greptimedb/blob/main/src/meta-srv/src/selector.rs
-
-There is a specific scenario in `Metasrv` service. When a request to create a table is sent to the `Metasrv` service, it creates a routing table (the details of table creation will not be described here). The `Metasrv` service needs to select the appropriate `Datanode` list when creating a routing table.
+When a table is created, Metasrv uses a `Selector` to choose Datanodes for its Regions. Selection uses the current node leases and, depending on the selector, Region statistics.
## Selector Type
The `Metasrv` service currently offers the following types of `Selectors`:
-### LeasebasedSelector
+### LeaseBasedSelector
-`LeasebasedSelector` randomly selects from all available (in lease) `Datanode`s, its characteristic is simplicity and fast.
+`LeaseBasedSelector` randomly selects from Datanodes with valid leases.
### LoadBasedSelector
The `LoadBasedSelector` load value is determined by the number of regions on each `Datanode`, fewer regions indicate lower load, and `LoadBasedSelector` prioritizes selecting low-load `Datanodes`.
### RoundRobinSelector [default]
-`RoundRobinSelector` selects `Datanode`s in a round-robin fashion. It is recommended and the default option in most cases. If you're unsure which to choose, it's usually the right choice.
+`RoundRobinSelector` selects `Datanode`s in a round-robin fashion. It is the default and recommended choice for most deployments.
## Configuration
You can configure the `Selector` by its name when starting the `Metasrv` service.
-- LeasebasedSelector: `lease_based` or `LeaseBased`
+- LeaseBasedSelector: `lease_based` or `LeaseBased`
- LoadBasedSelector: `load_based` or `LoadBased`
- RoundRobinSelector: `round_robin` or `RoundRobin`
diff --git a/versioned_docs/version-1.2/contributor-guide/overview.md b/versioned_docs/version-1.2/contributor-guide/overview.md
index 875b4b1e16..6e6ec669c9 100644
--- a/versioned_docs/version-1.2/contributor-guide/overview.md
+++ b/versioned_docs/version-1.2/contributor-guide/overview.md
@@ -5,9 +5,7 @@ description: Overview of GreptimeDB's architecture, key components, and how they
# Contributor Guide
-DeepWiki provides a detailed and clear explanation of GreptimeDB's architecture and implementation. Highly recommended:
-
-[https://deepwiki.com/GreptimeTeam/greptimedb](https://deepwiki.com/GreptimeTeam/greptimedb)
+This guide explains the internal design of GreptimeDB for contributors. Start with [Getting Started](/contributor-guide/getting-started.md) to build and run it from source. Submission requirements, including the CLA, license headers, formatting, and the checks a pull request must pass, are maintained in the source repository's [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md).
## Architecture
@@ -18,8 +16,13 @@ For more details on each component, see the following guides:
- [frontend][1]
- [datanode][2]
- [metasrv][3]
+- [flownode][4]
[1]: /contributor-guide/frontend/overview.md
[2]: /contributor-guide/datanode/overview.md
[3]: /contributor-guide/metasrv/overview.md
+[4]: /contributor-guide/flownode/overview.md
+
+## Additional reference
+[DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb) provides an automatically generated walkthrough of the GreptimeDB repository. It can help when exploring an unfamiliar area, but it is a secondary reference: verify version-sensitive behavior against the source code.
diff --git a/versioned_docs/version-1.2/contributor-guide/tests/integration-test.md b/versioned_docs/version-1.2/contributor-guide/tests/integration-test.md
index 5d5f6cb1a5..bc52ee4854 100644
--- a/versioned_docs/version-1.2/contributor-guide/tests/integration-test.md
+++ b/versioned_docs/version-1.2/contributor-guide/tests/integration-test.md
@@ -7,7 +7,14 @@ description: Guide on writing and running integration tests in GreptimeDB, cover
## Introduction
-Integration testing is written with Rust test harness (`#[test]`), unlike unit testing, they are placed separately
-[here](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration).
-It covers scenarios involving multiple components, in which one typical case is HTTP/gRPC-related features. You can check
-its [documentation](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md) for more information.
+Integration tests cover behavior that crosses crate or service boundaries, such as HTTP and gRPC handling, distributed components, or external storage. They use Rust's test harness and live in the [`tests-integration`](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration) package.
+
+Run the package with:
+
+```shell
+cargo nextest run -p tests-integration
+```
+
+Some cases require environment variables or fixtures for external services. Follow the package's [setup instructions](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md) before running those cases.
+
+Use an integration test when a crate-level test or a Sqlness case cannot exercise the required boundary. Keep isolated logic in unit tests so that failures remain fast to reproduce.
diff --git a/versioned_docs/version-1.2/contributor-guide/tests/overview.md b/versioned_docs/version-1.2/contributor-guide/tests/overview.md
index feeefe2b6d..6fe48b2c32 100644
--- a/versioned_docs/version-1.2/contributor-guide/tests/overview.md
+++ b/versioned_docs/version-1.2/contributor-guide/tests/overview.md
@@ -5,4 +5,13 @@ description: Overview of the testing methods used in GreptimeDB to ensure its be
# Tests
-Our team has conducted lots of tests to ensure the behaviours of `GreptimeDB` . This chapter will introduce several significant methods used to test `GreptimeDB`, and how to work with them.
+Choose the narrowest test that exercises the behavior you changed:
+
+| Test type | Use it for | Typical command |
+| --- | --- | --- |
+| [Unit test](unit-test.md) | Logic contained within one crate or component | `cargo nextest run -p ` |
+| [Sqlness test](sqlness-test.md) | SQL, protocol, planner, execution, and end-to-end regressions | `cargo sqlness bare -t ` |
+| [Integration test](integration-test.md) | Behavior that crosses components or requires external services | `cargo nextest run -p tests-integration` |
+| [Compatibility test](https://github.com/GreptimeTeam/greptimedb/blob/main/tests/compatibility/README.md) | Reading data or metadata written by an older release | `cargo sqlness compat --from-version ` |
+
+Run `make test` when a change needs the full Rust workspace test suite. Changes to persisted metadata, WAL records, SST files, or wire formats should also include a compatibility test when an older release may have produced the input.
diff --git a/versioned_docs/version-1.2/contributor-guide/tests/sqlness-test.md b/versioned_docs/version-1.2/contributor-guide/tests/sqlness-test.md
index b5fdcf2b0b..0262ff6851 100644
--- a/versioned_docs/version-1.2/contributor-guide/tests/sqlness-test.md
+++ b/versioned_docs/version-1.2/contributor-guide/tests/sqlness-test.md
@@ -7,42 +7,34 @@ description: Instructions for running SQL tests in GreptimeDB using the `sqlness
## Introduction
-SQL is an important user interface for `GreptimeDB`. We have a separate test suite for it (named `sqlness`).
+Sqlness is GreptimeDB's end-to-end regression suite for SQL and protocol behavior. A case sends statements to a running GreptimeDB instance and compares the output with a checked-in result file.
## Sqlness manual
### Case file
-Sqlness has two types of file
+Each case uses two files:
- `.sql`: test input, SQL only
- `.result`: expected test output, SQL and its results
-The `.result` file is the expected execution output. If you see `.result` files changed,
-it means the test gets a different result and indicates it may fail. You should
-check the change logs to solve the problem.
-
-You only need to write test SQL in the `.sql` file, and run the test.
+Write the input in the `.sql` file and run the test to generate or update `.result`. Review every result diff: accept it only when the behavior change is intended.
### Case organization
-The root dir of input cases is `tests/cases`. It contains several sub-directories stand for different test
-modes. E.g., `standalone/` contains all the tests to run under `greptimedb standalone start` mode.
+Input cases live under `tests/cases`. The first directory level selects an environment. For example, `standalone/` runs against a standalone GreptimeDB instance.
-Under the first level of sub-directory (e.g. the `cases/standalone`), you can organize your cases as you like.
-Sqlness walks through every file recursively and runs them.
+Within an environment, group a new case with the feature it exercises. Sqlness discovers case files recursively.
## Run the test
-Unlike other tests, this harness is in a binary target form. You can run it with
+Run the suite with:
```shell
-cargo run --bin sqlness-runner bare
+cargo sqlness bare
```
-It automatically finishes the following procedures: compile `GreptimeDB`, start it, grab tests and feed it to
-the server, then collect and compare the results. You only need to check whether any `.result` files changed.
-If not, congratulations, the test is passed 🥳!
+The command builds and starts GreptimeDB, runs the selected cases, and compares their output. A changed `.result` file is part of the review, not proof that the new output is correct.
### Run a specific test
diff --git a/versioned_docs/version-1.2/contributor-guide/tests/unit-test.md b/versioned_docs/version-1.2/contributor-guide/tests/unit-test.md
index 82e30bf5fa..183a72aab4 100644
--- a/versioned_docs/version-1.2/contributor-guide/tests/unit-test.md
+++ b/versioned_docs/version-1.2/contributor-guide/tests/unit-test.md
@@ -8,25 +8,26 @@ description: Guide on writing and running unit tests in GreptimeDB using Rust's
## Introduction
Unit tests are embedded into the codebase, usually placed next to the logic being tested.
-They are written using Rust's `#[test]` attribute and can run with `cargo nextest run`.
+They are written using Rust's `#[test]` attribute. GreptimeDB uses [`cargo-nextest`](https://nexte.st/) as its primary Rust test runner.
-The default test runner ships with `cargo` is not supported in GreptimeDB codebase. It's recommended
-to use [`nextest`](https://nexte.st/) instead. You can install it with
+Install it with:
```shell
cargo install cargo-nextest --locked
```
-And run the tests (here the `--workspace` is not necessary)
+Run the package you changed first:
```shell
-cargo nextest run
+cargo nextest run -p
```
-Notes if your Rust is installed via `rustup`, be sure to install `nextest` with `cargo` rather
-than the package manager like `homebrew`. Otherwise it will mess up your local environment.
+Use a test name or nextest filter to narrow the run further while developing. Before submitting a change with broad effects, run the full workspace suite:
+
+```shell
+make test
+```
## Coverage
-Our continuous integration (CI) jobs have a "coverage checking" step. It will report how many
-codes are covered by unit tests. Please add the necessary unit test to your patch.
+CI reports unit-test coverage. Add tests for changed behavior and failure cases that could otherwise regress; coverage percentage alone is not the goal.
diff --git a/versioned_docs/version-1.2/reference/sql/create.md b/versioned_docs/version-1.2/reference/sql/create.md
index 4eccb986d0..29a9cf666f 100644
--- a/versioned_docs/version-1.2/reference/sql/create.md
+++ b/versioned_docs/version-1.2/reference/sql/create.md
@@ -26,7 +26,7 @@ If the `db_name` database already exists, then GreptimeDB has the following beha
The database can also carry options similar to the `CREATE TABLE` statement by using the `WITH` keyword. The following options are available for databases:
- `ttl` - Time-To-Live for data in all tables within the database (cannot be set to `instant`)
-- `memtable.type` - Type of memtable (`time_series`, `partition_tree`)
+- `memtable.type` - Type of memtable (`bulk`, `time_series`)
- `append_mode` - Whether tables in the database should be append-only (`true`/`false`)
- `merge_mode` - Strategy for merging duplicate rows (`last_row`, `last_non_null`)
- `skip_wal` - Whether to disable Write-Ahead-Log for tables in the database (`'true'`/`'false'`)
@@ -74,7 +74,7 @@ Create a database with multiple options, including append mode and custom memtab
```sql
CREATE DATABASE test WITH (
ttl='30d',
- 'memtable.type'='partition_tree',
+ 'memtable.type'='bulk',
'append_mode'='true'
);
```
@@ -154,7 +154,7 @@ Users can add table options by using `WITH`. The valid options contain the follo
| `compaction.twcs.trigger_file_num` | Number of files in a specific time window to trigger a compaction | String value, such as '8'. Only available when `compaction.type` is `twcs`. You can refer to this [document](https://cassandra.apache.org/doc/latest/cassandra/managing/operating/compaction/twcs.html) to learn more about the `twcs` compaction strategy. |
| `compaction.twcs.time_window` | Compaction time window | String value, such as '1d' for 1 day. The table usually partitions rows into different time windows by their timestamps. Only available when `compaction.type` is `twcs`. |
| `compaction.twcs.max_output_file_size` | Maximum allowed output file size for TWCS compaction | String value, such as '1GB', '512MB'. Sets the maximum size for files produced by TWCS compaction. Only available when `compaction.type` is `twcs`. |
-| `memtable.type` | Type of the memtable. | String value, supports `time_series`, `partition_tree`. |
+| `memtable.type` | Type of the memtable | String value: `bulk` or `time_series`. If omitted, Mito selects the implementation from the SST format; the default flat format uses `bulk`. Setting `bulk` forces `sst_format=flat`, and flat SSTs use the bulk implementation even if `time_series` is specified. The legacy value `partition_tree` is accepted for compatibility and maps to the bulk and flat path. |
| `append_mode` | Whether the table is append-only | String value. Default is 'false', which removes duplicate rows by primary keys and timestamps according to the `merge_mode`. Setting it to 'true' to enable append mode and create an append-only table which keeps duplicate rows. |
| `merge_mode` | The strategy to merge duplicate rows | String value. Only available when `append_mode` is 'false'. Default is `last_row`, which keeps the last row for the same primary key and timestamp. Setting it to `last_non_null` to keep the last non-null field for the same primary key and timestamp. |
| `sst_format` | The format of SST files | String value, supports `primary_key`, `flat`. Default is `flat`. `flat` is recommended for tables which have a large number of unique primary keys. |
diff --git a/versioned_docs/version-1.2/user-guide/deployments-administration/configuration.md b/versioned_docs/version-1.2/user-guide/deployments-administration/configuration.md
index 1895b44b83..09cc360a7f 100644
--- a/versioned_docs/version-1.2/user-guide/deployments-administration/configuration.md
+++ b/versioned_docs/version-1.2/user-guide/deployments-administration/configuration.md
@@ -596,20 +596,9 @@ create_on_compaction = "auto"
apply_on_query = "auto"
mem_threshold_on_create = "64M"
intermediate_path = ""
-
-[region_engine.mito.memtable]
-type = "time_series"
```
-The `mito` engine provides an experimental memtable which optimizes for write performance and memory efficiency under large amounts of time-series. Its read performance might not as fast as the default `time_series` memtable.
-
-```toml
-[region_engine.mito.memtable]
-type = "partition_tree"
-index_max_keys_per_shard = 8192
-data_freeze_threshold = 32768
-fork_dictionary_bytes = "1GiB"
-```
+Mito selects the memtable implementation for each Region according to its table options and SST format. When `default_flat_format` is `true`, Regions without an explicit `sst_format` use flat SSTs and the bulk memtable. Configure `memtable.type` as a database or table option; `[region_engine.mito.memtable]` is not an engine setting. See [table options](/reference/sql/create.md#table-options).
Available options:
@@ -644,7 +633,7 @@ Available options:
| `scan_memory_on_exhausted` | String | `fail` | Behavior when scan memory is exhausted. Options: `fail` (fail fast), `wait` or `wait()` (wait for memory). |
| `min_compaction_interval` | String | `0m` | Minimum time interval between two compactions. Set to "0m" (default) to allow compactions to run immediately without restriction. |
| `schedule_compaction_after_edit` | Bool | `true` | Whether to allow scheduling a compaction after a successful region edit. Setting this to `true` is a necessary but not sufficient condition for scheduling compaction after a region edit. Other constraints, such as `min_compaction_interval`, may still prevent compaction from being scheduled. Setting this to `false` guarantees that compaction will not be scheduled after a region edit. |
-| `default_flat_format` | Bool | `true` | Whether to enable flat format as the default SST format. |
+| `default_flat_format` | Bool | `true` | Whether Regions without an explicit `sst_format` use flat SSTs. Flat SSTs use the bulk memtable. |
| `scan_parallelism` | Integer | `0` | (Deprecated, use `max_concurrent_scan_files` instead) Legacy option for scan parallelism. |
| `index` | -- | -- | The options for index in Mito engine. |
| `index.aux_path` | String | `""` | Auxiliary directory path for the index in the filesystem. This path is used to store intermediate files for creating the index and staging files for searching the index. It defaults to `{data_home}/index_intermediate`. The default name for this directory is `index_intermediate` for backward compatibility. This path contains two subdirectories: `__intm` for storing intermediate files used during index creation, and `staging` for storing staging files used during index searching. |
@@ -660,10 +649,6 @@ Available options:
| `inverted_index.apply_on_query` | String | `auto` | Whether to apply the index on query - `auto`: automatically - `disable`: never |
| `inverted_index.mem_threshold_on_create` | String | `64M` | Memory threshold for performing an external sort during index creation. Setting to empty will disable external sorting, forcing all sorting operations to happen in memory. |
| `inverted_index.intermediate_path` | String | `""` | File system path to store intermediate files for external sorting (default `{data_home}/index_intermediate`). |
-| `memtable.type` | String | `time_series` | Memtable type. - `time_series`: time-series memtable - `partition_tree`: partition tree memtable (experimental) |
-| `memtable.index_max_keys_per_shard` | Integer | `8192` | The max number of keys in one shard. Only available for `partition_tree` memtable. |
-| `memtable.data_freeze_threshold` | Integer | `32768` | The max rows of data inside the actively writing buffer in one shard. Only available for `partition_tree` memtable. |
-| `memtable.fork_dictionary_bytes` | String | `1GiB` | Max dictionary bytes. Only available for `partition_tree` memtable. |
The `metric` engine is optimized for handling metrics data with a large number of small tables.
diff --git a/versioned_sidebars/version-1.1-sidebars.json b/versioned_sidebars/version-1.1-sidebars.json
index f75b74f205..74da2398f0 100644
--- a/versioned_sidebars/version-1.1-sidebars.json
+++ b/versioned_sidebars/version-1.1-sidebars.json
@@ -732,6 +732,7 @@
"label": "Overview"
},
"contributor-guide/datanode/storage-engine",
+ "contributor-guide/datanode/memtable",
"contributor-guide/datanode/query-engine",
"contributor-guide/datanode/data-persistence-indexing",
"contributor-guide/datanode/wal",
diff --git a/versioned_sidebars/version-1.2-sidebars.json b/versioned_sidebars/version-1.2-sidebars.json
index 983fd9a25b..ecaeb12fdc 100644
--- a/versioned_sidebars/version-1.2-sidebars.json
+++ b/versioned_sidebars/version-1.2-sidebars.json
@@ -759,6 +759,7 @@
"label": "Overview"
},
"contributor-guide/datanode/storage-engine",
+ "contributor-guide/datanode/memtable",
"contributor-guide/datanode/query-engine",
"contributor-guide/datanode/data-persistence-indexing",
"contributor-guide/datanode/wal",
+
+
+An SST may span more than one compaction time window.
## Scan Pruning
diff --git a/docs/contributor-guide/datanode/wal.md b/docs/contributor-guide/datanode/wal.md
index 4ecb19ef02..8f898d91d6 100644
--- a/docs/contributor-guide/datanode/wal.md
+++ b/docs/contributor-guide/datanode/wal.md
@@ -7,30 +7,26 @@ description: Introduction to Write-Ahead Logging (WAL) in GreptimeDB, its purpos
## Introduction
-Our storage engine is inspired by the Log-structured Merge Tree (LSMT). Mutating operations are
-applied to a MemTable instead of persisting to disk, which significantly improves performance but
-also brings durability-related issues, especially when the Datanode crashes unexpectedly. Similar
-to all LSMT-like storage engines, GreptimeDB uses a write-ahead log (WAL) to ensure data durability
-and is safe from crashing.
+Mito buffers writes in [memtables](memtable.md) before flushing them to SST files. It first appends each Region's mutations to the write-ahead log (WAL), so data that has not reached an SST can be recovered.
-WAL is an append-only file group. All `INSERT` and `DELETE` operations are transformed into
-operation entries and then appended to WAL. Once operation entries are persisted to the underlying
-file, the operation can be further applied to MemTable.
+The WAL uses a common log-store abstraction with local raft-engine and remote Kafka providers.
-When the Datanode restarts, operation entries in WAL are replayed to reconstruct the correct
-in-memory state.
+## Write and Recovery Cycle
-
+The order of a normal write is:
+
+1. The Region worker assigns sequence numbers and a WAL entry ID.
+2. It appends the mutations to the WAL. If the append fails, the mutations are not applied to the memtable.
+3. After the append succeeds, Mito writes the mutations to the memtable and publishes the new committed sequence.
+4. A flush writes immutable SST files and persists a manifest edit containing the new files and `flushed_entry_id`.
+5. After the manifest edit is durable, WAL entries through `flushed_entry_id` are marked obsolete. The log store may reclaim them later.
+
+The manifest is the recovery boundary. On a normal reopen, Mito rebuilds the Region from the manifest and replays WAL entries starting at `flushed_entry_id + 1`. Region transitions may supply a later replay checkpoint, but they never replay entries before the persisted flush boundary.
## Namespace
-Namespace of WAL is used to separate entries from different tables (different regions). Append and
-read operations must provide a Namespace. Currently, region ID is used as the Namespace, because
-each region has a MemTable that needs to be reconstructed when Datanode restarts.
+WAL entries are isolated by Region, not by table. Each append and read identifies a Region namespace so one Region can be replayed or truncated independently. The local raft-engine provider uses the Region ID as its namespace ID. Kafka keeps Region identity within the provider's topic-backed log.
## Synchronous/Asynchronous flush
-By default, appending to WAL is asynchronous, which means the writer will not wait until entries are
-flushed to disk. This setting provides higher performance, but may lose data when running host shutdown unexpectedly. In the other hand, synchronous flush provides higher durability at the cost of performance.
-
-In v0.4 version, the new region worker architecture can use batching to alleviate the overhead of sync flush.
+For the local raft-engine provider, `sync_write` controls whether an append waits for the log to be synced to durable storage. It defaults to `false`. Asynchronous writes reduce latency but can lose recently acknowledged entries if the host fails before buffered data is synced. Kafka WAL durability is controlled by its producer and cluster settings instead of this local option.
diff --git a/docs/contributor-guide/flownode/arrangement.md b/docs/contributor-guide/flownode/arrangement.md
index aed75af777..8b472ea316 100644
--- a/docs/contributor-guide/flownode/arrangement.md
+++ b/docs/contributor-guide/flownode/arrangement.md
@@ -5,6 +5,8 @@ description: Details on the arrangement component in Flownode, which stores stat
# Arrangement
+This page describes state used by Flownode's legacy streaming mode. Batching mode does not use an Arrangement.
+
Arrangement stores the state in the dataflow's process. It stores the streams of update flows for further querying and updating.
The arrangement essentially stores key-value pairs with timestamps to mark their change time.
diff --git a/docs/contributor-guide/flownode/batching_mode.md b/docs/contributor-guide/flownode/batching_mode.md
index 37a695ce9c..aa8099d1ad 100644
--- a/docs/contributor-guide/flownode/batching_mode.md
+++ b/docs/contributor-guide/flownode/batching_mode.md
@@ -9,13 +9,13 @@ This guide provides a brief overview of the batching mode in `flownode`. It's in
## Overview
-The batching mode in `flownode` is designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. This is in contrast to the original streaming mode, now deprecated, where data was processed as it arrived.
+The batching mode in `flownode` is designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. This is in contrast to the legacy streaming path, which processes data as it arrives and is retained for compatibility but deprecated for new workloads.
The core idea is to:
1. Define a `flow` with a SQL query that aggregates data from a source table into a sink table.
2. The query typically includes a time window function (e.g., `date_bin`) on a timestamp column.
3. When new data is inserted into the source table, the system marks the corresponding time windows as "dirty."
-4. A background task periodically wakes up, identifies these dirty windows, and re-runs the aggregation query for those specific time ranges.
+4. A background task runs on its own cadence, consumes the pending dirty windows at its next evaluation, and re-runs the aggregation query for those time ranges.
5. The results are then inserted into the sink table, effectively updating the aggregated view.
## Architecture
@@ -39,15 +39,15 @@ A `BatchingTask` represents a single, independent data flow. Each task is associ
- **State (`TaskState`)**: This contains the dynamic, mutable state of the task, most importantly the `DirtyTimeWindows`.
- **Execution Loop**: The task runs an infinite loop (`start_executing_loop`) that:
1. Checks for a shutdown signal.
- 2. Waits for a scheduled interval or until it's woken up.
+ 2. Sleeps until its next evaluation time. A task with an evaluation schedule sleeps until the next scheduled time; an adaptive task sleeps for a polling interval derived from the time window size and the minimum refresh duration.
3. Generates a new query plan (`gen_insert_plan`) based on the current set of dirty time windows.
4. Executes the query (`execute_logical_plan`) against the database.
5. Cleans up the processed dirty windows.
### `TaskState` and `DirtyTimeWindows`
-- **`TaskState`**: This struct tracks the runtime state of a `BatchingTask`. It includes `dirty_time_windows`, which is crucial for determining what work needs to be done.
-- **`DirtyTimeWindows`**: This is a key data structure that keeps track of which time windows have received new data since the last query execution. It stores a set of non-overlapping time ranges. When a task's execution loop runs, it consults this structure to build a `WHERE` clause that filters the source table for only the dirty time windows.
+- **`TaskState`**: This struct tracks the runtime state of a `BatchingTask`, including the `dirty_time_windows` that determine its pending work.
+- **`DirtyTimeWindows`**: This data structure tracks which time windows have received new data since the last query execution. It stores a set of non-overlapping time ranges. The execution loop uses it to build a `WHERE` clause that selects only the dirty windows from the source table.
### `TimeWindowExpr`
@@ -56,15 +56,15 @@ The `TimeWindowExpr` is a helper utility for dealing with time window expression
- **Evaluation**: It can take a timestamp and evaluate the time window expression to determine the start and end of the window that the timestamp falls into.
- **Window Size**: It can also determine the size (duration) of the time window from the expression.
-This is essential for both marking windows as dirty and for generating the correct filter conditions when querying the source table.
+The same calculation is used to mark dirty windows and generate the source-table filters.
## Query Execution Flow
Here's a simplified step-by-step walkthrough of how a query is executed in batch mode:
1. **Data Ingestion**: New data is written to a source table.
-2. **Marking Dirty**: The `BatchingEngine` receives a notification about the new data. It uses the `TimeWindowExpr` associated with each relevant flow to determine which time windows are affected by the new data points. These windows are then added to the `DirtyTimeWindows` set in the corresponding `TaskState`.
-3. **Task Wake-up**: The `BatchingTask`'s execution loop wakes up, either due to its periodic schedule or because it was notified of a large backlog of dirty windows.
+2. **Marking Dirty**: The `BatchingEngine` receives a notification about the new data. It uses the `TimeWindowExpr` associated with each relevant flow to determine which time windows are affected by the new data points. These windows are then added to the `DirtyTimeWindows` set in the corresponding `TaskState`. Marking a window dirty does not wake the task.
+3. **Next Evaluation**: The `BatchingTask`'s execution loop reaches its next evaluation, either at a scheduled time or after its adaptive polling interval, and consumes the pending dirty windows.
4. **Plan Generation**: The task calls `gen_insert_plan`. This method:
- Inspects the `DirtyTimeWindows`.
- Generates a series of `OR`'d `WHERE` clauses (e.g., `(ts >= 't1' AND ts < 't2') OR (ts >= 't3' AND ts < 't4') ...`) that cover the dirty windows.
diff --git a/docs/contributor-guide/flownode/dataflow.md b/docs/contributor-guide/flownode/dataflow.md
index 000a65edb3..c876054d6d 100644
--- a/docs/contributor-guide/flownode/dataflow.md
+++ b/docs/contributor-guide/flownode/dataflow.md
@@ -1,17 +1,38 @@
---
-keywords: [dataflow module, SQL query transformation, execution plan, DAG, map and reduce operations]
-description: Explanation of the dataflow module in Flownode, its operations, internal data handling, and future enhancements.
+keywords: [Flownode, batching mode, streaming mode, dataflow, dirty time windows]
+description: How Flownode selects and runs its batching and legacy streaming execution paths.
---
# Dataflow
+Flownode has two internal execution paths:
+
+- **Batching mode** is the primary path for aggregation and TQL workloads. It evaluates queries over persisted source data and writes materialized results to a sink table.
+- **Streaming mode** is the legacy path retained for compatibility and deprecated for new workloads. It incrementally processes rows mirrored from Frontend as they arrive.
+
+Users do not select the mode directly. When a Flow is created, GreptimeDB chooses the path from the query and source-table properties. Aggregation, `DISTINCT`, and TQL queries use batching mode. Simple non-aggregation queries, and any Flow whose source table has `ttl = 'instant'`, currently use streaming mode. A Flow deferred because its source table does not yet exist starts as a pending batching Flow.
+
+## Batching mode
+
+Batching mode reuses GreptimeDB's query engine instead of maintaining an operator graph for every incoming row. For a time-windowed Flow, its main loop is:
+
+1. A source-table write marks the affected time windows as dirty.
+2. A `BatchingTask` runs on its evaluation schedule or adaptive polling cadence and collects the pending dirty windows at that evaluation. Marking a window dirty does not wake the task.
+3. The task adds time predicates for those windows to the Flow query and asks Frontend to execute it against the source tables.
+4. The query result is inserted into the sink table, updating the materialized result for windows that were evaluated.
+5. Successfully processed windows are removed from the dirty set. Failed work remains available for a later evaluation.
+
+Flows with an evaluation interval but without a time-window expression run the complete query on each scheduled evaluation. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Flownode Batching Mode Developer Guide](./batching_mode.md) for the task and dirty-window components.
+
+## Streaming mode
+
The `dataflow` module (see `flow::compute` module) is the core computing module of `flow`.
It takes a SQL query and transforms it into flow's internal execution plan.
This execution plan is then rendered into an actual dataflow, which is essentially a directed acyclic graph (DAG) of functions with input and output ports.
-The dataflow is triggered to run when needed.
+New row changes drive the graph incrementally.
-Currently, this dataflow only supports `map` and `reduce` operations. Support for `join` operations will be added in the future.
+The renderer supports map/filter/project and reduce operations. Join and union plan nodes exist, but their streaming renderers are not implemented.
Internally, the dataflow handles data in row format, using a tuple `(row, time, diff)`. Here, `row` represents the actual data being passed, which may contain multiple `Value` objects.
`time` is the system time which tracks the progress of the dataflow, and `diff` typically represents the insertion or deletion of the row (+1 or -1).
-Therefore, the tuple represents the insert/delete operation of the `row` at a given system `time`.
\ No newline at end of file
+Therefore, the tuple represents the insert/delete operation of the `row` at a given system `time`. Stateful operators keep indexed traces of these changes in an [Arrangement](./arrangement.md).
diff --git a/docs/contributor-guide/frontend/distributed-querying.md b/docs/contributor-guide/frontend/distributed-querying.md
index 21ee07d7e8..ca3822e113 100644
--- a/docs/contributor-guide/frontend/distributed-querying.md
+++ b/docs/contributor-guide/frontend/distributed-querying.md
@@ -5,29 +5,16 @@ description: Describes the process of distributed querying in GreptimeDB, focusi
# Distributed Querying
-Most steps of querying in frontend and datanode are identical. The only difference is that
-Frontend have a "special" step in planning phase to make the logical query plan distributed.
-Let's reference it as "dist planner" in the following text.
-
-The modified, distributed logical plan has multiple stages, each of them is executed in different
-server node.
+Frontend and Datanode use the same DataFusion-based query engine. In distributed mode, Frontend adds a planning step that separates work performed by Datanodes from work completed by Frontend.
## Dist Planner
-Planner will traverse the input logical plan, and split it into multiple stages by the "[commutativity
-rule](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md)".
+The distributed planner rewrites the logical plan. It pushes compatible operators toward table scans and wraps remote subplans in `MergeScan` nodes. Partition predicates are also used to prune Regions before the remote work is scheduled.
-This rule is under heavy development. At present it will consider things like:
-- whether the operator itself is commutative
-- how the partition rule is configured
-- etc...
+Whether an operator can be pushed down depends on the plan shape and the operator's properties. Unsupported parts remain on Frontend. The original design and its commutativity rules are described in the [distributed planner RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md).
## Dist Plan
-Except the first stage, which have to read data from files in storage. All other stages' leaf node
-are actually a gRPC call to its previous stage.
-
-Sub-plan in a stage is itself a complete logical plan, and can be executed independently without
-the follow up stages. The plan is encoded in [substrait format](https://substrait.io).
+A remote input is a complete logical subplan, not just a table scan. Frontend serializes the subplan in [Substrait](https://substrait.io) format and sends a Region-specific request to the Datanode that owns the data. The Datanode plans and executes it locally, then streams the result back. Frontend merges the remote streams and executes any operators that were not pushed down.
diff --git a/docs/contributor-guide/frontend/overview.md b/docs/contributor-guide/frontend/overview.md
index 3a2f63e7ae..18cabfc2b8 100644
--- a/docs/contributor-guide/frontend/overview.md
+++ b/docs/contributor-guide/frontend/overview.md
@@ -5,27 +5,47 @@ description: Overview of GreptimeDB's Frontend component - a stateless proxy ser
# Frontend
-The **Frontend** is a stateless service that serves as the entry point for client requests in GreptimeDB. It provides a unified interface for multiple database protocols and acts as a proxy that forwards read/write requests to appropriate Datanodes in the distributed system.
+Frontend is GreptimeDB's stateless request-orchestration service. The server layer terminates protocols and converts wire messages; Frontend supplies the database behavior behind those handlers, including permission checks, statement execution, routing, and distributed query planning.
+
+Frontend does not store table data. It caches catalog and route metadata obtained from Metasrv, and Metasrv invalidates those caches through heartbeat responses when metadata changes.
## Core Functions
-- **Protocol Support**: Multiple database protocols including SQL, PromQL, MySQL, and PostgreSQL. See [Protocols][1] for details
-- **Request Routing**: Routes requests to appropriate Datanodes based on metadata
-- **Query Distribution**: Splits distributed queries across multiple nodes
-- **Response Aggregation**: Combines results from multiple Datanodes
-- **Authorization**: Security and access control validation
+- Provide query and ingestion behavior for the supported [protocols][1].
+- Resolve catalogs, schemas, tables, and Region routes.
+- Validate permissions before executing a request.
+- Plan distributed queries and merge results from Datanodes.
+- Convert table-level writes and deletes into Region requests.
## Architecture
### Key Components
-- **Protocol Handlers**: Handle different database protocols
-- **Catalog Manager**: Caches metadata from Metasrv to enable efficient request routing and schema validation
-- **Dist Planner**: Converts logical plans to distributed execution plans
-- **Request Router**: Determines target Datanodes for each request
+
+- Protocol handlers adapt SQL, PromQL, gRPC ingestion, and observability protocols to Frontend's internal request interfaces.
+- The catalog and partition managers provide table metadata, partition rules, and Region routes.
+- The statement executor dispatches queries, DML, and DDL to their respective execution paths.
+- The distributed planner replaces table scans with `MergeScan` plans that can run across Datanodes.
### Request Flow
-
+The request path depends on the operation.
+
+#### Queries
+
+1. A protocol handler creates the query context and performs authentication and permission checks.
+2. The language-specific planner produces a logical plan. In distributed mode, the planner uses partition metadata to select Regions and constructs a distributed plan.
+3. Frontend sends Region subplans to the owning Datanodes. Datanodes execute them against local Region engines and return streams of Arrow record batches.
+4. Frontend runs the remaining operators, merges the streams, and formats the result for the client protocol.
+
+#### Writes and deletes
+
+1. Frontend validates the request against the table schema. Protocols that support schema-on-write may create a missing table or add columns before retrying the write.
+2. The partition rule assigns rows to Regions. Frontend builds one Region request per target and routes it to the current Region leader.
+3. The Datanode's Region server dispatches each request to the Region engine. In standalone mode, the same request is sent to an embedded Region server instead of over RPC.
+
+#### DDL
+
+The statement executor converts DDL into a task. In distributed mode, Metasrv runs that task as a persisted procedure, updates metadata, and coordinates Region operations on Datanodes. Standalone mode uses the same statement boundary with local implementations of the metadata and procedure services.
### Deployment
diff --git a/docs/contributor-guide/frontend/table-sharding.md b/docs/contributor-guide/frontend/table-sharding.md
index a60276d14e..beaece40c0 100644
--- a/docs/contributor-guide/frontend/table-sharding.md
+++ b/docs/contributor-guide/frontend/table-sharding.md
@@ -5,21 +5,15 @@ description: Explains how table data in GreptimeDB is sharded and distributed, i
# Table Sharding
-The sharding of stored data is essential to any distributed database. This document will describe how table's data in GreptimeDB is being sharded, and distributed.
+GreptimeDB shards a table into Regions. Partition expressions define which rows belong to each Region, while Region routes define which Datanode currently owns each Region.
## Partition
-For the syntax of creating a partitioned table, please refer to the [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) section in the User Guide.
+A partition is a logical row set described by an expression over one or more columns. The partition layout must cover the table's input domain so each row has one target Region. See [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) for the SQL syntax and supported expressions.
## Region
-The data within a table is logically split after creating partitions. You may ask the question "
-how are the data, after being logically partitioned, stored in the GreptimeDB? The answer is in "`Region`"s.
-
-Each region is corresponding to a partition, and stores the data in the partition. The regions are distributed among
-`Datanode`s. `Metasrv` manages the route information that maps regions to Datanodes.
-If the partition layout needs to change after table creation, GreptimeDB supports explicit
-[repartitioning](/user-guide/deployments-administration/manage-data/repartition.md) through split and merge operations.
+Each partition maps to one Region. Region IDs remain the storage and routing identity used by Frontend, Datanode, and Metasrv. Multiple Regions from the same table may be placed on one Datanode.
The relationship between partition and region can be viewed as the following diagram:
@@ -53,3 +47,14 @@ The relationship between partition and region can be viewed as the following dia
│ │
└──────────────────────────────────┘
Could be placed in one Datanode
+```
+
+## Routing and Pruning
+
+For writes, Frontend evaluates the partition rule for each row, groups rows by Region, and sends Region requests to the current leaders from the route table.
+
+For queries, the distributed planner compares query predicates with the partition expressions. It scans only Regions that can satisfy the predicates. If partition metadata is missing or cannot be interpreted safely, the planner falls back to all Regions rather than risk omitting data.
+
+## Changing the Partition Layout
+
+[Repartitioning](/user-guide/deployments-administration/manage-data/repartition.md) changes an existing layout through explicit split and merge operations. Metasrv runs the change as a persisted procedure, updates the Region routes and partition expressions, and invalidates stale table-route caches. New requests use the published layout after their Frontend refreshes that metadata.
diff --git a/docs/contributor-guide/getting-started.md b/docs/contributor-guide/getting-started.md
index b17184dfa8..7e8cca3ad8 100644
--- a/docs/contributor-guide/getting-started.md
+++ b/docs/contributor-guide/getting-started.md
@@ -15,14 +15,13 @@ At the moment, GreptimeDB supports Linux (both amd64 and arm64), macOS (both amd
### Build Dependencies
-- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) (optional)
+- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) (optional; needed to clone the repository, not to build it)
- C/C++ Toolchain: provides essential tools for compiling and linking. This is available either as `build-essential` on ubuntu or a similar name on other platforms.
-- Rust nightly toolchain ([guide][1])
- - Compile the source code
+- [Rustup][1]. The repository pins the required nightly toolchain in `rust-toolchain.toml`.
- Protobuf ([guide][2])
- Compile the proto file
- Note that the version needs to be >= 3.15. You can check it with `protoc --version`
-- Machine: Recommended memory is 16GB or more, or use the [mold](https://github.com/rui314/mold) tool to reduce memory usage during linking.
+- Machine: 16GB of memory or more is recommended. On a smaller machine, use [mold](https://github.com/rui314/mold) to reduce memory usage during linking.
[1]: 
+下面这张来自 Apache Parquet 规范的图进一步展示了物理文件布局：column chunk 按 row group 写入，文件元数据及其长度则保存在 footer 中。
+
+
+
+*来源：Apache Parquet [FileLayout.gif](https://github.com/apache/parquet-format/blob/master/doc/images/FileLayout.gif)。Copyright 2014 The Apache Software Foundation，依据 [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) 使用。*
## 数据持久化
@@ -26,18 +30,18 @@ GreptimeDB 提供了 `region_engine.mito.global_write_buffer_size` 的配置项
## SST 文件中的索引数据
-Apache Parquet 文件格式在列块和数据页的头部提供了内置的统计信息，用于剪枝和跳过。
+Parquet 在每个 column chunk 的元数据中保存 row group 级列统计信息，例如最小值、最大值和 null 数量。Page 元数据和可选的 column index 可以提供粒度更细的统计信息。
-
+
-例如，在上述 Parquet 文件中，如果你想要过滤 `name` 等于 `Emily` 的行，你可以轻松跳过行组 0，因为 `name` 字段的最大值是 `Charlie`。这些统计信息减少了 IO 操作。
+例如，查询 `name` 等于 `Emily` 的行时，可以跳过 row group 0，因为其中 `name` 的最大值是 `Charlie`，无需读取该 row group。
## 索引文件
-对于每个 SST 文件，GreptimeDB 不但维护 SST 文件内部索引，还会单独生成一个文件用于存储针对该 SST 文件的索引结构。
+当一个 SST 存在已配置且适用的索引输出时，GreptimeDB 将这些索引写入与该 SST 关联的 Puffin 文件。没有适用索引的 SST 不需要生成 Puffin 文件。
-索引文件采用 [Puffin][3] 格式，这种格式具有较大的灵活性，能够存储更多的元数据，并支持更多的索引结构。
+Puffin 是索引 Blob 及其元数据的容器，使不同索引结构可以共用一个文件。
@@ -58,13 +62,13 @@ GreptimeDB 会将多种索引结构作为 Blob 存储在 Puffin 文件中，包
-例如，上述查询使用倒排索引来定位数据段，数据段满足条件：`job` 等于 `apiserver`，`handler` 符合正则匹配 `.*users` 及 `status` 符合正则匹配 `4..`，然后扫描这些数据段以产生满足所有条件的最终结果，从而显着减少 IO 操作的次数。
+上述查询使用倒排索引定位 `job` 等于 `apiserver`、`handler` 匹配 `.*users` 且 `status` 匹配 `4..` 的数据段。Mito 只扫描这些数据段，再应用剩余过滤条件。
### 倒排索引格式
-
+
-GreptimeDB 按列构建倒排索引，每个倒排索引包含一个 FST 和多个 Bitmap。
+GreptimeDB 按列构建倒排索引。每个列索引包含一个 null bitmap、多个 posting bitmap 和一个 FST。Blob footer 记录定位和解码各列索引所需的 offset、size 和元数据。
FST（Finite State Transducer）允许 GreptimeDB 以紧凑的格式存储列值到 Bitmap 位置的映射，并且提供了优秀的搜索性能和支持复杂搜索（例如正则表达式匹配）；Bitmap 则维护了数据段 ID 列表，每个位表示一个数据段。
@@ -80,7 +84,7 @@ GreptimeDB 把一个 SST 文件分割成多个索引数据段，每个数据段
## 统一数据访问层：OpenDAL
-GreptimeDB 使用 [OpenDAL][2] 提供统一的数据访问层，因此，存储引擎无需与不同的存储 API 交互，数据可以无缝迁移到基于云的存储，如 AWS S3。
+GreptimeDB 使用 [OpenDAL][2] 为本地文件系统和对象存储提供统一访问层。修改配置的存储 backend 不会迁移已有数据。
[1]: https://parquet.apache.org
[2]: https://github.com/datafuselabs/opendal
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md
new file mode 100644
index 0000000000..3905a1aecd
--- /dev/null
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md
@@ -0,0 +1,100 @@
+---
+keywords: [memtable, Mito engine, 写缓冲, flush, 时间分区, BulkMemtable]
+description: 介绍 Mito 如何在 memtable 中组织 Region 的可变数据，以及如何将这些数据写入 SST 文件。
+---
+
+# Memtable 设计
+
+Memtable 是 Mito 为每个 Region 维护的内存写缓冲。数据 flush 为 SST 文件前，读取可以先从 memtable 获取这些数据。Region version 确定 scan 可以读取的 memtable 和 SST 文件；配合 committed sequence 上限，scan 可以在写入和 flush 推进当前 version 时保持一致性。
+
+## 写入和 flush 生命周期
+
+对于使用 WAL 的常规写入，Mito 按以下顺序处理：
+
+```text
+写请求
+ |
+ v
+追加 WAL -> mutable memtable -> 发布 committed sequence
+ |
+ freeze
+ v
+ immutable memtable -> 写入 SST -> manifest edit
+```
+
+Region worker 先分配 sequence number 和 WAL entry ID，再将 mutation 追加到[预写日志](wal.md)。如果追加失败，Mito 不会更新 memtable。Memtable 更新成功后，Mito 发布 committed sequence，新的读取随后可以看到这些数据。配置了 `skip_wal` 的 Region 会跳过 WAL，但 memtable 更新和可见性顺序不变。
+
+Flush 在启动后台 SST 写入前，先冻结 mutable memtable 并安装一组新的 mutable memtable。后续写入可以继续进行，也不会修改已经冻结的数据。Flush 将 immutable memtable 写为 SST 文件，再持久化包含新文件、flushed WAL checkpoint 和 sequence checkpoint 的 manifest edit。只有 manifest edit 持久化成功后，Mito 才会从当前 Region version 中移除已 flush 的 memtable。Flush 失败时，immutable memtable 会保留，供后续任务重试。
+
+## Region version 和时间分区
+
+每个 Region 包含一个 mutable `TimePartitions` 容器，其中可以有多个 memtable：
+
+```text
+Region version
+├─ mutable TimePartitions
+│ ├─ [t0, t1) -> memtable
+│ └─ [t1, t2) -> memtable
+├─ immutable memtables
+└─ SST files
+```
+
+Mito 根据 time index 的值把每行数据路由到对应分区。分区使用左闭右开的时间范围，并按照固定时长对齐。该时长跟随 Region 的 compaction time window；在取得 compaction time window 前，Mito 使用一天作为初始值。乱序写入可能在最新分区之外创建更早的分区。
+
+冻结 Region 时，Mito 会同时冻结所有 mutable 时间分区，把其中的 memtable 移入 immutable 列表，再创建新的 `TimePartitions` 容器。Flush 失败可能留下多代 immutable memtable，因此读取和后续 flush 不能假定列表中只有一个对象。
+
+## Memtable 实现
+
+Mito 根据 Region 的 SST format、primary key encoding 和 memtable 选项选择实现：
+
+```text
+flat SST format（默认）或 sparse primary-key encoding -> BulkMemtable
+memtable.type=bulk -> BulkMemtable，并强制 flat SST
+primary_key SST + dense encoding（遗留） -> 遗留实现
+```
+
+使用默认 engine 配置时，没有显式指定 SST format 的 Region 会使用 `flat`，因此通常走 `BulkMemtable` 路径，本页其余内容也以它为准。这些规则用于排除不兼容的组合：flat format 或 sparse primary key encoding 必须使用 `BulkMemtable`；显式选择 bulk 实现则会强制使用 flat format。
+
+### BulkMemtable
+
+`BulkMemtable` 使用 flat Arrow 布局把写入保存为 part，而不是将数据行插入按时间序列组织的缓冲区：
+
+```text
+BulkMemtable
+├─ unordered_part
+│ └─ 小批量 BulkPart
+└─ parts
+ ├─ BulkPart (Arrow RecordBatch)
+ ├─ MultiBulkPart (未编码的 RecordBatch)
+ └─ EncodedBulkPart (内存中的 Parquet 数据)
+```
+
+小 part 先积累在 `unordered_part` 中，较大的 part 则直接进入 `parts`。后台 memtable compaction 对符合条件的 part 执行 merge sort，生成 `MultiBulkPart` 或编码为 `EncodedBulkPart`。Scan 利用 part 的统计信息裁剪 range；flush 可以将已编码的 range 写入 SST，无需再次解码和编码数据行。设计动机和性能数据见 [Scaling Time Series to Millions of Cardinalities: GreptimeDB's Flat Format](https://greptime.cn/blogs/2025-12-22-flat-format)。
+
+### 遗留实现
+
+使用遗留 `primary_key` SST format 且为 dense primary key encoding 的 Region 仍然使用 `TimeSeriesMemtable`，它按编码后的 primary key 对数据行分组，而不是保存 flat part。如果 Region 没有 primary key 列，同一个 builder 会创建 `SimpleBulkMemtable`。两者都是为已有表保留的兼容代码，`primary_key` format 退役后可能一并移除；新的工作应面向 bulk 和 flat 路径。
+
+已经删除的 `partition_tree` memtable 不是第三种实现。Option parser 仍接受 `memtable.type=partition_tree` 以兼容旧配置，但不会恢复该实现。Region 最终使用 bulk 和 flat 路径。
+
+## 读取快照
+
+Scan 通过一次 `VersionControl` 快照同时取得 Region version 和 committed sequence，并先确定 version，再应用 sequence 上限。如果单独读取 sequence 后再获取 version，flush 或 compaction 可能在两次读取之间移除旧输入，使 scan 得到不完整的快照。
+
+选定的 version 提供 mutable memtable、immutable memtable 和 SST 文件。Mito 先按照时间范围裁剪数据源，再使用 scan 的 projection、predicate 和 sequence range 从各 memtable 获取 range。Scan 将这些 range 与 SST range 合并，并在所有数据源上应用相同的排序、删除和 merge 语义。即使新的 Region version 已经移除某个 memtable，scan 持有的引用也会让该 memtable 存活到本次读取结束。
+
+## 内存压力
+
+每个 memtable 通过 engine 的 write-buffer manager 记录估算的堆内存分配量。冻结 memtable 后，这部分内存不再计入 mutable memory，但在所有引用释放前仍计入总用量。因此，mutable memory 只反映仍可接收写入的数据，总用量仍包含活跃 scan 保留的内存。
+
+全局 write-buffer 达到限制后，worker 会选择 Region 执行 flush。如果内存用量持续超过配置限制，Mito 会阻塞写入，并在达到更高阈值后拒绝写入。可选的 Region 级限制会单独约束热点 Region，避免其阻塞无关 Region。定期任务、手动请求和 Region 生命周期操作也可以触发 flush。
+
+## 修改约束
+
+修改 memtable 代码时必须保持以下性质：
+
+- 对于使用 WAL 的 Region，先追加 WAL，再把数据安装到 memtable；只有安装成功后才能发布 committed sequence。
+- SST 文件和 manifest edit 持久化前，冻结的 memtable 必须保持可读，并能在 flush 失败后重试。
+- 从同一个 `VersionControl` 快照取得 Region version 和 committed sequence，不得先单独读取 sequence 再获取 version。
+- 保留 scan 和 flush 在 memtable range 与 SST range 之间执行统一删除、去重和 merge 所需的排序及元数据。
+- 通过 write-buffer manager 记录内存分配，并且只在底层内存不再可能被引用时释放计数。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md
index c0cbf681b9..57baecf400 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md
@@ -7,9 +7,9 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述
## 概述
-`Metric` 引擎是 GreptimeDB 的一个组件，属于存储引擎的一种实现，主要针对可观测 metrics 等存在大量小表的场景。
+`Metric` 引擎用于存储包含大量小型指标表的负载。
-它的主要特点是利用合成的物理宽表来存储大量的小表数据，实现相同列复用和元数据复用等效果，从而达到减少小表的存储开销以及提高列式压缩效率等目标。表这一概念在 `Metric` 引擎下变得更更加轻量。
+它将这些逻辑表映射到共享的物理宽表，使其复用列和元数据，从而降低每张表的存储开销并改善列式压缩。
## 概念
@@ -18,7 +18,7 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述
### 逻辑表
逻辑表，即用户定义的表。与普通的表都完全一样，逻辑表的定义包括表的名称、列的定义、索引的定义等。用户的查询、写入等操作都是基于逻辑表进行的。用户在使用过程中不需要关心逻辑表和普通表的区别。
-从实现层面来说，逻辑表是一个虚拟的表，它并不直接读写物理的数据，而是通过将读写请求映射成对应物理表的请求来实现数据的存储与查询。
+逻辑表是虚拟表，本身不直接存储数据。Metric 引擎将其读写请求映射为对应物理表的请求。
### 物理表
物理表是真实存储数据的表，它拥有若干个由分区规则定义的物理 Region。
@@ -27,16 +27,14 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述
`Metric` 引擎的主要设计架构如下：
-
+
-在目前版本的实现中，`Metric` 引擎复用了 `Mito` 引擎来实现物理数据的存储及查询能力，并在此之上同时提供物理表与逻辑表的访问能力。
+`Metric` 引擎将物理存储和查询交给 `Mito` 引擎。每个物理 Region 组包含一个数据 Region 和一个元数据 Region：数据 Region 保存映射到该 Region 组的逻辑表数据，元数据 Region 保存逻辑表及逻辑列的映射。
-在分区方面，逻辑表拥有与物理表完全一致的分区规则及 Region 分布。这是非常自然的，因为逻辑表的数据直接存储在物理表中，所以分区规则也是一致的。
+关联到同一物理表的逻辑表使用相同的分区布局。写入时，Metric 引擎为每行数据记录逻辑表身份；读取时，它在扫描物理 Region 前增加逻辑表过滤条件。
-在路由元数据方面，逻辑表的路由地址为逻辑地址，即该逻辑表所对应的物理表是什么，而后通过该物理表进行二次路由取得真正的物理地址。这一间接路由方式能够显著减少 `Metric` 引擎的 Region 发生迁移调度时所需要修改的元数据数量。
+逻辑表的路由只保存所属物理表的 ID，再由物理表路由解析出持有 Region 的 Datanode。逻辑路由本身不记录 peer，因此迁移物理 Region 只需改写一条物理路由，而不必改写映射到它的每一条逻辑路由。
-在操作方面，`Metric` 引擎支持对逻辑表进行标准的 DML 操作（INSERT、DELETE、SELECT）。然而，对物理表的操作进行了有限的支持以防止误操作，例如禁止直接写入物理表等操作防止影响用户逻辑表的数据。总体上可以认为物理表是对用户只读的。
+逻辑表支持普通的 INSERT、DELETE 和 SELECT 操作。直接写入物理 Region 会绕过逻辑表映射，因此会被拒绝；物理表仍然可以查询。
-为了提升对大量表同时进行 DDL（Data Definition Language，数据操作语言）操作时性能，如 Prometheus Remote Write 冷启动时大量 metrics 带来的自动建表请求，以及前面提到的迁移物理 Region 时大量路由表的修改请求等，`Metric` 引擎引入了一些批量 DDL 操作。这些批量 DDL 操作能够将大量的 DDL 操作合并成一个请求，从而减少了元数据的查询及修改次数，提升了性能。
-
-除了物理表的物理数据 Region 之外，`Metric` 引擎还额外为每一个物理数据 Region 创建了一个物理的元数据 Region，用于存储 `Metric` 引擎自身为了维护映射等状态所需要的一些元数据。这些元数据包括逻辑表与物理表的映射关系，逻辑列与物理列的映射关系等等。
+批量 DDL 用于减少大量逻辑表同时创建或更新时的元数据操作，例如 Prometheus Remote Write 自动建表或物理 Region 迁移。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md
index 3f44a4543b..c7684eb165 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md
@@ -7,22 +7,26 @@ description: 介绍了 Datanode 的主要职责和组件，包括 gRPC 服务、
## Introduction
-`Datanode` 主要的职责是为 GreptimeDB 存储数据，我们知道在 GreptimeDB 中一个 `table` 可以有一个或者多个 `Region`,
-而 `Datanode` 的职责便是管理这些 `Region` 的读写。`Datanode` 不感知 `table`，可以认为它是一个 `region server`。
-所以 `Frontend` 和 `Metasrv` 按照 `Region` 粒度来操作 `Datanode`。
+Datanode 存储并处理 Region 数据。一张表可以包含多个 Region，但 Datanode 不负责表级路由。Frontend 按 Region 发送数据请求，Metasrv 则控制 Region 的放置和生命周期。
-
+这个边界使同一个 Region server 可以承载不同的存储引擎，而不向 Frontend 或 Metasrv 暴露引擎实现。
+
+
## Components
-一个 datanode 包含了 region server 所需的全部组件。这里列出了比较重要的部分：
-
-- 一个 gRPC 服务来提供对 `Region` 数据的读写，`Frontend` 便是使用这个服务来从 `Datanode` 读写数据。
-- 一个 HTTP 服务，可以通过它来获得当前节点的 metrics、配置信息等
-- `Heartbeat Task` 用来向 `Metasrv` 发送心跳，心跳在 GreptimeDB 的分布式架构中发挥着至关重要的作用，
- 是分布式协调和调度的基础通信通道，心跳的上行消息中包含了重要信息比如 `Region` 的负载，如果 `Metasrv` 做出了调度
- 决定（比如 Region 转移），它会通过心跳的下行消息发送指令到 `Datanode`
-- `Datanode` 不负责解析用户 SQL 或进行分布式规划，用户对一个或多个 `Table` 的查询请求会在 `Frontend` 中被转换为
- `Region` 查询请求，`Datanode` 负责用本地 query engine 执行这些 `Region` 查询计划
-- 一个 `Region Manager` 用来管理 `Datanode` 上的所有 `Region`s
-- GreptimeDB 支持可插拔的多引擎架构，目前已有的 engine 包括 `File Engine` 和 `Mito Engine`
+Datanode 包含以下主要组件：
+
+- Region server 记录已打开的 Region，并把读写和生命周期请求分发给该 Region 注册的 engine。
+- `Mito` 是主要的时序 Region engine。`Metric` 将多个逻辑指标 Region 映射到共享的 Mito Region，`File` 通过 Region 接口访问外部文件。
+- 本地 query engine 执行 Region 查询计划。它不解析客户端 SQL，也不进行集群级规划。
+- Heartbeat task 向 Metasrv 上报节点和 Region 状态，并接收 open、close、upgrade、downgrade 和迁移步骤等指令。
+- gRPC 承载发往 Datanode 的 Region 请求；HTTP 提供 metrics 和配置等节点诊断信息。
+
+## Region 请求生命周期
+
+Mito 写入到达 Region server 后，Region server 根据 Region 元数据选择 Mito。Mito 将 mutation 追加到 WAL，写入 memtable，并在之后把 memtable flush 为 SST 文件。Metric 写入会先补充逻辑表标识，再委托给对应的物理 Mito Region。
+
+读取时，本地 query engine 在 Region engine 提供的 table provider 上执行 Region 计划。Mito scan 获取不可变的 Region version，读取相关 memtable 和 SST 文件，合并并去重数据，最后返回 Arrow record batch 流。
+
+Region 所有权可以在不重启 Datanode 的情况下改变。Metasrv 通过心跳流下发生命周期指令；Region server 将指令应用到对应 engine，并在后续心跳中上报新的 Region role 和统计信息。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/python-scripts.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/python-scripts.md
deleted file mode 100644
index 831278e80d..0000000000
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/python-scripts.md
+++ /dev/null
@@ -1,30 +0,0 @@
----
-keywords: [Python 脚本, 数据分析, CPython, RustPython]
-description: 介绍了在 GreptimeDB 中使用 Python 脚本进行数据分析的两种后端实现：CPython 和嵌入式 RustPython 解释器。
----
-
-# Python 脚本
-
-## 简介
-
-Python 脚本是分析本地数据库中的数据的便捷方式，
-通过将脚本直接在数据库内运行而不是从数据库拉取数据的方式，可以节省大量的数据传输时间。
-下图描述了 Python 脚本的工作原理。
-`RecordBatch`（基本上是表中的一列，带有类型和元数据）可以来自数据库中的任何地方，
-而返回的 `RecordBatch` 可以用 Python 语法注释以指示其元数据，例如类型或空。
-脚本将尽其所能将返回的对象转换为 `RecordBatch`，无论它是 Python 列表、从参数计算出的 `RecordBatch` 还是常量（它被扩展到与输入参数相同的长度）。
-
-
-
-## 两种可选的后端
-
-### CPython 后端
-
-该后端由 [PyO3](https://pyo3.rs/v0.18.1/) 提供支持，可以使用您最喜欢的 Python 库（如 NumPy、Pandas 等），并允许 Conda 管理您的 Python 环境。
-
-但是使用它也涉及一些复杂性。您必须设置正确的 Python 共享库，这可能有点棘手。一般来说，您只需要安装 `python-dev` 包。但是，如果您使用 Homebrew 在 macOS 上安装 Python，则必须创建一个适当的软链接到 `Library/Frameworks/Python.framework`。有关使用 PyO3 crate 与不同 Python 版本的详细说明，请参见 [这里](https://pyo3.rs/v0.18.1/building_and_distribution#configuring-the-python-version)
-
-### 嵌入式 RustPython 解释器
-
-可以运行脚本的实验性 [python 解释器](https://github.com/RustPython/RustPython)，它支持 Python 3.10 语法。您可以使用所有的 Python 语法，更多信息请参见 [Python 脚本的用户指南](/user-guide/python-scripts/overview.md).
-
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/query-engine.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/query-engine.md
index 62d7563bb2..84445759d4 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/query-engine.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/query-engine.md
@@ -7,33 +7,33 @@ description: 介绍了 GreptimeDB 的查询引擎架构，基于 Apache DataFusi
## 介绍
-GreptimeDB 的查询引擎是基于[Apache DataFusion][1]（属于[Apache Arrow][2]的子项目）构建的，它是一个用 Rust 编写的出色的查询引擎。它提供了一整套功能齐全的组件，从逻辑计划、物理计划到执行运行时。下面将解释每个组件如何被整合在一起，以及在执行过程中它们的位置。
+GreptimeDB 的查询引擎基于 [Apache DataFusion][1]。DataFusion 提供逻辑计划、物理计划、优化器框架和执行运行时；GreptimeDB 在此基础上增加各查询语言的 planner、存储相关优化规则、自定义计划节点和分布式执行。
-
+DDL 和其他控制面操作由 statement executor 分发。Query engine 接收数据处理计划，包括 `INSERT ... SELECT` 等操作中读取输入数据的部分。
-入口点是逻辑计划，它被用作查询或执行逻辑等的通用中间表示。逻辑计划的两个主要来源是：1. 用户查询，例如通过 SQL 解析器和规划器的 SQL；2. Frontend 的分布式查询，这将在下一节中详细解释。
+## 查询生命周期
-接下来是物理计划，或称执行计划。与包含所有逻辑计划变体（除特殊扩展计划节点外）的大型枚举的逻辑计划不同，物理计划实际上是一个定义了在执行过程中调用的一组方法的特性。所有数据处理逻辑都包装在实现该特性的相应结构中。它们是对数据执行的实际操作，如聚合器 `MIN` 或 `AVG` ，以及表扫描 `SELECT ... FROM`。
+1. SQL、PromQL 或日志查询 planner 通过 catalog 解析表，并生成 DataFusion logical plan。DataFusion 不直接支持的操作由 GreptimeDB plan extension 表示。
+2. DataFusion 的 analyzer 和 optimizer rule 与 GreptimeDB rule 共同运行。这些规则规范化表达式和类型、改写时间范围操作、将 projection 和 filter 下推到 scan，并在需要时引入分布式计划节点。
+3. Physical planner 将优化后的 logical plan 转换为流式 operator。GreptimeDB 随后应用 scan 并行度、排序和分布式执行相关的 physical rule。
+4. 执行阶段通过 physical plan 拉取 Arrow record batch。存储 scan 接收 projection 和 predicate，下游 operator 消费数据流，无需先物化完整结果。
-优化阶段通过转换逻辑计划和物理计划来提高执行性能，现在全部基于规则。它也被称为“基于规则的优化”。一些规则是 DataFusion 原生的，其他一些是在 GreptimeDB 中自定义的。在未来，我们计划添加更多规则，并利用数据统计进行基于成本的优化 (CBO)。
-
-最后一个阶段"执行"是一个动词，代表从存储读取数据、进行计算并生成预期结果的过程。虽然它比之前提到的概念更抽象，但你可以简单地将它想象为执行一个 Rust 异步函数，并且它确实是一个异步流。
-
-当你想知道 SQL 是如何通过逻辑计划或物理计划中表示时，`EXPLAIN [VERBOSE]