From 0703ef0f9987a0402c33e5815204e713f85b084a Mon Sep 17 00:00:00 2001 From: Dennis Zhuang Date: Tue, 25 Aug 2026 10:43:35 +0800 Subject: [PATCH 01/14] docs: refresh Nightly contributor guide --- .../datanode/data-persistence-indexing.md | 54 ++--- .../datanode/metric-engine.md | 35 ++-- docs/contributor-guide/datanode/overview.md | 33 +-- .../datanode/python-scripts.md | 35 ---- .../datanode/query-engine.md | 46 ++--- .../datanode/storage-engine.md | 79 ++----- docs/contributor-guide/datanode/wal.md | 28 +-- .../contributor-guide/flownode/arrangement.md | 18 +- .../flownode/batching_mode.md | 66 ++---- docs/contributor-guide/flownode/dataflow.md | 17 +- docs/contributor-guide/flownode/overview.md | 26 +-- .../frontend/distributed-querying.md | 27 +-- docs/contributor-guide/frontend/overview.md | 50 ++--- .../frontend/table-sharding.md | 12 +- docs/contributor-guide/getting-started.md | 51 ++--- .../how-to/how-to-trace-greptimedb.md | 88 ++++---- .../how-to/how-to-use-tokio-console.md | 26 +-- .../how-to/how-to-write-sdk.md | 47 ++--- docs/contributor-guide/metasrv/admin-api.md | 146 ++----------- docs/contributor-guide/metasrv/overview.md | 193 +++++------------- docs/contributor-guide/metasrv/selector.md | 33 ++- docs/contributor-guide/overview.md | 24 +-- .../tests/integration-test.md | 29 ++- docs/contributor-guide/tests/overview.md | 12 +- docs/contributor-guide/tests/sqlness-test.md | 36 ++-- docs/contributor-guide/tests/unit-test.md | 26 +-- .../datanode/data-persistence-indexing.md | 52 ++--- .../datanode/metric-engine.md | 35 ++-- .../contributor-guide/datanode/overview.md | 33 ++- .../datanode/python-scripts.md | 30 --- .../datanode/query-engine.md | 30 ++- .../datanode/storage-engine.md | 82 ++------ .../current/contributor-guide/datanode/wal.md | 18 +- .../contributor-guide/flownode/arrangement.md | 18 +- .../flownode/batching_mode.md | 66 ++---- .../contributor-guide/flownode/dataflow.md | 18 +- .../contributor-guide/flownode/overview.md | 27 +-- .../frontend/distributed-querying.md | 39 +--- .../contributor-guide/frontend/overview.md | 49 ++--- .../frontend/table-sharding.md | 9 +- .../contributor-guide/getting-started.md | 45 ++-- .../how-to/how-to-trace-greptimedb.md | 62 +++--- .../how-to/how-to-use-tokio-console.md | 26 +-- .../how-to/how-to-write-sdk.md | 43 ++-- .../contributor-guide/metasrv/admin-api.md | 140 ++----------- .../contributor-guide/metasrv/overview.md | 170 ++++----------- .../contributor-guide/metasrv/selector.md | 33 ++- .../current/contributor-guide/overview.md | 23 +-- .../tests/integration-test.md | 28 ++- .../contributor-guide/tests/overview.md | 11 +- .../contributor-guide/tests/sqlness-test.md | 30 ++- .../contributor-guide/tests/unit-test.md | 22 +- 52 files changed, 803 insertions(+), 1573 deletions(-) delete mode 100644 docs/contributor-guide/datanode/python-scripts.md delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/python-scripts.md diff --git a/docs/contributor-guide/datanode/data-persistence-indexing.md b/docs/contributor-guide/datanode/data-persistence-indexing.md index 6403ffc2df..61b7c2a4bb 100644 --- a/docs/contributor-guide/datanode/data-persistence-indexing.md +++ b/docs/contributor-guide/datanode/data-persistence-indexing.md @@ -5,80 +5,62 @@ description: Explanation of data persistence and indexing in GreptimeDB, includi # Data Persistence and Indexing -Similar to all LSMT-like storage engines, data in MemTables is persisted to durable storage, for example, the local disk file system or object storage service. GreptimeDB adopts [Apache Parquet][1] as its persistent file format. +Mito flushes data from memtables to durable local filesystems or object storage. SST files use [Apache Parquet][1] as their data format. ## SST File Format -Parquet is an open source columnar format that provides fast data querying and has already been adopted by many projects, such as Delta Lake. +Parquet is a columnar file format. Its hierarchy determines the units that Mito can read, cache, or prune during a scan. -Parquet has a hierarchical structure like "row groups-columns-data pages". Data in a Parquet file is horizontally partitioned into row groups, in which all values of the same column are stored together to form a data page. Data page is the minimal storage unit. This structure greatly improves performance. +Parquet organizes data as row groups, column chunks, and pages. A row group contains one column chunk for each column, and each column chunk contains one or more pages. Pages are the smallest encoded I/O units within a column chunk. -First, clustering data by column makes file scanning more efficient, especially when only a few columns are queried, which is very common in analytical systems. +Column chunks let a projected scan read only the requested columns. -Second, data of the same column tends to be homogeneous which helps with compression when apply techniques like dictionary and Run-Length Encoding (RLE). +Pages within one column also tend to compress well with encodings such as dictionary encoding and run-length encoding (RLE). Parquet file format ## Data Persistence -GreptimeDB provides a configuration item `region_engine.mito.global_write_buffer_size`, which is flush threshold of the total memory usage for all MemTables. +`region_engine.mito.global_write_buffer_size` sets the memory threshold shared by all Mito memtables on a Datanode. -When the size of data buffered in MemTables reaches that threshold, GreptimeDB will pick MemTables and flush them to SST files. +When memory usage reaches the threshold, the write-buffer manager selects memtables and schedules SST flushes through `src/mito2/src/flush.rs`. ## Indexing Data in SST Files -Apache Parquet file format provides inherent statistics in headers of column chunks and data pages, which are used for pruning and skipping. - -Column chunk header - -For example, in the above Parquet file, if you want to filter rows where `name` = `Emily`, you can easily skip row group 0 because the max value for `name` field is `Charlie`. This statistical information reduces IO operations. +Parquet records column statistics for row groups and pages. Mito converts compatible query predicates into Parquet pruning predicates so it can skip row groups whose min/max or null statistics cannot match. ## Index Files -For each SST file, GreptimeDB not only maintains an internal index but also generates a separate file to store the index structures specific to that SST file. - -The index files utilize the [Puffin][3] format, which offers significant flexibility, allowing for the storage of additional metadata and supporting a broader range of index structures. - -![Puffin](/puffin.png) +Mito stores index artifacts associated with an SST in versioned [Puffin][3] files. The Region manifest identifies the active index version; publishing or rebuilding an index must not make the manifest reference an incomplete artifact. -GreptimeDB stores several index structures in the Puffin file as Blobs, including the inverted index, the skipping index (backed by a bloom filter), and the full-text index. The inverted index was the first one supported and is described in detail below. +`src/mito2/src/sst/index/` integrates inverted, bloom-filter skipping, full-text, and feature-gated vector indexes with SST reads and writes. Their reusable index formats live under `src/index/src/`, while `puffin_manager.rs` manages the companion files. ## Inverted Index -In version 0.7, GreptimeDB introduced the inverted index to accelerate queries. - -The inverted index is a common index structure used for full-text searches, mapping each word in the document to a list of documents containing that word. GreptimeDB applies this search-engine technique to indexes over time-series data. - -Search engines and time series databases operate in separate domains, yet the principle behind the applied inverted index technology is similar. This similarity requires some conceptual adjustments: -1. Term: In GreptimeDB, it refers to the column value of the time series. -2. Document: In GreptimeDB, it refers to the data segment containing multiple time series. - -The inverted index enables GreptimeDB to skip data segments that do not meet query conditions, thus improving scanning efficiency. +For each indexed column, the inverted index maps encoded column values to the SST data segments that contain them. Applying a predicate produces candidate segment IDs; the normal scan still evaluates the complete predicate on rows from those segments. ![Inverted index searching](/inverted-index-searching.png) -For instance, the query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches the regex `.*users`, and `status` matches the regex `4...`. It then scans these data segments to produce the final results that meet all conditions, significantly reducing the number of IO operations. +The query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches the regex `.*users`, and `status` matches the regex `4...`. Mito scans those candidate segments and applies the complete query predicate to their rows. ### Inverted Index Format ![Inverted index format](/inverted-index-format.png) -GreptimeDB builds inverted indexes by column, with each inverted index consisting of an FST and multiple Bitmaps. - -The FST (Finite State Transducer) enables GreptimeDB to store mappings from column values to Bitmap positions in a compact format and provides excellent search performance and supports complex search capabilities (such as regular expression matching). The Bitmaps maintain a list of data segment IDs, with each bit representing a data segment. +Each column index contains a finite-state transducer (FST) and bitmaps. The FST maps encoded values to bitmap positions and supports lookups such as regular-expression matching. Each bitmap records the data segments that contain a value. ### Index Data Segments -GreptimeDB divides an SST file into multiple indexed data segments, with each segment housing an equal number of rows. This segmentation is designed to optimize query performance by scanning only the data segments that match the query conditions. +GreptimeDB divides an SST file into fixed-size indexed data segments. A matching bitmap becomes a Parquet row selection, so Mito reads only the candidate row ranges. -For example, if a data segment contains 1024 rows and the list of data segments identified through the inverted index for the query conditions is `[0, 2]`, then only the 0th and 2nd data segments in the SST file—from rows 0 to 1023 and 2048 to 3071, respectively—need to be scanned. +For example, with 1024 rows per segment and candidate segment IDs `[0, 2]`, Mito scans rows 0–1023 and 2048–3071 instead of all rows in the SST. -The number of rows in a data segment is controlled by the engine option `index.inverted_index.segment_row_count`, which defaults to `1024`. A smaller value means more precise indexing and often results in better query performance but increases the cost of index storage. By adjusting this option, a balance can be struck between storage costs and query performance. +The engine option `index.inverted_index.segment_row_count`, which defaults to `1024`, controls the target segment size. Smaller segments make pruning more precise but increase index size and build cost. ## Unified Data Access Layer: OpenDAL -GreptimeDB uses [OpenDAL][2] to provide a unified data access layer, thus, the storage engine does not need to interact with different storage APIs, and data can be migrated to cloud-based storage like AWS S3 seamlessly. +The `object-store` crate wraps [OpenDAL][2] for local filesystems and object stores. Mito performs SST and index I/O through `src/mito2/src/access_layer.rs`; storage-engine code should not add backend-specific paths around that boundary. Changing a configured backend does not migrate existing data. [1]: https://parquet.apache.org -[2]: https://github.com/datafuselabs/opendal +[2]: https://opendal.apache.org/ [3]: https://iceberg.apache.org/puffin-spec diff --git a/docs/contributor-guide/datanode/metric-engine.md b/docs/contributor-guide/datanode/metric-engine.md index 064872ce14..92f0ec7e70 100644 --- a/docs/contributor-guide/datanode/metric-engine.md +++ b/docs/contributor-guide/datanode/metric-engine.md @@ -1,44 +1,39 @@ --- -keywords: [Metric engine, small tables, logical table, physical table, storage optimization] -description: Overview of the Metric engine in GreptimeDB, its concepts, architecture, and design for handling small tables. +keywords: [Metric engine, logical table, physical table, Mito, Prometheus] +description: Metric Engine's logical-to-physical storage model for large numbers of metric tables. --- # Metric Engine ## Overview -The `Metric` engine is a component of GreptimeDB, and it's an implementation of the storage engine. It mainly targets scenarios with a large number of small tables for observable metrics. +Metric Engine is a `RegionEngine` implementation for Prometheus-style workloads with many small metric tables. It multiplexes logical tables into shared physical Mito Regions, reducing per-table metadata and storage overhead while retaining a table-level interface for reads and writes. -Its main feature is to use synthetic physical wide tables to store a large amount of small table data, achieving effects such as reuse of the same column and metadata. This reduces storage overhead for small tables and improves columnar compression efficiency. The concept of a table becomes even more lightweight under the `Metric` engine. +Metric Engine does not implement another on-disk format. It rewrites logical requests and delegates physical storage, indexing, and scans to Mito. ## Concepts -The `Metric` engine introduces two new concepts: "logical table" and "physical table". From the user's perspective, logical tables are exactly like ordinary ones. From a storage point-of-view, physical Regions are just regular Regions. - ### Logical Table -A logical table refers to user-defined tables. Just like any other ordinary table, its definition includes the name of the table, column definitions, index definitions etc. All operations such as queries or write-ins by users are based on these logical tables. Users don't need to worry about differences between logical and ordinary tables during usage. +A logical table is the table exposed to users. It has its own schema and table ID, and all user writes and queries address that table. Internally, each logical Region records the physical Region that stores its rows. -From an implementation standpoint, a logical table is virtual; it doesn't directly read or write physical data but maps read/write requests into corresponding requests for physical tables in order to implement data storage and querying. +On writes, Metric Engine injects the logical table identity into each row before forwarding it to the physical data Region. On reads, it adds a logical-table filter so only rows belonging to the requested table are returned. ### Physical Table -A physical table is a table that actually stores data, possessing several physical Regions defined by partition rules. - -## Architecture and Design +A physical table owns the shared Regions. Each physical Region is represented by a pair of Mito Regions: -The main design architecture of the `Metric` engine is as follows: +- a data Region containing rows from multiple logical tables; +- a metadata Region containing logical-table and logical-column mappings used by Metric Engine. -![Arch](/metric-engine-arch.png) +Direct writes to a physical Region are rejected because they would bypass the logical-table mapping. Queries against a physical table remain supported. -In the current version implementation, the `Metric` engine reuses the `Mito` engine to achieve storage and query capabilities for physical data. It also provides access to both physical tables and logical tables simultaneously. +## Architecture and Design -Regarding partitioning, logical tables have identical partition rules and Region distribution as physical tables. This makes sense because the data of logical tables are directly stored in physical tables, so their partition rules are consistent. +Logical tables associated with a physical table use the same partition layout. Their logical Region IDs map to the corresponding physical data and metadata Region IDs. The mapping is maintained by Metric Engine and by table-route metadata. -Concerning routing metadata, the routing address of a logical table is a logical address - what its corresponding physical table is - then through this physical table for secondary routing to obtain the real physical address. This indirect routing method can significantly reduce the number of metadata modifications required when Region migration scheduling occurs in Metric engines. +`row_modifier.rs` and `batch_modifier.rs` encode the logical table identity and time-series identity into Mito's internal columns. Depending on the physical Region's primary-key encoding, this uses `__table_id` and `__tsid` columns or the sparse `__primary_key` representation. The read path always applies the logical table ID before delegating the scan to Mito. -Operationally speaking, The `Metric` engine supports standard DML operations (INSERT, DELETE, SELECT) on logical tables. However, it only supports limited operations on physical tables to prevent misoperations - for example, writing directly to a physical table is prohibited as it could affect user's logical table data. Generally speaking, users can consider that they have read-only access to these physical tables. +Metric Engine provides batch DDL paths for operations that affect many logical tables. This avoids issuing a separate metadata update for every table during workloads such as Prometheus Remote Write auto-creation or physical Region migration. These are data definition language operations; ordinary logical-table inserts, deletes, and queries still use the standard Region request paths. -To improve performance during simultaneous DDL (Data Definition Language) operations on many tables, the 'Metric' engine has introduced some batch DDL operations. These batch DDL operations can merge lots of DDL actions into one request thereby reducing queries and modifications times for metadata thus enhancing performance. This feature is particularly beneficial in scenarios such as the automatic creation requests brought about by large amounts of metrics during Prometheus Remote Write cold start-up, as well as the modification requests for numerous route-tables mentioned earlier during migration of many physical regions. - -Apart from physical data regions belonging to physical tables, the 'Metric' engine creates an additional metadata region physically for each individual physical data region used in storing some metadata needed by itself while maintaining mapping and other states. This metadata includes the mapping relationship between logical tables and physical tables, the mapping relationship between logical columns and physical columns etc. +The main implementation is under `src/metric-engine/src/`. Changes to reserved columns, Region ID conversion, or metadata encoding affect persisted data and require backward-compatibility review. diff --git a/docs/contributor-guide/datanode/overview.md b/docs/contributor-guide/datanode/overview.md index d0afe21b34..b921f66de7 100644 --- a/docs/contributor-guide/datanode/overview.md +++ b/docs/contributor-guide/datanode/overview.md @@ -1,34 +1,21 @@ --- -keywords: [Datanode, region server, data storage, gRPC service, heartbeat task, region manager] -description: Overview of Datanode in GreptimeDB, its responsibilities, components, and interaction with other parts of the system. +keywords: [Datanode, RegionServer, storage engine, query engine, heartbeat] +description: Overview of Datanode's Region-level storage and query responsibilities. --- # Datanode ## Introduction -`Datanode` is mainly responsible for storing the actual data for GreptimeDB. As we know, in GreptimeDB, -a `table` can have one or more `Region`s, and `Datanode` is responsible for managing the reading and writing -of these `Region`s. `Datanode` is not aware of `table` and can be considered as a `region server`. Therefore, -`Frontend` and `Metasrv` operate `Datanode` at the granularity of `Region`. - -![Datanode](/datanode.png) +Datanode stores table data and executes queries against its local Regions. A table can contain multiple Regions, but Datanode does not manage tables as a metadata object. Frontend and Metasrv address it through Region-level requests, so its primary abstraction is a Region server. ## Components -A `Datanode` contains all the components needed for a `region server`. Here we list some of the vital parts: +- `RegionServer` in `src/datanode/src/region_server.rs` dispatches Region requests to the registered storage engine and exposes Regions to the query layer. +- The gRPC service accepts Region reads, writes, and lifecycle operations from Frontend and Metasrv. +- The local query engine plans and executes logical subplans received from Frontend. Datanode does not parse client SQL or coordinate a distributed query. +- The heartbeat task reports node and Region state to Metasrv and receives control instructions such as Region open, close, migration, and cache invalidation messages. +- HTTP handlers expose operational endpoints such as metrics and configuration. +- Datanode registers the Mito, Metric, and File Region engines. Mito is the primary time-series storage engine; Metric delegates physical storage to Mito for high-cardinality metric-table workloads; File exposes data in external files. -- A gRPC service is provided for reading and writing region data, and `Frontend` uses this service - to read and write data from `Datanode`s. -- An HTTP service, through which you can obtain metrics, configuration information, etc., of the current node. -- `Heartbeat Task` is used to send heartbeat to the `Metasrv`. The heartbeat plays a crucial role in the - distributed architecture of GreptimeDB and serves as a basic communication channel for distributed coordination. - The upstream heartbeat messages contain important information such as the workload of a `Region`. If the - `Metasrv `has made scheduling(such as `Region` migration) decisions, it will send instructions to the - `Datanode` via downstream heartbeat messages. -- The `Datanode` does not parse user SQL or perform distributed planning. The user's query requests for one or - more `Table`s will be transformed into `Region` query requests in the `Frontend`. The `Datanode` is responsible - for executing these `Region` query plans with its local query engine. -- A `Region Manager` is used to manage all `Region`s on a `Datanode`. -- GreptimeDB supports a pluggable multi-engine architecture, with existing engines including `File Engine` and - `Mito Engine`. +In standalone mode, the same Region server runs in-process without Metasrv coordination. In distributed mode, Region writability and lifecycle changes are coordinated through Metasrv leases and heartbeat messages. diff --git a/docs/contributor-guide/datanode/python-scripts.md b/docs/contributor-guide/datanode/python-scripts.md deleted file mode 100644 index 98909142a6..0000000000 --- a/docs/contributor-guide/datanode/python-scripts.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -keywords: [Python scripts, data analysis, CPython backend, RustPython interpreter, RecordBatch] -description: Guide on using Python scripts for data analysis in GreptimeDB, including backend options and setup instructions. ---- - -# Python Scripts - -## Introduction - -Python scripts are methods for analyzing data in GreptimeDB, -by running it in the database directly instead of fetching all the data from the database and running it locally. -This approach saves a lot of data transfer costs. -The image below depicts how the script works. -The `RecordBatch` (which is basically a column in a table with type and nullability metadata) -can come from anywhere in the database, -and the returned `RecordBatch` can be annotated in Python grammar to indicate its metadata, -such as type or nullability. -The script will do its best to convert the returned object to a `RecordBatch`, -whether it is a Python list, a `RecordBatch` computed from parameters, -or a constant (which is extended to the same length as the input arguments). - -![Python Coprocessor](/python-coprocessor.png) - -## Two optional backends - -### CPython Backend powered by PyO3 - -This backend is powered by [PyO3](https://pyo3.rs/v0.18.1/), enabling the use of your favourite Python libraries (such as NumPy, Pandas, etc.) and allowing Conda to manage your Python environment. - -But using it also involves some complications. You must set up the correct Python shared library, which can be a bit challenging. In general, you just need to install the `python-dev` package. However, if you are using Homebrew to install Python on macOS, you must create a proper soft link to `Library/Frameworks/Python.framework`. Detailed instructions on using PyO3 crate with different Python Version can be found [here](https://pyo3.rs/v0.18.1/building_and_distribution#configuring-the-python-version) - -### Embedded RustPython Interpreter - -An experiment [python interpreter](https://github.com/RustPython/RustPython) to run -the coprocessor script, it supports Python 3.10 grammar. You can use all the very Python syntax, see [User Guide/Python Coprocessor](/user-guide/python-scripts/overview.md) for more! diff --git a/docs/contributor-guide/datanode/query-engine.md b/docs/contributor-guide/datanode/query-engine.md index 5b83a30613..bc6cb1d450 100644 --- a/docs/contributor-guide/datanode/query-engine.md +++ b/docs/contributor-guide/datanode/query-engine.md @@ -1,57 +1,35 @@ --- -keywords: [query engine, Apache DataFusion, logical plan, physical plan, data representation, indexing] -description: Overview of GreptimeDB's query engine, its architecture, data representation, indexing, and extensibility. +keywords: [query engine, Apache DataFusion, logical plan, physical plan, Arrow, indexes] +description: Overview of GreptimeDB's DataFusion-based query planning and execution pipeline. --- # Query Engine ## Introduction -GreptimeDB's query engine is built on [Apache DataFusion][1] (subproject under [Apache -Arrow][2]), a brilliant query engine written in Rust. It provides a set of well functional components from -logical plan, physical plan and the execution runtime. Below explains how each component is orchestrated and their positions during execution. +GreptimeDB's query engine is built on [Apache DataFusion][1]. The `query` crate owns SQL, PromQL, and log planning, GreptimeDB optimizer rules, physical planning, and execution. ![Execution Procedure](/execution-procedure.png) -The entry point is the logical plan, which is used as the general intermediate representation of a -query or execution logic etc. Two noticeable sources of logical plan are from: 1. the user query, like -SQL through SQL parser and planner; 2. the Frontend's distributed query, which is explained in details in the following section. +A query first becomes a DataFusion logical plan. SQL and other query-language planners produce these plans, and Frontend also sends serialized logical subplans to Datanodes during distributed execution. -Next is the physical plan, or the execution plan. Unlike the logical plan which is a big -enumeration containing all the logical plan variants (except the special extension plan node), the -physical plan is in fact a trait that defines a group of methods invoked during -execution. All data processing logics are packed in corresponding structures that -implement the trait. They are the actual operations performed on the data, like -aggregator `MIN` or `AVG`, and table scan `SELECT ... FROM`. +Analyzer and optimizer rules normalize the plan, push filters and projections, prune Regions, and introduce GreptimeDB extension nodes such as `MergeScan`. Both DataFusion rules and rules under `src/query/src/optimizer/` participate in this phase. -The optimization phase which improves execution performance by transforming both logical and physical plans, is now all based on rules. It is also called, "Rule Based Optimization". Some of the rules are DataFusion native and others are customized in Greptime DB. In the future, we plan to add more -rules and leverage the data statistics for Cost Based Optimization/CBO. - -The last phase "execute" is a verb, stands for the procedure that reads data from storage, performs -calculations and generates the expected results. Although it's more abstract than previously mentioned concepts, you can just -simply imagine it as executing a Rust async function. And it's indeed a future (stream). - -`EXPLAIN [VERBOSE] ` is very useful if you want to see how your SQL is represented in the logical or physical plan. +The physical planner converts the optimized logical plan into DataFusion `ExecutionPlan` implementations. Executing the root plan returns an asynchronous stream of Arrow `RecordBatch` values. Use `EXPLAIN` or `EXPLAIN VERBOSE` to inspect the plans produced for a SQL statement. ## Data Representation -GreptimeDB uses [Apache Arrow][2] as the in-memory data representation. It's column-oriented, in -cross-platform format, and also contains many high-performance data operators. These features -make it easy to share data in many different environments and implement calculation logic. +GreptimeDB uses [Apache Arrow][2] arrays and `RecordBatch` values for in-memory data exchange. The columnar representation is shared by storage scans, query operators, RPC streams, and result encoders, avoiding row-by-row conversion inside the execution pipeline. ## Indexing -In time series data, there are two important dimensions: timestamp and tag columns (or like -primary key in a general relational database). GreptimeDB groups data in time buckets, so it's efficient -to locate and extract data within the expected time range at a very low cost. The mainly used persistent file format [Apache Parquet][3] in GreptimeDB helps a lot -- it -provides multi-level indices and filters that make it easy to prune data during querying. In the future, we -will make more use of this feature, and develop our separated index to handle more complex use cases. +Index construction and persistent index formats belong to the storage engine, not the query engine. Mito uses Parquet statistics and inverted, skipping, and full-text indexes to prune SST files, row groups, and data segments. A feature-gated vector index supplies candidate rows for vector search. See [Data Persistence and Indexing](./data-persistence-indexing.md). + +The query layer contributes predicates and projections to the scan. An index can reduce the data read by a compatible predicate, but it does not replace the remaining filter operators in the query plan. ## Distributed Execution -Covered in [Distributed Querying][6]. +Frontend rewrites compatible logical-plan fragments into remote `MergeScan` inputs, serializes them with Substrait, and sends Region-specific requests to Datanodes. See [Distributed Querying](../frontend/distributed-querying.md). -[1]: https://github.com/apache/arrow-datafusion +[1]: https://datafusion.apache.org/ [2]: https://arrow.apache.org/ -[3]: https://parquet.apache.org -[6]: ../frontend/distributed-querying.md diff --git a/docs/contributor-guide/datanode/storage-engine.md b/docs/contributor-guide/datanode/storage-engine.md index c220c72208..9b67236485 100644 --- a/docs/contributor-guide/datanode/storage-engine.md +++ b/docs/contributor-guide/datanode/storage-engine.md @@ -7,36 +7,18 @@ description: Overview of the storage engine in GreptimeDB, its architecture, com ## Introduction -The `storage engine` is responsible for storing the data of the database. Mito, based on [LSMT][1] (Log-structured Merge-tree), is the storage engine we use by default. We have made significant optimizations for handling time-series data scenarios, so mito engine is not suitable for general purposes. +Mito is GreptimeDB's primary time-series Region engine. It implements the `RegionEngine` trait and uses an [LSM tree][1] write path: WAL and memtables absorb writes, immutable Parquet SST files hold persisted data, and background compaction reorganizes those files. ## Architecture -The picture below shows the architecture and process procedure of the storage engine. - -![Architecture](/storage-engine-arch.png) - -The architecture is the same as a traditional LSMT engine: - -- [WAL][2] - - Guarantees high durability for data that is not yet being flushed. - - Based on the `Log Store` API, thus it doesn't care about the underlying storage - media. - - Log records of the WAL can be stored on the local disk, or in a remote log service such as - Kafka (remote WAL) that implements the `Log Store` API. -- Memtables: - - Data is written into the `active memtable`, aka `mutable memtable` first. - - When a `mutable memtable` is full, it will be changed to a `read-only memtable`, aka `immutable memtable`. -- SST - - The full name of SST, aka SSTable is `Sorted String Table`. - - `Immutable memtable` is flushed to persistent storage and produces an SST file. - - Rows in an SST are sorted by primary key and time index; see [Data Layout in SST Files](#data-layout-in-sst-files). -- Compactor - - Small `SST` is merged into large `SST` by the compactor via compaction. - - The default compaction strategy is [TWCS][3]. Compaction groups SST files into time windows and, together with TTL, removes expired data. See [Compaction](/user-guide/deployments-administration/manage-data/compaction.md). -- Manifest - - The manifest stores the metadata of the engine, such as the metadata of the `SST`. -- Cache - - Speed up queries. +The implementation is under `src/mito2/src/`. `engine.rs` dispatches Region requests, `worker/` owns the per-Region write loop, `read/` builds scans, and `flush.rs`, `compaction/`, `manifest/`, and `sst/` implement the persistent lifecycle. + +- **WAL** records writes that have not reached an SST so a Region can recover its memtable state. It uses the `LogStore` API with local raft-engine and remote Kafka providers. The acknowledgement durability boundary depends on provider configuration; see [Write-Ahead Logging](./wal.md). +- **Memtables** receive writes in a mutable active memtable. A flush freezes it into an immutable memtable that remains readable until its rows have been written to an SST. +- **SST files** are immutable Parquet files whose rows are sorted by primary key and time index; see [Data Layout in SST Files](#data-layout-in-sst-files). +- **Compaction** merges SST files and removes expired data. The default strategy is [TWCS][3], which groups files by time window. See [Compaction](/user-guide/deployments-administration/manage-data/compaction.md). +- **Manifest** stores versioned Region metadata and SST file changes used during recovery. +- **Caches** retain file metadata, data pages, and other reusable scan state. [1]: https://en.wikipedia.org/wiki/Log-structured_merge-tree [2]: https://en.wikipedia.org/wiki/Write-ahead_logging @@ -44,26 +26,11 @@ The architecture is the same as a traditional LSMT engine: ## Data Model -The data model provided by the storage engine is between the `key-value` model and the tabular model. - -```txt -tag-1, ..., tag-m, timestamp -> field-1, ..., field-n -``` - -Each row of data contains multiple tag columns, one timestamp column, and multiple field columns. -- `0 ~ m` tag columns - - Tag columns can be nullable. - - Specified during table creation using `PRIMARY KEY`. -- Must include one timestamp column - - Timestamp column cannot be null. - - Specified during table creation using `TIME INDEX`. -- `0 ~ n` field columns - - Field columns can be nullable. -- Data is sorted by tag columns and timestamp column. +Mito receives a `RegionMetadata` schema with a primary-key column list, one non-null time-index column, and field columns. The SQL layer exposes primary-key columns as tags, but Mito operates on column IDs and semantic types rather than SQL table definitions. ### Region -Data in the storage engine is stored in `regions`, which are logical isolated storage units within the engine. Rows within a `region` must have the same `schema`, which defines the tag columns, timestamp column, and field columns within the `region`. The data of tables in the database is stored in one or multiple `regions`. +A Region is Mito's isolation, recovery, and request unit. Every row in a Region follows its Region metadata. A table can span several Regions, while table routing and placement remain outside the storage engine. ## Data Layout in SST Files @@ -71,28 +38,6 @@ When a memtable is flushed, Mito writes its rows into immutable [Apache Parquet] Within an SST file, rows are sorted by `(primary key, time index)`. Rows that share the same primary key (the tag columns) belong to the same time-series and are stored contiguously, ordered by timestamp. This locality is what makes scanning a single series cheap and improves compression. For append-only tables without a primary key, rows are sorted by the time index alone. -For example, consider a table that stores host metrics: - -```sql -CREATE TABLE host_metrics ( - host STRING, - region STRING, - ts TIMESTAMP TIME INDEX, - cpu DOUBLE, - memory DOUBLE, - PRIMARY KEY (host, region) -); -``` - -Mito groups rows by primary key and orders them by time, so the data within an SST conceptually looks like: - -| host | region | ts | cpu | memory | -| --- | --- | --- | --- | --- | -| host-a | us-east | 10:00 | 0.42 | 7.1 | -| host-a | us-east | 10:01 | 0.47 | 7.4 | -| host-a | us-west | 10:00 | 0.31 | 6.8 | -| host-b | us-east | 10:00 | 0.80 | 8.6 | - Besides the table columns, Mito stores three internal columns in each SST file so it can merge, deduplicate, and apply deletes correctly when reading from multiple memtables and SST files: - `__primary_key`: the encoded primary key (tags) of the row. @@ -111,6 +56,6 @@ Mito avoids reading data that cannot match a query by combining several pruning 1. **Time-range pruning.** Files and memtables whose time range does not intersect the query's time range are skipped before opening any reader. This is usually the cheapest and most effective step for time-series queries. 2. **Row-group statistics.** If a row group's min-max statistics prove that no row can match a predicate, the whole row group is skipped. -3. **Indexes.** Inverted, skipping, and full-text indexes provide more selective pruning for predicates that statistics cannot resolve. See [Data Persistence and Indexing](data-persistence-indexing.md). +3. **Indexes.** Inverted, skipping, and full-text indexes provide more selective pruning for predicates that statistics cannot resolve. The feature-gated vector index selects candidate rows for vector search. See [Data Persistence and Indexing](data-persistence-indexing.md). Scan pruning pipeline diff --git a/docs/contributor-guide/datanode/wal.md b/docs/contributor-guide/datanode/wal.md index 4ecb19ef02..7f33e756f4 100644 --- a/docs/contributor-guide/datanode/wal.md +++ b/docs/contributor-guide/datanode/wal.md @@ -1,36 +1,24 @@ --- -keywords: [write-ahead logging, WAL, data durability, LSMT, synchronous flush, asynchronous flush] -description: Introduction to Write-Ahead Logging (WAL) in GreptimeDB, its purpose, architecture, and operational modes. +keywords: [write-ahead log, WAL, recovery, raft-engine, Kafka] +description: Mito's write-ahead log abstraction, recovery path, and durability settings. --- # Write-Ahead Logging ## Introduction -Our storage engine is inspired by the Log-structured Merge Tree (LSMT). Mutating operations are -applied to a MemTable instead of persisting to disk, which significantly improves performance but -also brings durability-related issues, especially when the Datanode crashes unexpectedly. Similar -to all LSMT-like storage engines, GreptimeDB uses a write-ahead log (WAL) to ensure data durability -and is safe from crashing. +Mito applies writes to an in-memory memtable before they are flushed to SST files. To recover data that has not reached an SST, it appends each Region's write operations to a write-ahead log (WAL) before applying them to the memtable. -WAL is an append-only file group. All `INSERT` and `DELETE` operations are transformed into -operation entries and then appended to WAL. Once operation entries are persisted to the underlying -file, the operation can be further applied to MemTable. +On Region open or Datanode restart, Mito replays WAL entries after the last persisted sequence and rebuilds the in-memory state. Sequence numbers are assigned per Region and are also used for deduplication and snapshot reads. -When the Datanode restarts, operation entries in WAL are replayed to reconstruct the correct -in-memory state. - -![WAL in Datanode](/wal.png) +The WAL is accessed through the `LogStore` abstraction. Datanode supports a local `raft_engine` provider and a remote Kafka provider; the storage engine does not assume that the log is a local file. Provider construction is in `src/datanode/src/datanode.rs`, while Mito's WAL integration is in `src/mito2/src/wal.rs` and its write worker. ## Namespace -Namespace of WAL is used to separate entries from different tables (different regions). Append and -read operations must provide a Namespace. Currently, region ID is used as the Namespace, because -each region has a MemTable that needs to be reconstructed when Datanode restarts. +WAL entries are isolated by Region. Append and read operations use the Region ID as their namespace, allowing recovery to replay exactly the log for the Region being opened. A table may contain several Regions, so the WAL namespace is not a table identifier. ## Synchronous/Asynchronous flush -By default, appending to WAL is asynchronous, which means the writer will not wait until entries are -flushed to disk. This setting provides higher performance, but may lose data when running host shutdown unexpectedly. In the other hand, synchronous flush provides higher durability at the cost of performance. +For the local `raft_engine` provider, `sync_write` controls whether an append waits for the log to be synced to durable storage. It defaults to `false`. Asynchronous writes reduce latency but can lose recently acknowledged entries if the host or storage fails before the buffered log is synced. Setting `sync_write = true` strengthens that durability boundary at the cost of additional write latency. -In v0.4 version, the new region worker architecture can use batching to alleviate the overhead of sync flush. +Kafka WAL durability depends on the Kafka producer and cluster settings rather than the local `sync_write` option. Code that acknowledges a write must preserve the ordering between WAL append and memtable mutation for every provider. diff --git a/docs/contributor-guide/flownode/arrangement.md b/docs/contributor-guide/flownode/arrangement.md index aed75af777..28551db10c 100644 --- a/docs/contributor-guide/flownode/arrangement.md +++ b/docs/contributor-guide/flownode/arrangement.md @@ -1,20 +1,14 @@ --- -keywords: [arrangement component, state storage, update streams, key-value pairs, querying and updating] -description: Details on the arrangement component in Flownode, which stores state and update streams for querying and updating. +keywords: [legacy streaming mode, Arrangement, state, differential updates, watermark] +description: In-memory Arrangement state used by Flownode's legacy streaming path. --- # Arrangement -Arrangement stores the state in the dataflow's process. It stores the streams of update flows for further querying and updating. +`Arrangement` is an in-memory state index used by Flownode's legacy streaming path. It is implemented in `src/flow/src/utils.rs`; batching mode does not use it. -The arrangement essentially stores key-value pairs with timestamps to mark their change time. +An Arrangement stores updates as `((key row, value row), timestamp, diff)`. The timestamp orders changes in dataflow time, and the differential `diff` adds or removes a value. `get(now: Timestamp, key: &Row)` returns the value visible for a key at the requested time. -Internally, the arrangement receives tuples like -`((Key Row, Value Row), timestamp, diff)` and stores them in memory. One can query key-value pairs at a certain time using the `get(now: Timestamp, key: Row)` method. -The arrangement also assumes that everything older than a certain time (also known as the low watermark) has already been ingested to the sink tables and does not keep a history for them. +The low watermark is the earliest time for which history may still be needed. State older than that watermark is assumed to have reached the sink and can be compacted. Advancing it too far would make later differential updates impossible to reconcile. -:::tip NOTE - -The arrangement allows for the removal of keys by setting the `diff` to -1 in incoming tuples. Moreover, if a row has been previously added to the arrangement and the same key is inserted with a different value, the original value is overwritten with the new value. - -::: +For the current implementation, a `diff` of `-1` removes a key. Inserting the same key with a different value replaces the previous value. These semantics are part of the legacy streaming state model and must not be applied to batching-mode sink writes. diff --git a/docs/contributor-guide/flownode/batching_mode.md b/docs/contributor-guide/flownode/batching_mode.md index 37a695ce9c..07a36186d1 100644 --- a/docs/contributor-guide/flownode/batching_mode.md +++ b/docs/contributor-guide/flownode/batching_mode.md @@ -1,74 +1,50 @@ --- -keywords: [batching mode, flow management, Flownode components, Flownode limitations, continuous aggregation] -description: Overview of Flownode's batching mode, the active execution mode for continuous data aggregation, including its architecture and query execution flow. +keywords: [batching mode, BatchingEngine, dirty time windows, checkpoints, continuous aggregation] +description: Batching mode task lifecycle, dirty-window processing, and recovery invariants. --- # Flownode Batching Mode Developer Guide -This guide provides a brief overview of the batching mode in `flownode`. It's intended for developers who want to understand the internal workings of this mode. +Batching mode maintains a sink table by rerunning a Flow query for source data that may have changed. It is the actively developed Flownode execution path. Mode selection remains internal; see the [Flownode overview](./overview.md). ## Overview -The batching mode in `flownode` is designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. This is in contrast to the original streaming mode, now deprecated, where data was processed as it arrived. +For a time-windowed Flow, writes to a source table mark the corresponding windows as dirty. A background task consumes those windows, adds time predicates to the Flow query when the query shape permits it, and sends an insert plan to Frontend. Frontend executes the query and writes the result to the sink table. -The core idea is to: -1. Define a `flow` with a SQL query that aggregates data from a source table into a sink table. -2. The query typically includes a time window function (e.g., `date_bin`) on a timestamp column. -3. When new data is inserted into the source table, the system marks the corresponding time windows as "dirty." -4. A background task periodically wakes up, identifies these dirty windows, and re-runs the aggregation query for those specific time ranges. -5. The results are then inserted into the sink table, effectively updating the aggregated view. +Evaluation-interval and TQL flows can require an unfiltered execution rather than a dirty-window filter. The batching path therefore treats a dirty window either as an exact range to recompute or as a signal that a full query is required, depending on the Flow definition. ## Architecture -The batching mode consists of several key components that work together to achieve this continuous aggregation. As shown in the diagram below: - -![batching mode architecture](/batching_mode_arch.png) - ### `BatchingEngine` -The `BatchingEngine` is the heart of the batching mode. It's a central component that manages all active flows. Its primary responsibilities are: +`BatchingEngine` in `src/flow/src/batching_mode/engine.rs` owns the map from `FlowId` to `BatchingTask`. It creates and removes tasks, handles flush requests, and dispatches dirty-window notifications to every Flow that reads the affected source table. -- **Task Management**: It maintains a map of `FlowId` to `BatchingTask`. It handles the creation, deletion, and retrieval of these tasks. -- **Event Dispatching**: When new data arrives (via `handle_inserts_inner`) or when time windows are explicitly marked as dirty (`handle_mark_dirty_time_window`), the `BatchingEngine` identifies which flows are affected and forwards the information to the corresponding `BatchingTask`s. +Task creation parses the Flow query, records source and sink tables, creates the sink table when needed, and initializes the execution state. Metadata for the Flow itself is persisted by `common-meta`. ### `BatchingTask` -A `BatchingTask` represents a single, independent data flow. Each task is associated with one `flow` definition and runs in its own asynchronous loop. +One `BatchingTask` represents one Flow. `TaskConfig` contains immutable query, table, window, expiration, and scheduling data. `TaskState` contains the mutable execution state. -- **Configuration (`TaskConfig`)**: This struct holds the immutable configuration for a flow, such as the SQL query, source and sink table names, and time window expression. -- **State (`TaskState`)**: This contains the dynamic, mutable state of the task, most importantly the `DirtyTimeWindows`. -- **Execution Loop**: The task runs an infinite loop (`start_executing_loop`) that: - 1. Checks for a shutdown signal. - 2. Waits for a scheduled interval or until it's woken up. - 3. Generates a new query plan (`gen_insert_plan`) based on the current set of dirty time windows. - 4. Executes the query (`execute_logical_plan`) against the database. - 5. Cleans up the processed dirty windows. +The background loop waits for its schedule or a notification, generates the next insert plan, executes it through `FrontendClient`, and records the result. An execution lock serializes background execution, manual flush, plan generation, and checkpoint updates so two runs cannot consume the same state concurrently. ### `TaskState` and `DirtyTimeWindows` -- **`TaskState`**: This struct tracks the runtime state of a `BatchingTask`. It includes `dirty_time_windows`, which is crucial for determining what work needs to be done. -- **`DirtyTimeWindows`**: This is a key data structure that keeps track of which time windows have received new data since the last query execution. It stores a set of non-overlapping time ranges. When a task's execution loop runs, it consults this structure to build a `WHERE` clause that filters the source table for only the dirty time windows. +`DirtyTimeWindows` stores non-overlapping ranges that need recomputation. Plan generation removes a bounded set of ranges from the queue. If planning or execution fails, those ranges are restored; they are not discarded merely because a run started. -### `TimeWindowExpr` +`TaskState` also stores per-Region checkpoints for the experimental incremental-read path. Incremental mode advances a checkpoint only when the result reports a complete watermark proof for the participating Regions. A scoped full-snapshot repair freezes a high watermark while it drains dirty windows; new writes stay in the live queue. If the repair fails or its watermark proof is incomplete, pending windows return to the queue. -The `TimeWindowExpr` is a helper utility for dealing with time window expressions like `date_bin`. +Incremental reads are disabled by default through `experimental_enable_incremental_read`. When disabled or when the query shape is incompatible, the task uses full-snapshot execution. -- **Evaluation**: It can take a timestamp and evaluate the time window expression to determine the start and end of the window that the timestamp falls into. -- **Window Size**: It can also determine the size (duration) of the time window from the expression. +### `TimeWindowExpr` -This is essential for both marking windows as dirty and for generating the correct filter conditions when querying the source table. +`TimeWindowExpr` in `src/flow/src/batching_mode/time_window.rs` evaluates window expressions such as `date_bin`. It maps an input timestamp to its window bounds and provides the window size used to merge ranges and generate predicates. ## Query Execution Flow -Here's a simplified step-by-step walkthrough of how a query is executed in batch mode: - -1. **Data Ingestion**: New data is written to a source table. -2. **Marking Dirty**: The `BatchingEngine` receives a notification about the new data. It uses the `TimeWindowExpr` associated with each relevant flow to determine which time windows are affected by the new data points. These windows are then added to the `DirtyTimeWindows` set in the corresponding `TaskState`. -3. **Task Wake-up**: The `BatchingTask`'s execution loop wakes up, either due to its periodic schedule or because it was notified of a large backlog of dirty windows. -4. **Plan Generation**: The task calls `gen_insert_plan`. This method: - - Inspects the `DirtyTimeWindows`. - - Generates a series of `OR`'d `WHERE` clauses (e.g., `(ts >= 't1' AND ts < 't2') OR (ts >= 't3' AND ts < 't4') ...`) that cover the dirty windows. - - Rewrites the original SQL query to include this new filter, ensuring that only the necessary data is processed. -5. **Execution**: The modified query plan is sent to the `Frontend` for execution. The database processes the aggregation on the filtered data. -6. **Upsert**: The results are inserted into the sink table. The sink table is typically defined with a primary key that includes the time window column, so new results for an existing window will overwrite (upsert) the old ones. -7. **State Update**: The `DirtyTimeWindows` set is cleared of the windows that were just processed. The task then goes back to sleep until the next interval. +1. A source write or explicit mark-dirty request identifies the affected Flow and time range. +2. `BatchingEngine` adds the range to the task's `DirtyTimeWindows` and wakes the task when required. +3. `BatchingTask` consumes a bounded group of windows and builds a filtered insert plan, or chooses an unfiltered full query for a Flow that cannot be scoped safely. +4. `FrontendClient` sends the serialized logical plan to Frontend. Frontend executes the query and writes rows to the sink table. +5. On success, the task commits the execution state and advances only checkpoints justified by returned watermarks. On failure, it restores consumed windows before the next retry. + +Changes to plan coverage, checkpoint advancement, or dirty-window restoration affect correctness. A run must never clear work that has not been reflected in the sink table, and a checkpoint must never move beyond data proven to be included in the result. diff --git a/docs/contributor-guide/flownode/dataflow.md b/docs/contributor-guide/flownode/dataflow.md index 000a65edb3..cec6366273 100644 --- a/docs/contributor-guide/flownode/dataflow.md +++ b/docs/contributor-guide/flownode/dataflow.md @@ -1,17 +1,14 @@ --- -keywords: [dataflow module, SQL query transformation, execution plan, DAG, map and reduce operations] -description: Explanation of the dataflow module in Flownode, its operations, internal data handling, and future enhancements. +keywords: [legacy streaming mode, dataflow, DFIR, differential rows, Flow] +description: Internal compute graph used by Flownode's legacy streaming execution path. --- # Dataflow -The `dataflow` module (see `flow::compute` module) is the core computing module of `flow`. -It takes a SQL query and transforms it into flow's internal execution plan. -This execution plan is then rendered into an actual dataflow, which is essentially a directed acyclic graph (DAG) of functions with input and output ports. -The dataflow is triggered to run when needed. +This page describes the compute graph used by Flownode's legacy streaming mode. New continuous-aggregation work uses [batching mode](./batching_mode.md); do not use this page to infer batching behavior. -Currently, this dataflow only supports `map` and `reduce` operations. Support for `join` operations will be added in the future. +The streaming path converts a Flow definition through `src/flow/src/transform.rs` into a typed plan in `plan.rs`. `src/flow/src/compute/render.rs` renders supported plan nodes into a DFIR-style dataflow graph, and workers under `src/flow/src/adapter/` own and execute those graphs. -Internally, the dataflow handles data in row format, using a tuple `(row, time, diff)`. Here, `row` represents the actual data being passed, which may contain multiple `Value` objects. -`time` is the system time which tracks the progress of the dataflow, and `diff` typically represents the insertion or deletion of the row (+1 or -1). -Therefore, the tuple represents the insert/delete operation of the `row` at a given system `time`. \ No newline at end of file +The internal record is a differential row `(row, timestamp, diff)`. `row` contains the values, `timestamp` tracks dataflow progress, and `diff` records multiplicity changes such as insertion (`+1`) and deletion (`-1`). Operators propagate those changes so aggregates and sink output can be updated incrementally. + +The typed plan represents map/filter/project and reduce operations, along with join and union nodes. The streaming renderer currently executes map/filter/project and reduce; join and union rendering still return a not-implemented error. Check both `plan.rs` and `compute/render.rs` before adding an operator, because being representable in a plan does not mean it is executable. diff --git a/docs/contributor-guide/flownode/overview.md b/docs/contributor-guide/flownode/overview.md index d1317bfc1c..5677f41c22 100644 --- a/docs/contributor-guide/flownode/overview.md +++ b/docs/contributor-guide/flownode/overview.md @@ -1,26 +1,28 @@ --- -keywords: [continuous aggregation, flow management, standalone mode, Flownode components, Flownode limitations] -description: Overview of Flownode, a component providing Flow computation capabilities to the database, including batching mode, deprecated streaming mode, and core components. +keywords: [Flownode, continuous aggregation, batching mode, streaming mode, Flow] +description: Flownode's execution modes, routing boundary, and implementation layout. --- # Flownode ## Introduction +Flownode is the execution component behind GreptimeDB Flow, which maintains continuously computed results from source tables in a sink table. It runs in-process in standalone mode and as a separate service in distributed mode. -`Flownode` provides Flow computation capabilities to the database. -`Flownode` manages `flows` which are tasks that receive data from the `source` and send data to the `sink`. +Flownode has two execution paths: -`Flownode` support both `standalone` and `distributed` mode. In `standalone` mode, `Flownode` runs in the same process as the database. In `distributed` mode, `Flownode` runs in a separate process and communicates with the database through the network. +- **Batching mode** is the actively developed path. It tracks affected time windows and periodically runs an aggregation query through Frontend. See the [batching mode guide](./batching_mode.md). +- **Streaming mode** is the legacy incremental-dataflow path. It processes row-level changes through worker-owned compute graphs and remains for compatibility. -There are two execution modes for a flow: -- **Batching Mode**: The active mode for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. Aggregation and TQL queries use this mode. For more details, see the [Batching Mode Developer Guide](./batching_mode.md). -- **Streaming Mode (deprecated)**: The original mode where data is processed as it arrives. It is kept for legacy compatibility and is not recommended for new workloads. +Users do not select an execution mode directly. `flow_type` is reserved internal metadata. `StatementExecutor::determine_flow_type` in `src/operator/src/statement/ddl.rs` chooses the mode when a Flow is created, and `FlowDualEngine` handles compatibility routing inside Flownode. ## Components -A `Flownode` contains all the components needed to execute a flow. The specific components involved depend on the execution mode. At a high level, the key parts are: +- `FlowEngine` in `src/flow/src/engine.rs` defines the create, remove, flush, and insert lifecycle shared by both paths. +- `FlowDualEngine` in `src/flow/src/adapter/flownode_impl.rs` routes each Flow to the batching or streaming engine. +- `src/flow/src/batching_mode/` contains time-window tracking, task scheduling, Frontend RPC, sink-table creation, and checkpoint logic. +- `src/flow/src/adapter/`, `compute/`, `expr/`, and `plan.rs` implement the legacy streaming path. +- `src/flow/src/server.rs` exposes the Flownode gRPC service; `heartbeat.rs` reports Flownode state to Metasrv. +- Persisted Flow metadata and DDL procedures live in `src/common/meta/`, not in the `flow` crate. -- **Flow Manager**: A central component responsible for managing the lifecycle of all flows. -- **Task Executor**: The runtime environment where the flow logic is executed. In batching mode, this is a `BatchingTask`; in the deprecated streaming mode, this is typically a `FlowWorker`. -- **Flow Task**: Represents a single, independent data flow, containing the logic for transforming data from a source to a sink. +Mode-specific changes must be reviewed against `FlowDualEngine` and the shared metadata contract. Do not assume that a fix in one execution path applies to the other. diff --git a/docs/contributor-guide/frontend/distributed-querying.md b/docs/contributor-guide/frontend/distributed-querying.md index 21ee07d7e8..ebc490b80e 100644 --- a/docs/contributor-guide/frontend/distributed-querying.md +++ b/docs/contributor-guide/frontend/distributed-querying.md @@ -1,33 +1,24 @@ --- -keywords: [distributed querying, dist planner, dist plan, logical plan, substrait format] -description: Describes the process of distributed querying in GreptimeDB, focusing on the dist planner and dist plan. +keywords: [distributed querying, DistPlannerAnalyzer, MergeScan, Substrait, Region pruning] +description: How GreptimeDB turns a logical query plan into local and remote execution stages. --- # Distributed Querying -Most steps of querying in frontend and datanode are identical. The only difference is that -Frontend have a "special" step in planning phase to make the logical query plan distributed. -Let's reference it as "dist planner" in the following text. - -The modified, distributed logical plan has multiple stages, each of them is executed in different -server node. +Frontend and Datanode use the same DataFusion-based query engine. In distributed mode, Frontend adds a planning step that separates work executed on Datanodes from work completed by Frontend. ![Frontend query](/frontend-query.png) ## Dist Planner -Planner will traverse the input logical plan, and split it into multiple stages by the "[commutativity -rule](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md)". +`DistPlannerAnalyzer` in `src/query/src/dist_plan/analyzer.rs` rewrites the DataFusion logical plan. It pushes compatible operators toward table scans and wraps remote subplans in `MergeScan` nodes. The planner uses operator commutativity and plan-shape rules to decide which work is safe to execute on each Datanode; unsupported shapes remain on Frontend or use the configured fallback path. + +Filters on partition columns are also used to prune Regions. Frontend resolves each selected Region to a Datanode through `FrontendRegionQueryHandler` before execution. -This rule is under heavy development. At present it will consider things like: -- whether the operator itself is commutative -- how the partition rule is configured -- etc... +The original design and its commutativity rules are documented in the [distributed planner RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md). ## Dist Plan -Except the first stage, which have to read data from files in storage. All other stages' leaf node -are actually a gRPC call to its previous stage. +A `MergeScan` remote input is a complete logical subplan. Frontend serializes that subplan with Substrait and sends a Region-specific query request to the selected Datanode. The Datanode plans and executes the subplan against its local Regions and streams Arrow record batches back. -Sub-plan in a stage is itself a complete logical plan, and can be executed independently without -the follow up stages. The plan is encoded in [substrait format](https://substrait.io). +Frontend merges the remote streams and executes any operators that could not be pushed down. This boundary is not limited to the logical `TableScan` node: filters, projections, partial aggregates, and other compatible operators may be part of a remote subplan. diff --git a/docs/contributor-guide/frontend/overview.md b/docs/contributor-guide/frontend/overview.md index 3a2f63e7ae..170124650f 100644 --- a/docs/contributor-guide/frontend/overview.md +++ b/docs/contributor-guide/frontend/overview.md @@ -1,44 +1,46 @@ --- -keywords: [frontend, proxy, protocol, routing, distributed query, tenant management, authorization, flow control, cloud deployment, endpoints] -description: Overview of GreptimeDB's Frontend component - a stateless proxy service for client requests. +keywords: [frontend, protocols, request routing, distributed query, authorization] +description: Overview of Frontend, GreptimeDB's stateless request entry point and query coordinator. --- # Frontend -The **Frontend** is a stateless service that serves as the entry point for client requests in GreptimeDB. It provides a unified interface for multiple database protocols and acts as a proxy that forwards read/write requests to appropriate Datanodes in the distributed system. +Frontend is GreptimeDB's stateless request entry point and orchestration layer. It implements the business logic behind the protocol servers, plans queries, routes writes and Region reads, and coordinates distributed query execution. -## Core Functions +Network listeners and wire formats belong to the `servers` crate. The `frontend` crate implements handler traits for SQL, gRPC, MySQL, PostgreSQL, InfluxDB, OpenTelemetry, Prometheus, OpenTSDB, Jaeger, and other supported interfaces. -- **Protocol Support**: Multiple database protocols including SQL, PromQL, MySQL, and PostgreSQL. See [Protocols][1] for details -- **Request Routing**: Routes requests to appropriate Datanodes based on metadata -- **Query Distribution**: Splits distributed queries across multiple nodes -- **Response Aggregation**: Combines results from multiple Datanodes -- **Authorization**: Security and access control validation + + +## Responsibilities + +- Parse and plan SQL, PromQL, and log queries. +- Check permissions and carry session context through request processing. +- Route inserts, deletes, and Region queries using catalog and route metadata. +- Dispatch distributed query fragments to Datanodes and merge their results. + +See the [protocol overview](/user-guide/protocols/overview.md) for the user-facing interfaces. ## Architecture ### Key Components -- **Protocol Handlers**: Handle different database protocols -- **Catalog Manager**: Caches metadata from Metasrv to enable efficient request routing and schema validation -- **Dist Planner**: Converts logical plans to distributed execution plans -- **Request Router**: Determines target Datanodes for each request + +- `Instance` in `src/frontend/src/instance.rs` is the main business-logic container and implements the server handler traits. +- Modules under `src/frontend/src/instance/` handle individual request types and protocols. +- `StatementExecutor` in the `operator` crate handles statements and write-side operations. +- The `query` crate owns logical planning, optimization, and distributed plans. +- `FrontendRegionQueryHandler` in `instance/region_query.rs` resolves Region targets and sends query requests to Datanodes. ### Request Flow -![request flow](/request_flow.png) +In standalone mode, Frontend accesses an embedded Datanode through a local `RegionServer` adapter. In distributed mode, it uses metadata from Metasrv and RPC clients to reach remote Datanodes. ### Deployment -The following picture shows a typical deployment of GreptimeDB in the cloud. The `Frontend` instances -form a cluster to serve the requests from clients: - -![frontend](/frontend.png) +Frontend instances do not own table data. Multiple instances can serve requests against the same Metasrv and Datanode cluster. -## Details + -- [Table Sharding][2] -- [Distributed Querying][3] +## Implementation guides -[1]: /user-guide/protocols/overview.md -[2]: ./table-sharding.md -[3]: ./distributed-querying.md +- [Table Sharding](./table-sharding.md) +- [Distributed Querying](./distributed-querying.md) diff --git a/docs/contributor-guide/frontend/table-sharding.md b/docs/contributor-guide/frontend/table-sharding.md index a60276d14e..1e15c2a4f9 100644 --- a/docs/contributor-guide/frontend/table-sharding.md +++ b/docs/contributor-guide/frontend/table-sharding.md @@ -5,23 +5,19 @@ description: Explains how table data in GreptimeDB is sharded and distributed, i # Table Sharding -The sharding of stored data is essential to any distributed database. This document will describe how table's data in GreptimeDB is being sharded, and distributed. +GreptimeDB shards a table into partitions and stores each partition in a Region. This page describes the implementation-level relationship between those objects. ## Partition -For the syntax of creating a partitioned table, please refer to the [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) section in the User Guide. +The [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) section in the User Guide documents the partition syntax. ## Region -The data within a table is logically split after creating partitions. You may ask the question " -how are the data, after being logically partitioned, stored in the GreptimeDB? The answer is in "`Region`"s. - -Each region is corresponding to a partition, and stores the data in the partition. The regions are distributed among -`Datanode`s. `Metasrv` manages the route information that maps regions to Datanodes. +Each partition maps to one Region, which is the storage and scheduling unit managed by Datanodes. Metasrv stores the route that maps each Region to its Datanode. If the partition layout needs to change after table creation, GreptimeDB supports explicit [repartitioning](/user-guide/deployments-administration/manage-data/repartition.md) through split and merge operations. -The relationship between partition and region can be viewed as the following diagram: +The relationship is shown below: ```text ┌───────┐ diff --git a/docs/contributor-guide/getting-started.md b/docs/contributor-guide/getting-started.md index b17184dfa8..0bfbd91e81 100644 --- a/docs/contributor-guide/getting-started.md +++ b/docs/contributor-guide/getting-started.md @@ -1,35 +1,30 @@ --- -keywords: [setup, running from source, prerequisites, build dependencies, unit tests] -description: Instructions for setting up and running GreptimeDB from source, including prerequisites, build dependencies, and running unit tests. +keywords: [setup, build from source, Rust toolchain, unit tests] +description: Set up a development environment and build, run, and test GreptimeDB from source. --- # Getting started -This page describes how to run GreptimeDB from source in your local environment. +This page covers the minimum setup for building and running GreptimeDB from source. -## Prerequisite + + +## Prerequisites ### System & Architecture -At the moment, GreptimeDB supports Linux (both amd64 and arm64), macOS (both amd64 and Apple Silicon), and Windows. +GreptimeDB supports Linux and macOS on x86-64 and Arm64, as well as Windows. ### Build Dependencies -- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) (optional) -- C/C++ Toolchain: provides essential tools for compiling and linking. This is available either as `build-essential` on ubuntu or a similar name on other platforms. -- Rust nightly toolchain ([guide][1]) - - Compile the source code -- Protobuf ([guide][2]) - - Compile the proto file - - Note that the version needs to be >= 3.15. You can check it with `protoc --version` -- Machine: Recommended memory is 16GB or more, or use the [mold](https://github.com/rui314/mold) tool to reduce memory usage during linking. - -[1]: -[2]: +- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line). +- A C/C++ build toolchain, such as `build-essential` on Ubuntu or Xcode Command Line Tools on macOS. +- [Rustup](https://rustup.rs/). The repository's `rust-toolchain.toml` selects the required nightly toolchain automatically. +- [Protocol Buffers compiler](https://grpc.io/docs/protoc-installation/) 3.15 or later. Check the installed version with `protoc --version`. ## Compile and Run -Start GreptimeDB standalone instance in just a few commands! +Clone the repository and start a standalone instance: ```shell git clone https://github.com/GreptimeTeam/greptimedb.git @@ -37,34 +32,32 @@ cd greptimedb cargo run -- standalone start ``` -Next, you can choose the protocol you like to interact with in GreptimeDB. - -Or if you just want to build the server without running it: +To build without starting the server, run: ```shell -cargo build # --release +cargo build ``` -The artifacts can be found under `$REPO/target/debug` or `$REPO/target/release`, depending on the build mode (whether the `--release` option is passed) +Add `--release` for an optimized build. Artifacts are written to `target/debug` or `target/release`. -## Unit test + -GreptimeDB is well-tested, the entire unit test suite is shipped with source code. To test them, run with [nextest](https://nexte.st/index.html). +## Unit tests -To install nextest using cargo, run: +GreptimeDB uses [cargo-nextest](https://nexte.st/) as its standard Rust test runner. Install it with: ```shell cargo install cargo-nextest --locked ``` -Or you can check their [docs](https://nexte.st/docs/installation/pre-built-binaries/) for other ways to install. - -After nextest is ready, you can run the test suite with: +Run the workspace test suite with the features used by CI: ```shell cargo nextest run --workspace --features pg_kvbackend,mysql_kvbackend ``` +For package-scoped tests and other test types, see the [testing guide](./tests/overview.md). + ## Docker -We also provide prebuilt binaries via Docker, available on Docker Hub: [https://hub.docker.com/r/greptime/greptimedb](https://hub.docker.com/r/greptime/greptimedb) +Prebuilt images are published to [Docker Hub](https://hub.docker.com/r/greptime/greptimedb). They are useful for running GreptimeDB, but do not replace the source build when developing or testing code changes. diff --git a/docs/contributor-guide/how-to/how-to-trace-greptimedb.md b/docs/contributor-guide/how-to/how-to-trace-greptimedb.md index 75941aa0be..ee7a1f6744 100644 --- a/docs/contributor-guide/how-to/how-to-trace-greptimedb.md +++ b/docs/contributor-guide/how-to/how-to-trace-greptimedb.md @@ -1,80 +1,88 @@ --- -keywords: [tracing, distributed tracing, trace_id, RPC, instrument, span, runtime] -description: Describes how to use Rust's tracing framework in GreptimeDB for distributed tracing, including defining tracing context in RPC, passing it, and instrumenting code. +keywords: [tracing, W3C Trace Context, RPC, instrument, runtime] +description: Propagate and instrument distributed traces in GreptimeDB code. --- # How to trace GreptimeDB -GreptimeDB uses Rust's [tracing](https://docs.rs/tracing/latest/tracing/) framework for code instrument. For the specific details and usage of tracing, please refer to the official documentation of tracing. +GreptimeDB uses the Rust [`tracing`](https://docs.rs/tracing/latest/tracing/) ecosystem and OpenTelemetry context propagation. Local spans are connected automatically only while their tracing context is carried through the same asynchronous execution path. RPC and runtime boundaries require explicit propagation. -By transparently transmitting `trace_id` and other information on the entire distributed system, we can record the function call chain of the entire distributed link, know the time of each tracked function take and other related information, so as to monitor the entire system. +The shared implementation is [`TracingContext`](https://github.com/GreptimeTeam/greptimedb/blob/main/src/common/telemetry/src/tracing_context.rs) in `common-telemetry`. It converts the active span context to and from W3C Trace Context fields. -## Define tracing context in RPC + -Because the tracing framework does not natively support distributed tracing, we need to manually pass information such as `trace_id` in the RPC message to correctly identify the function calling relationship. We use standards based on [w3c](https://www.w3.org/TR/trace-context/#traceparent-header-field-values) to encode relevant information into `tracing_context` and attach the message to the RPC header. Mainly defined in: +## RPC context fields -- `frontend` interacts with `datanode`: `tracing_context` is defined in [`RegionRequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/region/server.proto) -- `frontend` interacts with `metasrv`: `tracing_context` is defined in [`RequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/meta/common.proto) -- Client interacts with `frontend`: `tracing_context` is defined in [`RequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/common.proto) +GreptimeDB protobuf headers store W3C trace fields in a `map tracing_context` field: -## Pass tracing context in RPC call +- Frontend to Datanode: [`RegionRequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/region/server.proto) +- Meta clients and services: [`RequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/meta/common.proto) +- Client to Frontend database RPC: [`RequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/common.proto) -We build a `TracingContext` structure that encapsulates operations related to the tracing context. [Related code](https://github.com/GreptimeTeam/greptimedb/blob/main/src/common/telemetry/src/tracing_context.rs) +When adding an internal RPC, use the existing header type when possible. A separate tracing field with a different encoding creates a propagation path that the common helpers cannot handle. -GreptimeDB uses `TracingContext::from_current_span()` to obtain the current tracing context, uses the `to_w3c()` method to encode the tracing context into a w3c-compliant format, and attaches it to the RPC message, so that the tracing context is correctly distributed passed within the component. + -The following example illustrates how to obtain the current tracing context and pass the parameters correctly when constructing the RPC message, so that the tracing context is correctly passed among the distributed components. +## Propagate context across an RPC +Capture the current context when constructing an outbound request: ```rust let request = RegionRequest { - header: Some(RegionRequestHeader { - tracing_context: TracingContext::from_current_span().to_w3c(), - ..Default::default() - }), - body: Some(region_request::Body::Alter(request)), + header: Some(RegionRequestHeader { + tracing_context: TracingContext::from_current_span().to_w3c(), + ..Default::default() + }), + body: Some(region_request::Body::Alter(request)), }; ``` -On the receiver side of the RPC message, the tracing context needs to be correctly decoded and used to build the first `span` to trace the function call. For example, the following code will correctly decode the `tracing_context` in the received RPC message using the `TracingContext::from_w3c` method. And use the `attach` method to attach the context message to the newly created `info_span!("RegionServer::handle_read")`, so that the call can be tracked across distributed components. +At the receiver, decode the header and attach a new local span as a child of that context: ```rust -... let tracing_context = request - .header - .as_ref() - .map(|h| TracingContext::from_w3c(&h.tracing_context)) - .unwrap_or_default(); + .header + .as_ref() + .map(|header| TracingContext::from_w3c(&header.tracing_context)) + .unwrap_or_default(); + let result = self - .handle_read(request) - .trace(tracing_context.attach(info_span!("RegionServer::handle_read"))) - .await?; -... + .handle_read(request) + .trace(tracing_context.attach(info_span!("RegionServer::handle_read"))) + .await?; ``` -## Use `tracing::instrument` to instrument the code +An absent or invalid context becomes an empty context, so request handling still works without a parent trace. Do not reuse one request's context for unrelated work. + + + +## Instrument local work -We use the `instrument` macro provided by tracing to instrument the code. We only need to annotate the `instrument` macro in the function that needs to be instrument. The `instrument` macro will print every function parameter on each function call into the span in the form of `Debug`. For parameters that do not implement the `Debug` trait, or the structure is too large and has too many parameters, resulting in a span that is too large. If you want to avoid these situations, you need to use `skip_all` to skip printing all parameters. +Use `#[tracing::instrument]` at asynchronous or expensive boundaries where a span helps correlate latency and errors. The macro records arguments through `Debug` by default. Skip credentials, tokens, large batches, query payloads, and any argument whose full value is not safe or useful in telemetry. ```rust -#[tracing::instrument(skip_all)] -async fn instrument_function(....) { - ... +#[tracing::instrument(skip_all, fields(region_id = %region_id))] +async fn handle_region(region_id: RegionId, request: RegionRequest) { + region_server.handle(request).await; } ``` -## Code instrument across runtime +Prefer a small set of stable identifiers in `fields(...)` to recording a complete request. Instrumenting every helper function creates high-volume traces without improving the request-level call graph. + + + +## Propagate context across runtimes -Rust's tracing library will automatically handle the nested relationship between instrument functions in the same runtime, but if a function call across the runtime, tracing library cannot automatically trace such calls, and we need to manually pass the context across the runtime. +Moving a future to another runtime or spawning work outside the current instrumented future can lose the active parent. Capture the context before crossing that boundary and attach it to a new span inside the spawned future: ```rust let tracing_context = TracingContext::from_current_span(); let handle = runtime.spawn(async move { - handler - .handle(query) - .trace(tracing_context.attach(info_span!("xxxxx"))) - ... + handler + .handle(query) + .trace(tracing_context.attach(info_span!("background_query"))) + .await }); ``` -For example, the above code needs to perform tracing across runtimes. We first obtain the current tracing context through `TracingContext::from_current_span()`, create a span in another runtime, and attach the span to the current context, and we are done. The hidden code points that span the runtime are eliminated, and the call chain is correctly traced. \ No newline at end of file +The context must be captured before the spawn. Keep the attached span scoped to the spawned operation so unrelated tasks do not inherit the same parent. diff --git a/docs/contributor-guide/how-to/how-to-use-tokio-console.md b/docs/contributor-guide/how-to/how-to-use-tokio-console.md index 81cff8f7ce..0cb1b3a7fd 100644 --- a/docs/contributor-guide/how-to/how-to-use-tokio-console.md +++ b/docs/contributor-guide/how-to/how-to-use-tokio-console.md @@ -1,34 +1,30 @@ --- -keywords: [tokio-console, GreptimeDB, tokio_unstable, build, connect, subscriber] -description: Guides on using tokio-console in GreptimeDB, including building with specific features and connecting to the tokio console subscriber. +keywords: [tokio-console, tokio_unstable, asynchronous tasks, diagnostics] +description: Build GreptimeDB with tokio-console support and inspect its Tokio runtime. --- # How to use tokio-console in GreptimeDB -This document introduces how to use the [tokio-console](https://github.com/tokio-rs/console) in GreptimeDB. +[`tokio-console`](https://github.com/tokio-rs/console) displays live Tokio tasks and resources. GreptimeDB compiles the subscriber behind the `cmd/tokio-console` feature and also requires Tokio's unstable instrumentation cfg. -First, build GreptimeDB with feature `cmd/tokio-console`. Also the `tokio_unstable` cfg must be enabled: +Build GreptimeDB with both enabled: ```bash RUSTFLAGS="--cfg tokio_unstable" cargo build -F cmd/tokio-console ``` -Then start GreptimeDB with the tokio console binding address config: `--tokio-console-addr`. For example: +Start the component with a full socket address for the console subscriber: ```bash -greptime --tokio-console-addr="127.0.0.1:6669" standalone start +./target/debug/greptime --tokio-console-addr="127.0.0.1:6669" standalone start ``` -Now you can use `tokio-console` to connect to GreptimeDB's tokio console subscriber: +The option is global and can also be used with `frontend`, `datanode`, `metasrv`, or `flownode` commands built with the same feature. + +Install the console client as described in the [tokio-console repository](https://github.com/tokio-rs/console#installing-the-console) and connect to the configured address: ```bash -tokio-console [TARGET_ADDR] +tokio-console http://127.0.0.1:6669 ``` -"TARGET_ADDR" defaults to "\". - -:::tip Note - -You can refer to [tokio-console](https://github.com/tokio-rs/console) to see the installation of `tokio-console`. - -::: +Keep the subscriber on a loopback or otherwise protected address. It is a diagnostic endpoint, not a public GreptimeDB protocol. The feature and `tokio_unstable` instrumentation add runtime diagnostics and should be enabled deliberately when investigating task stalls, wakeups, or resource contention. diff --git a/docs/contributor-guide/how-to/how-to-write-sdk.md b/docs/contributor-guide/how-to/how-to-write-sdk.md index 40d0bc3713..4d468a14c6 100644 --- a/docs/contributor-guide/how-to/how-to-write-sdk.md +++ b/docs/contributor-guide/how-to/how-to-write-sdk.md @@ -1,42 +1,31 @@ --- -keywords: [gRPC SDK, GreptimeDatabase, Handle, HandleRequests, GreptimeRequest, GreptimeResponse] -description: Explains how to write a gRPC SDK for GreptimeDB, focusing on the GreptimeDatabase service, its methods, and the structure of requests and responses. +keywords: [gRPC ingester SDK, GreptimeDatabase, RowInsertRequests, streaming RPC] +description: Protocol and reliability requirements for a GreptimeDB gRPC ingester SDK. --- # How to write a gRPC SDK for GreptimeDB -A GreptimeDB gRPC SDK only needs to handle the writes. The reads are standard SQL and PromQL, can be handled by any JDBC -client or Prometheus client. This is also why GreptimeDB gRPC SDKs are all named -like "`greptimedb-ingester-`". Please make sure your GreptimeDB SDK follow the same naming convention. +This guide covers an **ingester SDK** built on GreptimeDB's native gRPC database service. Query drivers and clients are outside its scope. Official ingester libraries use the `greptimedb-ingester-` naming pattern. -## `GreptimeDatabase` Service +Generate message and client code from the versioned [greptime-proto](https://github.com/GreptimeTeam/greptime-proto) definitions rather than copying message layouts into an SDK. Keep the generated protocol package separate from the ergonomic row and batch APIs exposed to application code. -GreptimeDB defines a custom gRPC service called `GreptimeDatabase`. All you need to do in your SDK are implement it. You -can find its Protobuf -definitions [here](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto). +## `GreptimeDatabase` Service -The service contains two RPC methods: +[`database.proto`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto) defines two RPC methods: ```protobuf service GreptimeDatabase { rpc Handle(GreptimeRequest) returns (GreptimeResponse); - rpc HandleRequests(stream GreptimeRequest) returns (GreptimeResponse); } ``` -The `Handle` method is for unary call: when a `GreptimeRequest` is received and processed by a GreptimeDB -server, it responds with a `GreptimeResponse` immediately. +`Handle` is a unary RPC. `HandleRequests` is a [client-streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc): the client sends a stream of requests, closes its send side, and receives one summarized response. A production SDK should apply bounded buffering and gRPC flow control rather than accumulating an unbounded batch in memory. -The `HandleRequests` acts in -a "[Client streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc)" style. It ingests a -stream of `GreptimeRequest`, and handles them on the fly. After all the requests have been handled, it returns a -summarized `GreptimeResponse`. Through `HandleRequests`, we can achieve a very high throughput of requests handling. +The protocol has no request-level idempotency key. An SDK must not promise exactly-once ingestion. If a transport failure leaves the server outcome unknown, an automatic retry can duplicate data for table configurations that retain duplicate rows; make retry behavior explicit to callers. ### `GreptimeRequest` -The `GreptimeRequest` is a Protobuf message defined like this: - ```protobuf message GreptimeRequest { RequestHeader header = 1; @@ -51,27 +40,21 @@ message GreptimeRequest { } ``` -A `RequestHeader` is needed, it includes some context, authentication and others. The "oneof" field contains the request -to the GreptimeDB server. +For ingestion, prefer `RowInsertRequests`. Each `RowInsertRequest` names one table and carries a `Rows` schema plus row values. Validate column count, data type, semantic type, and null representation before sending so client-side construction errors do not become opaque server errors. The older column-oriented `InsertRequests` remains part of the protocol for compatibility. -Note that we have two types of insertions, one is in the form of "column" (the `InsertRequests`), and the other is " -row" (`RowInsertRequests`). It's generally recommended to use the "row" form, since it's more natural for insertions on -a table, and easier to use. However, if there's a need to insert a large number of columns at once, or there're plenty -of "null" values to insert, the "column" form is better to be used. +Every request includes a `RequestHeader`. Populate the target catalog and schema, authentication header, timezone, and W3C tracing context when the corresponding SDK option is set. Do not silently replace an explicitly selected catalog or schema with a client default. ### `GreptimeResponse` -The `GreptimeResponse` is a Protobuf message defined like this: - ```protobuf message GreptimeResponse { ResponseHeader header = 1; - oneof response {AffectedRows affected_rows = 2;} + oneof response { + AffectedRows affected_rows = 2; + } } ``` -The `ResponseHeader` contains the response's status code, and error message (if there's any). The "oneof" response only -contains the affected rows for now. +Successful gRPC transport does not by itself mean the database operation succeeded. Inspect `ResponseHeader.status`, map non-success status codes and `err_msg` into the SDK's error type, and return `affected_rows` only after that check. Preserve the underlying gRPC status separately from a GreptimeDB response status so callers can distinguish transport failures from server-side request errors. -GreptimeDB has a lot of SDKs now, you can refer to -them [here](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=) for some examples. +Protobuf clients must also tolerate unknown fields and an unset response variant. Add compatibility tests using serialized messages from the supported protocol versions, plus integration tests for unary writes, client streaming, authentication errors, partial stream failure, and server status propagation. diff --git a/docs/contributor-guide/metasrv/admin-api.md b/docs/contributor-guide/metasrv/admin-api.md index b091b48b70..5e635cf6b3 100644 --- a/docs/contributor-guide/metasrv/admin-api.md +++ b/docs/contributor-guide/metasrv/admin-api.md @@ -1,157 +1,53 @@ --- keywords: [admin api, health check, leader query, heartbeat, maintenance mode, RESTful API] -description: Details the Admin API for Metasrv, including endpoints for health checks, leader queries, heartbeat data, maintenance mode, and Procedure Manager controls. +description: Maintainer reference for Metasrv's unauthenticated Admin API router and state-changing endpoints. --- # Admin API -:::tip -Note that all Admin API endpoints in this document listen on Metasrv's `HTTP_PORT`, which defaults to `4000`. -::: +The Axum router is assembled in `src/meta-srv/src/service/admin.rs` and mounted under `/admin` on Metasrv's HTTP server. The default HTTP port is `4000`. -The Admin API provides a simple way to view and manage cluster information, including metasrv health detection, metasrv leader query, datanode heartbeat detection, maintenance mode, and Procedure Manager controls. +The router does not add authentication. Some endpoints change cluster behavior, so deployments must protect this port with network-level controls. When adding a route, define its HTTP method explicitly, keep read and mutation handlers separate, and add handler-level tests in `src/meta-srv/src/service/admin/`. -The Admin API is an HTTP service that provides a set of RESTful APIs that can be called through HTTP requests. The Admin API is simple, user-friendly and safe. -This page covers the following APIs: +## /health HTTP endpoint -- /health -- /leader -- /heartbeat -- /maintenance -- /procedure-manager - -All these APIs are under the parent resource `/admin`. - -In the following sections, we assume that your metasrv instance is running on localhost port 4000. - -## /health HTTP endpoint - -The `/health` endpoint accepts GET HTTP requests and you can use this endpoint to check the health of your metasrv instance. - -### Definition - -```bash -curl -X GET http://localhost:4000/admin/health -``` - -### Examples - -#### Request - -```bash -curl -X GET http://localhost:4000/admin/health -``` - -#### Response - -```json -OK -``` +`GET /admin/health` returns `OK` when the HTTP service is running. It does not prove that this node is the current leader or that external dependencies are reachable. The handler is in `health.rs`. ## /leader HTTP endpoint -The `/leader` endpoint accepts GET HTTP requests and you can use this endpoint to query the leader's addr of your metasrv instance. - -### Definition - -```bash -curl -X GET http://localhost:4000/admin/leader -``` - -### Examples - -#### Request - -```bash -curl -X GET http://localhost:4000/admin/leader -``` - -#### Response - -```json -127.0.0.1:4000 -``` +`GET /admin/leader` reads the elected Metasrv leader address through the configured election backend. The handler is in `leader.rs`. ## /heartbeat HTTP endpoint -The `/heartbeat` endpoint accepts GET HTTP requests and you can use this endpoint to query the heartbeat of all datanodes. - -You can also query the heartbeat data of the datanode for a specified `addr`, however, specifying `addr` in the path is optional. - -### Definition - -```bash -curl -X GET http://localhost:4000/admin/heartbeat -``` - -| Query String Parameter | Type | Optional/Required | Definition | -|:-----------------------|:-------|:------------------|:--------------------------| -| addr | String | Optional | The addr of the datanode. | - -### Examples - -#### Request - -```bash -curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100' -``` - -#### Response - -```json -[ - [ - { - "timestamp_millis": 1677049348651, - "id": 1, - "addr": "127.0.0.1:4100", - "rcus": 0, - "wcus": 0, - "region_num": 2, - "region_stats": [], - "topic_stats": [], - "node_epoch": 0, - "datanode_workloads": { - "types": [] - }, - "gc_stat": null - } - ] -] -``` +`GET /admin/heartbeat` returns Datanode heartbeat records. The optional `addr` query parameter filters by Datanode address, and `GET /admin/heartbeat/help` shows the supported query forms. The handler is in `heartbeat.rs` and reads through `MetaPeerClient`. ## /maintenance HTTP endpoint -Cluster Maintenance Mode is a safety feature in GreptimeDB that temporarily disables automatic cluster management operations. This mode is particularly useful during cluster upgrades, planned downtime, and any operation that might temporarily affect cluster stability. For more details, please refer to [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md). - -The `/maintenance` endpoint supports the following HTTP requests: +Maintenance mode disables selected automatic cluster-management work. Its user-facing behavior is documented under [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md). The router exposes: - `GET /admin/maintenance` or `GET /admin/maintenance/status`: query the maintenance mode status. - `POST /admin/maintenance/enable`: enable maintenance mode. - `POST /admin/maintenance/disable`: disable maintenance mode. -The response body uses the following format: - -```json -{ - "enabled": true -} -``` +The implementation is in `maintenance.rs` and updates `RuntimeSwitchManager`. ## /procedure-manager HTTP endpoint -This endpoint is used to manage the Procedure Manager status. For more details, please refer to [Prevent Metadata Changes](/user-guide/deployments-administration/maintenance/prevent-metadata-changes.md). - -The `/procedure-manager` endpoint supports the following HTTP requests: +These routes pause or resume Procedure Manager scheduling. See [Prevent Metadata Changes](/user-guide/deployments-administration/maintenance/prevent-metadata-changes.md) for user-facing behavior. The router exposes: - `GET /admin/procedure-manager/status`: query the Procedure Manager status. - `POST /admin/procedure-manager/pause`: pause the Procedure Manager. - `POST /admin/procedure-manager/resume`: resume the Procedure Manager. -The response body uses the following format: +The implementation is in `procedure.rs` and also updates `RuntimeSwitchManager`. + +## Other internal endpoints + +The router also exposes these maintainer-facing endpoints: + +- `GET /admin/node-lease` returns the active Datanode lease records. +- `GET /admin/recovery/status` and `POST /admin/recovery/{enable,disable}` read or change recovery mode. +- `GET /admin/sequence/table/next-id` reads the next table ID without allocating it. +- `POST /admin/sequence/table/set-next-id` changes the allocator's next table ID. The handler rejects this operation unless recovery mode is enabled. -```json -{ - "status": "running" -} -``` +The recovery and sequence routes can change cluster state and are intended for controlled repair procedures. Read their handlers and tests before changing or invoking them; this page does not define a general recovery workflow. diff --git a/docs/contributor-guide/metasrv/overview.md b/docs/contributor-guide/metasrv/overview.md index 7aa052dd7e..e2f1efd930 100644 --- a/docs/contributor-guide/metasrv/overview.md +++ b/docs/contributor-guide/metasrv/overview.md @@ -1,161 +1,70 @@ --- -keywords: [metasrv, metadata, request-router, load balancing, election, high availability, heartbeat] -description: Provides an overview of the Metasrv service, its components, interactions with the Frontend, architecture, and key functionalities like distributed consensus and heartbeat management. +keywords: [metasrv, metadata, routing, leader election, heartbeat, distributed procedures] +description: Overview of Metasrv's metadata, coordination, and cluster-management responsibilities. --- # Metasrv -![meta](/meta.png) + -## What's in Metasrv +## Responsibilities -- Store metadata (Catalog, Schema, Table, Region, etc.) -- Request-Router. It tells the Frontend where to write and read data. -- Load balancing for Datanode, determines who should handle new table creation requests, more precisely, it makes resource allocation decisions. -- Election & High Availability, GreptimeDB is designed in a Leader-Follower architecture, only Leader nodes can write while Follower nodes can read, the number of Follower nodes is usually >= 1, and Follower nodes need to be able to switch to Leader quickly when Leader is not available. -- Statistical data collection (reported via Heartbeats on each node), such as CPU, Load, number of Tables on the node, average/peak data read/write size, etc., can be used as the basis for distributed scheduling. +Metasrv is the metadata and coordination service for distributed deployments. It: -## How the Frontend interacts with Metasrv +- persists Catalog, Schema, Table, Region, route, and node metadata through the KV backend; +- uses leader election so coordination and metadata-changing work runs on one leader; +- tracks node leases and Region statistics through heartbeat streams; +- selects Datanodes for Regions when tables are created; +- runs recoverable distributed procedures for DDL, Region migration, failover, repartitioning, and related maintenance work; +- publishes cache invalidations and other control messages to Frontend and Datanode. -First, the routing table in Request-Router is in the following structure (note that this is only the logical structure, the actual storage structure varies, for example, endpoints may have dictionary compression). +The data models, KV abstraction, election interfaces, key encoding, and DDL manager are implemented in `src/common/meta/`. The `src/meta-srv/` crate provides the server, state machine, heartbeat handlers, and control procedures. -``` - table_A - table_name - table_schema // for physical plan - regions - region_1 - mutate_endpoint - select_endpoint_1, select_endpoint_2 - region_2 - mutate_endpoint - select_endpoint_1, select_endpoint_2, select_endpoint_3 - region_xxx - table_B - ... -``` + + +## Frontend interaction + +Frontend uses the `meta-client` crate to obtain table metadata and Region routes and to submit metadata-changing operations. It caches metadata locally; Metasrv sends invalidation messages when a procedure changes metadata. ### Create Table -1. The Frontend sends `CREATE TABLE` requests to Metasrv. -2. Plan the number of Regions according to the partition rules contained in the request. -3. Check the global view of resources available to Datanodes (collected by Heartbeats) and assign one node to each region. -4. The Frontend creates the table and stores the `Schema` to Metasrv after successful creation. +1. Frontend submits the DDL request to the Metasrv leader. +2. The DDL manager validates the request, derives the Regions from the partition rules, and selects Datanodes for those Regions. +3. A persisted procedure creates the Regions and records the table and route metadata. Persisted procedure state makes the operation recoverable after a restart or leader change. +4. Metasrv invalidates affected caches after the metadata change is committed. ### Insert -1. The Frontend fetches the routes of the specified table from Metasrv. Note that the smallest routing unit is the route of the table (several regions), i.e., it contains the addresses of all regions of this table. -2. The best practice is that the Frontend first fetches the routes from its local cache and forwards the request to the Datanode. If the route is no longer valid, then Datanode is obliged to return an `Invalid Route` error, and the Frontend re-fetches the latest data from Metasrv and updates its cache. Route information does not change frequently, thus, it's sufficient for Frontend uses the Lazy policy to maintain the cache. -3. The Frontend processes a batch of writes that may contain multiple tables and multiple regions, so the Frontend needs to split user requests based on the 'route table'. +Frontend resolves the table route, splits rows by partition, and sends Region write requests to the corresponding Datanodes. Route metadata is cached, but cache invalidation or a stale-route error causes Frontend to refresh it from Metasrv. ### Select -1. As with `Insert`, the Frontend first fetches the route table from the local cache. -2. Unlike `Insert`, for `Select`, the Frontend needs to extract the read-only node (follower) from the route table, then dispatch the request to the leader or follower node depending on the priority. -3. The distributed query engine in the Frontend distributes multiple sub-query tasks based on the routing information and aggregates the query results. - -## Metasrv Architecture - -![metasrv-architecture](/metasrv-architecture.png) - -## Distributed Consensus - -As you can see, Metasrv has a dependency on distributed consensus because: - -1. First, Metasrv has to elect a leader, Datanode only sends heartbeats to the leader, and we only use a single metasrv node to receive heartbeats, which makes it easy to do some calculations or scheduling accurately and quickly based on global information. As for how the Datanode connects to the leader, this is for MetaClient to decide (using a redirect, Heartbeat requests becomes a gRPC stream, and using redirect will be less error-prone than forwarding), and it is transparent to the Datanode. -2. Second, Metasrv must provide an election API for Datanode to elect "write" and "read-only" nodes and help Datanode achieve high availability. -3. Finally, `Metadata`, `Schema` and other data must be reliably and consistently stored on Metasrv. Therefore, consensus-based algorithms are the ideal approach for storing them. - -For the first version of Metasrv, we choose Etcd as the consensus algorithm component (Metasrv is designed to consider adapting different implementations and even creating a new wheel) for the following reasons: - -1. Etcd provides exactly the API we need, such as `Watch`, `Election`, `KV`, etc. -2. We only perform two tasks with distributed consensus: elections (using the `Watch` mechanism) and storing (a small amount of metadata), and neither of them requires us to customize our own state machine, nor do we need to customize our own state machine based on raft; the small amount of data also does not require multi-raft-group support. -3. The initial version of Metasrv uses Etcd, which allows us to focus on the capabilities of Metasrv and not spend too much effort on distributed consensus algorithms, which improves the design of the system (avoiding coupling with consensus algorithms) and helps with rapid development at the beginning, as well as allows easy access to good consensus algorithm implementations in the future through good architectural designs. - -## Heartbeat Management - -The primary means of communication between Datanode and Metasrv is the Heartbeat Request/Response Stream, and we want this to be the only way to communicate. This idea is inspired by the design of [TiKV PD](https://github.com/tikv/pd), and we have practical experience in [RheaKV](https://github.com/sofastack/sofa-jraft/tree/master/jraft-rheakv/rheakv-pd). The request sends its state, while Metasrv sends different scheduling instructions via Heartbeat Response. - -A heartbeat will probably carry the data listed below, but this is not the final design, and we are still discussing and exploring exactly which data should be mostly collected. - -``` -service Heartbeat { - // Heartbeat, there may be many contents of the heartbeat, such as: - // 1. Metadata to be registered to metasrv and discoverable by other nodes. - // 2. Some performance metrics, such as Load, CPU usage, etc. - // 3. The number of computing tasks being executed. - rpc Heartbeat(stream HeartbeatRequest) returns (stream HeartbeatResponse) {} -} - -message HeartbeatRequest { - RequestHeader header = 1; - - // Self peer - Peer peer = 2; - // Leader node - bool is_leader = 3; - // Actually reported time interval - TimeInterval report_interval = 4; - // Node stat - NodeStat node_stat = 5; - // Region stats in this node - repeated RegionStat region_stats = 6; - // Follower nodes and stats, empty on follower nodes - repeated ReplicaStat replica_stats = 7; -} - -message NodeStat { - // The read capacity units during this period - uint64 rcus = 1; - // The write capacity units during this period - uint64 wcus = 2; - // Table number in this node - uint64 table_num = 3; - // Region number in this node - uint64 region_num = 4; - - double cpu_usage = 5; - double load = 6; - // Read disk I/O in the node - double read_io_rate = 7; - // Write disk I/O in the node - double write_io_rate = 8; - - // Others - map attrs = 100; -} - -message RegionStat { - uint64 region_id = 1; - TableName table_name = 2; - // The read capacity units during this period - uint64 rcus = 3; - // The write capacity units during this period - uint64 wcus = 4; - // Approximate region size - uint64 approximate_size = 5; - // Approximate number of rows - uint64 approximate_rows = 6; - - // Others - map attrs = 100; -} - -message ReplicaStat { - Peer peer = 1; - bool in_sync = 2; - bool is_learner = 3; -} -``` - -## Central Nervous System (CNS) - -We are to build an algorithmic system, which relies on real-time and historical heartbeat data from each node, should make some smarter scheduling decisions and send them to Metasrv's Autoadmin unit, which distributes the scheduling decisions, either by the Datanode itself or more likely by the PaaS platform. - -## Abstraction of Workloads - -The level of workload abstraction determines the efficiency and quality of the scheduling strategy generated by Metasrv such as resource allocation. - -DynamoDB defines RCUs & WCUs (Read Capacity Units / Write Capacity Units), explaining that a RCU is a read request of 4KB data, and a WCU is a write request of 1KB data. When using RCU and WCU to describe workloads, it's easier to achieve performance measurability and get more informative resource preallocation because we can abstract different hardware capabilities as a combination of RCU and WCU. - -However, GreptimeDB still faces a more complex situation than DynamoDB, in particular, RCU doesn't fit to describe GreptimeDB's read workloads which require a lot of computation. We are working on that. +Frontend uses table and Region metadata while planning a query. Partition predicates prune Regions, and the distributed query engine sends remote subplans to the Datanodes that own the selected Regions. See [Distributed Querying](../frontend/distributed-querying.md). + + + +## Source layout + +The main implementation areas are: + +- `src/meta-srv/src/service/`: gRPC services and the HTTP Admin API. +- `src/meta-srv/src/handler/`: the heartbeat handler chain. +- `src/meta-srv/src/procedure/`: Region migration, repartition, WAL pruning, and other distributed procedures. +- `src/meta-srv/src/region/`: Region leases, supervision, and failover triggers. +- `src/meta-srv/src/selector/`: Datanode selection for Region placement. + + + +## Leadership and persistence + +Metasrv separates leader election and durable metadata storage behind interfaces in `common-meta`. Coordination and metadata-changing operations run on the leader; a non-leader returns a not-leader response so the client can reconnect to the current leader. + +Anything required after a leader change must be stored in the KV backend. In-memory caches and leader-local state are rebuilt or cleared during a transition. Distributed procedures persist their state and must keep each step idempotent so execution can resume safely. + + + +## Heartbeat invariants + +Datanodes and Frontends maintain heartbeat streams to the Metasrv leader. Requests report node identity, leases, Region statistics, and other state. The handler chain under `src/meta-srv/src/handler/` checks leadership, updates leases and statistics, and handles mailbox messages. + +Heartbeat responses carry control messages such as Region lifecycle instructions and cache invalidations. Region supervision uses lease state to detect unavailable Regions and trigger failover procedures. Changes to heartbeat intervals must remain consistent with lease and supervisor timing in `common-meta` and `meta-srv`. diff --git a/docs/contributor-guide/metasrv/selector.md b/docs/contributor-guide/metasrv/selector.md index 790ffb849b..5eb862af6e 100644 --- a/docs/contributor-guide/metasrv/selector.md +++ b/docs/contributor-guide/metasrv/selector.md @@ -1,47 +1,40 @@ --- -keywords: [selector, metasrv, datanode, leasebased, loadbased, roundrobin] -description: Describes the different types of selectors in the Metasrv service, their characteristics, and how to configure them. +keywords: [selector, metasrv, datanode, lease based, load based, round robin] +description: Region placement selectors used by Metasrv and their configuration names. --- # Selector ## Introduction -What is the `Selector`? As its name suggests, it allows users to select specific items from a given `namespace` and `context`. There is a related trait, also named `Selector`, whose definition can be found [below][0]. - -[0]: https://github.com/GreptimeTeam/greptimedb/blob/main/src/meta-srv/src/selector.rs - -There is a specific scenario in `Metasrv` service. When a request to create a table is sent to the `Metasrv` service, it creates a routing table (the details of table creation will not be described here). The `Metasrv` service needs to select the appropriate `Datanode` list when creating a routing table. +When a table is created, Metasrv must choose Datanodes for its Regions. The [`Selector` trait](https://github.com/GreptimeTeam/greptimedb/blob/main/src/meta-srv/src/selector.rs) receives the required number of peers and a selection context, then returns candidate Datanodes from the current lease and statistics data. ## Selector Type -The `Metasrv` service currently offers the following types of `Selectors`: +Metasrv provides three selector implementations: -### LeasebasedSelector +### LeaseBasedSelector -`LeasebasedSelector` randomly selects from all available (in lease) `Datanode`s, its characteristic is simplicity and fast. +`LeaseBasedSelector` chooses randomly from Datanodes with valid leases. It does not use Region counts to rank candidates. ### LoadBasedSelector -The `LoadBasedSelector` load value is determined by the number of regions on each `Datanode`, fewer regions indicate lower load, and `LoadBasedSelector` prioritizes selecting low-load `Datanodes`. +`LoadBasedSelector` treats the number of Regions on a Datanode as its load and prefers nodes with fewer Regions. ### RoundRobinSelector [default] -`RoundRobinSelector` selects `Datanode`s in a round-robin fashion. It is recommended and the default option in most cases. If you're unsure which to choose, it's usually the right choice. + +`RoundRobinSelector` rotates through available Datanodes. It is the default selector. ## Configuration -You can configure the `Selector` by its name when starting the `Metasrv` service. +Set the selector when starting Metasrv. The accepted names are: -- LeasebasedSelector: `lease_based` or `LeaseBased` -- LoadBasedSelector: `load_based` or `LoadBased` -- RoundRobinSelector: `round_robin` or `RoundRobin` +- `lease_based` or `LeaseBased` +- `load_based` or `LoadBased` +- `round_robin` or `RoundRobin` For example: ```shell cargo run -- metasrv start --selector round_robin ``` - -```shell -cargo run -- metasrv start --selector RoundRobin -``` diff --git a/docs/contributor-guide/overview.md b/docs/contributor-guide/overview.md index 875b4b1e16..c9142d332c 100644 --- a/docs/contributor-guide/overview.md +++ b/docs/contributor-guide/overview.md @@ -1,25 +1,19 @@ --- -keywords: [architecture, key components, user requests, data processing, database components] -description: Overview of GreptimeDB's architecture, key components, and how they interact to process user requests. +keywords: [contributor guide, architecture, frontend, datanode, metasrv, flownode] +description: Entry point for contributors who want to understand and develop GreptimeDB. --- # Contributor Guide -DeepWiki provides a detailed and clear explanation of GreptimeDB's architecture and implementation. Highly recommended: - -[https://deepwiki.com/GreptimeTeam/greptimedb](https://deepwiki.com/GreptimeTeam/greptimedb) +This guide describes GreptimeDB's internal architecture and points contributors to the code that implements each subsystem. For build, test, and contribution requirements, start with the repository's [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md). ## Architecture -For the architecture and components of GreptimeDB, please see the [Architecture](/user-guide/concepts/architecture.md) document in the user guide. - -For more details on each component, see the following guides: - -- [frontend][1] -- [datanode][2] -- [metasrv][3] +The [architecture overview](/user-guide/concepts/architecture.md) explains the components and request paths from a user's perspective. The contributor guides below cover their implementation boundaries: -[1]: /contributor-guide/frontend/overview.md -[2]: /contributor-guide/datanode/overview.md -[3]: /contributor-guide/metasrv/overview.md +- [Frontend](./frontend/overview.md): protocol handling, request orchestration, routing, and distributed query planning. +- [Datanode](./datanode/overview.md): Region management, query execution, and storage engines. +- [Metasrv](./metasrv/overview.md): metadata, cluster coordination, and distributed procedures. +- [Flownode](./flownode/overview.md): continuous aggregation in standalone and distributed deployments. +To build GreptimeDB locally, continue with [Getting started](./getting-started.md). diff --git a/docs/contributor-guide/tests/integration-test.md b/docs/contributor-guide/tests/integration-test.md index 5d5f6cb1a5..ad71f80563 100644 --- a/docs/contributor-guide/tests/integration-test.md +++ b/docs/contributor-guide/tests/integration-test.md @@ -1,13 +1,30 @@ --- -keywords: [integration tests, Rust test harness, multiple components, HTTP testing, gRPC testing] -description: Guide on writing and running integration tests in GreptimeDB, covering scenarios involving multiple components. +keywords: [integration tests, Rust test harness, storage backend, Kafka, protocols] +description: Run multi-component and external-service tests from tests-integration. --- # Integration Test ## Introduction -Integration testing is written with Rust test harness (`#[test]`), unlike unit testing, they are placed separately -[here](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration). -It covers scenarios involving multiple components, in which one typical case is HTTP/gRPC-related features. You can check -its [documentation](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md) for more information. +The `tests-integration/` crate contains Rust test-harness cases that need several GreptimeDB components or an external service. Typical cases exercise HTTP or gRPC behavior, object-storage backends, Kafka WAL, and TLS-enabled dependencies. + +Use an integration test when the behavior cannot be established through a crate-local unit test or a sqlness query case. Keep protocol assertions at the public boundary and use the fixtures under `tests-integration/fixtures/` rather than introducing a second environment setup. + +The authoritative setup and command list is in [`tests-integration/README.md`](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md). Tests that require credentials or endpoints read them from a repository-root `.env` file created from `.env.example`; do not commit credentials. + +Run the general integration group from the repository root: + +```shell +cargo test integration +``` + +Backend-specific groups use their own filters, for example: + +```shell +cargo test s3 +cargo test oss +cargo test azblob +``` + +Kafka and TLS cases require the Docker Compose services documented in the integration README. Start only the dependencies required by the selected test, and clean up those services after the run. diff --git a/docs/contributor-guide/tests/overview.md b/docs/contributor-guide/tests/overview.md index feeefe2b6d..564eac9e97 100644 --- a/docs/contributor-guide/tests/overview.md +++ b/docs/contributor-guide/tests/overview.md @@ -1,8 +1,14 @@ --- -keywords: [testing methods, behavior testing, performance testing, test overview, GreptimeDB tests] -description: Overview of the testing methods used in GreptimeDB to ensure its behavior and performance. +keywords: [tests, unit tests, sqlness, integration tests, regression] +description: Choose and run the GreptimeDB test suite that matches a code change. --- # Tests -Our team has conducted lots of tests to ensure the behaviours of `GreptimeDB` . This chapter will introduce several significant methods used to test `GreptimeDB`, and how to work with them. +GreptimeDB uses several test layers. Choose the narrowest layer that exercises the behavior being changed, then add a broader regression test when the behavior crosses component boundaries or is visible through a public interface. + +- [Unit tests](./unit-test.md) cover crate-local logic, invariants, and error paths. They live next to the Rust implementation and run with cargo-nextest. +- [Sqlness tests](./sqlness-test.md) cover user-visible SQL and query behavior against standalone or distributed test environments. Cases and expected results live under `tests/cases/`. +- [Integration tests](./integration-test.md) cover interactions that require multiple components or external services, including storage backends and protocol-level behavior. They live under `tests-integration/`. + +The repository also contains specialized suites such as `tests-fuzz/`, `tests/compatibility/`, and `tests/perf/`. Use their local README or `AGENTS.md` instructions when a change affects input robustness, persisted-format compatibility, or performance. Passing one layer does not replace a test at the layer where the regression would be observed. diff --git a/docs/contributor-guide/tests/sqlness-test.md b/docs/contributor-guide/tests/sqlness-test.md index b5fdcf2b0b..781335aba3 100644 --- a/docs/contributor-guide/tests/sqlness-test.md +++ b/docs/contributor-guide/tests/sqlness-test.md @@ -1,53 +1,45 @@ --- -keywords: [SQL tests, sqlness, test suite, test cases, test output] -description: Instructions for running SQL tests in GreptimeDB using the `sqlness` test suite, including file types, case organization, and running tests. +keywords: [SQL tests, sqlness, golden files, standalone, distributed] +description: Add and run sqlness regression cases for user-visible query behavior. --- # Sqlness Test ## Introduction -SQL is an important user interface for `GreptimeDB`. We have a separate test suite for it (named `sqlness`). +Sqlness is GreptimeDB's golden-file test harness for SQL and query behavior. It builds and starts the requested GreptimeDB environment, executes case files, and compares the output with checked-in results. The harness and its current options are documented in [`tests/README.md`](https://github.com/GreptimeTeam/greptimedb/blob/main/tests/README.md). ## Sqlness manual ### Case file -Sqlness has two types of file +Each case has two files: -- `.sql`: test input, SQL only -- `.result`: expected test output, SQL and its results +- `.sql` contains the statements and sqlness directives. +- `.result` contains the expected statements and output. -The `.result` file is the expected execution output. If you see `.result` files changed, -it means the test gets a different result and indicates it may fail. You should -check the change logs to solve the problem. - -You only need to write test SQL in the `.sql` file, and run the test. +Edit the `.sql` input first, run sqlness, and review the resulting `.result` diff. A changed result can be the intended new behavior or a regression; the harness cannot decide which one. Commit a result change only after checking every changed row and error message. ### Case organization -The root dir of input cases is `tests/cases`. It contains several sub-directories stand for different test -modes. E.g., `standalone/` contains all the tests to run under `greptimedb standalone start` mode. +Cases live under `tests/cases/`. The first directory level selects an environment, such as `standalone/`; directories below it organize related cases. Sqlness discovers case files recursively. -Under the first level of sub-directory (e.g. the `cases/standalone`), you can organize your cases as you like. -Sqlness walks through every file recursively and runs them. +Place a regression in the environment where the behavior is observable. Distributed planning, routing, and multi-node metadata behavior require a distributed case even when an equivalent standalone query also succeeds. ## Run the test -Unlike other tests, this harness is in a binary target form. You can run it with +The repository defines a cargo alias for the harness: ```shell -cargo run --bin sqlness-runner bare +cargo sqlness bare ``` -It automatically finishes the following procedures: compile `GreptimeDB`, start it, grab tests and feed it to -the server, then collect and compare the results. You only need to check whether any `.result` files changed. -If not, congratulations, the test is passed 🥳! +This command builds GreptimeDB, starts the test environment, runs the cases, and updates or compares `.result` files. Inspect both the command result and `git diff`. ### Run a specific test ```shell -cargo sqlness bare -t your_test +cargo sqlness bare -t 'standalone:your_case' ``` -The `-t` or `--test-filter` option accepts a regex string. Sqlness examines case names in the format of `env:case`. +`-t`/`--test-filter` accepts a regular expression and matches case names in `env:case` form. Use a narrow filter while iterating, then run the affected environment or full suite before submission. diff --git a/docs/contributor-guide/tests/unit-test.md b/docs/contributor-guide/tests/unit-test.md index 82e30bf5fa..3d2831521c 100644 --- a/docs/contributor-guide/tests/unit-test.md +++ b/docs/contributor-guide/tests/unit-test.md @@ -1,32 +1,34 @@ --- -keywords: [unit tests, Rust, cargo nextest, test runner, coverage] -description: Guide on writing and running unit tests in GreptimeDB using Rust's `#[test]` attribute and `cargo nextest`. +keywords: [unit tests, Rust, cargo-nextest, package tests, coverage] +description: Write and run crate-local Rust tests with cargo-nextest. --- # Unit Test ## Introduction -Unit tests are embedded into the codebase, usually placed next to the logic being tested. -They are written using Rust's `#[test]` attribute and can run with `cargo nextest run`. +Rust unit tests normally live in the module they exercise or in a nearby `*_test.rs` file. Use them for local invariants, boundary conditions, error handling, and behavior that does not require a running GreptimeDB cluster. -The default test runner ships with `cargo` is not supported in GreptimeDB codebase. It's recommended -to use [`nextest`](https://nexte.st/) instead. You can install it with +GreptimeDB's standard runner is [cargo-nextest](https://nexte.st/). Install it with: ```shell cargo install cargo-nextest --locked ``` -And run the tests (here the `--workspace` is not necessary) +During development, run the affected package first: ```shell -cargo nextest run +cargo nextest run -p ``` -Notes if your Rust is installed via `rustup`, be sure to install `nextest` with `cargo` rather -than the package manager like `homebrew`. Otherwise it will mess up your local environment. +Run the workspace configuration used by CI before submitting a change that can affect several crates: + +```shell +cargo nextest run --workspace --features pg_kvbackend,mysql_kvbackend +``` + +Feature-gated code requires the corresponding feature in the test command. Check the crate's `Cargo.toml`, local `AGENTS.md`, and CI workflow before assuming the default feature set covers the path. ## Coverage -Our continuous integration (CI) jobs have a "coverage checking" step. It will report how many -codes are covered by unit tests. Please add the necessary unit test to your patch. +CI records Rust test coverage. Add tests that protect the changed behavior and credible failure cases; do not add assertions solely to increase the percentage. Query-language behavior and cross-component flows usually need a sqlness or integration test in addition to a unit test. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md index e0ef9e8e54..9505e72101 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md @@ -5,83 +5,65 @@ description: 介绍了 GreptimeDB 的数据持久化和索引机制，包括 SST # 数据持久化与索引 -与所有类似 LSMT 的存储引擎一样，MemTables 中的数据被持久化到耐久性存储，例如本地磁盘文件系统或对象存储服务。GreptimeDB 采用 [Apache Parquet][1] 作为其持久文件格式。 +Mito 将 memtable 中的数据 flush 到本地文件系统或对象存储，SST 文件使用 [Apache Parquet][1] 作为数据格式。 ## SST 文件格式 -Parquet 是一种提供快速数据查询的开源列式存储格式，已经被许多项目采用，例如 Delta Lake。 +Parquet 是一种列式文件格式，其层级结构决定 Mito 扫描时可以读取、缓存或裁剪的单元。 -Parquet 具有层次结构，类似于“行组 - 列-数据页”。Parquet 文件中的数据被水平分区为行组（row group），在其中相同列的所有值一起存储以形成数据页（data pages）。数据页是最小的存储单元。这种结构极大地提高了性能。 +Parquet 按 row group、column chunk 和 page 组织数据。每个 row group 为每一列保存一个 column chunk，每个 column chunk 再包含一个或多个 page。Page 是 column chunk 内最小的编码 I/O 单元。 -首先，数据按列聚集，这使得文件扫描更加高效，特别是当查询只涉及少数列时，这在分析系统中非常常见。 +Column chunk 使投影扫描只读取查询需要的列。 -其次，相同列的数据往往是同质的（比如具备近似的值），这有助于在采用字典和 Run-Length Encoding（RLE）等技术进行压缩。 +同一列的 page 也适合使用字典编码、run-length encoding（RLE）等方式压缩。 Parquet file format ## 数据持久化 -GreptimeDB 提供了 `region_engine.mito.global_write_buffer_size` 的配置项来设置全局的 Memtable 大小阈值。当数据库所有 MemTable 中的数据量之和达到阈值时将自动触发持久化操作，将 MemTable 的数据 flush 到 SST 文件中。 +`region_engine.mito.global_write_buffer_size` 设置一个 Datanode 上所有 Mito memtable 共享的内存阈值。内存使用达到阈值后，write-buffer manager 选择 memtable，并通过 `src/mito2/src/flush.rs` 调度 SST flush。 ## SST 文件中的索引数据 -Apache Parquet 文件格式在列块和数据页的头部提供了内置的统计信息，用于剪枝和跳过。 - -Column chunk header - -例如，在上述 Parquet 文件中，如果你想要过滤 `name` 等于 `Emily` 的行，你可以轻松跳过行组 0，因为 `name` 字段的最大值是 `Charlie`。这些统计信息减少了 IO 操作。 +Parquet 为 row group 和 page 保存列统计信息。Mito 将兼容的查询谓词转换为 Parquet pruning predicate，利用 min/max 和 null 统计信息跳过不可能匹配的 row group。 ## 索引文件 -对于每个 SST 文件，GreptimeDB 不但维护 SST 文件内部索引，还会单独生成一个文件用于存储针对该 SST 文件的索引结构。 - -索引文件采用 [Puffin][3] 格式，这种格式具有较大的灵活性，能够存储更多的元数据，并支持更多的索引结构。 - -![Puffin](/puffin.png) +Mito 将 SST 对应的索引 artifact 保存在带版本的 [Puffin][3] 文件中，Region manifest 记录当前生效的索引版本。发布或重建索引时，不能让 manifest 引用尚未完整写入的 artifact。 -GreptimeDB 会将多种索引结构作为 Blob 存储在 Puffin 文件中，包括倒排索引、跳数索引（基于 bloom filter）和全文索引。倒排索引是最早支持的索引结构，下面将详细介绍。 +`src/mito2/src/sst/index/` 负责将倒排索引、基于 bloom filter 的 skipping index、全文索引及 feature-gated vector index 接入 SST 读写。可复用的索引格式位于 `src/index/src/`，companion file 由 `puffin_manager.rs` 管理。 ## 倒排索引 -在 v0.7 版本中，GreptimeDB 引入了倒排索引（Inverted Index）来加速查询。 - -倒排索引是全文搜索中常见的索引结构，它将文档中的每个单词映射到包含该单词的文档列表。GreptimeDB 将这项搜索引擎技术用于时序数据索引。 - -搜索引擎和时间序列数据库虽然运行在不同的领域，但是应用的倒排索引技术背后的原理是相似的。这种相似性需要一些概念上的调整： -1. 单词：在 GreptimeDB 中，指时间线的列值。 -2. 文档：在 GreptimeDB 中，指包含多个时间线的数据段。 - -倒排索引的引入，使得 GreptimeDB 可以跳过不符合查询条件的数据段，从而提高扫描效率。 +倒排索引按列把编码后的列值映射到包含该值的 SST 数据段。应用谓词后得到候选 segment ID；正常扫描仍会对候选数据段中的行执行完整谓词。 ![Inverted index searching](/inverted-index-searching.png) -例如，上述查询使用倒排索引来定位数据段，数据段满足条件：`job` 等于 `apiserver`，`handler` 符合正则匹配 `.*users` 及 `status` 符合正则匹配 `4..`，然后扫描这些数据段以产生满足所有条件的最终结果，从而显着减少 IO 操作的次数。 +上图中的查询使用倒排索引找出 `job` 等于 `apiserver`、`handler` 匹配正则表达式 `.*users` 且 `status` 匹配正则表达式 `4...` 的候选数据段。Mito 扫描这些数据段，并对数据行应用完整查询谓词。 ### 倒排索引格式 ![Inverted index format](/inverted-index-format.png) -GreptimeDB 按列构建倒排索引，每个倒排索引包含一个 FST 和多个 Bitmap。 - -FST（Finite State Transducer）允许 GreptimeDB 以紧凑的格式存储列值到 Bitmap 位置的映射，并且提供了优秀的搜索性能和支持复杂搜索（例如正则表达式匹配）；Bitmap 则维护了数据段 ID 列表，每个位表示一个数据段。 +每个列索引包含一个 FST（Finite State Transducer）和多个 bitmap。FST 把编码后的列值映射到 bitmap 位置，并支持正则表达式匹配等查询。每个 bitmap 记录包含该值的数据段。 ### 索引数据段 -GreptimeDB 把一个 SST 文件分割成多个索引数据段，每个数据段包含相同行数的数据。这种分段的目的是通过只扫描符合查询条件的数据段来优化查询性能。 +GreptimeDB 把 SST 文件分割成固定大小的索引数据段。匹配的 bitmap 会转换为 Parquet row selection，使 Mito 只读取候选行范围。 -例如，当数据段的行数为 1024，如果查询条件应用倒排索引后，得到的数据段列表为 `[0, 2]`，那么只需扫描 SST 文件中的第 0 和第 2 个数据段（即第 0 行到第 1023 行和第 2048 行到第 3071 行）即可。 +例如，每个数据段包含 1024 行且候选数据段 ID 为 `[0, 2]` 时，Mito 只扫描第 0–1023 行和第 2048–3071 行，不需要读取 SST 中的全部数据行。 -数据段的行数由引擎选项 `index.inverted_index.segment_row_count` 控制，默认为 `1024`。较小的值意味着更精确的索引，往往会得到更好的查询性能，但会增加索引存储成本。通过调整该选项，可以在存储成本和查询性能之间进行权衡。 +引擎选项 `index.inverted_index.segment_row_count` 控制目标 segment 大小，默认值为 `1024`。较小的 segment 可以提高裁剪精度，但会增加索引大小和构建成本。 ## 统一数据访问层：OpenDAL -GreptimeDB 使用 [OpenDAL][2] 提供统一的数据访问层，因此，存储引擎无需与不同的存储 API 交互，数据可以无缝迁移到基于云的存储，如 AWS S3。 +`object-store` crate 基于 [OpenDAL][2] 封装本地文件系统和对象存储。Mito 通过 `src/mito2/src/access_layer.rs` 执行 SST 与索引 I/O；存储引擎代码不应绕过该边界增加 backend-specific 路径。修改配置的 backend 不会迁移已有数据。 [1]: https://parquet.apache.org -[2]: https://github.com/datafuselabs/opendal +[2]: https://opendal.apache.org/ [3]: https://iceberg.apache.org/puffin-spec diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md index c0cbf681b9..e7d0547432 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md @@ -1,42 +1,39 @@ --- -keywords: [Metric 引擎, 逻辑表, 物理表, DDL 操作] -description: 介绍了 Metric 引擎的概念、架构及设计，重点描述了逻辑表与物理表的区别和批量 DDL 操作的实现。 +keywords: [Metric 引擎, 逻辑表, 物理表, Mito, Prometheus] +description: 介绍 Metric 引擎面向大量指标表的逻辑到物理存储模型。 --- # Metric 引擎 ## 概述 -`Metric` 引擎是 GreptimeDB 的一个组件，属于存储引擎的一种实现，主要针对可观测 metrics 等存在大量小表的场景。 +Metric 引擎是一种 `RegionEngine` 实现，面向包含大量小指标表的 Prometheus 类 workload。它把多个逻辑表复用到共享的 Mito 物理 Region 中，在保留表级读写接口的同时，降低每张表的元数据和存储开销。 -它的主要特点是利用合成的物理宽表来存储大量的小表数据，实现相同列复用和元数据复用等效果，从而达到减少小表的存储开销以及提高列式压缩效率等目标。表这一概念在 `Metric` 引擎下变得更更加轻量。 +Metric 引擎不实现另一套磁盘格式。它重写逻辑请求，再将物理存储、索引和扫描委托给 Mito。 ## 概念 -`Metric` 引擎引入了两个新的概念，分别是逻辑表与物理表。从用户视角看，逻辑表与普通表完全一样。从存储视角看，物理 Region 就是一个普通的 Region。 - ### 逻辑表 -逻辑表，即用户定义的表。与普通的表都完全一样，逻辑表的定义包括表的名称、列的定义、索引的定义等。用户的查询、写入等操作都是基于逻辑表进行的。用户在使用过程中不需要关心逻辑表和普通表的区别。 -从实现层面来说，逻辑表是一个虚拟的表，它并不直接读写物理的数据，而是通过将读写请求映射成对应物理表的请求来实现数据的存储与查询。 +逻辑表是对用户暴露的表，拥有独立的 Schema 和 Table ID。用户写入和查询都以逻辑表为目标；在内部，每个逻辑 Region 会记录实际存储数据的物理 Region。 -### 物理表 -物理表是真实存储数据的表，它拥有若干个由分区规则定义的物理 Region。 +写入时，Metric 引擎把逻辑表身份写入每一行，再将请求转发到物理数据 Region。读取时，它添加逻辑表过滤条件，只返回属于目标逻辑表的数据。 -## 架构及设计 +### 物理表 -`Metric` 引擎的主要设计架构如下： +物理表持有共享 Region。每个物理 Region 由一对 Mito Region 表示： -![Arch](/metric-engine-arch.png) +- 数据 Region，保存多个逻辑表的数据行； +- 元数据 Region，保存 Metric 引擎使用的逻辑表和逻辑列映射。 -在目前版本的实现中，`Metric` 引擎复用了 `Mito` 引擎来实现物理数据的存储及查询能力，并在此之上同时提供物理表与逻辑表的访问能力。 +直接写入物理 Region 会绕过逻辑表映射，因此会被拒绝；查询物理表仍然受支持。 -在分区方面，逻辑表拥有与物理表完全一致的分区规则及 Region 分布。这是非常自然的，因为逻辑表的数据直接存储在物理表中，所以分区规则也是一致的。 +## 架构及设计 -在路由元数据方面，逻辑表的路由地址为逻辑地址，即该逻辑表所对应的物理表是什么，而后通过该物理表进行二次路由取得真正的物理地址。这一间接路由方式能够显著减少 `Metric` 引擎的 Region 发生迁移调度时所需要修改的元数据数量。 +关联到同一物理表的逻辑表使用相同的分区布局。逻辑 Region ID 映射到对应的物理数据 Region 和元数据 Region，映射关系由 Metric 引擎及表路由元数据共同维护。 -在操作方面，`Metric` 引擎支持对逻辑表进行标准的 DML 操作（INSERT、DELETE、SELECT）。然而，对物理表的操作进行了有限的支持以防止误操作，例如禁止直接写入物理表等操作防止影响用户逻辑表的数据。总体上可以认为物理表是对用户只读的。 +`row_modifier.rs` 和 `batch_modifier.rs` 将逻辑表身份与时间序列身份编码到 Mito 内部列中。根据物理 Region 的主键编码方式，具体表示为 `__table_id` 与 `__tsid` 列，或稀疏编码的 `__primary_key`。读取路径在委托 Mito 扫描前始终添加逻辑 Table ID 条件。 -为了提升对大量表同时进行 DDL（Data Definition Language，数据操作语言）操作时性能，如 Prometheus Remote Write 冷启动时大量 metrics 带来的自动建表请求，以及前面提到的迁移物理 Region 时大量路由表的修改请求等，`Metric` 引擎引入了一些批量 DDL 操作。这些批量 DDL 操作能够将大量的 DDL 操作合并成一个请求，从而减少了元数据的查询及修改次数，提升了性能。 +Metric 引擎为影响大量逻辑表的操作提供批量 DDL 路径，避免在 Prometheus Remote Write 自动建表或物理 Region 迁移时为每张表单独修改元数据。这里的 DDL 指数据定义语言操作；逻辑表的普通插入、删除和查询仍使用标准 Region 请求路径。 -除了物理表的物理数据 Region 之外，`Metric` 引擎还额外为每一个物理数据 Region 创建了一个物理的元数据 Region，用于存储 `Metric` 引擎自身为了维护映射等状态所需要的一些元数据。这些元数据包括逻辑表与物理表的映射关系，逻辑列与物理列的映射关系等等。 +主要实现位于 `src/metric-engine/src/`。修改保留列、Region ID 转换或元数据编码会影响持久化数据，必须审查向后兼容性。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md index 3f44a4543b..6b97110cf9 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md @@ -1,28 +1,25 @@ --- -keywords: [Datanode, gRPC 服务, HTTP 服务, Heartbeat Task, Region Manager] -description: 介绍了 Datanode 的主要职责和组件，包括 gRPC 服务、HTTP 服务、Heartbeat Task 和 Region Manager。 +keywords: [Datanode, RegionServer, 存储引擎, 查询引擎, 心跳] +description: 介绍 Datanode 在 Region 级别的存储和查询职责。 --- # Datanode -## Introduction + -`Datanode` 主要的职责是为 GreptimeDB 存储数据，我们知道在 GreptimeDB 中一个 `table` 可以有一个或者多个 `Region`, -而 `Datanode` 的职责便是管理这些 `Region` 的读写。`Datanode` 不感知 `table`，可以认为它是一个 `region server`。 -所以 `Frontend` 和 `Metasrv` 按照 `Region` 粒度来操作 `Datanode`。 +## 介绍 -![Datanode](/datanode.png) +Datanode 存储表数据，并在本地 Region 上执行查询。一张表可以包含多个 Region，但 Datanode 不把表作为元数据对象管理。Frontend 和 Metasrv 通过 Region 级请求访问 Datanode，因此它的核心抽象是 Region server。 -## Components + -一个 datanode 包含了 region server 所需的全部组件。这里列出了比较重要的部分： +## 组件 -- 一个 gRPC 服务来提供对 `Region` 数据的读写，`Frontend` 便是使用这个服务来从 `Datanode` 读写数据。 -- 一个 HTTP 服务，可以通过它来获得当前节点的 metrics、配置信息等 -- `Heartbeat Task` 用来向 `Metasrv` 发送心跳，心跳在 GreptimeDB 的分布式架构中发挥着至关重要的作用， - 是分布式协调和调度的基础通信通道，心跳的上行消息中包含了重要信息比如 `Region` 的负载，如果 `Metasrv` 做出了调度 - 决定（比如 Region 转移），它会通过心跳的下行消息发送指令到 `Datanode` -- `Datanode` 不负责解析用户 SQL 或进行分布式规划，用户对一个或多个 `Table` 的查询请求会在 `Frontend` 中被转换为 - `Region` 查询请求，`Datanode` 负责用本地 query engine 执行这些 `Region` 查询计划 -- 一个 `Region Manager` 用来管理 `Datanode` 上的所有 `Region`s -- GreptimeDB 支持可插拔的多引擎架构，目前已有的 engine 包括 `File Engine` 和 `Mito Engine` +- `src/datanode/src/region_server.rs` 中的 `RegionServer` 将 Region 请求分发到已注册的存储引擎，并向查询层提供 Region 数据。 +- gRPC 服务接收 Frontend 和 Metasrv 发出的 Region 读写及生命周期操作。 +- 本地查询引擎规划并执行 Frontend 发送的逻辑子计划。Datanode 不解析客户端 SQL，也不负责协调分布式查询。 +- 心跳任务向 Metasrv 报告节点和 Region 状态，并接收 Region 打开、关闭、迁移及缓存失效等控制消息。 +- HTTP handler 提供指标、配置等运维端点。 +- Datanode 注册 Mito、Metric 和 File 三种 Region engine。Mito 是主要的时序存储引擎；Metric 面向大量指标表的场景，并将物理存储委托给 Mito；File 用于访问外部文件中的数据。 + +单机模式下，同一个 Region server 在进程内运行，不需要 Metasrv 协调。分布式模式下，Region 的可写状态和生命周期变更由 Metasrv 租约及心跳消息协调。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/python-scripts.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/python-scripts.md deleted file mode 100644 index 831278e80d..0000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/python-scripts.md +++ /dev/null @@ -1,30 +0,0 @@ ---- -keywords: [Python 脚本, 数据分析, CPython, RustPython] -description: 介绍了在 GreptimeDB 中使用 Python 脚本进行数据分析的两种后端实现：CPython 和嵌入式 RustPython 解释器。 ---- - -# Python 脚本 - -## 简介 - -Python 脚本是分析本地数据库中的数据的便捷方式， -通过将脚本直接在数据库内运行而不是从数据库拉取数据的方式，可以节省大量的数据传输时间。 -下图描述了 Python 脚本的工作原理。 -`RecordBatch`（基本上是表中的一列，带有类型和元数据）可以来自数据库中的任何地方， -而返回的 `RecordBatch` 可以用 Python 语法注释以指示其元数据，例如类型或空。 -脚本将尽其所能将返回的对象转换为 `RecordBatch`，无论它是 Python 列表、从参数计算出的 `RecordBatch` 还是常量（它被扩展到与输入参数相同的长度）。 - -![Python Coprocessor](/python-coprocessor.png) - -## 两种可选的后端 - -### CPython 后端 - -该后端由 [PyO3](https://pyo3.rs/v0.18.1/) 提供支持，可以使用您最喜欢的 Python 库（如 NumPy、Pandas 等），并允许 Conda 管理您的 Python 环境。 - -但是使用它也涉及一些复杂性。您必须设置正确的 Python 共享库，这可能有点棘手。一般来说，您只需要安装 `python-dev` 包。但是，如果您使用 Homebrew 在 macOS 上安装 Python，则必须创建一个适当的软链接到 `Library/Frameworks/Python.framework`。有关使用 PyO3 crate 与不同 Python 版本的详细说明，请参见 [这里](https://pyo3.rs/v0.18.1/building_and_distribution#configuring-the-python-version) - -### 嵌入式 RustPython 解释器 - -可以运行脚本的实验性 [python 解释器](https://github.com/RustPython/RustPython)，它支持 Python 3.10 语法。您可以使用所有的 Python 语法，更多信息请参见 [Python 脚本的用户指南](/user-guide/python-scripts/overview.md). - diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/query-engine.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/query-engine.md index 62d7563bb2..b3aba35c70 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/query-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/query-engine.md @@ -1,39 +1,35 @@ --- -keywords: [查询引擎, DataFusion, 逻辑计划, 物理计划] -description: 介绍了 GreptimeDB 的查询引擎架构，基于 Apache DataFusion 构建，涵盖逻辑计划、物理计划、优化和执行过程。 +keywords: [查询引擎, Apache DataFusion, 逻辑计划, 物理计划, Arrow, 索引] +description: 介绍 GreptimeDB 基于 DataFusion 的查询规划和执行链路。 --- # Query Engine ## 介绍 -GreptimeDB 的查询引擎是基于[Apache DataFusion][1]（属于[Apache Arrow][2]的子项目）构建的，它是一个用 Rust 编写的出色的查询引擎。它提供了一整套功能齐全的组件，从逻辑计划、物理计划到执行运行时。下面将解释每个组件如何被整合在一起，以及在执行过程中它们的位置。 +GreptimeDB 查询引擎基于 [Apache DataFusion][1] 构建。`query` crate 负责 SQL、PromQL 和日志查询规划，以及 GreptimeDB optimizer rule、物理计划和执行。 -![Execution Procedure](/execution-procedure.png) +![执行流程](/execution-procedure.png) -入口点是逻辑计划，它被用作查询或执行逻辑等的通用中间表示。逻辑计划的两个主要来源是：1. 用户查询，例如通过 SQL 解析器和规划器的 SQL；2. Frontend 的分布式查询，这将在下一节中详细解释。 +查询首先转换为 DataFusion 逻辑计划。SQL 及其他查询语言的 planner 会生成逻辑计划；分布式执行期间，Frontend 也会把序列化后的逻辑子计划发送给 Datanode。 -接下来是物理计划，或称执行计划。与包含所有逻辑计划变体（除特殊扩展计划节点外）的大型枚举的逻辑计划不同，物理计划实际上是一个定义了在执行过程中调用的一组方法的特性。所有数据处理逻辑都包装在实现该特性的相应结构中。它们是对数据执行的实际操作，如聚合器 `MIN` 或 `AVG` ，以及表扫描 `SELECT ... FROM`。 +Analyzer 和 optimizer rule 会规范化计划、下推过滤与投影、裁剪 Region，并插入 `MergeScan` 等 GreptimeDB extension node。该阶段同时使用 DataFusion 原生规则和 `src/query/src/optimizer/` 下的自定义规则。 -优化阶段通过转换逻辑计划和物理计划来提高执行性能，现在全部基于规则。它也被称为“基于规则的优化”。一些规则是 DataFusion 原生的，其他一些是在 GreptimeDB 中自定义的。在未来，我们计划添加更多规则，并利用数据统计进行基于成本的优化 (CBO)。 - -最后一个阶段"执行"是一个动词，代表从存储读取数据、进行计算并生成预期结果的过程。虽然它比之前提到的概念更抽象，但你可以简单地将它想象为执行一个 Rust 异步函数，并且它确实是一个异步流。 - -当你想知道 SQL 是如何通过逻辑计划或物理计划中表示时，`EXPLAIN [VERBOSE] ` 是非常有用的。 +物理 planner 将优化后的逻辑计划转换为 DataFusion `ExecutionPlan` 实现。执行根计划会返回异步 Arrow `RecordBatch` stream。可以使用 `EXPLAIN` 或 `EXPLAIN VERBOSE` 查看 SQL 语句对应的计划。 ## 数据表示 -GreptimeDB 使用 [Apache Arrow][2]作为内存中的数据表示格式。它是面向列的，以跨平台格式，也包含许多高性能的基础操作。这些特性使得在许多不同的环境中共享数据和实现计算逻辑变得容易。 +GreptimeDB 使用 [Apache Arrow][2] array 和 `RecordBatch` 在内存中交换数据。存储扫描、查询算子、RPC stream 和结果编码器共享同一种列式表示，避免在执行链路中逐行转换。 ## 索引 -在时序数据中，有两个重要的维度：时间戳和标签列（或者类似于关系数据库中的主键）。GreptimeDB 将数据分组到时间桶中，因此能在非常低的成本下定位和提取预期时间范围内的数据。GreptimeDB 中主要使用的持久文件格式 [Apache Parquet][3] 提供了多级索引和过滤器，使得在查询过程中很容易修剪数据。在未来，我们将更多地利用这个特性，并开发我们的分离索引来处理更复杂的用例。 +索引构建和持久化格式属于存储引擎，而不是查询引擎。Mito 使用 Parquet 统计信息、倒排索引、跳数索引和全文索引裁剪 SST 文件、row group 及数据段；feature-gated vector index 为向量搜索提供候选行。参见[数据持久化与索引](./data-persistence-indexing.md)。 + +查询层向扫描提供谓词和投影。兼容的谓词可以通过索引减少读取的数据量，但查询计划仍需执行其余过滤算子。 ## 分布式查询 -参考 [Distributed Querying][6]. +Frontend 将兼容的逻辑计划片段重写为远端 `MergeScan` 输入，使用 Substrait 序列化，再向 Datanode 发送 Region 级请求。参见[分布式查询](../frontend/distributed-querying.md)。 -[1]: https://github.com/apache/arrow-datafusion +[1]: https://datafusion.apache.org/ [2]: https://arrow.apache.org/ -[3]: https://parquet.apache.org -[6]: ../frontend/distributed-querying.md diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md index 67f98216fe..22599cd49e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md @@ -1,39 +1,24 @@ --- keywords: [存储引擎, Mito, LSMT, 数据模型, Region] -description: 详细介绍了 GreptimeDB 的存储引擎架构、数据模型和 region 的概念，重点描述了 Mito 存储引擎的优化和组件。 +description: 介绍 Mito 存储引擎的核心组件、Region 模型和 SST 数据布局。 --- # 存储引擎 ## 概述 -`存储引擎` 负责存储数据库的数据。Mito 是我们默认使用的存储引擎，基于 [LSMT][1]（Log-structured Merge-tree）。我们针对处理时间序列数据的场景做了很多优化，因此 mito 这个存储引擎并不适用于通用用途。 +Mito 是 GreptimeDB 主要的时序 Region engine，实现了 `RegionEngine` trait，并使用 [LSM tree][1] 写入链路：WAL 和 memtable 接收写入，不可变的 Parquet SST 文件保存持久化数据，后台 compaction 负责重组这些文件。 ## 架构 -下图展示了存储引擎的架构和处理数据的流程。 - -![Architecture](/storage-engine-arch.png) - -该架构与传统的 LSMT 引擎相同： - -- [WAL][2] - - 为尚未刷盘的数据提供高持久性保证。 - - 基于 `LogStore` API 实现，不关心底层存储介质。 - - WAL 的日志记录可以存储在本地磁盘上，也可以存储在实现了 `LogStore` API 的远程日志服务中，例如 Kafka（remote WAL）。 -- Memtable - - 数据首先写入 `active memtable`，又称 `mutable memtable`。 - - 当 `mutable memtable` 已满时，它将变为只读的 `immutable memtable`。 -- SST - - SST 的全名为有序字符串表（`Sorted String Table`）。 - - `immutable memtable` 刷到持久存储后形成一个 SST 文件。 - - SST 中的行按照主键和时间索引排序；详见 [SST 文件中的数据布局](#sst-文件中的数据布局)。 -- Compactor - - `Compactor` 通过 compaction 操作将小的 SST 合并为大的 SST。 - - 默认使用 [TWCS][3] 策略进行合并。Compaction 会按照时间窗口组织 SST 文件，并结合 TTL 清理过期数据。详见 [Compaction](/user-guide/deployments-administration/manage-data/compaction.md)。 -- Manifest - - `Manifest` 存储引擎的元数据，例如 SST 的元数据。 -- Cache - - 加速查询操作。 + +实现代码位于 `src/mito2/src/`。`engine.rs` 分发 Region 请求，`worker/` 负责每个 Region 的写入循环，`read/` 构建扫描，`flush.rs`、`compaction/`、`manifest/` 和 `sst/` 共同实现持久化生命周期。 + +- **WAL** 记录尚未进入 SST 的写入，用于恢复 Region 的 memtable 状态。它通过 `LogStore` API 支持本地 raft-engine 和远端 Kafka provider。写入确认对应的持久性边界取决于 provider 配置；参见[预写日志](./wal.md)。 +- **Memtable** 通过可变的 active memtable 接收写入。Flush 会将其冻结为 immutable memtable；在数据写入 SST 前，immutable memtable 仍参与读取。 +- **SST 文件**是不可变的 Parquet 文件，其中的数据按照 primary key 和 time index 排序；详见 [SST 文件中的数据布局](#sst-文件中的数据布局)。 +- **Compaction** 合并 SST 文件并清理过期数据。默认策略为 [TWCS][3]，按时间窗口组织文件。详见 [Compaction](/user-guide/deployments-administration/manage-data/compaction.md)。 +- **Manifest** 保存带版本的 Region 元数据和 SST 文件变更，用于恢复。 +- **Cache** 保存文件元数据、数据页及其他可复用的扫描状态。 [1]: https://en.wikipedia.org/wiki/Log-structured_merge-tree [2]: https://en.wikipedia.org/wiki/Write-ahead_logging @@ -41,26 +26,11 @@ description: 详细介绍了 GreptimeDB 的存储引擎架构、数据模型和 ## 数据模型 -存储引擎提供的数据模型介于 `key-value` 模型和表模型之间 - -```txt -tag-1, ..., tag-m, timestamp -> field-1, ..., field-n -``` - -每一行数据包含多个 tag 列，一个 timestamp 列和多个 field 列 -- `0 ~ m` 个 tag 列 - - tag 列是可空的 - - 在建表时通过 `PRIMARY KEY` 指定 -- 必须包含一个 timestamp 列 - - timestamp 列非空 - - 在建表时通过 `TIME INDEX` 指定 -- `0 ~ n` 个 field 列 - - field 列是可空的 -- 数据按照 tag 列和 timestamp 列有序存储 +Mito 接收由 `RegionMetadata` 描述的 schema，其中包含 primary-key column list、一个非空 time-index column 和 field columns。SQL 层把 primary-key column 暴露为 tag，Mito 本身依据 column ID 和 semantic type 工作，不解析 SQL 表定义。 -## Region +### Region -数据在存储引擎中以 `region` 的形式存储，`region` 是引擎中的一个逻辑隔离存储单元。`region` 中的行必须具有相同的 `schema`（模式），该 `schema` 定义了 `region` 中的 tag 列，timestamp 列和 field 列。数据库中表的数据存储在一到多个 `region` 中。 +Region 是 Mito 的隔离、恢复和请求单元，其中每一行都遵循该 Region 的元数据。一张表可以跨多个 Region，但表路由和放置不属于存储引擎职责。 ## SST 文件中的数据布局 @@ -68,28 +38,6 @@ tag-1, ..., tag-m, timestamp -> field-1, ..., field-n 在一个 SST 文件内，行按照 `(primary key, time index)` 排序。具有相同 primary key（tag 列）的行属于同一条时间序列，会连续存储并按时间戳排序。这种局部性使得扫描单条时间序列的成本更低，也有助于提升压缩效果。对于没有 primary key 的 append-only 表，行仅按 time index 排序。 -例如，考虑一个存储主机指标的表： - -```sql -CREATE TABLE host_metrics ( - host STRING, - region STRING, - ts TIMESTAMP TIME INDEX, - cpu DOUBLE, - memory DOUBLE, - PRIMARY KEY (host, region) -); -``` - -Mito 会按 primary key 对行分组，并按时间排序，因此 SST 中的数据在概念上类似于： - -| host | region | ts | cpu | memory | -| --- | --- | --- | --- | --- | -| host-a | us-east | 10:00 | 0.42 | 7.1 | -| host-a | us-east | 10:01 | 0.47 | 7.4 | -| host-a | us-west | 10:00 | 0.31 | 6.8 | -| host-b | us-east | 10:00 | 0.80 | 8.6 | - 除了表中的列，Mito 还会在每个 SST 文件中存储三个内部列，以便在从多个 memtable 和 SST 文件读取时正确地合并、去重并应用删除操作： - `__primary_key`：行的编码后 primary key（tags）。 @@ -108,6 +56,6 @@ Mito 会组合多个从粗到细的裁剪步骤，避免读取不可能匹配查 1. **时间范围裁剪。** 如果文件和 memtable 的时间范围与查询时间范围不相交，就会在打开 reader 之前被跳过。对于时间序列查询，这通常是成本最低且最有效的步骤。 2. **Row group 统计信息。** 如果 row group 的 min-max 统计信息能够证明没有任何行匹配谓词，则会跳过整个 row group。 -3. **索引。** 倒排索引、跳数索引和全文索引可以针对统计信息无法处理的谓词提供更精细的裁剪。详见[数据持久化和索引](data-persistence-indexing.md)。 +3. **索引。** 倒排索引、跳数索引和全文索引可以针对统计信息无法处理的谓词提供更精细的裁剪；feature-gated vector index 为向量搜索选择候选行。详见[数据持久化和索引](data-persistence-indexing.md)。 Scan pruning pipeline diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/wal.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/wal.md index 529adcdf50..0cc48c0778 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/wal.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/wal.md @@ -1,6 +1,6 @@ --- -keywords: [预写日志, WAL, 数据持久化, 同步刷盘, 异步刷盘] -description: 介绍了 GreptimeDB 的预写日志（WAL）机制，包括其命名空间、同步/异步刷盘策略和在数据节点重启时的重放功能。 +keywords: [预写日志, WAL, 恢复, raft-engine, Kafka] +description: 介绍 Mito 的 WAL 抽象、恢复路径和持久性配置。 --- # 预写日志 @@ -9,20 +9,18 @@ description: 介绍了 GreptimeDB 的预写日志（WAL）机制，包括其命 ## 介绍 -我们的存储引擎受到了日志结构合并树（Log-structured Merge Tree，LSMT）的启发。对数据的变更操作直接应用于 MemTable 而不是持久化到磁盘上的数据页，这显著提高了性能，但也带来了持久化相关的问题，特别是在 Datanode 意外崩溃时。与所有类似 LSMT 的存储引擎一样，GreptimeDB 使用预写日志（Write-Ahead Log，WAL）来确保数据被可靠地持久化，并且保证崩溃时的数据完整性。 +Mito 在把数据刷写为 SST 文件前，先将写入应用到内存中的 memtable。为了恢复尚未进入 SST 的数据，每个 Region 的写操作会先追加到预写日志（WAL），再写入 memtable。 -预写日志是一个仅提供追加写的文件组。所有的 INSERT 和 DELETE 操作都被转换为操作日志，然后追加到 WAL。一旦操作日志被持久化到底层文件，该操作才可以进一步应用到 MemTable。 +打开 Region 或重启 Datanode 时，Mito 从已持久化的最后一个 sequence 之后开始重放 WAL，重建内存状态。Sequence number 在 Region 内分配，同时用于去重和 snapshot read。 -当数据节点重新启动时，WAL 中的操作条目将被重放，以重建正确的 MemTable 状态。 - -![WAL in Datanode](/wal.png) +存储引擎通过 `LogStore` 抽象访问 WAL。Datanode 支持本地 `raft_engine` provider 和远端 Kafka provider，因此 WAL 并不等同于本地文件。Provider 在 `src/datanode/src/datanode.rs` 中构建，Mito 的 WAL 接入位于 `src/mito2/src/wal.rs` 及写入 worker。 ## 命名空间 -WAL 的命名空间用于区分来自不同 region 的条目。追加和读取操作必须提供一个命名空间。目前，region ID 被用作命名空间，因为每个 region 都有一个在数据节点重新启动时需要重构的 MemTable。 +WAL 按 Region 隔离。追加和读取操作使用 Region ID 作为 namespace，使恢复过程只重放当前 Region 的日志。一张表可以包含多个 Region，因此 WAL namespace 不是 Table ID。 ## 同步/异步刷盘 -默认情况下，WAL 的追加写是异步的，这意味着写入方不会等待操作日志被刷入到磁盘并持久化。这个默认设置提供了更高的性能，但在服务器意外关闭时可能会丢失数据。另一方面，同步刷新提供了更高的可靠性，但其代价是性能更低。 +对于本地 `raft_engine` provider，`sync_write` 控制追加操作是否等待日志同步到持久化存储，默认值为 `false`。异步写入延迟较低，但主机或存储在日志同步前发生故障时，最近已确认的 entry 可能丢失。设置 `sync_write = true` 可以加强这一持久性边界，同时会增加写入延迟。 -在 v0.4 版本中，新的 region worker 架构可以使用批处理来减轻同步刷盘的开销。 +Kafka WAL 的持久性取决于 Kafka producer 和集群配置，而不是本地 `sync_write` 选项。无论使用哪一种 provider，确认写入的代码都必须保持先追加 WAL、再修改 memtable 的顺序。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/arrangement.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/arrangement.md index dd3b6de090..b7aae65d6d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/arrangement.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/arrangement.md @@ -1,18 +1,14 @@ --- -keywords: [Arrangement, 状态存储, 键值对] -description: 描述了 Arrangement 在数据流进程中的状态存储功能，包括键值对存储、查询和删除操作的实现。 +keywords: [旧流处理模式, Arrangement, 状态, 差分更新, watermark] +description: 介绍 Flownode 旧 streaming 路径使用的内存 Arrangement 状态。 --- # Arrangement -Arrangement 存储数据流进程中的状态，存储 flow 的更新流（stream）以供进一步查询和更新。 +`Arrangement` 是 Flownode 旧 streaming 路径使用的内存状态索引，实现在 `src/flow/src/utils.rs` 中；batching 模式不使用它。 -Arrangement 本质上存储的是带有时间戳的键值对。 -在内部，Arrangement 接收类似 `((Key Row, Value Row), timestamp, diff)` 的 tuple，并将其存储在内存中。 -你可以使用 `get(now: Timestamp, key: Row)` 查询某个时间的键值对。 -Arrangement 假定早于某个时间（也称为 Low Watermark）的所有内容都已被写入到 sink 表中，不会为其保留历史记录。 +Arrangement 以 `((key row, value row), timestamp, diff)` 保存更新。`timestamp` 按 dataflow 时间排列变更，差分值 `diff` 用于添加或删除 value。`get(now: Timestamp, key: &Row)` 返回指定时间对该 key 可见的 value。 -:::tip 注意 -Arrangement 允许通过将传入 tuple 的 `diff` 设置为 -1 来删除键。 -此外，如果已将行数据添加到 Arrangement 并且使用不同的值插入相同的键，则原始值将被新值覆盖。 -::: +Low watermark 表示仍可能需要保留历史状态的最早时间。早于该 watermark 的状态被视为已经写入 sink，可以进行压缩。过早推进 watermark 会使后续差分更新无法正确合并。 + +在当前实现中，`diff` 为 `-1` 时删除 key；以不同 value 再次插入同一个 key 时，会替换原 value。这些语义属于旧 streaming 状态模型，不能套用到 batching 模式的 sink 写入。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md index bec092b0af..9e2610c0dd 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md @@ -1,74 +1,50 @@ --- -keywords: [批处理模式, flow 管理, Flownode 组件, Flownode 限制, 持续聚合] -description: Flownode 批处理模式概述，这是持续数据聚合当前使用的执行模式，包括其架构和查询执行流程。 +keywords: [批处理模式, BatchingEngine, 脏时间窗口, checkpoint, 持续聚合] +description: 介绍批处理模式的任务生命周期、脏窗口处理和恢复不变量。 --- # Flownode 批处理模式开发者指南 -本指南简要概述了 `flownode` 中的批处理模式。它旨在帮助希望了解此模式内部工作原理的开发人员。 +批处理模式通过重新执行可能受源数据变更影响的 Flow 查询来维护 sink 表，是当前持续开发的 Flownode 执行路径。模式选择属于内部行为，参见 [Flownode 概览](./overview.md)。 ## 概述 -`flownode` 中的批处理模式专为持续数据聚合而设计。它在离散的、微小的时间窗口上周期性地执行用户定义的 SQL 查询。这与原始的流处理模式形成对比；流处理模式现在已经废弃，在该模式下数据会在到达时即被处理。 +对于按时间窗口计算的 Flow，源表写入会把对应窗口标记为脏。后台任务消费这些窗口；如果查询形态允许，任务会给 Flow 查询添加时间谓词，然后把 insert plan 发送到 Frontend。Frontend 执行查询并将结果写入 sink 表。 -其核心思想是： -1. 定义一个带有 SQL 查询的 `flow`，该查询将数据从源表聚合到目标表。 -2. 查询通常在时间戳列上包含一个时间窗口函数（例如 `date_bin`）。 -3. 当新数据插入源表时，系统会将相应的时间窗口标记为“脏”(dirty)。 -4. 一个后台任务会周期性地唤醒，识别这些脏窗口，并为那些特定的时间范围重新运行聚合查询。 -5. 然后将结果插入到目标表中，从而有效地更新聚合视图。 +按固定间隔执行的 Flow 和 TQL Flow 可能需要执行完整查询，不能使用脏窗口过滤。因此，batching 路径会根据 Flow 定义，把脏窗口视为需要重算的精确范围，或视为需要执行完整查询的信号。 ## 架构 -批处理模式由几个协同工作的关键组件组成，以实现这种持续聚合。如下图所示： - -![batching mode architecture](/batching_mode_arch.png) - ### `BatchingEngine` -`BatchingEngine` 是批处理模式的核心。它是一个管理所有活动 flow 的中心组件。其主要职责是： +`src/flow/src/batching_mode/engine.rs` 中的 `BatchingEngine` 持有 `FlowId` 到 `BatchingTask` 的映射。它负责创建和删除任务、处理 flush 请求，并将脏窗口通知分发给所有读取相关源表的 Flow。 -- **任务管理**: 维护一个从 `FlowId` 到 `BatchingTask` 的映射。它处理这些任务的创建、删除和检索。 -- **事件分发**: 当新数据到达（通过 `handle_inserts_inner`）或当时间窗口被显式标记为脏（`handle_mark_dirty_time_window`）时，`BatchingEngine` 会识别受影响的 flow，并将信息转发给相应的 `BatchingTask`。 +创建任务时会解析 Flow 查询，记录源表和 sink 表，在需要时创建 sink 表，并初始化执行状态。Flow 自身的元数据由 `common-meta` 持久化。 ### `BatchingTask` -`BatchingTask` 代表一个独立的、单个的数据流。每个任务都与一个 `flow` 定义相关联，并在其自己的异步循环中运行。 +每个 `BatchingTask` 对应一个 Flow。`TaskConfig` 保存不可变的查询、表、窗口、过期时间及调度配置，`TaskState` 保存可变的执行状态。 -- **配置 (`TaskConfig`)**: 此结构体持有 flow 的不可变配置，例如 SQL 查询、源表和目标表名以及时间窗口表达式。 -- **状态 (`TaskState`)**: 包含任务的动态、可变状态，最重要的是 `DirtyTimeWindows`。 -- **执行循环**: 任务运行一个无限循环 (`start_executing_loop`)，该循环： - 1. 检查关闭信号。 - 2. 等待一个预定的时间间隔或直到被唤醒。 - 3. 基于当前的脏时间窗口集合生成一个新的查询计划 (`gen_insert_plan`)。 - 4. 对数据库执行查询 (`execute_logical_plan`)。 - 5. 清理已处理的脏窗口。 +后台循环等待调度时间或通知，生成下一次 insert plan，通过 `FrontendClient` 执行并记录结果。Execution lock 会串行化后台执行、手动 flush、计划生成和 checkpoint 更新，避免两个执行过程同时消费同一份状态。 ### `TaskState` 和 `DirtyTimeWindows` -- **`TaskState`**: 此结构体跟踪 `BatchingTask` 的运行时状态。它包括 `dirty_time_windows`，这对于确定需要完成哪些操作至关重要。 -- **`DirtyTimeWindows`**: 这是一个关键的数据结构，用于跟踪自上次查询执行以来哪些时间窗口接收到了新数据。它存储一组不重叠的时间范围。当任务的执行循环运行时，它会参考此结构来构建一个 `WHERE` 子句，该子句仅过滤源表中的脏时间窗口。 +`DirtyTimeWindows` 保存需要重新计算且互不重叠的时间范围。生成计划时会从队列中取出数量受限的一组范围。如果计划生成或执行失败，这些范围会重新放回队列；任务不会因为一次执行已经开始就直接丢弃它们。 -### `TimeWindowExpr` +`TaskState` 还为实验性的增量读取路径保存每个 Region 的 checkpoint。只有执行结果为参与查询的 Region 提供完整 watermark 证明时，增量模式才会推进 checkpoint。按范围执行 full-snapshot repair 时，任务会冻结 high watermark 并逐步处理脏窗口；期间的新写入仍保留在 live queue。Repair 失败或 watermark 证明不完整时，尚未完成的窗口会返回队列。 -`TimeWindowExpr` 是一个用于处理像 `date_bin` 这样的时间窗口表达式的辅助工具。 +增量读取通过 `experimental_enable_incremental_read` 控制，默认关闭。关闭该选项或查询形态不兼容时，任务使用 full-snapshot 执行。 -- **求值**: 它可以接受一个时间戳并对时间窗口表达式求值，以确定该时间戳所属窗口的开始和结束。 -- **窗口大小**: 它还可以从表达式中确定时间窗口的大小（持续时间）。 +### `TimeWindowExpr` -这对于标记窗口为脏以及在查询源表时生成正确的过滤条件都至关重要。 +`src/flow/src/batching_mode/time_window.rs` 中的 `TimeWindowExpr` 负责计算 `date_bin` 等窗口表达式。它把输入时间戳映射到窗口边界，并提供合并范围及生成谓词所需的窗口大小。 ## 查询执行流程 -以下是批处理模式下查询执行的简化分步演练： - -1. **数据摄取**: 新数据被写入源表。 -2. **标记为脏**: `BatchingEngine` 收到有关新数据的通知。它使用与每个相关 flow 关联的 `TimeWindowExpr` 来确定哪些时间窗口受到新数据点的影响。然后将这些窗口添加到相应 `TaskState` 中的 `DirtyTimeWindows` 集合中。 -3. **任务唤醒**: `BatchingTask` 的执行循环被唤醒，原因可能是其周期性调度，也可能是因为它被通知有大量积压的脏窗口。 -4. **计划生成**: 任务调用 `gen_insert_plan`。此方法： - - 检查 `DirtyTimeWindows`。 - - 生成一系列 `OR` 连接的 `WHERE` 子句（例如 `(ts >= 't1' AND ts < 't2') OR (ts >= 't3' AND ts < 't4') ...`），覆盖所有脏窗口。 - - 重写原始 SQL 查询以包含此新过滤器，确保只处理必要的数据。 -5. **执行**: 修改后的查询计划被发送到 `Frontend` 执行。数据库处理已过滤数据的聚合。 -6. **Upsert**: 结果被插入到目标表中。目标表通常定义了一个包含时间窗口列的主键，因此现有窗口的新结果将覆盖（upsert）旧结果。 -7. **状态更新**: `DirtyTimeWindows` 集合中刚刚处理过的窗口被清除。然后任务返回睡眠状态，直到下一个时间间隔。 +1. 源表写入或显式 mark-dirty 请求确定受影响的 Flow 和时间范围。 +2. `BatchingEngine` 将范围加入任务的 `DirtyTimeWindows`，并在需要时唤醒任务。 +3. `BatchingTask` 取出数量受限的一组窗口，构建带过滤条件的 insert plan；无法安全限定范围的 Flow 则执行完整查询。 +4. `FrontendClient` 将序列化逻辑计划发送到 Frontend。Frontend 执行查询并把结果写入 sink 表。 +5. 执行成功后，任务提交执行状态，并且只推进有返回 watermark 证明的 checkpoint；执行失败时，先恢复已取出的窗口，再等待下一次重试。 + +修改 plan coverage、checkpoint 推进或脏窗口恢复逻辑会影响正确性。尚未反映到 sink 表的工作不能被清除，checkpoint 也不能越过结果已证明包含的数据范围。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md index 9d07922542..edb39ba7f1 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md @@ -1,18 +1,14 @@ --- -keywords: [Dataflow, SQL 查询, 执行计划, 数据流, map, reduce] -description: 解释了 Dataflow 模块的核心计算功能，包括 SQL 查询转换、内部执行计划、数据流的触发运行和支持的操作。 +keywords: [旧流处理模式, Dataflow, DFIR, 差分数据, Flow] +description: 介绍 Flownode 旧 streaming 执行路径使用的内部计算图。 --- # 数据流 -Dataflow 模块（参见 `flow::compute` 模块）是 `flow` 的核心计算模块。 -它接收 SQL 查询并将其转换为 `flow` 的内部执行计划。 -然后，该执行计划被转化为实际的数据流，而数据流本质上是一个由带有输入和输出端口的函数组成的有向无环图（DAG）。 -数据流会在需要时被触发运行。 +本文说明 Flownode 旧 streaming 模式使用的计算图。新的持续聚合工作使用[批处理模式](./batching_mode.md)，不能根据本页推断 batching 行为。 -目前该数据流只支持 `map`和 `reduce` 操作，未来将添加对 `join` 等操作的支持。 +Streaming 路径通过 `src/flow/src/transform.rs` 将 Flow 定义转换为 `plan.rs` 中的 typed plan。`src/flow/src/compute/render.rs` 把受支持的 plan node 渲染为 DFIR 风格的 dataflow graph，`src/flow/src/adapter/` 下的 worker 持有并执行这些 graph。 -在内部，数据流使用 `tuple(row, time, diff)` 以行格式处理数据。 -这里 `row` 表示实际传递的数据，可能包含多个 `value` 对象。 -`time` 是系统时间，用于跟踪数据流的进度，`diff` 通常表示行的插入或删除（+1 或 -1）。 -因此，`tuple` 表示给定系统时间的 `row` 的插入/删除操作。 +内部记录使用差分行 `(row, timestamp, diff)`。`row` 保存值，`timestamp` 跟踪 dataflow 进度，`diff` 表示插入（`+1`）、删除（`-1`）等 multiplicity 变更。算子沿执行图传递这些变更，从而增量更新聚合状态和 sink 输出。 + +Typed plan 可以表示 map/filter/project、reduce、join 和 union 节点。当前 streaming renderer 可以执行 map/filter/project 和 reduce；join 与 union 的渲染仍返回 not-implemented 错误。添加算子时必须同时检查 `plan.rs` 和 `compute/render.rs`，因为能出现在计划中并不等于已经可以执行。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/overview.md index a093e564b4..e2ff865515 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/overview.md @@ -1,25 +1,28 @@ --- -keywords: [持续聚合, flow 管理, 单机模式, Flownode 组件, Flownode 限制] -description: Flownode 概览，一个为数据库提供 Flow 计算能力的组件，包括 batching mode、已废弃的 streaming mode 和核心组件。 +keywords: [Flownode, 持续聚合, 批处理模式, 流处理模式, Flow] +description: 介绍 Flownode 的执行模式、路由边界和实现目录。 --- # Flownode ## 简介 -`Flownode` 为数据库提供 Flow 计算能力。 -`Flownode` 管理 `flow`，这些 `flow` 是从 `source` 接收数据并将数据发送到 `sink` 的任务。 +Flownode 是 GreptimeDB Flow 的执行组件，负责根据源表持续计算结果并写入 sink 表。单机模式下它运行在 GreptimeDB 进程内，分布式模式下则作为独立服务运行。 -`Flownode` 支持 `standalone`（单机）和 `distributed`（分布式）两种模式。在 `standalone` 模式下，`Flownode` 与数据库运行在同一进程中。在 `distributed` 模式下，`Flownode` 运行在单独的进程中，并通过网络与数据库通信。 +Flownode 包含两条执行路径： -一个 flow 有两种执行模式： -- **批处理模式 (Batching Mode)**: 持续数据聚合当前使用的模式。它在离散的、微小的时间窗口上周期性地执行用户定义的 SQL 查询。聚合和 TQL 查询使用此模式。更多详情，请参阅[批处理模式开发者指南](./batching_mode.md)。 -- **流处理模式 (Streaming Mode，已废弃)**: 原始的模式，数据在到达时即被处理。该模式保留用于兼容旧 workload，不推荐新 workload 使用。 +- **批处理模式**是当前持续开发的路径。它跟踪受影响的时间窗口，并定期通过 Frontend 执行聚合查询。参见[批处理模式开发者指南](./batching_mode.md)。 +- **流处理模式**是旧的增量 dataflow 路径。它在 worker 持有的计算图中处理行级变更，目前仅为兼容性保留。 + +用户不能直接选择执行模式。`flow_type` 是保留的内部元数据。创建 Flow 时，`src/operator/src/statement/ddl.rs` 中的 `StatementExecutor::determine_flow_type` 决定执行模式，Flownode 内部再由 `FlowDualEngine` 完成兼容路由。 ## 组件 -`Flownode` 包含了执行一个 flow 所需的所有组件。所涉及的具体组件取决于执行模式。在较高的层面上，关键部分包括： +- `src/flow/src/engine.rs` 中的 `FlowEngine` 定义两条路径共用的创建、删除、flush 和 insert 生命周期。 +- `src/flow/src/adapter/flownode_impl.rs` 中的 `FlowDualEngine` 把每个 Flow 路由到 batching engine 或 streaming engine。 +- `src/flow/src/batching_mode/` 包含时间窗口跟踪、任务调度、Frontend RPC、sink 表创建和 checkpoint 逻辑。 +- `src/flow/src/adapter/`、`compute/`、`expr/` 和 `plan.rs` 实现旧的 streaming 路径。 +- `src/flow/src/server.rs` 提供 Flownode gRPC 服务；`heartbeat.rs` 向 Metasrv 报告 Flownode 状态。 +- 持久化 Flow 元数据和 DDL Procedure 位于 `src/common/meta/`，不属于 `flow` crate。 -- **Flow Manager**: 一个负责管理所有 flow生命周期的中心组件。 -- **Task Executor**: flow 逻辑执行的运行时环境。在批处理模式下，它是一个 `BatchingTask`；在已废弃的流处理模式下，这通常是一个 `FlowWorker`。 -- **Flow Task**: 代表一个独立的、单个的数据流，包含将数据从 source 转换为 sink 的逻辑。 +修改某一种执行模式时，必须同时检查 `FlowDualEngine` 和共享元数据契约，不能默认一条路径的修复也适用于另一条路径。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/distributed-querying.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/distributed-querying.md index 612174662b..e5734b5945 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/distributed-querying.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/distributed-querying.md @@ -1,41 +1,24 @@ --- -keywords: [分布式查询, 查询拆分, 查询合并, TableScan, 物理计划] -description: 介绍 GreptimeDB 中的分布式查询方法，包括查询的拆分和合并过程，以及 TableScan 节点的作用。 +keywords: [分布式查询, DistPlannerAnalyzer, MergeScan, Substrait, Region 裁剪] +description: GreptimeDB 如何将逻辑查询计划划分为本地和远端执行阶段。 --- # 分布式查询 -我们知道在 GreptimeDB 中数据是如何分布的（参见“[表分片][1]”），那么如何查询呢？在 GreptimeDB 中，分布式查询非常简单。简单来说，我们只需将查询拆分为子查询，每个子查询负责查询表数据的一个部分，然后将所有结果合并为最终结果。这是一种典型的“拆分 - 合并”方法。具体来说，让我们从查询到达 `frontend` 开始。 +Frontend 和 Datanode 使用同一套基于 DataFusion 的查询引擎。在分布式模式下，Frontend 会增加一个规划步骤，将 Datanode 上执行的工作与 Frontend 上完成的工作分开。 -当查询到达 `frontend` 时，它首先被解析为 SQL 抽象语法树（AST）。我们遍历 AST，并从中生成逻辑计划。顾名思义，逻辑计划只是如何“逻辑地”执行查询的“提示”，它不能被直接运行，因此我们进一步从中生成可执行的物理计划。物理计划是一种类似树形的数据结构，每个节点实际上表示查询的执行方法。一旦我们从上到下运行物理计划树，结果数据将从叶子到根流动，被合并或计算。最终，我们在根节点的输出处得到了查询的结果。 +![Frontend query](/frontend-query.png) -到目前为止，这只是一个典型的“volcano”查询执行模型，你可以在几乎每个 SQL 数据库中看到这种模型。那么“分布式”是在哪里发生的呢？这全部发生在一个名为“TableScan”的物理计划节点中。TableScan 是物理计划树中的一个叶子节点，它负责扫描表的数据（就像它的名称所暗示的）。当 `frontend` 即将扫描表时，它首先需要根据每个 `region` 的数据范围将表扫描拆分为较小的扫描。 +## 分布式规划器 -[1]: ./table-sharding.md +`src/query/src/dist_plan/analyzer.rs` 中的 `DistPlannerAnalyzer` 会重写 DataFusion 逻辑计划。它将可下推的算子移向表扫描，并用 `MergeScan` 节点包装远端子计划。规划器根据算子的交换律和计划形态判断哪些工作可以安全地在各 Datanode 执行；不支持的计划形态保留在 Frontend，或使用配置允许的 fallback 路径。 -表的所有 `region` 都有它们存储数据的范围。以下表为例： +分区列上的过滤条件同时用于裁剪 Region。执行前，Frontend 通过 `FrontendRegionQueryHandler` 将每个入选 Region 解析到对应 Datanode。 -```sql -CREATE TABLE my_table ( - a INT, - others STRING, - ts TIMESTAMP TIME INDEX, -) -PARTITION ON COLUMNS (a) ( - a < 10, - a >= 10 AND a < 20, - a >= 20 -); -``` +初始设计及交换律规则参见[分布式规划器 RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md)。 -`my_table` 表创建时被设定了 3 个分区。在 GreptimeDB 的当前实现中，将为该表创建 3 个 `region`（分区与 `region` 的比例为 1:1）。这 3 个区域将分别包含以下范围："[-∞, 10)", "[10, 20)" 和 "[20, +∞)"。例如，如果提供了值 "42"，我们将搜索这些范围，并找到包含该值的相应的 `region`（在此示例中为第 3 个 `region`）。 +## 分布式计划 -对于查询，我们使用“过滤器”来查找 `region`。 "过滤器"是 "WHERE" 子句中的条件。例如，查询 `SELECT * FROM my_table WHERE a < 10 AND others = 'x'`，其“过滤器”为“a < 10 AND others = 'x'”。然后我们检查这些范围，找出包含满足过滤器条件的值的所有 `region`。 +`MergeScan` 的远端输入是一个完整的逻辑子计划。Frontend 使用 Substrait 对子计划进行序列化，并向选定的 Datanode 发送 Region 级查询请求。Datanode 针对本地 Region 规划并执行该子计划，再以 Arrow RecordBatch stream 返回结果。 -> 如果某个查询没有任何过滤器，则将其视为全表扫描。 - -找到所需的区域后，我们只需在其中组装子扫描。通过这种方式，我们将查询拆分为子查询，每个子查询都获取表数据的一部分。子查询在 `datanode` 中执行，并在 `frontend` 中等待完成。它们的结果将合并为表扫描请求的最终返回。 - -下面这张图片总结了分布式查询执行的过程： - -![Distributed Querying](/distributed-querying.png) +Frontend 合并远端数据流，并执行无法下推的算子。这个边界并不局限于逻辑计划中的 `TableScan` 节点：过滤、投影、部分聚合以及其他兼容算子都可能进入远端子计划。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/overview.md index ee48da3ce3..adbf0e5b13 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/overview.md @@ -1,43 +1,46 @@ --- -keywords: [frontend, proxy, protocol, routing, distributed query, tenant management, authorization, flow control, cloud deployment, endpoints] -description: GreptimeDB Frontend 组件概述 - 为客户端请求提供服务的无状态代理服务。 +keywords: [Frontend, 协议, 请求路由, 分布式查询, 权限校验] +description: GreptimeDB 无状态请求入口和查询协调组件 Frontend 的实现概览。 --- # Frontend -**Frontend** 是一个无状态服务，作为 GreptimeDB 中客户端请求的入口点。它为多种数据库协议提供统一接口，并充当代理，将读写请求转发到分布式系统中的相应 Datanode。 +Frontend 是 GreptimeDB 的无状态请求入口和编排层。它实现协议服务背后的业务逻辑，负责查询规划、写入与 Region 读取路由，以及分布式查询协调。 -## 核心功能 +网络监听和 wire format 属于 `servers` crate。`frontend` crate 为 SQL、gRPC、MySQL、PostgreSQL、InfluxDB、OpenTelemetry、Prometheus、OpenTSDB、Jaeger 等接口实现对应的 handler trait。 -- **协议支持**：支持多种数据库协议，包括 SQL、PromQL、MySQL 和 PostgreSQL。详见[协议][1] -- **请求路由**：基于元数据将请求路由到相应的 Datanode -- **查询分发**：将分布式查询拆分到多个节点 -- **响应聚合**：合并来自多个 Datanode 的结果 -- **认证授权**：安全和访问控制验证 + + +## 职责 + +- 解析和规划 SQL、PromQL 及日志查询。 +- 校验权限，并在请求处理链路中传递 session context。 +- 使用 Catalog 和路由元数据分发插入、删除及 Region 查询。 +- 将分布式查询片段发送到 Datanode，并合并执行结果。 + +面向用户的接口参见[协议概览](/user-guide/protocols/overview.md)。 ## 架构 ### 关键组件 -- **协议处理器**：处理不同的数据库协议 -- **目录管理器**：缓存来自 Metasrv 的元数据以实现高效的请求路由和 Schema 校验 -- **分布式规划器**：将逻辑计划转换为分布式执行计划 -- **请求路由器**：为每个请求确定目标 Datanodes + +- `src/frontend/src/instance.rs` 中的 `Instance` 是主要业务逻辑容器，实现各类 server handler trait。 +- `src/frontend/src/instance/` 下的模块处理不同请求类型和协议。 +- `operator` crate 中的 `StatementExecutor` 负责语句及写入侧操作。 +- `query` crate 负责逻辑计划、优化和分布式计划。 +- `instance/region_query.rs` 中的 `FrontendRegionQueryHandler` 解析 Region 目标并向 Datanode 发送查询请求。 ### 请求流程 -![request flow](/request_flow.png) +单机模式下，Frontend 通过本地 `RegionServer` adapter 访问内嵌的 Datanode。分布式模式下，Frontend 使用 Metasrv 提供的元数据和 RPC client 访问远端 Datanode。 ### 部署 -下图是 GreptimeDB 在云上的一个典型的部署。`Frontend` 实例组成了一个集群处理来自客户端的请求： - -![frontend](/frontend.png) +Frontend 不持有表数据。多个 Frontend 实例可以共同服务同一组 Metasrv 和 Datanode。 -## 详细信息 + -- [表分片][2] -- [分布式查询][3] +## 实现指南 -[1]: /user-guide/protocols/overview.md -[2]: ./table-sharding.md -[3]: ./distributed-querying.md +- [表分片](./table-sharding.md) +- [分布式查询](./distributed-querying.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/table-sharding.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/table-sharding.md index 63a8ac10a6..b24605d670 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/table-sharding.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/table-sharding.md @@ -5,7 +5,7 @@ description: 介绍 GreptimeDB 中表数据的分片方法，包括分区和 Reg # 表分片 -对于任何分布式数据库来说，数据的分片都是必不可少的。本文将描述 GreptimeDB 中的表数据如何进行分片。 +GreptimeDB 将表拆分为分区，并把每个分区存储在一个 Region 中。本文说明这两个对象在实现上的关系。 @@ -15,10 +15,7 @@ description: 介绍 GreptimeDB 中表数据的分片方法，包括分区和 Reg ## Region -在创建分区后，表中的数据被逻辑上分割。你可能会问："在 GreptimeDB 中，被逻辑上分区的数据是如何存储的？" 答案是保存在 `Region` 当中。 - -每个 `Region` 对应一个分区，并保存分区的数据。所有的 `Region` 分布在各个 `Datanode` 之中。 -`Metasrv` 管理 `Region` 到 `Datanode` 的路由信息。如果建表后需要调整分区布局， +每个分区对应一个 Region。Region 是由 Datanode 管理的存储和调度单元，Metasrv 保存 Region 到 Datanode 的路由信息。如果建表后需要调整分区布局， GreptimeDB 支持通过显式的 [repartition](/user-guide/deployments-administration/manage-data/repartition.md) 操作拆分或合并分区。 分区和 Region 的关系参见下图： @@ -41,7 +38,7 @@ GreptimeDB 支持通过显式的 [repartition](/user-guide/deployments-administr │ P0 │ │ P1 │ │ Px │ └─────┬─────┘ └─────┬─────┘ └─────┬─────┘ │ │ │ - │ │ │ + │ │ │ ┌───────┼──────────────────┼───────┐ │ Partition 和 Region 是一一对应的 │ │ │ │ │ │ ┌─────▼─────┐ ┌─────▼─────┐ │ ┌─────▼─────┐ diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/getting-started.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/getting-started.md index e5aa2ca855..6335f80fbb 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/getting-started.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/getting-started.md @@ -1,35 +1,28 @@ --- -keywords: [编译, 运行, 源代码, 系统要求, 依赖项, Docker] -description: 介绍如何在本地环境中从源代码编译和运行 GreptimeDB，包括系统要求和依赖项。 +keywords: [开发环境, 源码构建, Rust 工具链, 单元测试] +description: 配置开发环境，并从源码构建、运行和测试 GreptimeDB。 --- # 立即开始 -本页面介绍如何在本地环境中从源代码运行 GreptimeDB。 +本页说明从源码构建和运行 GreptimeDB 所需的基本环境。 ## 先决条件 ### 系统和架构 -目前，GreptimeDB 支持 Linux（amd64 和 arm64）、macOS（amd64 和 Apple Silicon）和 Windows。 +GreptimeDB 支持 x86-64 和 Arm64 架构的 Linux 与 macOS，也支持 Windows。 ### 构建依赖项 -- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line)（可选） -- C/C++ 工具链：提供编译和链接的基本工具。在 Ubuntu 上，这可用作 `build-essential`。在其他平台上，也有类似的命令。 -- Rust nightly 工具链（[指南][1]） - - 编译源代码 -- Protobuf（[指南][2]） - - 编译 proto 文件 - - 请注意，版本需要 >= 3.15。你可以使用 `protoc --version` 检查它。 -- 机器：建议内存在 16GB 以上 或者 使用[mold](https://github.com/rui314/mold)工具以降低链接时的内存使用。 - -[1]: -[2]: +- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line)。 +- C/C++ 构建工具链，例如 Ubuntu 上的 `build-essential` 或 macOS 上的 Xcode Command Line Tools。 +- [Rustup](https://rustup.rs/)。仓库中的 `rust-toolchain.toml` 会自动选择项目要求的 nightly 工具链。 +- 3.15 或更高版本的 [Protocol Buffers 编译器](https://grpc.io/docs/protoc-installation/)。使用 `protoc --version` 检查版本。 ## 编译和运行 -只需几个命令即可使用以 Standalone 模式启动 GreptimeDB 实例： +克隆仓库并启动单机实例： ```shell git clone https://github.com/GreptimeTeam/greptimedb.git @@ -37,34 +30,30 @@ cd greptimedb cargo run -- standalone start ``` -接下来，你可以选择与 GreptimeDB 交互的协议。 - -如果你只想构建服务器而不运行它： +只构建、不启动服务时运行： ```shell -cargo build # --release +cargo build ``` -根据构建的模式（是否传递了 `--release` 选项），构建后的文件可以在 `$REPO/target/debug` 或 `$REPO/target/release` 目录下找到。 +优化构建请添加 `--release`。构建产物位于 `target/debug` 或 `target/release`。 ## 单元测试 -GreptimeDB 经过了充分的测试，整个单元测试套件都随源代码一起提供。要测试它们，请使用 [nextest](https://nexte.st/index.html)。 - -要使用 cargo 安装 nextest，请运行： +GreptimeDB 使用 [cargo-nextest](https://nexte.st/) 作为标准 Rust 测试运行器。安装命令如下： ```shell cargo install cargo-nextest --locked ``` -或者，你可以查看他们的[文档](https://nexte.st/docs/installation/pre-built-binaries/)以了解其他安装方式。 - -安装好 nextest 后，你可以使用以下命令运行测试套件： +使用 CI 对应的 feature 运行 workspace 测试： ```shell cargo nextest run --workspace --features pg_kvbackend,mysql_kvbackend ``` +按 crate 运行测试以及其他测试类型参见[测试指南](./tests/overview.md)。 + ## Docker -我们还通过 Docker 提供预构建二进制文件，可以在 [Docker Hub 上获取](https://hub.docker.com/r/greptime/greptimedb)。 +预构建镜像发布在 [Docker Hub](https://hub.docker.com/r/greptime/greptimedb)。镜像适合直接运行 GreptimeDB；开发和验证代码改动时仍应使用源码构建。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-trace-greptimedb.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-trace-greptimedb.md index 6c1d922ac6..27c85cc9b2 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-trace-greptimedb.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-trace-greptimedb.md @@ -1,30 +1,31 @@ --- -keywords: [tracing, 分布式追踪, tracing 上下文, RPC 调用, 代码埋点] -description: 介绍如何在 GreptimeDB 中使用 Rust 的 tracing 框架进行代码埋点，包括在 RPC 中定义和传递 tracing 上下文的方法。 +keywords: [tracing, W3C Trace Context, RPC, instrument, runtime] +description: 介绍 GreptimeDB 代码中的分布式 trace 传递和埋点方法。 --- # How to trace GreptimeDB -GreptimeDB 使用 Rust 的 [tracing](https://docs.rs/tracing/latest/tracing/) 框架进行代码埋点，tracing 的具体原理和使用方法参见 tracing 的官方文档。 +GreptimeDB 使用 Rust [`tracing`](https://docs.rs/tracing/latest/tracing/) 生态和 OpenTelemetry context propagation。只有 tracing context 沿同一条异步执行路径传递时，本地 span 才会自动建立父子关系；跨 RPC 或 runtime 时必须显式传递。 -通过将 `trace_id` 等信息在整个分布式数据链路上透传，使得我们能够记录整个分布式链路的函数调用链，知道每个被追踪函数的调用时间等相关信息，从而对整个系统进行诊断。 +公共实现在 `common-telemetry` 的 [`TracingContext`](https://github.com/GreptimeTeam/greptimedb/blob/main/src/common/telemetry/src/tracing_context.rs) 中，负责在当前 span context 和 W3C Trace Context 字段之间转换。 -## 在 RPC 中定义 tracing 上下文 + -因为 tracing 框架并没有原生支持分布式追踪，我们需要手动将 `trace_id` 等信息在 RPC 消息中传递，从而正确的识别函数的调用关系。我们使用基于 [w3c 的标准](https://www.w3.org/TR/trace-context/#traceparent-header-field-values) 将相关信息编码为 `tracing_context` ，将消息附在 RPC 的 header 中。主要定义在： +## RPC 中的 context 字段 -- `frontend` 与 `datanode` 交互：`tracing_context` 定义在 [`RegionRequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/region/server.proto) 中 -- `frontend` 与 `metasrv` 交互：`tracing_context` 定义在 [`RequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/meta/common.proto) 中 -- Client 与 `frontend` 交互：`tracing_context` 定义在 [`RequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/common.proto) 中 +GreptimeDB 的 protobuf header 使用 `map tracing_context` 保存 W3C trace 字段： -## 在 RPC 调用中传递 tracing 上下文 +- Frontend 到 Datanode：[`RegionRequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/region/server.proto) +- Meta client 和 service：[`RequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/meta/common.proto) +- Client 到 Frontend database RPC：[`RequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/common.proto) -我们构建了一个 `TracingContext` 结构体，封装了与 tracing 上下文有关的操作。[相关代码](https://github.com/GreptimeTeam/greptimedb/blob/main/src/common/telemetry/src/tracing_context.rs) +增加内部 RPC 时应尽量复用已有 header type。另建 tracing 字段或采用不同编码，会产生公共 helper 无法处理的传递路径。 -GreptimeDB 在使用 `TracingContext::from_current_span()` 获取当前 tracing 上下文，使用 `to_w3c()` 方法将 tracing 上下文编码为符合 w3c 的格式，并将其附在 RPC 消息中，从而使 tracing 上下文正确的在分布式组件之中传递。 + -下面的例子说明了如何获取当前 tracing 上下文，并在构造 RPC 消息时正确传递参数，从而使 tracing 上下文正确的在分布式组件之中传递。 +## 跨 RPC 传递 context +构造出站请求时获取当前 context： ```rust let request = RegionRequest { @@ -36,45 +37,52 @@ let request = RegionRequest { }; ``` -在 RPC 消息的接收方，需要将 tracing 上下文正确解码，并且使用该上下文构建第一个 `span` 对函数调用进行追踪。比如下面的代码就将接收到的 RPC 消息中的 `tracing_context` 使用 `TracingContext::from_w3c` 方法正确解码。并使用 `attach` 方法将新建的 `info_span!("RegionServer::handle_read")`  附上了上下文消息，从而能够跨分布式组件对调用进行追踪。 +接收端解析 header，并把新的本地 span 挂到该 context 下： ```rust -... let tracing_context = request .header .as_ref() - .map(|h| TracingContext::from_w3c(&h.tracing_context)) + .map(|header| TracingContext::from_w3c(&header.tracing_context)) .unwrap_or_default(); + let result = self .handle_read(request) .trace(tracing_context.attach(info_span!("RegionServer::handle_read"))) .await?; -... ``` -## 使用 `tracing::instrument` 对监测代码进行埋点 +Header 缺失或 context 无效时会得到空 context，请求仍可在没有 parent trace 的情况下执行。不能把一个请求的 context 复用于无关工作。 + + + +## 使用 `tracing::instrument` 创建 span -我们使用 tracing 提供的 `instrument` 宏对代码进行埋点，只要将 `instrument` 宏标记在需要进行埋点的函数即可。 `instrument` 宏会每次将函数调用的参数以 `Debug` 的形式打印到 span 中。对于没有实现 `Debug` trait 的参数，或者结构体过大、参数过多，最后导致 span 过大，希望避免这些情况就需要使用 `skip_all`，跳过所有的参数打印。 +在异步边界或开销较大的操作上使用 `#[tracing::instrument]`，便于关联延迟和错误。该宏默认通过 `Debug` 记录参数。凭据、token、大 batch、查询 payload 以及不适合进入 telemetry 的完整参数必须跳过。 ```rust -#[tracing::instrument(skip_all)] -async fn instrument_function(....) { - ... +#[tracing::instrument(skip_all, fields(region_id = %region_id))] +async fn handle_region(region_id: RegionId, request: RegionRequest) { + region_server.handle(request).await; } ``` -## 跨越 runtime 的代码埋点 +`fields(...)` 中应记录少量稳定标识符，不要记录完整请求。为每个 helper 都添加 span 会增加大量 trace 数据，却不能改善请求级调用链。 + + + +## 跨 runtime 传递 context -Rust 的 tracing 库会自动处理埋点函数间的嵌套关系，但如果某个函数的调用跨越 runtime 的话，tracing 不能自动对这类调用进行追踪，我们需要手动跨越 runtime 去传递上下文。 +把 future 移到另一个 runtime，或在当前 instrumented future 之外 spawn 任务时，可能丢失当前 parent。跨越边界前先获取 context，再在新 future 中挂载 span： ```rust let tracing_context = TracingContext::from_current_span(); let handle = runtime.spawn(async move { handler .handle(query) - .trace(tracing_context.attach(info_span!("xxxxx"))) - ... + .trace(tracing_context.attach(info_span!("background_query"))) + .await }); ``` -比如上面这段代码需要跨越 runtime 去进行 tracing，我们先通过 `TracingContext::from_current_span()` 获取当前 tracing 上下文，通过在另外一个 runtime 里新建一个 span，并将 span 附着在当前上下文中，我们就完成了跨越 runtime 的代码埋点，正确追踪到了调用链。 +Context 必须在 spawn 前获取。挂载的 span 只应覆盖该异步操作，避免无关任务继承同一个 parent。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-use-tokio-console.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-use-tokio-console.md index e32c4f3a55..c66257fc70 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-use-tokio-console.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-use-tokio-console.md @@ -1,34 +1,30 @@ --- -keywords: [tokio-console, GreptimeDB, 构建配置, 启动配置, 调试工具] -description: 介绍如何在 GreptimeDB 中启用 tokio-console，包括构建和启动时的配置方法。 +keywords: [tokio-console, tokio_unstable, 异步任务, 诊断] +description: 构建启用 tokio-console 的 GreptimeDB 并检查 Tokio runtime。 --- # 如何在 GreptimeDB 中启用 tokio-console -本文介绍了如何在 GreptimeDB 中启用 [tokio-console](https://github.com/tokio-rs/console)。 +[`tokio-console`](https://github.com/tokio-rs/console) 用于查看实时 Tokio task 和 resource。GreptimeDB 通过 `cmd/tokio-console` feature 编译 subscriber，同时要求启用 Tokio 的 unstable instrumentation cfg。 -首先，在构建 GreptimeDB 时带上 feature `cmd/tokio-console`。同时 `tokio_unstable` cfg 也必须开启： +使用以下命令构建： ```bash RUSTFLAGS="--cfg tokio_unstable" cargo build -F cmd/tokio-console ``` -启动 GreptimeDB，可设置 tokio console 绑定的地址，配置是 `--tokio-console-addr`。例如： +启动组件时为 console subscriber 指定完整 socket address： ```bash -greptime --tokio-console-addr="127.0.0.1:6669" standalone start +./target/debug/greptime --tokio-console-addr="127.0.0.1:6669" standalone start ``` -这样就可以使用 `tokio-console` 命令去连接 GreptimeDB 的 tokio console 服务了： +该参数是全局参数，也可以用于以相同 feature 构建的 `frontend`、`datanode`、`metasrv` 或 `flownode` 命令。 + +按照 [tokio-console 仓库](https://github.com/tokio-rs/console#installing-the-console)的说明安装 client，再连接到配置地址： ```bash -tokio-console [TARGET_ADDR] +tokio-console http://127.0.0.1:6669 ``` -"`TARGET_ADDR`" 默认是 "\"。 - -:::tip Note - -`tokio-console` 命令的安装方法参见 [tokio-console](https://github.com/tokio-rs/console)。 - -::: +Subscriber 应绑定到 loopback 或其他受保护的地址。它是诊断端点，不是公开的 GreptimeDB 协议。该 feature 和 `tokio_unstable` instrumentation 会增加 runtime 诊断信息，只应在排查 task 阻塞、唤醒或资源争用时按需启用。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-write-sdk.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-write-sdk.md index 28ff757763..abb783a704 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-write-sdk.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-write-sdk.md @@ -1,42 +1,31 @@ --- -keywords: [gRPC SDK, GreptimeDatabase, GreptimeRequest, GreptimeResponse, 插入请求] -description: 介绍如何为 GreptimeDB 开发一个 gRPC SDK，包括 GreptimeDatabase 服务的定义、GreptimeRequest 和 GreptimeResponse 的结构。 +keywords: [gRPC ingester SDK, GreptimeDatabase, RowInsertRequests, streaming RPC] +description: 介绍 GreptimeDB gRPC ingester SDK 的协议和可靠性要求。 --- # 如何为 GreptimeDB 开发一个 gRPC SDK -GreptimeDB 的 gRPC SDK 只需要处理写请求即可。读请求是标准 SQL 或 PromQL，可以由任何 JDBC 客户端或 Prometheus -客户端处理。这也是为什么所有的 GreptimeDB SDK 都命名为 "`greptimedb-ingester-`"。请确保你的 GreptimeDB SDK -遵循相同的命名约定。 +本文面向基于 GreptimeDB 原生 gRPC database service 的 **ingester SDK**，不涵盖查询 driver 和 client。官方写入库统一采用 `greptimedb-ingester-` 命名。 -## `GreptimeDatabase` 服务 +消息和 client 代码应由版本化的 [greptime-proto](https://github.com/GreptimeTeam/greptime-proto) 定义生成，不要在 SDK 中复制 message layout。生成的协议 package 应与提供给应用代码的 row 和 batch API 分离。 -GreptimeDB 自定义了一个 gRPC 服务：`GreptimeDatabase` -。你只需要实现这个服务即可。你可以在[这里](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto) -找到它的 Protobuf 定义。 +## `GreptimeDatabase` 服务 -`GreptimeDatabase` 有 2 个 RPC 方法： +[`database.proto`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto) 定义了两个 RPC 方法： ```protobuf service GreptimeDatabase { rpc Handle(GreptimeRequest) returns (GreptimeResponse); - rpc HandleRequests(stream GreptimeRequest) returns (GreptimeResponse); } ``` -`Handle` 方法是一个 unary 调用：当 GreptimeDB 服务接收到一个 `GreptimeRequest` 请求后，它立刻处理该请求并返回一个相应的 -`GreptimeResponse`。 +`Handle` 是 unary RPC。`HandleRequests` 是 [client-streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc)：client 发送一组 request stream，关闭发送端，再接收一个汇总 response。生产 SDK 应使用有界缓冲和 gRPC flow control，不能在内存中无限累积 batch。 -`HandleRequests` 方法则是一个 "[Client Streaming RPC][3]" 方式的调用。 -它可以接受一个连续的 `GreptimeRequest` 请求流，持续地发给 GreptimeDB 服务。 -GreptimeDB 服务会在收到流中的每个请求时立刻进行处理，并最终（流结束时）返回一个总结性的 `GreptimeResponse`。 -通过 `HandleRequests`，我们可以获得一个非常高的请求吞吐量。 +协议没有 request 级 idempotency key，因此 SDK 不能承诺 exactly-once ingestion。传输故障导致服务端结果未知时，自动重试可能在保留重复行的表配置中写入重复数据；重试行为必须显式暴露给调用方。 ### `GreptimeRequest` -`GreptimeRequest` 是一个 Protobuf 消息，定义如下： - ```protobuf message GreptimeRequest { RequestHeader header = 1; @@ -51,23 +40,21 @@ message GreptimeRequest { } ``` -`RequestHeader` 是必需，它包含了一些上下文，鉴权和其他信息。"oneof" 的字段包含了发往 GreptimeDB 服务的请求。 +写入优先使用 `RowInsertRequests`。每个 `RowInsertRequest` 指定一张表，并携带一个 `Rows` schema 和对应数据行。发送前应校验列数、数据类型、semantic type 及 null 表示，避免 client 构造错误变成难以定位的服务端错误。较早的列式 `InsertRequests` 仍作为兼容协议保留。 -注意我们有两种类型的插入请求，一种是以 "列" 的形式（`InsertRequests`），另一种是以 "行" 的形式（`RowInsertRequests` -）。通常我们建议使用 "行" 的形式，因为它对于表的插入更自然，更容易使用。但是，如果需要一次插入大量列，或者有大量的 "null" -值需要插入，那么最好使用 "列" 的形式。 +每个 request 都包含 `RequestHeader`。SDK 配置指定相关值时，应填写目标 Catalog、Schema、认证 header、时区和 W3C tracing context。调用方显式选择 Catalog 或 Schema 后，不能静默替换为 client 默认值。 ### `GreptimeResponse` -`GreptimeResponse` 是一个 Protobuf 消息，定义如下： - ```protobuf message GreptimeResponse { ResponseHeader header = 1; - oneof response {AffectedRows affected_rows = 2;} + oneof response { + AffectedRows affected_rows = 2; + } } ``` -`ResponseHeader` 包含了返回值的状态码，以及错误信息（如果有的话）。"oneof" 的字段目前只有 "affected rows"。 +gRPC 传输成功不代表数据库操作成功。SDK 必须检查 `ResponseHeader.status`，把非成功 status code 和 `err_msg` 转换为 SDK error，随后才能返回 `affected_rows`。底层 gRPC status 应与 GreptimeDB response status 分开保留，使调用方能够区分传输故障和服务端请求错误。 -GreptimeDB 现在有很多 SDK，你可以参考[这里](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=)获取一些示例。 +Protobuf client 还必须容忍未知字段及未设置的 response variant。兼容性测试应覆盖支持版本的序列化消息；集成测试应覆盖 unary 写入、client streaming、认证错误、stream 中途失败和服务端 status 传递。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/admin-api.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/admin-api.md index 3ac3d44f3b..640d58d5a7 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/admin-api.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/admin-api.md @@ -1,153 +1,53 @@ --- keywords: [Admin API, 健康检查, leader 查询, 心跳检测, 维护模式] -description: 介绍 Metasrv 的 Admin API，包括健康检查、leader 查询、心跳检测、维护模式和 Procedure Manager 控制等功能。 +description: 面向维护者的 Metasrv Admin API router 及状态修改端点参考。 --- # Admin API -Admin 提供了一种简单的方法来查看和管理集群信息，包括 metasrv 健康检测、metasrv leader 查询、数据节点心跳检测、维护模式和 Procedure Manager 控制。 +Axum router 在 `src/meta-srv/src/service/admin.rs` 中组装，并挂载到 Metasrv HTTP server 的 `/admin` 路径下。默认 HTTP 端口为 `4000`。 -Admin API 是一个 HTTP 服务，提供一组可以通过 HTTP 请求调用的 RESTful API。Admin API 简单、用户友好且安全。 -本页介绍以下 API： - -- /health -- /leader -- /heartbeat -- /maintenance -- /procedure-manager - -所有这些 API 都在父资源 `/admin` 下。 - -在以下部分中，我们假设你的 metasrv 实例运行在本地主机的 4000 端口。 +Router 本身不增加认证层。部分端点会改变集群行为，部署时必须通过网络策略保护该端口。增加路由时应显式指定 HTTP method，分离读取与修改 handler，并在 `src/meta-srv/src/service/admin/` 中添加 handler-level 测试。 ## /health HTTP 端点 -`/health` 端点接受 GET HTTP 请求，你可以使用此端点检查你的 metasrv 实例的健康状况。 - -### 定义 - -```bash -curl -X GET http://localhost:4000/admin/health -``` - -### 示例 - -#### 请求 - -```bash -curl -X GET http://localhost:4000/admin/health -``` - -#### 响应 - -```json -OK -``` +`GET /admin/health` 在 HTTP service 正常运行时返回 `OK`，但不能证明当前节点是 Leader，也不能证明外部依赖可用。Handler 位于 `health.rs`。 ## /leader HTTP 端点 -`/leader` 端点接受 GET HTTP 请求，你可以使用此端点查询你的 metasrv 实例的 leader 地址。 - -### 定义 - -```bash -curl -X GET http://localhost:4000/admin/leader -``` - -### 示例 - -#### 请求 - -```bash -curl -X GET http://localhost:4000/admin/leader -``` - -#### 响应 - -```json -127.0.0.1:4000 -``` +`GET /admin/leader` 通过已配置的 election backend 读取当前 Metasrv Leader 地址。Handler 位于 `leader.rs`。 ## /heartbeat HTTP 端点 -`/heartbeat` 端点接受 GET HTTP 请求，你可以使用此端点查询所有数据节点的心跳。 - -你还可以查询指定 `addr` 的数据节点的心跳数据，但在路径中指定 `addr` 是可选的。 - -### 定义 - -```bash -curl -X GET http://localhost:4000/admin/heartbeat -``` - -| 查询字符串参数 | 类型 | 可选/必选 | 定义 | -|:---------------|:-------|:----------|:--------------------| -| addr | String | 可选 | 数据节点的地址。 | - -### 示例 - -#### 请求 - -```bash -curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100' -``` - -#### 响应 - -```json -[ - [ - { - "timestamp_millis": 1677049348651, - "id": 1, - "addr": "127.0.0.1:4100", - "rcus": 0, - "wcus": 0, - "region_num": 2, - "region_stats": [], - "topic_stats": [], - "node_epoch": 0, - "datanode_workloads": { - "types": [] - }, - "gc_stat": null - } - ] -] -``` +`GET /admin/heartbeat` 返回 Datanode 心跳记录，可通过 `addr` query parameter 按 Datanode 地址过滤。`GET /admin/heartbeat/help` 展示支持的查询形式。Handler 位于 `heartbeat.rs`，并通过 `MetaPeerClient` 读取数据。 ## /maintenance HTTP 端点 -集群维护模式是 GreptimeDB 中的一项安全功能，它可以临时禁用自动集群管理操作。此模式在集群升级、计划停机以及任何可能暂时影响集群稳定性的操作期间特别有用。有关更多详细信息，请参阅[集群维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。 - -`/maintenance` 端点支持以下 HTTP 请求： +维护模式会禁用部分自动集群管理操作，面向用户的行为参见[集群维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。Router 提供： - `GET /admin/maintenance` 或 `GET /admin/maintenance/status`：查询维护模式状态。 - `POST /admin/maintenance/enable`：启用维护模式。 - `POST /admin/maintenance/disable`：禁用维护模式。 -响应体使用以下格式： - -```json -{ - "enabled": true -} -``` +实现位于 `maintenance.rs`，通过 `RuntimeSwitchManager` 修改状态。 ## /procedure-manager HTTP 端点 -该端点用于管理 Procedure Manager 状态。有关更多详细信息，请参阅[防止元数据变更](/user-guide/deployments-administration/maintenance/prevent-metadata-changes.md)。 - -`/procedure-manager` 端点支持以下 HTTP 请求： +这些路由用于暂停或恢复 Procedure Manager 调度，面向用户的行为参见[防止元数据变更](/user-guide/deployments-administration/maintenance/prevent-metadata-changes.md)。Router 提供： - `GET /admin/procedure-manager/status`：查询 Procedure Manager 状态。 - `POST /admin/procedure-manager/pause`：暂停 Procedure Manager。 - `POST /admin/procedure-manager/resume`：恢复 Procedure Manager。 -响应体使用以下格式： +实现位于 `procedure.rs`，同样通过 `RuntimeSwitchManager` 修改状态。 + +## 其他内部端点 + +Router 还提供以下维护端点： + +- `GET /admin/node-lease` 返回当前 Datanode 租约记录。 +- `GET /admin/recovery/status` 和 `POST /admin/recovery/{enable,disable}` 查询或修改 recovery mode。 +- `GET /admin/sequence/table/next-id` 读取下一个 Table ID，但不执行分配。 +- `POST /admin/sequence/table/set-next-id` 修改 allocator 的下一个 Table ID。未启用 recovery mode 时，handler 会拒绝该操作。 -```json -{ - "status": "running" -} -``` +Recovery 和 sequence 路由会改变集群状态，只能用于受控的修复流程。修改或调用前必须阅读对应 handler 和测试；本文不提供通用恢复流程。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md index 2e3c525baa..d60ea36d24 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md @@ -1,162 +1,70 @@ --- -keywords: [Metasrv, 元数据存储, 请求路由, 负载均衡, 高可用性] -description: 介绍 Metasrv 的功能、架构和与前端的交互方式。 +keywords: [Metasrv, 元数据, 路由, Leader 选举, 心跳, 分布式 Procedure] +description: 介绍 Metasrv 的元数据、协调和集群管理职责。 --- # Metasrv -![meta](/meta.png) + -## Metasrv 包含什么 +## 职责 -- 存储元数据（Catalog, Schema, Table, Region 等） -- 请求路由器。它告诉前端在哪里写入和读取数据。 -- 数据节点的负载均衡，决定谁应该处理新的表创建请求，更准确地说，它做出资源分配决策。 -- 选举与高可用性，GreptimeDB 设计为 Leader-Follower 架构，只有 leader 节点可以写入，而 follower 节点可以读取，follower 节点的数量通常 >= 1，当 leader 不可用时，follower 节点需要能够快速切换为 leader。 -- 统计数据收集（通过每个节点上的心跳报告），如 CPU、负载、节点上的表数量、平均/峰值数据读写大小等，可用作分布式调度的基础。 +Metasrv 是分布式部署中的元数据和协调服务，负责： -## 前端如何与 Metasrv 交互 +- 通过 KV backend 持久化 Catalog、Schema、Table、Region、路由和节点元数据； +- 通过 Leader 选举保证协调操作和元数据修改只在一个 Leader 上执行； +- 通过心跳流跟踪节点租约和 Region 统计信息； +- 建表时为 Region 选择 Datanode； +- 执行可恢复的 DDL、Region 迁移、故障转移、repartition 等分布式 Procedure； +- 向 Frontend 和 Datanode 发布缓存失效及其他控制消息。 -首先，请求路由器中的路由表结构如下（注意这只是逻辑结构，实际存储结构可能不同，例如端点可能有字典压缩）。 +数据模型、KV 抽象、选举接口、key 编码和 DDL manager 位于 `src/common/meta/`。`src/meta-srv/` crate 实现服务端、状态机、心跳 handler 和控制 Procedure。 -```txt - table_A - table_name - table_schema // 用于物理计划 - regions - region_1 - mutate_endpoint - select_endpoint_1, select_endpoint_2 - region_2 - mutate_endpoint - select_endpoint_1, select_endpoint_2, select_endpoint_3 - region_xxx - table_B - ... -``` + + +## Frontend 与 Metasrv 的交互 + +Frontend 通过 `meta-client` crate 获取表元数据和 Region 路由，并提交修改元数据的操作。Frontend 在本地缓存元数据；Procedure 修改元数据后，Metasrv 会发送缓存失效消息。 ### 创建表 -1. 前端发送 `CREATE TABLE` 请求到 Metasrv。 -2. 根据请求中包含的分区规则规划 Region 数量。 -3. 检查数据节点可用资源的全局视图（通过心跳收集）并为每个 Region 分配一个节点。 -4. 前端创建表并在成功创建后将 `Schema` 存储到 Metasrv。 +1. Frontend 向 Metasrv Leader 提交 DDL 请求。 +2. DDL manager 校验请求，根据分区规则生成 Region，并为 Region 选择 Datanode。 +3. 持久化的 Procedure 创建 Region，随后记录表和路由元数据。Procedure 状态持久化后，可以在服务重启或 Leader 切换后恢复执行。 +4. 元数据提交后，Metasrv 使相关缓存失效。 ### `Insert` -1. 前端从 Metasrv 获取指定表的路由。注意，最小的路由单元是表的路由（多个 Region），即包含该表所有 Region 的地址。 -2. 最佳实践是前端首先从本地缓存中获取路由并将请求转发到数据节点。如果路由不再有效，则数据节点有义务返回 `Invalid Route` 错误，前端重新从 Metasrv 获取最新数据并更新其缓存。路由信息不经常变化，因此，前端使用惰性策略维护缓存是足够的。 -3. 前端处理可能包含多个表和多个 Region 的一批写入，因此前端需要根据“路由表”拆分用户请求。 +Frontend 获取表路由，按分区拆分数据行，并把 Region 写请求发送到对应 Datanode。路由元数据保存在本地缓存中；收到缓存失效消息或 stale-route 错误时，Frontend 会从 Metasrv 刷新路由。 ### `Select` -1. 与 `Insert` 类似，前端首先从本地缓存中获取路由表。 -2. 与 `Insert` 不同，对于 `Select`，前端需要从路由表中提取只读节点（follower），然后根据优先级将请求分发到 leader 或 follower 节点。 -3. 前端的分布式查询引擎根据路由信息分发多个子查询任务并聚合查询结果。 - -## Metasrv 架构 - -![metasrv-architecture](/metasrv-architecture.png) - -## 分布式共识 - -如你所见，Metasrv 依赖于分布式共识，因为： - -1. 首先，Metasrv 必须选举一个 leader，数据节点只向 leader 发送心跳，我们只使用单个 Metasrv 节点接收心跳，这使得基于全局信息进行一些计算或调度变得容易且快速。至于数据节点如何连接到 leader，这由 MetaClient 决定（使用重定向，心跳请求变为 gRPC 流，使用重定向比转发更不容易出错），这对数据节点是透明的。 -2. 其次，Metasrv 必须为数据节点提供选举 API，以选举“写入”和“只读”节点，并帮助数据节点实现高可用性。 -3. 最后，`Metadata`、`Schema` 和其他数据必须在 Metasrv 上可靠且一致地存储。因此，基于共识的算法是存储它们的理想方法。 - -对于 Metasrv 的第一个版本，我们选择 Etcd 作为共识算法组件（Metasrv 设计时考虑适应不同的实现，甚至创建一个新的轮子），原因如下： - -1. Etcd 提供了我们需要的 API，例如 `Watch`、`Election`、`KV` 等。 -2. 我们只执行两个分布式共识任务：选举（使用 `Watch` 机制）和存储（少量元数据），这两者都不需要我们定制自己的状态机，也不需要基于 raft 定制自己的状态机；少量数据也不需要多 raft 组支持。 -3. Metasrv 的初始版本使用 Etcd，使我们能够专注于 Metasrv 的功能，而不需要在分布式共识算法上花费太多精力，这提高了系统设计（避免与共识算法耦合）并有助于初期的快速开发，同时通过良好的架构设计，未来可以轻松接入优秀的共识算法实现。 - -## 心跳管理 - -数据节点与 Metasrv 之间的主要通信方式是心跳请求/响应流，我们希望这是唯一的通信方式。这个想法受到 [TiKV PD](https://github.com/tikv/pd) 设计的启发，我们在 [RheaKV](https://github.com/sofastack/sofa-jraft/tree/master/jraft-rheakv/rheakv-pd) 中有实际经验。请求发送其状态，而 Metasrv 通过心跳响应发送不同的调度指令。 - -心跳可能携带以下数据，但这不是最终设计，我们仍在讨论和探索究竟应该收集哪些数据。 - -``` -service Heartbeat { - // 心跳，心跳可能有很多内容，例如： - // 1. 要注册到 Metasrv 并可被其他节点发现的元数据。 - // 2. 一些性能指标，例如负载、CPU 使用率等。 - // 3. 正在执行的计算任务数量。 - rpc Heartbeat(stream HeartbeatRequest) returns (stream HeartbeatResponse) {} -} +Frontend 在查询规划期间使用表和 Region 元数据。分区谓词用于裁剪 Region，分布式查询引擎再将远端子计划发送到持有这些 Region 的 Datanode。参见[分布式查询](../frontend/distributed-querying.md)。 -message HeartbeatRequest { - RequestHeader header = 1; + - // 自身节点 - Peer peer = 2; - // leader 节点 - bool is_leader = 3; - // 实际报告时间间隔 - TimeInterval report_interval = 4; - // 节点状态 - NodeStat node_stat = 5; - // 此节点中的 Region 状态 - repeated RegionStat region_stats = 6; - // follower 节点和状态，在 follower 节点上为空 - repeated ReplicaStat replica_stats = 7; -} +## 源码结构 -message NodeStat { - // 此期间的读取容量单位 - uint64 rcus = 1; - // 此期间的写入容量单位 - uint64 wcus = 2; - // 此节点中的表数量 - uint64 table_num = 3; - // 此节点中的 Region 数量 - uint64 region_num = 4; +主要实现目录如下： - double cpu_usage = 5; - double load = 6; - // 节点中的读取磁盘 I/O - double read_io_rate = 7; - // 节点中的写入磁盘 I/O - double write_io_rate = 8; +- `src/meta-srv/src/service/`：gRPC 服务和 HTTP Admin API。 +- `src/meta-srv/src/handler/`：心跳 handler chain。 +- `src/meta-srv/src/procedure/`：Region 迁移、repartition、WAL 清理等分布式 Procedure。 +- `src/meta-srv/src/region/`：Region 租约、监控和故障转移触发逻辑。 +- `src/meta-srv/src/selector/`：为 Region 选择 Datanode。 - // 其他 - map attrs = 100; -} + -message RegionStat { - uint64 region_id = 1; - TableName table_name = 2; - // 此期间的读取容量单位 - uint64 rcus = 3; - // 此期间的写入容量单位 - uint64 wcus = 4; - // 近似 Region 大小 - uint64 approximate_size = 5; - // 近似行数 - uint64 approximate_rows = 6; +## Leader 与持久化 - // 其他 - map attrs = 100; -} +Metasrv 通过 `common-meta` 中的接口隔离 Leader 选举与持久化元数据存储。协调操作和元数据修改在 Leader 上执行；非 Leader 节点返回 not-leader 响应，client 随后连接到当前 Leader。 -message ReplicaStat { - Peer peer = 1; - bool in_sync = 2; - bool is_learner = 3; -} -``` +Leader 切换后仍需保留的数据必须写入 KV backend。进程内缓存和 Leader 本地状态会在切换时重建或清空。分布式 Procedure 会持久化状态，其每个执行步骤必须保持幂等，才能安全恢复。 -## Central Nervous System (CNS) + -我们要构建一个算法系统，该系统依赖于每个节点的实时和历史心跳数据，应该做出一些更智能的调度决策并将其发送到 Metasrv 的 Autoadmin 单元，该单元分发调度决策，由数据节点本身或更可能由 PaaS 平台执行。 - -## 工作负载抽象 +## 心跳不变量 -工作负载抽象的级别决定了 Metasrv 生成的调度策略（如资源分配）的效率和质量。 - -DynamoDB 定义了 RCUs 和 WCUs（读取容量单位/写入容量单位），解释说 RCU 是一个 4KB 数据的读取请求，WCU 是一个 1KB 数据的写入请求。当使用 RCU 和 WCU 描述工作负载时，更容易实现性能可测量性并获得更有信息量的资源预分配，因为我们可以将不同的硬件能力抽象为 RCU 和 WCU 的组合。 - -然而，GreptimeDB 面临比 DynamoDB 更复杂的情况，特别是 RCU 不适合描述需要大量计算的 GreptimeDB 读取工作负载。我们正在努力解决这个问题。 +Datanode 和 Frontend 与 Metasrv Leader 保持心跳流。请求携带节点身份、租约、Region 统计信息及其他状态。`src/meta-srv/src/handler/` 下的 handler chain 负责检查 Leader、更新租约与统计信息，并处理 mailbox 消息。 +心跳响应携带 Region 生命周期指令、缓存失效等控制消息。Region supervisor 根据租约状态发现不可用 Region，并触发故障转移 Procedure。修改心跳间隔时，必须同步检查 `common-meta` 和 `meta-srv` 中的租约与 supervisor 时序。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/selector.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/selector.md index 9048e0daa4..ade57eefa5 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/selector.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/selector.md @@ -1,49 +1,42 @@ --- -keywords: [Selector, Metasrv, Datanode, 路由表, 负载均衡] -description: 介绍 Metasrv 中的 Selector，包括其类型和配置方法。 +keywords: [Selector, Metasrv, Datanode, Region 放置, 负载均衡] +description: 介绍 Metasrv 的 Region 放置 Selector 及其配置名称。 --- # Selector ## 介绍 -什么是 `Selector`？顾名思义，它允许用户从给定的 `namespace` 和 `context` 中选择 `Item`s。有一个相关的 `trait`，也叫做 `Selector`，其定义可以在[这里][0]找到。 - -[0]: https://github.com/GreptimeTeam/greptimedb/blob/main/src/meta-srv/src/selector.rs - -在 `Metasrv` 中存在一个特定的场景。当 `Frontend` 向 `Metasrv` 发送建表请求时，`Metasrv` 会创建一个路由表（表的创建细节不在这里赘述）。在创建路由表时，`Metasrv` 需要选择适当的 `Datanode`s，这时候就需要用到 `Selector`。 +建表时，Metasrv 需要为各 Region 选择 Datanode。[`Selector` trait](https://github.com/GreptimeTeam/greptimedb/blob/main/src/meta-srv/src/selector.rs) 接收所需 peer 数量和 selection context，再根据当前租约及统计信息返回候选 Datanode。 ## Selector 类型 -`Metasrv` 目前提供以下几种类型的 `Selectors`: +Metasrv 提供三种 Selector 实现： -### LeasebasedSelector +### LeaseBasedSelector -`LeasebasedSelector` 从所有可用的（也就是在租约期间内）`Datanode` 中随机选择，其特点是简单和快速。 +`LeaseBasedSelector` 从租约有效的 Datanode 中随机选择，不使用 Region 数量为候选节点排序。 ### LoadBasedSelector -`LoadBasedSelector` 按照负载来选择，负载值则由每个 `Datanode` 上的 region 数量决定，较少的 region 表示较低的负载，`LoadBasedSelector` 优先选择低负载的 `Datanode`。 +`LoadBasedSelector` 使用 Datanode 上的 Region 数量表示负载，优先选择 Region 较少的节点。 ### RoundRobinSelector [默认选项] -`RoundRobinSelector` 以轮询的方式选择 `Datanode`。在大多数情况下，这是默认的且推荐的选项。如果你不确定选择哪个，通常它就是正确的选择。 + +`RoundRobinSelector` 依次轮询可用 Datanode，是默认的 Selector。 ## 配置 -您可以在启动 `Metasrv` 服务时通过名称配置 `Selector`。 +启动 Metasrv 时可以指定 Selector。可用名称如下： -- LeasebasedSelector: `lease_based` 或 `LeaseBased` -- LoadBasedSelector: `load_based` 或 `LoadBased` -- RoundRobinSelector: `round_robin` 或 `RoundRobin` +- `lease_based` 或 `LeaseBased` +- `load_based` 或 `LoadBased` +- `round_robin` 或 `RoundRobin` 例如： ```shell cargo run -- metasrv start --selector round_robin ``` - -```shell -cargo run -- metasrv start --selector RoundRobin -``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/overview.md index 9f82a72171..72d35c86fa 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/overview.md @@ -1,24 +1,19 @@ --- -keywords: [架构, 关键概念, 数据处理, 组件交互, 数据库] -description: 介绍 GreptimeDB 的架构、关键概念和工作原理，包括各组件的交互方式和数据处理流程。 +keywords: [贡献者指南, 架构, Frontend, Datanode, Metasrv, Flownode] +description: 介绍 GreptimeDB 内部架构及各子系统源码入口的贡献者文档。 --- # 贡献者指南 -DeepWiki 对 GreptimeDB 的架构和实现进行了详细且清晰的描述，强烈推荐阅读： - -[https://deepwiki.com/GreptimeTeam/greptimedb](https://deepwiki.com/GreptimeTeam/greptimedb) +本指南介绍 GreptimeDB 的内部架构，并提供各子系统的源码入口。构建、测试及贡献要求以源码仓库的 [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md) 为准。 ## 架构 -有关 GreptimeDB 的架构和组件，请参阅用户指南中的 [架构](/user-guide/concepts/architecture.md) 文档。 - -有关每个组件的更多详细信息，请参阅以下指南： +[架构概览](/user-guide/concepts/architecture.md) 从用户视角说明系统组件和请求链路。以下贡献者文档进一步说明各组件的实现边界： -- [frontend][1] -- [datanode][2] -- [metasrv][3] +- [Frontend](./frontend/overview.md)：协议处理、请求编排、路由和分布式查询规划。 +- [Datanode](./datanode/overview.md)：Region 管理、查询执行和存储引擎。 +- [Metasrv](./metasrv/overview.md)：元数据、集群协调和分布式 Procedure。 +- [Flownode](./flownode/overview.md)：单机及分布式部署中的持续聚合。 -[1]: /contributor-guide/frontend/overview.md -[2]: /contributor-guide/datanode/overview.md -[3]: /contributor-guide/metasrv/overview.md +本地构建 GreptimeDB 请继续阅读[快速开始](./getting-started.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/integration-test.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/integration-test.md index 63ffa56bc0..074296e671 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/integration-test.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/integration-test.md @@ -1,14 +1,30 @@ --- -keywords: [集成测试, Rust, HTTP, gRPC, 测试工具] -description: 介绍 GreptimeDB 的集成测试，包括测试范围和如何运行这些测试。 +keywords: [集成测试, Rust test harness, 存储 backend, Kafka, 协议] +description: 运行 tests-integration 中的多组件和外部服务测试。 --- # 集成测试 ## 介绍 -集成测试使用 Rust 测试工具（`#[test]`）编写，与单元测试不同，它们被单独放置在 -[这里](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration)。 -它涵盖了涉及多个组件的场景，其中一个典型案例是与 HTTP/gRPC 相关的功能。你可以查看 -其[文档](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md)以获取更多信息。 +`tests-integration/` crate 包含需要多个 GreptimeDB 组件或外部服务的 Rust test-harness case。常见场景包括 HTTP 或 gRPC 行为、对象存储 backend、Kafka WAL，以及启用 TLS 的依赖服务。 +如果某项行为无法通过 crate 内单元测试或 sqlness 查询 case 验证，应使用集成测试。协议断言应放在公共接口边界，并复用 `tests-integration/fixtures/` 中的 fixture，不要另建一套环境配置。 + +环境准备和命令以 [`tests-integration/README.md`](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md) 为准。需要凭据或 endpoint 的测试会读取仓库根目录下由 `.env.example` 创建的 `.env` 文件；不要提交凭据。 + +在仓库根目录运行通用集成测试组： + +```shell +cargo test integration +``` + +特定 backend 使用对应的 filter，例如： + +```shell +cargo test s3 +cargo test oss +cargo test azblob +``` + +Kafka 和 TLS case 需要集成测试 README 中记录的 Docker Compose 服务。只启动当前测试需要的依赖，并在测试结束后清理这些服务。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/overview.md index 81b6defa45..7fb89ac1fb 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/overview.md @@ -1,9 +1,14 @@ --- -keywords: [测试] -description: GreptimeDB 的测试 +keywords: [测试, 单元测试, sqlness, 集成测试, 回归] +description: 根据代码改动选择并运行相应的 GreptimeDB 测试套件。 --- # 测试 -我们的团队进行了大量测试，以确保 GreptimeDB 的行为。本章将介绍几种用于测试 GreptimeDB 的重要方法，以及如何使用它们。 +GreptimeDB 使用多层测试。首先选择能够覆盖当前改动的最窄测试；如果行为跨组件或通过公共接口暴露，再增加对应层级的回归测试。 +- [单元测试](./unit-test.md)覆盖 crate 内部逻辑、不变量和错误路径。测试与 Rust 实现放在一起，通过 cargo-nextest 运行。 +- [Sqlness 测试](./sqlness-test.md)覆盖单机或分布式测试环境下用户可见的 SQL 和查询行为。测试输入及预期结果位于 `tests/cases/`。 +- [集成测试](./integration-test.md)覆盖需要多个组件或外部服务的交互，包括存储 backend 和协议级行为。测试位于 `tests-integration/`。 + +仓库还包含 `tests-fuzz/`、`tests/compatibility/` 和 `tests/perf/` 等专项测试。改动涉及输入健壮性、持久化格式兼容性或性能时，应遵循相应目录中的 README 或 `AGENTS.md`。通过某一层测试，不能替代最接近回归可观察位置的测试。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/sqlness-test.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/sqlness-test.md index d29b69e835..b9985eda89 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/sqlness-test.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/sqlness-test.md @@ -1,47 +1,45 @@ --- -keywords: [Sqlness 测试, SQL, 测试套件, 测试文件, 测试案例] -description: 介绍 GreptimeDB 的 Sqlness 测试，包括测试文件类型、组织测试案例和运行测试的方法。 +keywords: [SQL 测试, sqlness, golden file, 单机, 分布式] +description: 为用户可见的查询行为添加并运行 sqlness 回归测试。 --- # Sqlness 测试 ## 介绍 -SQL 是 `GreptimeDB` 的一个重要用户接口。我们为它提供了一个单独的测试套件（名为 `sqlness`）。 +Sqlness 是 GreptimeDB 用于验证 SQL 和查询行为的 golden-file 测试框架。它会构建并启动指定的 GreptimeDB 环境，执行测试文件，再将输出与仓库中的预期结果比较。测试框架及当前参数参见 [`tests/README.md`](https://github.com/GreptimeTeam/greptimedb/blob/main/tests/README.md)。 ## Sqlness 手册 ### 测试文件 -Sqlness 有两种类型的文件 +每个 case 包含两类文件： -- `.sql`：测试输入，仅包含 SQL -- `.result`：预期的测试输出，包含 SQL 和其结果 +- `.sql` 保存测试语句和 sqlness directive。 +- `.result` 保存预期语句和输出。 -`.result` 文件是预期的执行输出。如果 `.result` 文件发生变化，意味着测试结果不同，测试可能失败。你应该检查变更日志来解决问题。 - -你只需要在 `.sql` 文件中编写测试 SQL，然后运行测试。 +先修改 `.sql` 输入，再运行 sqlness 并审查生成的 `.result` diff。结果变化可能是预期的新行为，也可能是回归，测试框架无法替你判断。只有逐项确认变更的数据行和错误信息后，才能提交 `.result` 改动。 ### 组织测试案例 -输入案例的根目录是 `tests/cases`。它包含几个子目录，代表不同的测试模式。例如，`standalone/` 包含所有在 `greptimedb standalone start` 模式下运行的测试。 +测试位于 `tests/cases/`。第一层目录选择运行环境，例如 `standalone/`；后续目录用于组织相关 case。Sqlness 会递归发现测试文件。 -在第一级子目录下（例如 `cases/standalone`），你可以随意组织你的测试案例。Sqlness 会递归地遍历每个文件并运行它们。 +回归测试应放在能够观察到该行为的环境中。分布式规划、路由和多节点元数据行为需要分布式 case，即使同一查询在单机模式下也能成功。 ## 运行测试 -与其他测试不同，这个测试工具是以二进制目标形式存在的。你可以用以下命令运行它 +仓库为测试框架定义了 cargo alias： ```shell -cargo run --bin sqlness-runner bare +cargo sqlness bare ``` -它会自动完成以下步骤：编译 `GreptimeDB`，启动它，抓取测试并将其发送到服务器，然后收集和比较结果。你只需要检查是否有 `.result` 文件发生变化。如果没有，恭喜你，测试通过了 🥳！ +该命令会构建 GreptimeDB、启动测试环境、执行 case，并更新或比较 `.result` 文件。需要同时检查命令结果和 `git diff`。 ### 运行特定测试 ```shell -cargo sqlness bare -t your_test +cargo sqlness bare -t 'standalone:your_case' ``` -`-t` 或 `--test-filter` 选项接受正则表达式字符串。Sqlness 会检查格式为 `env:case` 的案例名称。 +`-t`/`--test-filter` 接收正则表达式，并匹配 `env:case` 格式的 case 名称。开发时可以使用窄过滤器，提交前仍应运行受影响环境或完整测试套件。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/unit-test.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/unit-test.md index 79c73775b4..e56d0cafb6 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/unit-test.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/unit-test.md @@ -1,28 +1,34 @@ --- -keywords: [单元测试, Rust, nextest, 测试覆盖率, CI] -description: 介绍 GreptimeDB 的单元测试，包括如何编写、运行和检查测试覆盖率。 +keywords: [单元测试, Rust, cargo-nextest, crate 测试, 覆盖率] +description: 使用 cargo-nextest 编写并运行 crate 内的 Rust 测试。 --- # 单元测试 ## 介绍 -单元测试嵌入在代码库中，通常放置在被测试逻辑的旁边。它们使用 Rust 的 `#[test]` 属性编写，并可以使用 `cargo nextest run` 运行。 +Rust 单元测试通常位于被测模块内或相邻的 `*_test.rs` 文件中。适合覆盖 crate 内部不变量、边界条件、错误处理，以及不需要启动 GreptimeDB 集群的行为。 -GreptimeDB 代码库不支持默认的 `cargo` 测试运行器。推荐使用 [`nextest`](https://nexte.st/)。你可以通过以下命令安装它： +GreptimeDB 的标准测试运行器是 [cargo-nextest](https://nexte.st/)。安装命令如下： ```shell cargo install cargo-nextest --locked ``` -然后运行测试（这里 `--workspace` 不是必须的） +开发期间先运行受影响 crate 的测试： ```shell -cargo nextest run +cargo nextest run -p ``` -注意，如果你的 Rust 是通过 `rustup` 安装的，请确保使用 `cargo` 安装 `nextest`，而不是像 `homebrew` 这样的包管理器，否则会弄乱你的本地环境。 +可能影响多个 crate 的改动，在提交前应运行 CI 对应的 workspace 配置： + +```shell +cargo nextest run --workspace --features pg_kvbackend,mysql_kvbackend +``` + +Feature-gated 代码需要在测试命令中启用对应 feature。不要假设默认 feature set 已覆盖相关路径，应检查 crate 的 `Cargo.toml`、本地 `AGENTS.md` 和 CI workflow。 ## 覆盖率 -我们的持续集成（CI）作业有一个“覆盖率检查”步骤。它会报告有多少代码被单元测试覆盖。请在你的补丁中添加必要的单元测试。 +CI 会记录 Rust 测试覆盖率。测试应保护实际改动的行为和可信的失败场景，不要只为提高百分比增加断言。查询语言行为和跨组件流程通常还需要 sqlness 或集成测试。 From fb207bbcaf31b2fabae7cab2f700fe552349a3d1 Mon Sep 17 00:00:00 2001 From: Dennis Zhuang Date: Tue, 25 Aug 2026 11:07:38 +0800 Subject: [PATCH 02/14] docs: focus contributor guide on design concepts --- .../datanode/data-persistence-indexing.md | 52 ++++--- .../datanode/metric-engine.md | 31 ++-- docs/contributor-guide/datanode/overview.md | 32 ++-- .../datanode/query-engine.md | 42 +++-- .../datanode/storage-engine.md | 79 ++++++++-- docs/contributor-guide/datanode/wal.md | 18 +-- .../contributor-guide/flownode/arrangement.md | 20 ++- .../flownode/batching_mode.md | 66 +++++--- docs/contributor-guide/flownode/dataflow.md | 17 ++- docs/contributor-guide/flownode/overview.md | 26 ++-- .../frontend/distributed-querying.md | 16 +- docs/contributor-guide/frontend/overview.md | 50 +++--- .../frontend/table-sharding.md | 12 +- docs/contributor-guide/getting-started.md | 49 +++--- .../how-to/how-to-trace-greptimedb.md | 88 +++++------ .../how-to/how-to-use-tokio-console.md | 26 ++-- .../how-to/how-to-write-sdk.md | 47 ++++-- docs/contributor-guide/metasrv/admin-api.md | 144 +++++++++++++++--- docs/contributor-guide/metasrv/overview.md | 72 ++++----- docs/contributor-guide/metasrv/selector.md | 27 ++-- docs/contributor-guide/overview.md | 23 +-- .../tests/integration-test.md | 29 +--- docs/contributor-guide/tests/overview.md | 12 +- docs/contributor-guide/tests/sqlness-test.md | 34 +++-- docs/contributor-guide/tests/unit-test.md | 26 ++-- .../datanode/data-persistence-indexing.md | 50 ++++-- .../datanode/metric-engine.md | 33 ++-- .../contributor-guide/datanode/overview.md | 33 ++-- .../datanode/query-engine.md | 30 ++-- .../datanode/storage-engine.md | 82 ++++++++-- .../current/contributor-guide/datanode/wal.md | 16 +- .../contributor-guide/flownode/arrangement.md | 18 ++- .../flownode/batching_mode.md | 66 +++++--- .../contributor-guide/flownode/dataflow.md | 18 ++- .../contributor-guide/flownode/overview.md | 27 ++-- .../frontend/distributed-querying.md | 16 +- .../contributor-guide/frontend/overview.md | 49 +++--- .../frontend/table-sharding.md | 9 +- .../contributor-guide/getting-started.md | 43 +++--- .../how-to/how-to-trace-greptimedb.md | 62 ++++---- .../how-to/how-to-use-tokio-console.md | 26 ++-- .../how-to/how-to-write-sdk.md | 43 ++++-- .../contributor-guide/metasrv/admin-api.md | 138 ++++++++++++++--- .../contributor-guide/metasrv/overview.md | 74 ++++----- .../contributor-guide/metasrv/selector.md | 27 ++-- .../current/contributor-guide/overview.md | 23 +-- .../tests/integration-test.md | 28 +--- .../contributor-guide/tests/overview.md | 11 +- .../contributor-guide/tests/sqlness-test.md | 28 ++-- .../contributor-guide/tests/unit-test.md | 22 +-- 50 files changed, 1220 insertions(+), 790 deletions(-) diff --git a/docs/contributor-guide/datanode/data-persistence-indexing.md b/docs/contributor-guide/datanode/data-persistence-indexing.md index 61b7c2a4bb..86ddb7b667 100644 --- a/docs/contributor-guide/datanode/data-persistence-indexing.md +++ b/docs/contributor-guide/datanode/data-persistence-indexing.md @@ -5,62 +5,80 @@ description: Explanation of data persistence and indexing in GreptimeDB, includi # Data Persistence and Indexing -Mito flushes data from memtables to durable local filesystems or object storage. SST files use [Apache Parquet][1] as their data format. +Similar to all LSMT-like storage engines, data in MemTables is persisted to durable storage, for example, the local disk file system or object storage service. GreptimeDB adopts [Apache Parquet][1] as its persistent file format. ## SST File Format -Parquet is a columnar file format. Its hierarchy determines the units that Mito can read, cache, or prune during a scan. +Parquet is an open source columnar format that provides fast data querying and has already been adopted by many projects, such as Delta Lake. Parquet organizes data as row groups, column chunks, and pages. A row group contains one column chunk for each column, and each column chunk contains one or more pages. Pages are the smallest encoded I/O units within a column chunk. -Column chunks let a projected scan read only the requested columns. +First, clustering data by column makes file scanning more efficient, especially when only a few columns are queried, which is very common in analytical systems. -Pages within one column also tend to compress well with encodings such as dictionary encoding and run-length encoding (RLE). +Second, data of the same column tends to be homogeneous which helps with compression when apply techniques like dictionary and Run-Length Encoding (RLE). Parquet file format ## Data Persistence -`region_engine.mito.global_write_buffer_size` sets the memory threshold shared by all Mito memtables on a Datanode. +GreptimeDB provides a configuration item `region_engine.mito.global_write_buffer_size`, which is flush threshold of the total memory usage for all MemTables. -When memory usage reaches the threshold, the write-buffer manager selects memtables and schedules SST flushes through `src/mito2/src/flush.rs`. +When the size of data buffered in MemTables reaches that threshold, GreptimeDB will pick MemTables and flush them to SST files. ## Indexing Data in SST Files -Parquet records column statistics for row groups and pages. Mito converts compatible query predicates into Parquet pruning predicates so it can skip row groups whose min/max or null statistics cannot match. +Apache Parquet file format provides inherent statistics in headers of column chunks and data pages, which are used for pruning and skipping. + +Column chunk header + +For example, in the above Parquet file, if you want to filter rows where `name` = `Emily`, you can easily skip row group 0 because the max value for `name` field is `Charlie`. This statistical information reduces IO operations. ## Index Files -Mito stores index artifacts associated with an SST in versioned [Puffin][3] files. The Region manifest identifies the active index version; publishing or rebuilding an index must not make the manifest reference an incomplete artifact. +For each SST file, GreptimeDB not only maintains an internal index but also generates a separate file to store the index structures specific to that SST file. + +The index files utilize the [Puffin][3] format, which offers significant flexibility, allowing for the storage of additional metadata and supporting a broader range of index structures. + +![Puffin](/puffin.png) -`src/mito2/src/sst/index/` integrates inverted, bloom-filter skipping, full-text, and feature-gated vector indexes with SST reads and writes. Their reusable index formats live under `src/index/src/`, while `puffin_manager.rs` manages the companion files. +GreptimeDB stores several index structures in the Puffin file as Blobs, including the inverted index, the skipping index (backed by a bloom filter), and the full-text index. The inverted index was the first one supported and is described in detail below. ## Inverted Index -For each indexed column, the inverted index maps encoded column values to the SST data segments that contain them. Applying a predicate produces candidate segment IDs; the normal scan still evaluates the complete predicate on rows from those segments. +In version 0.7, GreptimeDB introduced the inverted index to accelerate queries. + +The inverted index is a common index structure used for full-text searches, mapping each word in the document to a list of documents containing that word. GreptimeDB applies this search-engine technique to indexes over time-series data. + +Search engines and time series databases operate in separate domains, yet the principle behind the applied inverted index technology is similar. This similarity requires some conceptual adjustments: +1. Term: In GreptimeDB, it refers to the column value of the time series. +2. Document: In GreptimeDB, it refers to the data segment containing multiple time series. + +The inverted index enables GreptimeDB to skip data segments that do not meet query conditions, thus improving scanning efficiency. ![Inverted index searching](/inverted-index-searching.png) -The query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches the regex `.*users`, and `status` matches the regex `4...`. Mito scans those candidate segments and applies the complete query predicate to their rows. +For instance, the query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches the regex `.*users`, and `status` matches the regex `4...`. It then scans these data segments to produce the final results that meet all conditions, significantly reducing the number of IO operations. ### Inverted Index Format ![Inverted index format](/inverted-index-format.png) -Each column index contains a finite-state transducer (FST) and bitmaps. The FST maps encoded values to bitmap positions and supports lookups such as regular-expression matching. Each bitmap records the data segments that contain a value. +GreptimeDB builds inverted indexes by column, with each inverted index consisting of an FST and multiple Bitmaps. + +The FST (Finite State Transducer) enables GreptimeDB to store mappings from column values to Bitmap positions in a compact format and provides excellent search performance and supports complex search capabilities (such as regular expression matching). The Bitmaps maintain a list of data segment IDs, with each bit representing a data segment. ### Index Data Segments -GreptimeDB divides an SST file into fixed-size indexed data segments. A matching bitmap becomes a Parquet row selection, so Mito reads only the candidate row ranges. +GreptimeDB divides an SST file into multiple indexed data segments, with each segment housing an equal number of rows. This segmentation is designed to optimize query performance by scanning only the data segments that match the query conditions. -For example, with 1024 rows per segment and candidate segment IDs `[0, 2]`, Mito scans rows 0–1023 and 2048–3071 instead of all rows in the SST. +For example, if a data segment contains 1024 rows and the list of data segments identified through the inverted index for the query conditions is `[0, 2]`, then only the 0th and 2nd data segments in the SST file—from rows 0 to 1023 and 2048 to 3071, respectively—need to be scanned. -The engine option `index.inverted_index.segment_row_count`, which defaults to `1024`, controls the target segment size. Smaller segments make pruning more precise but increase index size and build cost. +The number of rows in a data segment is controlled by the engine option `index.inverted_index.segment_row_count`, which defaults to `1024`. A smaller value means more precise indexing and often results in better query performance but increases the cost of index storage. By adjusting this option, a balance can be struck between storage costs and query performance. ## Unified Data Access Layer: OpenDAL -The `object-store` crate wraps [OpenDAL][2] for local filesystems and object stores. Mito performs SST and index I/O through `src/mito2/src/access_layer.rs`; storage-engine code should not add backend-specific paths around that boundary. Changing a configured backend does not migrate existing data. +GreptimeDB uses [OpenDAL][2] to provide a common access layer for local filesystems and object stores. Changing the configured storage backend does not migrate existing data. [1]: https://parquet.apache.org -[2]: https://opendal.apache.org/ +[2]: https://github.com/datafuselabs/opendal [3]: https://iceberg.apache.org/puffin-spec diff --git a/docs/contributor-guide/datanode/metric-engine.md b/docs/contributor-guide/datanode/metric-engine.md index 92f0ec7e70..59d536392e 100644 --- a/docs/contributor-guide/datanode/metric-engine.md +++ b/docs/contributor-guide/datanode/metric-engine.md @@ -1,39 +1,40 @@ --- -keywords: [Metric engine, logical table, physical table, Mito, Prometheus] -description: Metric Engine's logical-to-physical storage model for large numbers of metric tables. +keywords: [Metric engine, small tables, logical table, physical table, storage optimization] +description: Overview of the Metric engine in GreptimeDB, its concepts, architecture, and design for handling small tables. --- # Metric Engine ## Overview -Metric Engine is a `RegionEngine` implementation for Prometheus-style workloads with many small metric tables. It multiplexes logical tables into shared physical Mito Regions, reducing per-table metadata and storage overhead while retaining a table-level interface for reads and writes. +The `Metric` engine is a component of GreptimeDB, and it's an implementation of the storage engine. It mainly targets scenarios with a large number of small tables for observable metrics. -Metric Engine does not implement another on-disk format. It rewrites logical requests and delegates physical storage, indexing, and scans to Mito. +Its main feature is to use synthetic physical wide tables to store a large amount of small table data, achieving effects such as reuse of the same column and metadata. This reduces storage overhead for small tables and improves columnar compression efficiency. The concept of a table becomes even more lightweight under the `Metric` engine. ## Concepts +The `Metric` engine introduces two new concepts: "logical table" and "physical table". From the user's perspective, logical tables are exactly like ordinary ones. From a storage point-of-view, physical Regions are just regular Regions. + ### Logical Table -A logical table is the table exposed to users. It has its own schema and table ID, and all user writes and queries address that table. Internally, each logical Region records the physical Region that stores its rows. +A logical table refers to user-defined tables. Just like any other ordinary table, its definition includes the name of the table, column definitions, index definitions etc. All operations such as queries or write-ins by users are based on these logical tables. Users don't need to worry about differences between logical and ordinary tables during usage. -On writes, Metric Engine injects the logical table identity into each row before forwarding it to the physical data Region. On reads, it adds a logical-table filter so only rows belonging to the requested table are returned. +From an implementation standpoint, a logical table is virtual; it doesn't directly read or write physical data but maps read/write requests into corresponding requests for physical tables in order to implement data storage and querying. ### Physical Table -A physical table owns the shared Regions. Each physical Region is represented by a pair of Mito Regions: +A physical table is a table that actually stores data, possessing several physical Regions defined by partition rules. -- a data Region containing rows from multiple logical tables; -- a metadata Region containing logical-table and logical-column mappings used by Metric Engine. +## Architecture and Design -Direct writes to a physical Region are rejected because they would bypass the logical-table mapping. Queries against a physical table remain supported. +The main design architecture of the `Metric` engine is as follows: -## Architecture and Design +![Arch](/metric-engine-arch.png) -Logical tables associated with a physical table use the same partition layout. Their logical Region IDs map to the corresponding physical data and metadata Region IDs. The mapping is maintained by Metric Engine and by table-route metadata. +The `Metric` engine delegates physical storage and queries to the `Mito` engine. Each physical Region is represented by a data Region, which stores rows from many logical tables, and a metadata Region, which stores the logical-table and logical-column mappings. -`row_modifier.rs` and `batch_modifier.rs` encode the logical table identity and time-series identity into Mito's internal columns. Depending on the physical Region's primary-key encoding, this uses `__table_id` and `__tsid` columns or the sparse `__primary_key` representation. The read path always applies the logical table ID before delegating the scan to Mito. +Logical tables associated with the same physical table share its partition layout. During writes, the engine records the logical table identity with each row. During reads, it adds a logical-table filter before scanning the physical Region. -Metric Engine provides batch DDL paths for operations that affect many logical tables. This avoids issuing a separate metadata update for every table during workloads such as Prometheus Remote Write auto-creation or physical Region migration. These are data definition language operations; ordinary logical-table inserts, deletes, and queries still use the standard Region request paths. +Logical tables support normal INSERT, DELETE, and SELECT operations. Direct writes to a physical Region are rejected because they would bypass the logical-table mapping; querying a physical table remains supported. -The main implementation is under `src/metric-engine/src/`. Changes to reserved columns, Region ID conversion, or metadata encoding affect persisted data and require backward-compatibility review. +Batch DDL operations reduce metadata work when many logical tables are created or updated together, such as during Prometheus Remote Write auto-creation or physical Region migration. diff --git a/docs/contributor-guide/datanode/overview.md b/docs/contributor-guide/datanode/overview.md index b921f66de7..83e9d3b889 100644 --- a/docs/contributor-guide/datanode/overview.md +++ b/docs/contributor-guide/datanode/overview.md @@ -1,21 +1,33 @@ --- -keywords: [Datanode, RegionServer, storage engine, query engine, heartbeat] -description: Overview of Datanode's Region-level storage and query responsibilities. +keywords: [Datanode, region server, data storage, gRPC service, heartbeat task, region manager] +description: Overview of Datanode in GreptimeDB, its responsibilities, components, and interaction with other parts of the system. --- # Datanode ## Introduction -Datanode stores table data and executes queries against its local Regions. A table can contain multiple Regions, but Datanode does not manage tables as a metadata object. Frontend and Metasrv address it through Region-level requests, so its primary abstraction is a Region server. +`Datanode` is mainly responsible for storing the actual data for GreptimeDB. As we know, in GreptimeDB, +a `table` can have one or more `Region`s, and `Datanode` is responsible for managing the reading and writing +of these `Region`s. `Datanode` is not aware of `table` and can be considered as a `region server`. Therefore, +`Frontend` and `Metasrv` operate `Datanode` at the granularity of `Region`. + +![Datanode](/datanode.png) ## Components -- `RegionServer` in `src/datanode/src/region_server.rs` dispatches Region requests to the registered storage engine and exposes Regions to the query layer. -- The gRPC service accepts Region reads, writes, and lifecycle operations from Frontend and Metasrv. -- The local query engine plans and executes logical subplans received from Frontend. Datanode does not parse client SQL or coordinate a distributed query. -- The heartbeat task reports node and Region state to Metasrv and receives control instructions such as Region open, close, migration, and cache invalidation messages. -- HTTP handlers expose operational endpoints such as metrics and configuration. -- Datanode registers the Mito, Metric, and File Region engines. Mito is the primary time-series storage engine; Metric delegates physical storage to Mito for high-cardinality metric-table workloads; File exposes data in external files. +A `Datanode` contains all the components needed for a `region server`. Here we list some of the vital parts: -In standalone mode, the same Region server runs in-process without Metasrv coordination. In distributed mode, Region writability and lifecycle changes are coordinated through Metasrv leases and heartbeat messages. +- A gRPC service is provided for reading and writing region data, and `Frontend` uses this service + to read and write data from `Datanode`s. +- An HTTP service, through which you can obtain metrics, configuration information, etc., of the current node. +- `Heartbeat Task` is used to send heartbeat to the `Metasrv`. The heartbeat plays a crucial role in the + distributed architecture of GreptimeDB and serves as a basic communication channel for distributed coordination. + The upstream heartbeat messages contain important information such as the workload of a `Region`. If the + `Metasrv `has made scheduling(such as `Region` migration) decisions, it will send instructions to the + `Datanode` via downstream heartbeat messages. +- The `Datanode` does not parse user SQL or perform distributed planning. The user's query requests for one or + more `Table`s will be transformed into `Region` query requests in the `Frontend`. The `Datanode` is responsible + for executing these `Region` query plans with its local query engine. +- A Region server manages the lifecycle of all `Region`s on a `Datanode` and dispatches requests to the appropriate storage engine. +- GreptimeDB supports multiple Region engines. `Mito` is the primary time-series storage engine, `Metric` stores many logical metric tables in shared Mito Regions, and `File` provides access to external files. diff --git a/docs/contributor-guide/datanode/query-engine.md b/docs/contributor-guide/datanode/query-engine.md index bc6cb1d450..cceec7a766 100644 --- a/docs/contributor-guide/datanode/query-engine.md +++ b/docs/contributor-guide/datanode/query-engine.md @@ -1,35 +1,53 @@ --- -keywords: [query engine, Apache DataFusion, logical plan, physical plan, Arrow, indexes] -description: Overview of GreptimeDB's DataFusion-based query planning and execution pipeline. +keywords: [query engine, Apache DataFusion, logical plan, physical plan, data representation, indexing] +description: Overview of GreptimeDB's query engine, its architecture, data representation, indexing, and extensibility. --- # Query Engine ## Introduction -GreptimeDB's query engine is built on [Apache DataFusion][1]. The `query` crate owns SQL, PromQL, and log planning, GreptimeDB optimizer rules, physical planning, and execution. +GreptimeDB's query engine is built on [Apache DataFusion][1] (subproject under [Apache +Arrow][2]), a brilliant query engine written in Rust. It provides a set of well functional components from +logical plan, physical plan and the execution runtime. Below explains how each component is orchestrated and their positions during execution. ![Execution Procedure](/execution-procedure.png) -A query first becomes a DataFusion logical plan. SQL and other query-language planners produce these plans, and Frontend also sends serialized logical subplans to Datanodes during distributed execution. +The entry point is the logical plan, which is used as the general intermediate representation of a +query or execution logic etc. Two noticeable sources of logical plan are from: 1. the user query, like +SQL through SQL parser and planner; 2. the Frontend's distributed query, which is explained in details in the following section. -Analyzer and optimizer rules normalize the plan, push filters and projections, prune Regions, and introduce GreptimeDB extension nodes such as `MergeScan`. Both DataFusion rules and rules under `src/query/src/optimizer/` participate in this phase. +Next is the physical plan, or the execution plan. Unlike the logical plan which is a big +enumeration containing all the logical plan variants (except the special extension plan node), the +physical plan is in fact a trait that defines a group of methods invoked during +execution. All data processing logics are packed in corresponding structures that +implement the trait. They are the actual operations performed on the data, like +aggregator `MIN` or `AVG`, and table scan `SELECT ... FROM`. -The physical planner converts the optimized logical plan into DataFusion `ExecutionPlan` implementations. Executing the root plan returns an asynchronous stream of Arrow `RecordBatch` values. Use `EXPLAIN` or `EXPLAIN VERBOSE` to inspect the plans produced for a SQL statement. +The optimization phase which improves execution performance by transforming both logical and physical plans, is now all based on rules. It is also called, "Rule Based Optimization". Some of the rules are DataFusion native and others are customized in Greptime DB. In the future, we plan to add more +rules and leverage the data statistics for Cost Based Optimization/CBO. + +The last phase "execute" is a verb, stands for the procedure that reads data from storage, performs +calculations and generates the expected results. Although it's more abstract than previously mentioned concepts, you can just +simply imagine it as executing a Rust async function. And it's indeed a future (stream). + +`EXPLAIN [VERBOSE] ` is very useful if you want to see how your SQL is represented in the logical or physical plan. ## Data Representation -GreptimeDB uses [Apache Arrow][2] arrays and `RecordBatch` values for in-memory data exchange. The columnar representation is shared by storage scans, query operators, RPC streams, and result encoders, avoiding row-by-row conversion inside the execution pipeline. +GreptimeDB uses [Apache Arrow][2] as the in-memory data representation. It's column-oriented, in +cross-platform format, and also contains many high-performance data operators. These features +make it easy to share data in many different environments and implement calculation logic. ## Indexing -Index construction and persistent index formats belong to the storage engine, not the query engine. Mito uses Parquet statistics and inverted, skipping, and full-text indexes to prune SST files, row groups, and data segments. A feature-gated vector index supplies candidate rows for vector search. See [Data Persistence and Indexing](./data-persistence-indexing.md). - -The query layer contributes predicates and projections to the scan. An index can reduce the data read by a compatible predicate, but it does not replace the remaining filter operators in the query plan. +Index construction and persistent index formats belong to the storage engine. The query layer supplies predicates and projections to a scan; Mito then uses time ranges, Parquet statistics, and indexes to avoid reading data that cannot match. See [Data Persistence and Indexing](./data-persistence-indexing.md). ## Distributed Execution -Frontend rewrites compatible logical-plan fragments into remote `MergeScan` inputs, serializes them with Substrait, and sends Region-specific requests to Datanodes. See [Distributed Querying](../frontend/distributed-querying.md). +Covered in [Distributed Querying][6]. -[1]: https://datafusion.apache.org/ +[1]: https://github.com/apache/arrow-datafusion [2]: https://arrow.apache.org/ +[3]: https://parquet.apache.org +[6]: ../frontend/distributed-querying.md diff --git a/docs/contributor-guide/datanode/storage-engine.md b/docs/contributor-guide/datanode/storage-engine.md index 9b67236485..c220c72208 100644 --- a/docs/contributor-guide/datanode/storage-engine.md +++ b/docs/contributor-guide/datanode/storage-engine.md @@ -7,18 +7,36 @@ description: Overview of the storage engine in GreptimeDB, its architecture, com ## Introduction -Mito is GreptimeDB's primary time-series Region engine. It implements the `RegionEngine` trait and uses an [LSM tree][1] write path: WAL and memtables absorb writes, immutable Parquet SST files hold persisted data, and background compaction reorganizes those files. +The `storage engine` is responsible for storing the data of the database. Mito, based on [LSMT][1] (Log-structured Merge-tree), is the storage engine we use by default. We have made significant optimizations for handling time-series data scenarios, so mito engine is not suitable for general purposes. ## Architecture -The implementation is under `src/mito2/src/`. `engine.rs` dispatches Region requests, `worker/` owns the per-Region write loop, `read/` builds scans, and `flush.rs`, `compaction/`, `manifest/`, and `sst/` implement the persistent lifecycle. - -- **WAL** records writes that have not reached an SST so a Region can recover its memtable state. It uses the `LogStore` API with local raft-engine and remote Kafka providers. The acknowledgement durability boundary depends on provider configuration; see [Write-Ahead Logging](./wal.md). -- **Memtables** receive writes in a mutable active memtable. A flush freezes it into an immutable memtable that remains readable until its rows have been written to an SST. -- **SST files** are immutable Parquet files whose rows are sorted by primary key and time index; see [Data Layout in SST Files](#data-layout-in-sst-files). -- **Compaction** merges SST files and removes expired data. The default strategy is [TWCS][3], which groups files by time window. See [Compaction](/user-guide/deployments-administration/manage-data/compaction.md). -- **Manifest** stores versioned Region metadata and SST file changes used during recovery. -- **Caches** retain file metadata, data pages, and other reusable scan state. +The picture below shows the architecture and process procedure of the storage engine. + +![Architecture](/storage-engine-arch.png) + +The architecture is the same as a traditional LSMT engine: + +- [WAL][2] + - Guarantees high durability for data that is not yet being flushed. + - Based on the `Log Store` API, thus it doesn't care about the underlying storage + media. + - Log records of the WAL can be stored on the local disk, or in a remote log service such as + Kafka (remote WAL) that implements the `Log Store` API. +- Memtables: + - Data is written into the `active memtable`, aka `mutable memtable` first. + - When a `mutable memtable` is full, it will be changed to a `read-only memtable`, aka `immutable memtable`. +- SST + - The full name of SST, aka SSTable is `Sorted String Table`. + - `Immutable memtable` is flushed to persistent storage and produces an SST file. + - Rows in an SST are sorted by primary key and time index; see [Data Layout in SST Files](#data-layout-in-sst-files). +- Compactor + - Small `SST` is merged into large `SST` by the compactor via compaction. + - The default compaction strategy is [TWCS][3]. Compaction groups SST files into time windows and, together with TTL, removes expired data. See [Compaction](/user-guide/deployments-administration/manage-data/compaction.md). +- Manifest + - The manifest stores the metadata of the engine, such as the metadata of the `SST`. +- Cache + - Speed up queries. [1]: https://en.wikipedia.org/wiki/Log-structured_merge-tree [2]: https://en.wikipedia.org/wiki/Write-ahead_logging @@ -26,11 +44,26 @@ The implementation is under `src/mito2/src/`. `engine.rs` dispatches Region requ ## Data Model -Mito receives a `RegionMetadata` schema with a primary-key column list, one non-null time-index column, and field columns. The SQL layer exposes primary-key columns as tags, but Mito operates on column IDs and semantic types rather than SQL table definitions. +The data model provided by the storage engine is between the `key-value` model and the tabular model. + +```txt +tag-1, ..., tag-m, timestamp -> field-1, ..., field-n +``` + +Each row of data contains multiple tag columns, one timestamp column, and multiple field columns. +- `0 ~ m` tag columns + - Tag columns can be nullable. + - Specified during table creation using `PRIMARY KEY`. +- Must include one timestamp column + - Timestamp column cannot be null. + - Specified during table creation using `TIME INDEX`. +- `0 ~ n` field columns + - Field columns can be nullable. +- Data is sorted by tag columns and timestamp column. ### Region -A Region is Mito's isolation, recovery, and request unit. Every row in a Region follows its Region metadata. A table can span several Regions, while table routing and placement remain outside the storage engine. +Data in the storage engine is stored in `regions`, which are logical isolated storage units within the engine. Rows within a `region` must have the same `schema`, which defines the tag columns, timestamp column, and field columns within the `region`. The data of tables in the database is stored in one or multiple `regions`. ## Data Layout in SST Files @@ -38,6 +71,28 @@ When a memtable is flushed, Mito writes its rows into immutable [Apache Parquet] Within an SST file, rows are sorted by `(primary key, time index)`. Rows that share the same primary key (the tag columns) belong to the same time-series and are stored contiguously, ordered by timestamp. This locality is what makes scanning a single series cheap and improves compression. For append-only tables without a primary key, rows are sorted by the time index alone. +For example, consider a table that stores host metrics: + +```sql +CREATE TABLE host_metrics ( + host STRING, + region STRING, + ts TIMESTAMP TIME INDEX, + cpu DOUBLE, + memory DOUBLE, + PRIMARY KEY (host, region) +); +``` + +Mito groups rows by primary key and orders them by time, so the data within an SST conceptually looks like: + +| host | region | ts | cpu | memory | +| --- | --- | --- | --- | --- | +| host-a | us-east | 10:00 | 0.42 | 7.1 | +| host-a | us-east | 10:01 | 0.47 | 7.4 | +| host-a | us-west | 10:00 | 0.31 | 6.8 | +| host-b | us-east | 10:00 | 0.80 | 8.6 | + Besides the table columns, Mito stores three internal columns in each SST file so it can merge, deduplicate, and apply deletes correctly when reading from multiple memtables and SST files: - `__primary_key`: the encoded primary key (tags) of the row. @@ -56,6 +111,6 @@ Mito avoids reading data that cannot match a query by combining several pruning 1. **Time-range pruning.** Files and memtables whose time range does not intersect the query's time range are skipped before opening any reader. This is usually the cheapest and most effective step for time-series queries. 2. **Row-group statistics.** If a row group's min-max statistics prove that no row can match a predicate, the whole row group is skipped. -3. **Indexes.** Inverted, skipping, and full-text indexes provide more selective pruning for predicates that statistics cannot resolve. The feature-gated vector index selects candidate rows for vector search. See [Data Persistence and Indexing](data-persistence-indexing.md). +3. **Indexes.** Inverted, skipping, and full-text indexes provide more selective pruning for predicates that statistics cannot resolve. See [Data Persistence and Indexing](data-persistence-indexing.md). Scan pruning pipeline diff --git a/docs/contributor-guide/datanode/wal.md b/docs/contributor-guide/datanode/wal.md index 7f33e756f4..92523d756c 100644 --- a/docs/contributor-guide/datanode/wal.md +++ b/docs/contributor-guide/datanode/wal.md @@ -1,24 +1,24 @@ --- -keywords: [write-ahead log, WAL, recovery, raft-engine, Kafka] -description: Mito's write-ahead log abstraction, recovery path, and durability settings. +keywords: [write-ahead logging, WAL, data durability, LSMT, synchronous flush, asynchronous flush] +description: Introduction to Write-Ahead Logging (WAL) in GreptimeDB, its purpose, architecture, and operational modes. --- # Write-Ahead Logging ## Introduction -Mito applies writes to an in-memory memtable before they are flushed to SST files. To recover data that has not reached an SST, it appends each Region's write operations to a write-ahead log (WAL) before applying them to the memtable. +Mito applies writes to an in-memory MemTable before the data is flushed to SST files. It first appends each Region's write operations to the write-ahead log (WAL), so data that has not reached an SST can be recovered. -On Region open or Datanode restart, Mito replays WAL entries after the last persisted sequence and rebuilds the in-memory state. Sequence numbers are assigned per Region and are also used for deduplication and snapshot reads. +When a Region is reopened after a Datanode restart, Mito replays WAL entries after the last persisted sequence to rebuild its in-memory state. The WAL is accessed through a common log-store abstraction and can use local raft-engine storage or a remote Kafka cluster. -The WAL is accessed through the `LogStore` abstraction. Datanode supports a local `raft_engine` provider and a remote Kafka provider; the storage engine does not assume that the log is a local file. Provider construction is in `src/datanode/src/datanode.rs`, while Mito's WAL integration is in `src/mito2/src/wal.rs` and its write worker. +![WAL in Datanode](/wal.png) ## Namespace -WAL entries are isolated by Region. Append and read operations use the Region ID as their namespace, allowing recovery to replay exactly the log for the Region being opened. A table may contain several Regions, so the WAL namespace is not a table identifier. +Namespace of WAL is used to separate entries from different tables (different regions). Append and +read operations must provide a Namespace. Currently, region ID is used as the Namespace, because +each region has a MemTable that needs to be reconstructed when Datanode restarts. ## Synchronous/Asynchronous flush -For the local `raft_engine` provider, `sync_write` controls whether an append waits for the log to be synced to durable storage. It defaults to `false`. Asynchronous writes reduce latency but can lose recently acknowledged entries if the host or storage fails before the buffered log is synced. Setting `sync_write = true` strengthens that durability boundary at the cost of additional write latency. - -Kafka WAL durability depends on the Kafka producer and cluster settings rather than the local `sync_write` option. Code that acknowledges a write must preserve the ordering between WAL append and memtable mutation for every provider. +For the local raft-engine provider, `sync_write` controls whether an append waits for the log to be synced to durable storage. It defaults to `false`. Asynchronous writes reduce latency but can lose recently acknowledged entries if the host fails before buffered data is synced. Kafka WAL durability is controlled by its producer and cluster settings instead of this local option. diff --git a/docs/contributor-guide/flownode/arrangement.md b/docs/contributor-guide/flownode/arrangement.md index 28551db10c..8b472ea316 100644 --- a/docs/contributor-guide/flownode/arrangement.md +++ b/docs/contributor-guide/flownode/arrangement.md @@ -1,14 +1,22 @@ --- -keywords: [legacy streaming mode, Arrangement, state, differential updates, watermark] -description: In-memory Arrangement state used by Flownode's legacy streaming path. +keywords: [arrangement component, state storage, update streams, key-value pairs, querying and updating] +description: Details on the arrangement component in Flownode, which stores state and update streams for querying and updating. --- # Arrangement -`Arrangement` is an in-memory state index used by Flownode's legacy streaming path. It is implemented in `src/flow/src/utils.rs`; batching mode does not use it. +This page describes state used by Flownode's legacy streaming mode. Batching mode does not use an Arrangement. -An Arrangement stores updates as `((key row, value row), timestamp, diff)`. The timestamp orders changes in dataflow time, and the differential `diff` adds or removes a value. `get(now: Timestamp, key: &Row)` returns the value visible for a key at the requested time. +Arrangement stores the state in the dataflow's process. It stores the streams of update flows for further querying and updating. -The low watermark is the earliest time for which history may still be needed. State older than that watermark is assumed to have reached the sink and can be compacted. Advancing it too far would make later differential updates impossible to reconcile. +The arrangement essentially stores key-value pairs with timestamps to mark their change time. -For the current implementation, a `diff` of `-1` removes a key. Inserting the same key with a different value replaces the previous value. These semantics are part of the legacy streaming state model and must not be applied to batching-mode sink writes. +Internally, the arrangement receives tuples like +`((Key Row, Value Row), timestamp, diff)` and stores them in memory. One can query key-value pairs at a certain time using the `get(now: Timestamp, key: Row)` method. +The arrangement also assumes that everything older than a certain time (also known as the low watermark) has already been ingested to the sink tables and does not keep a history for them. + +:::tip NOTE + +The arrangement allows for the removal of keys by setting the `diff` to -1 in incoming tuples. Moreover, if a row has been previously added to the arrangement and the same key is inserted with a different value, the original value is overwritten with the new value. + +::: diff --git a/docs/contributor-guide/flownode/batching_mode.md b/docs/contributor-guide/flownode/batching_mode.md index 07a36186d1..37a695ce9c 100644 --- a/docs/contributor-guide/flownode/batching_mode.md +++ b/docs/contributor-guide/flownode/batching_mode.md @@ -1,50 +1,74 @@ --- -keywords: [batching mode, BatchingEngine, dirty time windows, checkpoints, continuous aggregation] -description: Batching mode task lifecycle, dirty-window processing, and recovery invariants. +keywords: [batching mode, flow management, Flownode components, Flownode limitations, continuous aggregation] +description: Overview of Flownode's batching mode, the active execution mode for continuous data aggregation, including its architecture and query execution flow. --- # Flownode Batching Mode Developer Guide -Batching mode maintains a sink table by rerunning a Flow query for source data that may have changed. It is the actively developed Flownode execution path. Mode selection remains internal; see the [Flownode overview](./overview.md). +This guide provides a brief overview of the batching mode in `flownode`. It's intended for developers who want to understand the internal workings of this mode. ## Overview -For a time-windowed Flow, writes to a source table mark the corresponding windows as dirty. A background task consumes those windows, adds time predicates to the Flow query when the query shape permits it, and sends an insert plan to Frontend. Frontend executes the query and writes the result to the sink table. +The batching mode in `flownode` is designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. This is in contrast to the original streaming mode, now deprecated, where data was processed as it arrived. -Evaluation-interval and TQL flows can require an unfiltered execution rather than a dirty-window filter. The batching path therefore treats a dirty window either as an exact range to recompute or as a signal that a full query is required, depending on the Flow definition. +The core idea is to: +1. Define a `flow` with a SQL query that aggregates data from a source table into a sink table. +2. The query typically includes a time window function (e.g., `date_bin`) on a timestamp column. +3. When new data is inserted into the source table, the system marks the corresponding time windows as "dirty." +4. A background task periodically wakes up, identifies these dirty windows, and re-runs the aggregation query for those specific time ranges. +5. The results are then inserted into the sink table, effectively updating the aggregated view. ## Architecture +The batching mode consists of several key components that work together to achieve this continuous aggregation. As shown in the diagram below: + +![batching mode architecture](/batching_mode_arch.png) + ### `BatchingEngine` -`BatchingEngine` in `src/flow/src/batching_mode/engine.rs` owns the map from `FlowId` to `BatchingTask`. It creates and removes tasks, handles flush requests, and dispatches dirty-window notifications to every Flow that reads the affected source table. +The `BatchingEngine` is the heart of the batching mode. It's a central component that manages all active flows. Its primary responsibilities are: -Task creation parses the Flow query, records source and sink tables, creates the sink table when needed, and initializes the execution state. Metadata for the Flow itself is persisted by `common-meta`. +- **Task Management**: It maintains a map of `FlowId` to `BatchingTask`. It handles the creation, deletion, and retrieval of these tasks. +- **Event Dispatching**: When new data arrives (via `handle_inserts_inner`) or when time windows are explicitly marked as dirty (`handle_mark_dirty_time_window`), the `BatchingEngine` identifies which flows are affected and forwards the information to the corresponding `BatchingTask`s. ### `BatchingTask` -One `BatchingTask` represents one Flow. `TaskConfig` contains immutable query, table, window, expiration, and scheduling data. `TaskState` contains the mutable execution state. +A `BatchingTask` represents a single, independent data flow. Each task is associated with one `flow` definition and runs in its own asynchronous loop. -The background loop waits for its schedule or a notification, generates the next insert plan, executes it through `FrontendClient`, and records the result. An execution lock serializes background execution, manual flush, plan generation, and checkpoint updates so two runs cannot consume the same state concurrently. +- **Configuration (`TaskConfig`)**: This struct holds the immutable configuration for a flow, such as the SQL query, source and sink table names, and time window expression. +- **State (`TaskState`)**: This contains the dynamic, mutable state of the task, most importantly the `DirtyTimeWindows`. +- **Execution Loop**: The task runs an infinite loop (`start_executing_loop`) that: + 1. Checks for a shutdown signal. + 2. Waits for a scheduled interval or until it's woken up. + 3. Generates a new query plan (`gen_insert_plan`) based on the current set of dirty time windows. + 4. Executes the query (`execute_logical_plan`) against the database. + 5. Cleans up the processed dirty windows. ### `TaskState` and `DirtyTimeWindows` -`DirtyTimeWindows` stores non-overlapping ranges that need recomputation. Plan generation removes a bounded set of ranges from the queue. If planning or execution fails, those ranges are restored; they are not discarded merely because a run started. +- **`TaskState`**: This struct tracks the runtime state of a `BatchingTask`. It includes `dirty_time_windows`, which is crucial for determining what work needs to be done. +- **`DirtyTimeWindows`**: This is a key data structure that keeps track of which time windows have received new data since the last query execution. It stores a set of non-overlapping time ranges. When a task's execution loop runs, it consults this structure to build a `WHERE` clause that filters the source table for only the dirty time windows. -`TaskState` also stores per-Region checkpoints for the experimental incremental-read path. Incremental mode advances a checkpoint only when the result reports a complete watermark proof for the participating Regions. A scoped full-snapshot repair freezes a high watermark while it drains dirty windows; new writes stay in the live queue. If the repair fails or its watermark proof is incomplete, pending windows return to the queue. +### `TimeWindowExpr` -Incremental reads are disabled by default through `experimental_enable_incremental_read`. When disabled or when the query shape is incompatible, the task uses full-snapshot execution. +The `TimeWindowExpr` is a helper utility for dealing with time window expressions like `date_bin`. -### `TimeWindowExpr` +- **Evaluation**: It can take a timestamp and evaluate the time window expression to determine the start and end of the window that the timestamp falls into. +- **Window Size**: It can also determine the size (duration) of the time window from the expression. -`TimeWindowExpr` in `src/flow/src/batching_mode/time_window.rs` evaluates window expressions such as `date_bin`. It maps an input timestamp to its window bounds and provides the window size used to merge ranges and generate predicates. +This is essential for both marking windows as dirty and for generating the correct filter conditions when querying the source table. ## Query Execution Flow -1. A source write or explicit mark-dirty request identifies the affected Flow and time range. -2. `BatchingEngine` adds the range to the task's `DirtyTimeWindows` and wakes the task when required. -3. `BatchingTask` consumes a bounded group of windows and builds a filtered insert plan, or chooses an unfiltered full query for a Flow that cannot be scoped safely. -4. `FrontendClient` sends the serialized logical plan to Frontend. Frontend executes the query and writes rows to the sink table. -5. On success, the task commits the execution state and advances only checkpoints justified by returned watermarks. On failure, it restores consumed windows before the next retry. - -Changes to plan coverage, checkpoint advancement, or dirty-window restoration affect correctness. A run must never clear work that has not been reflected in the sink table, and a checkpoint must never move beyond data proven to be included in the result. +Here's a simplified step-by-step walkthrough of how a query is executed in batch mode: + +1. **Data Ingestion**: New data is written to a source table. +2. **Marking Dirty**: The `BatchingEngine` receives a notification about the new data. It uses the `TimeWindowExpr` associated with each relevant flow to determine which time windows are affected by the new data points. These windows are then added to the `DirtyTimeWindows` set in the corresponding `TaskState`. +3. **Task Wake-up**: The `BatchingTask`'s execution loop wakes up, either due to its periodic schedule or because it was notified of a large backlog of dirty windows. +4. **Plan Generation**: The task calls `gen_insert_plan`. This method: + - Inspects the `DirtyTimeWindows`. + - Generates a series of `OR`'d `WHERE` clauses (e.g., `(ts >= 't1' AND ts < 't2') OR (ts >= 't3' AND ts < 't4') ...`) that cover the dirty windows. + - Rewrites the original SQL query to include this new filter, ensuring that only the necessary data is processed. +5. **Execution**: The modified query plan is sent to the `Frontend` for execution. The database processes the aggregation on the filtered data. +6. **Upsert**: The results are inserted into the sink table. The sink table is typically defined with a primary key that includes the time window column, so new results for an existing window will overwrite (upsert) the old ones. +7. **State Update**: The `DirtyTimeWindows` set is cleared of the windows that were just processed. The task then goes back to sleep until the next interval. diff --git a/docs/contributor-guide/flownode/dataflow.md b/docs/contributor-guide/flownode/dataflow.md index cec6366273..090ac870e9 100644 --- a/docs/contributor-guide/flownode/dataflow.md +++ b/docs/contributor-guide/flownode/dataflow.md @@ -1,14 +1,19 @@ --- -keywords: [legacy streaming mode, dataflow, DFIR, differential rows, Flow] -description: Internal compute graph used by Flownode's legacy streaming execution path. +keywords: [dataflow module, SQL query transformation, execution plan, DAG, map and reduce operations] +description: Explanation of the dataflow module in Flownode, its operations, internal data handling, and future enhancements. --- # Dataflow -This page describes the compute graph used by Flownode's legacy streaming mode. New continuous-aggregation work uses [batching mode](./batching_mode.md); do not use this page to infer batching behavior. +This page describes the compute graph used by Flownode's legacy streaming mode. New continuous-aggregation work uses [batching mode](./batching_mode.md). -The streaming path converts a Flow definition through `src/flow/src/transform.rs` into a typed plan in `plan.rs`. `src/flow/src/compute/render.rs` renders supported plan nodes into a DFIR-style dataflow graph, and workers under `src/flow/src/adapter/` own and execute those graphs. +The `dataflow` module (see `flow::compute` module) is the core computing module of `flow`. +It takes a SQL query and transforms it into flow's internal execution plan. +This execution plan is then rendered into an actual dataflow, which is essentially a directed acyclic graph (DAG) of functions with input and output ports. +The dataflow is triggered to run when needed. -The internal record is a differential row `(row, timestamp, diff)`. `row` contains the values, `timestamp` tracks dataflow progress, and `diff` records multiplicity changes such as insertion (`+1`) and deletion (`-1`). Operators propagate those changes so aggregates and sink output can be updated incrementally. +Currently, this dataflow only supports `map` and `reduce` operations. Support for `join` operations will be added in the future. -The typed plan represents map/filter/project and reduce operations, along with join and union nodes. The streaming renderer currently executes map/filter/project and reduce; join and union rendering still return a not-implemented error. Check both `plan.rs` and `compute/render.rs` before adding an operator, because being representable in a plan does not mean it is executable. +Internally, the dataflow handles data in row format, using a tuple `(row, time, diff)`. Here, `row` represents the actual data being passed, which may contain multiple `Value` objects. +`time` is the system time which tracks the progress of the dataflow, and `diff` typically represents the insertion or deletion of the row (+1 or -1). +Therefore, the tuple represents the insert/delete operation of the `row` at a given system `time`. diff --git a/docs/contributor-guide/flownode/overview.md b/docs/contributor-guide/flownode/overview.md index 5677f41c22..d1317bfc1c 100644 --- a/docs/contributor-guide/flownode/overview.md +++ b/docs/contributor-guide/flownode/overview.md @@ -1,28 +1,26 @@ --- -keywords: [Flownode, continuous aggregation, batching mode, streaming mode, Flow] -description: Flownode's execution modes, routing boundary, and implementation layout. +keywords: [continuous aggregation, flow management, standalone mode, Flownode components, Flownode limitations] +description: Overview of Flownode, a component providing Flow computation capabilities to the database, including batching mode, deprecated streaming mode, and core components. --- # Flownode ## Introduction -Flownode is the execution component behind GreptimeDB Flow, which maintains continuously computed results from source tables in a sink table. It runs in-process in standalone mode and as a separate service in distributed mode. -Flownode has two execution paths: +`Flownode` provides Flow computation capabilities to the database. +`Flownode` manages `flows` which are tasks that receive data from the `source` and send data to the `sink`. -- **Batching mode** is the actively developed path. It tracks affected time windows and periodically runs an aggregation query through Frontend. See the [batching mode guide](./batching_mode.md). -- **Streaming mode** is the legacy incremental-dataflow path. It processes row-level changes through worker-owned compute graphs and remains for compatibility. +`Flownode` support both `standalone` and `distributed` mode. In `standalone` mode, `Flownode` runs in the same process as the database. In `distributed` mode, `Flownode` runs in a separate process and communicates with the database through the network. -Users do not select an execution mode directly. `flow_type` is reserved internal metadata. `StatementExecutor::determine_flow_type` in `src/operator/src/statement/ddl.rs` chooses the mode when a Flow is created, and `FlowDualEngine` handles compatibility routing inside Flownode. +There are two execution modes for a flow: +- **Batching Mode**: The active mode for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. Aggregation and TQL queries use this mode. For more details, see the [Batching Mode Developer Guide](./batching_mode.md). +- **Streaming Mode (deprecated)**: The original mode where data is processed as it arrives. It is kept for legacy compatibility and is not recommended for new workloads. ## Components -- `FlowEngine` in `src/flow/src/engine.rs` defines the create, remove, flush, and insert lifecycle shared by both paths. -- `FlowDualEngine` in `src/flow/src/adapter/flownode_impl.rs` routes each Flow to the batching or streaming engine. -- `src/flow/src/batching_mode/` contains time-window tracking, task scheduling, Frontend RPC, sink-table creation, and checkpoint logic. -- `src/flow/src/adapter/`, `compute/`, `expr/`, and `plan.rs` implement the legacy streaming path. -- `src/flow/src/server.rs` exposes the Flownode gRPC service; `heartbeat.rs` reports Flownode state to Metasrv. -- Persisted Flow metadata and DDL procedures live in `src/common/meta/`, not in the `flow` crate. +A `Flownode` contains all the components needed to execute a flow. The specific components involved depend on the execution mode. At a high level, the key parts are: -Mode-specific changes must be reviewed against `FlowDualEngine` and the shared metadata contract. Do not assume that a fix in one execution path applies to the other. +- **Flow Manager**: A central component responsible for managing the lifecycle of all flows. +- **Task Executor**: The runtime environment where the flow logic is executed. In batching mode, this is a `BatchingTask`; in the deprecated streaming mode, this is typically a `FlowWorker`. +- **Flow Task**: Represents a single, independent data flow, containing the logic for transforming data from a source to a sink. diff --git a/docs/contributor-guide/frontend/distributed-querying.md b/docs/contributor-guide/frontend/distributed-querying.md index ebc490b80e..ca3822e113 100644 --- a/docs/contributor-guide/frontend/distributed-querying.md +++ b/docs/contributor-guide/frontend/distributed-querying.md @@ -1,24 +1,20 @@ --- -keywords: [distributed querying, DistPlannerAnalyzer, MergeScan, Substrait, Region pruning] -description: How GreptimeDB turns a logical query plan into local and remote execution stages. +keywords: [distributed querying, dist planner, dist plan, logical plan, substrait format] +description: Describes the process of distributed querying in GreptimeDB, focusing on the dist planner and dist plan. --- # Distributed Querying -Frontend and Datanode use the same DataFusion-based query engine. In distributed mode, Frontend adds a planning step that separates work executed on Datanodes from work completed by Frontend. +Frontend and Datanode use the same DataFusion-based query engine. In distributed mode, Frontend adds a planning step that separates work performed by Datanodes from work completed by Frontend. ![Frontend query](/frontend-query.png) ## Dist Planner -`DistPlannerAnalyzer` in `src/query/src/dist_plan/analyzer.rs` rewrites the DataFusion logical plan. It pushes compatible operators toward table scans and wraps remote subplans in `MergeScan` nodes. The planner uses operator commutativity and plan-shape rules to decide which work is safe to execute on each Datanode; unsupported shapes remain on Frontend or use the configured fallback path. +The distributed planner rewrites the logical plan. It pushes compatible operators toward table scans and wraps remote subplans in `MergeScan` nodes. Partition predicates are also used to prune Regions before the remote work is scheduled. -Filters on partition columns are also used to prune Regions. Frontend resolves each selected Region to a Datanode through `FrontendRegionQueryHandler` before execution. - -The original design and its commutativity rules are documented in the [distributed planner RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md). +Whether an operator can be pushed down depends on the plan shape and the operator's properties. Unsupported parts remain on Frontend. The original design and its commutativity rules are described in the [distributed planner RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md). ## Dist Plan -A `MergeScan` remote input is a complete logical subplan. Frontend serializes that subplan with Substrait and sends a Region-specific query request to the selected Datanode. The Datanode plans and executes the subplan against its local Regions and streams Arrow record batches back. - -Frontend merges the remote streams and executes any operators that could not be pushed down. This boundary is not limited to the logical `TableScan` node: filters, projections, partial aggregates, and other compatible operators may be part of a remote subplan. +A remote input is a complete logical subplan, not just a table scan. Frontend serializes the subplan in [Substrait](https://substrait.io) format and sends a Region-specific request to the Datanode that owns the data. The Datanode plans and executes it locally, then streams the result back. Frontend merges the remote streams and executes any operators that were not pushed down. diff --git a/docs/contributor-guide/frontend/overview.md b/docs/contributor-guide/frontend/overview.md index 170124650f..3a2f63e7ae 100644 --- a/docs/contributor-guide/frontend/overview.md +++ b/docs/contributor-guide/frontend/overview.md @@ -1,46 +1,44 @@ --- -keywords: [frontend, protocols, request routing, distributed query, authorization] -description: Overview of Frontend, GreptimeDB's stateless request entry point and query coordinator. +keywords: [frontend, proxy, protocol, routing, distributed query, tenant management, authorization, flow control, cloud deployment, endpoints] +description: Overview of GreptimeDB's Frontend component - a stateless proxy service for client requests. --- # Frontend -Frontend is GreptimeDB's stateless request entry point and orchestration layer. It implements the business logic behind the protocol servers, plans queries, routes writes and Region reads, and coordinates distributed query execution. +The **Frontend** is a stateless service that serves as the entry point for client requests in GreptimeDB. It provides a unified interface for multiple database protocols and acts as a proxy that forwards read/write requests to appropriate Datanodes in the distributed system. -Network listeners and wire formats belong to the `servers` crate. The `frontend` crate implements handler traits for SQL, gRPC, MySQL, PostgreSQL, InfluxDB, OpenTelemetry, Prometheus, OpenTSDB, Jaeger, and other supported interfaces. +## Core Functions - - -## Responsibilities - -- Parse and plan SQL, PromQL, and log queries. -- Check permissions and carry session context through request processing. -- Route inserts, deletes, and Region queries using catalog and route metadata. -- Dispatch distributed query fragments to Datanodes and merge their results. - -See the [protocol overview](/user-guide/protocols/overview.md) for the user-facing interfaces. +- **Protocol Support**: Multiple database protocols including SQL, PromQL, MySQL, and PostgreSQL. See [Protocols][1] for details +- **Request Routing**: Routes requests to appropriate Datanodes based on metadata +- **Query Distribution**: Splits distributed queries across multiple nodes +- **Response Aggregation**: Combines results from multiple Datanodes +- **Authorization**: Security and access control validation ## Architecture ### Key Components - -- `Instance` in `src/frontend/src/instance.rs` is the main business-logic container and implements the server handler traits. -- Modules under `src/frontend/src/instance/` handle individual request types and protocols. -- `StatementExecutor` in the `operator` crate handles statements and write-side operations. -- The `query` crate owns logical planning, optimization, and distributed plans. -- `FrontendRegionQueryHandler` in `instance/region_query.rs` resolves Region targets and sends query requests to Datanodes. +- **Protocol Handlers**: Handle different database protocols +- **Catalog Manager**: Caches metadata from Metasrv to enable efficient request routing and schema validation +- **Dist Planner**: Converts logical plans to distributed execution plans +- **Request Router**: Determines target Datanodes for each request ### Request Flow -In standalone mode, Frontend accesses an embedded Datanode through a local `RegionServer` adapter. In distributed mode, it uses metadata from Metasrv and RPC clients to reach remote Datanodes. +![request flow](/request_flow.png) ### Deployment -Frontend instances do not own table data. Multiple instances can serve requests against the same Metasrv and Datanode cluster. +The following picture shows a typical deployment of GreptimeDB in the cloud. The `Frontend` instances +form a cluster to serve the requests from clients: + +![frontend](/frontend.png) - +## Details -## Implementation guides +- [Table Sharding][2] +- [Distributed Querying][3] -- [Table Sharding](./table-sharding.md) -- [Distributed Querying](./distributed-querying.md) +[1]: /user-guide/protocols/overview.md +[2]: ./table-sharding.md +[3]: ./distributed-querying.md diff --git a/docs/contributor-guide/frontend/table-sharding.md b/docs/contributor-guide/frontend/table-sharding.md index 1e15c2a4f9..a60276d14e 100644 --- a/docs/contributor-guide/frontend/table-sharding.md +++ b/docs/contributor-guide/frontend/table-sharding.md @@ -5,19 +5,23 @@ description: Explains how table data in GreptimeDB is sharded and distributed, i # Table Sharding -GreptimeDB shards a table into partitions and stores each partition in a Region. This page describes the implementation-level relationship between those objects. +The sharding of stored data is essential to any distributed database. This document will describe how table's data in GreptimeDB is being sharded, and distributed. ## Partition -The [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) section in the User Guide documents the partition syntax. +For the syntax of creating a partitioned table, please refer to the [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) section in the User Guide. ## Region -Each partition maps to one Region, which is the storage and scheduling unit managed by Datanodes. Metasrv stores the route that maps each Region to its Datanode. +The data within a table is logically split after creating partitions. You may ask the question " +how are the data, after being logically partitioned, stored in the GreptimeDB? The answer is in "`Region`"s. + +Each region is corresponding to a partition, and stores the data in the partition. The regions are distributed among +`Datanode`s. `Metasrv` manages the route information that maps regions to Datanodes. If the partition layout needs to change after table creation, GreptimeDB supports explicit [repartitioning](/user-guide/deployments-administration/manage-data/repartition.md) through split and merge operations. -The relationship is shown below: +The relationship between partition and region can be viewed as the following diagram: ```text ┌───────┐ diff --git a/docs/contributor-guide/getting-started.md b/docs/contributor-guide/getting-started.md index 0bfbd91e81..bf6ec2a8fb 100644 --- a/docs/contributor-guide/getting-started.md +++ b/docs/contributor-guide/getting-started.md @@ -1,30 +1,33 @@ --- -keywords: [setup, build from source, Rust toolchain, unit tests] -description: Set up a development environment and build, run, and test GreptimeDB from source. +keywords: [setup, running from source, prerequisites, build dependencies, unit tests] +description: Instructions for setting up and running GreptimeDB from source, including prerequisites, build dependencies, and running unit tests. --- # Getting started -This page covers the minimum setup for building and running GreptimeDB from source. +This page describes how to run GreptimeDB from source in your local environment. - - -## Prerequisites +## Prerequisite ### System & Architecture -GreptimeDB supports Linux and macOS on x86-64 and Arm64, as well as Windows. +At the moment, GreptimeDB supports Linux (both amd64 and arm64), macOS (both amd64 and Apple Silicon), and Windows. ### Build Dependencies -- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line). -- A C/C++ build toolchain, such as `build-essential` on Ubuntu or Xcode Command Line Tools on macOS. -- [Rustup](https://rustup.rs/). The repository's `rust-toolchain.toml` selects the required nightly toolchain automatically. -- [Protocol Buffers compiler](https://grpc.io/docs/protoc-installation/) 3.15 or later. Check the installed version with `protoc --version`. +- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) +- C/C++ Toolchain: provides essential tools for compiling and linking. This is available either as `build-essential` on ubuntu or a similar name on other platforms. +- [Rustup][1]. The repository pins the required nightly toolchain in `rust-toolchain.toml`. +- Protobuf ([guide][2]) + - Compile the proto file + - Note that the version needs to be >= 3.15. You can check it with `protoc --version` + +[1]: +[2]: ## Compile and Run -Clone the repository and start a standalone instance: +Start GreptimeDB standalone instance in just a few commands! ```shell git clone https://github.com/GreptimeTeam/greptimedb.git @@ -32,32 +35,34 @@ cd greptimedb cargo run -- standalone start ``` -To build without starting the server, run: +Next, you can choose the protocol you like to interact with in GreptimeDB. + +Or if you just want to build the server without running it: ```shell -cargo build +cargo build # --release ``` -Add `--release` for an optimized build. Artifacts are written to `target/debug` or `target/release`. +The artifacts can be found under `$REPO/target/debug` or `$REPO/target/release`, depending on the build mode (whether the `--release` option is passed) - +## Unit test -## Unit tests +GreptimeDB is well-tested, the entire unit test suite is shipped with source code. To test them, run with [nextest](https://nexte.st/index.html). -GreptimeDB uses [cargo-nextest](https://nexte.st/) as its standard Rust test runner. Install it with: +To install nextest using cargo, run: ```shell cargo install cargo-nextest --locked ``` -Run the workspace test suite with the features used by CI: +Or you can check their [docs](https://nexte.st/docs/installation/pre-built-binaries/) for other ways to install. + +After nextest is ready, you can run the test suite with: ```shell cargo nextest run --workspace --features pg_kvbackend,mysql_kvbackend ``` -For package-scoped tests and other test types, see the [testing guide](./tests/overview.md). - ## Docker -Prebuilt images are published to [Docker Hub](https://hub.docker.com/r/greptime/greptimedb). They are useful for running GreptimeDB, but do not replace the source build when developing or testing code changes. +We also provide prebuilt binaries via Docker, available on Docker Hub: [https://hub.docker.com/r/greptime/greptimedb](https://hub.docker.com/r/greptime/greptimedb) diff --git a/docs/contributor-guide/how-to/how-to-trace-greptimedb.md b/docs/contributor-guide/how-to/how-to-trace-greptimedb.md index ee7a1f6744..75941aa0be 100644 --- a/docs/contributor-guide/how-to/how-to-trace-greptimedb.md +++ b/docs/contributor-guide/how-to/how-to-trace-greptimedb.md @@ -1,88 +1,80 @@ --- -keywords: [tracing, W3C Trace Context, RPC, instrument, runtime] -description: Propagate and instrument distributed traces in GreptimeDB code. +keywords: [tracing, distributed tracing, trace_id, RPC, instrument, span, runtime] +description: Describes how to use Rust's tracing framework in GreptimeDB for distributed tracing, including defining tracing context in RPC, passing it, and instrumenting code. --- # How to trace GreptimeDB -GreptimeDB uses the Rust [`tracing`](https://docs.rs/tracing/latest/tracing/) ecosystem and OpenTelemetry context propagation. Local spans are connected automatically only while their tracing context is carried through the same asynchronous execution path. RPC and runtime boundaries require explicit propagation. +GreptimeDB uses Rust's [tracing](https://docs.rs/tracing/latest/tracing/) framework for code instrument. For the specific details and usage of tracing, please refer to the official documentation of tracing. -The shared implementation is [`TracingContext`](https://github.com/GreptimeTeam/greptimedb/blob/main/src/common/telemetry/src/tracing_context.rs) in `common-telemetry`. It converts the active span context to and from W3C Trace Context fields. +By transparently transmitting `trace_id` and other information on the entire distributed system, we can record the function call chain of the entire distributed link, know the time of each tracked function take and other related information, so as to monitor the entire system. - +## Define tracing context in RPC -## RPC context fields +Because the tracing framework does not natively support distributed tracing, we need to manually pass information such as `trace_id` in the RPC message to correctly identify the function calling relationship. We use standards based on [w3c](https://www.w3.org/TR/trace-context/#traceparent-header-field-values) to encode relevant information into `tracing_context` and attach the message to the RPC header. Mainly defined in: -GreptimeDB protobuf headers store W3C trace fields in a `map tracing_context` field: +- `frontend` interacts with `datanode`: `tracing_context` is defined in [`RegionRequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/region/server.proto) +- `frontend` interacts with `metasrv`: `tracing_context` is defined in [`RequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/meta/common.proto) +- Client interacts with `frontend`: `tracing_context` is defined in [`RequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/common.proto) -- Frontend to Datanode: [`RegionRequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/region/server.proto) -- Meta clients and services: [`RequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/meta/common.proto) -- Client to Frontend database RPC: [`RequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/common.proto) +## Pass tracing context in RPC call -When adding an internal RPC, use the existing header type when possible. A separate tracing field with a different encoding creates a propagation path that the common helpers cannot handle. +We build a `TracingContext` structure that encapsulates operations related to the tracing context. [Related code](https://github.com/GreptimeTeam/greptimedb/blob/main/src/common/telemetry/src/tracing_context.rs) - +GreptimeDB uses `TracingContext::from_current_span()` to obtain the current tracing context, uses the `to_w3c()` method to encode the tracing context into a w3c-compliant format, and attaches it to the RPC message, so that the tracing context is correctly distributed passed within the component. -## Propagate context across an RPC +The following example illustrates how to obtain the current tracing context and pass the parameters correctly when constructing the RPC message, so that the tracing context is correctly passed among the distributed components. -Capture the current context when constructing an outbound request: ```rust let request = RegionRequest { - header: Some(RegionRequestHeader { - tracing_context: TracingContext::from_current_span().to_w3c(), - ..Default::default() - }), - body: Some(region_request::Body::Alter(request)), + header: Some(RegionRequestHeader { + tracing_context: TracingContext::from_current_span().to_w3c(), + ..Default::default() + }), + body: Some(region_request::Body::Alter(request)), }; ``` -At the receiver, decode the header and attach a new local span as a child of that context: +On the receiver side of the RPC message, the tracing context needs to be correctly decoded and used to build the first `span` to trace the function call. For example, the following code will correctly decode the `tracing_context` in the received RPC message using the `TracingContext::from_w3c` method. And use the `attach` method to attach the context message to the newly created `info_span!("RegionServer::handle_read")`, so that the call can be tracked across distributed components. ```rust +... let tracing_context = request - .header - .as_ref() - .map(|header| TracingContext::from_w3c(&header.tracing_context)) - .unwrap_or_default(); - + .header + .as_ref() + .map(|h| TracingContext::from_w3c(&h.tracing_context)) + .unwrap_or_default(); let result = self - .handle_read(request) - .trace(tracing_context.attach(info_span!("RegionServer::handle_read"))) - .await?; + .handle_read(request) + .trace(tracing_context.attach(info_span!("RegionServer::handle_read"))) + .await?; +... ``` -An absent or invalid context becomes an empty context, so request handling still works without a parent trace. Do not reuse one request's context for unrelated work. - - - -## Instrument local work +## Use `tracing::instrument` to instrument the code -Use `#[tracing::instrument]` at asynchronous or expensive boundaries where a span helps correlate latency and errors. The macro records arguments through `Debug` by default. Skip credentials, tokens, large batches, query payloads, and any argument whose full value is not safe or useful in telemetry. +We use the `instrument` macro provided by tracing to instrument the code. We only need to annotate the `instrument` macro in the function that needs to be instrument. The `instrument` macro will print every function parameter on each function call into the span in the form of `Debug`. For parameters that do not implement the `Debug` trait, or the structure is too large and has too many parameters, resulting in a span that is too large. If you want to avoid these situations, you need to use `skip_all` to skip printing all parameters. ```rust -#[tracing::instrument(skip_all, fields(region_id = %region_id))] -async fn handle_region(region_id: RegionId, request: RegionRequest) { - region_server.handle(request).await; +#[tracing::instrument(skip_all)] +async fn instrument_function(....) { + ... } ``` -Prefer a small set of stable identifiers in `fields(...)` to recording a complete request. Instrumenting every helper function creates high-volume traces without improving the request-level call graph. - - - -## Propagate context across runtimes +## Code instrument across runtime -Moving a future to another runtime or spawning work outside the current instrumented future can lose the active parent. Capture the context before crossing that boundary and attach it to a new span inside the spawned future: +Rust's tracing library will automatically handle the nested relationship between instrument functions in the same runtime, but if a function call across the runtime, tracing library cannot automatically trace such calls, and we need to manually pass the context across the runtime. ```rust let tracing_context = TracingContext::from_current_span(); let handle = runtime.spawn(async move { - handler - .handle(query) - .trace(tracing_context.attach(info_span!("background_query"))) - .await + handler + .handle(query) + .trace(tracing_context.attach(info_span!("xxxxx"))) + ... }); ``` -The context must be captured before the spawn. Keep the attached span scoped to the spawned operation so unrelated tasks do not inherit the same parent. +For example, the above code needs to perform tracing across runtimes. We first obtain the current tracing context through `TracingContext::from_current_span()`, create a span in another runtime, and attach the span to the current context, and we are done. The hidden code points that span the runtime are eliminated, and the call chain is correctly traced. \ No newline at end of file diff --git a/docs/contributor-guide/how-to/how-to-use-tokio-console.md b/docs/contributor-guide/how-to/how-to-use-tokio-console.md index 0cb1b3a7fd..81cff8f7ce 100644 --- a/docs/contributor-guide/how-to/how-to-use-tokio-console.md +++ b/docs/contributor-guide/how-to/how-to-use-tokio-console.md @@ -1,30 +1,34 @@ --- -keywords: [tokio-console, tokio_unstable, asynchronous tasks, diagnostics] -description: Build GreptimeDB with tokio-console support and inspect its Tokio runtime. +keywords: [tokio-console, GreptimeDB, tokio_unstable, build, connect, subscriber] +description: Guides on using tokio-console in GreptimeDB, including building with specific features and connecting to the tokio console subscriber. --- # How to use tokio-console in GreptimeDB -[`tokio-console`](https://github.com/tokio-rs/console) displays live Tokio tasks and resources. GreptimeDB compiles the subscriber behind the `cmd/tokio-console` feature and also requires Tokio's unstable instrumentation cfg. +This document introduces how to use the [tokio-console](https://github.com/tokio-rs/console) in GreptimeDB. -Build GreptimeDB with both enabled: +First, build GreptimeDB with feature `cmd/tokio-console`. Also the `tokio_unstable` cfg must be enabled: ```bash RUSTFLAGS="--cfg tokio_unstable" cargo build -F cmd/tokio-console ``` -Start the component with a full socket address for the console subscriber: +Then start GreptimeDB with the tokio console binding address config: `--tokio-console-addr`. For example: ```bash -./target/debug/greptime --tokio-console-addr="127.0.0.1:6669" standalone start +greptime --tokio-console-addr="127.0.0.1:6669" standalone start ``` -The option is global and can also be used with `frontend`, `datanode`, `metasrv`, or `flownode` commands built with the same feature. - -Install the console client as described in the [tokio-console repository](https://github.com/tokio-rs/console#installing-the-console) and connect to the configured address: +Now you can use `tokio-console` to connect to GreptimeDB's tokio console subscriber: ```bash -tokio-console http://127.0.0.1:6669 +tokio-console [TARGET_ADDR] ``` -Keep the subscriber on a loopback or otherwise protected address. It is a diagnostic endpoint, not a public GreptimeDB protocol. The feature and `tokio_unstable` instrumentation add runtime diagnostics and should be enabled deliberately when investigating task stalls, wakeups, or resource contention. +"TARGET_ADDR" defaults to "\". + +:::tip Note + +You can refer to [tokio-console](https://github.com/tokio-rs/console) to see the installation of `tokio-console`. + +::: diff --git a/docs/contributor-guide/how-to/how-to-write-sdk.md b/docs/contributor-guide/how-to/how-to-write-sdk.md index 4d468a14c6..40d0bc3713 100644 --- a/docs/contributor-guide/how-to/how-to-write-sdk.md +++ b/docs/contributor-guide/how-to/how-to-write-sdk.md @@ -1,31 +1,42 @@ --- -keywords: [gRPC ingester SDK, GreptimeDatabase, RowInsertRequests, streaming RPC] -description: Protocol and reliability requirements for a GreptimeDB gRPC ingester SDK. +keywords: [gRPC SDK, GreptimeDatabase, Handle, HandleRequests, GreptimeRequest, GreptimeResponse] +description: Explains how to write a gRPC SDK for GreptimeDB, focusing on the GreptimeDatabase service, its methods, and the structure of requests and responses. --- # How to write a gRPC SDK for GreptimeDB -This guide covers an **ingester SDK** built on GreptimeDB's native gRPC database service. Query drivers and clients are outside its scope. Official ingester libraries use the `greptimedb-ingester-` naming pattern. - -Generate message and client code from the versioned [greptime-proto](https://github.com/GreptimeTeam/greptime-proto) definitions rather than copying message layouts into an SDK. Keep the generated protocol package separate from the ergonomic row and batch APIs exposed to application code. +A GreptimeDB gRPC SDK only needs to handle the writes. The reads are standard SQL and PromQL, can be handled by any JDBC +client or Prometheus client. This is also why GreptimeDB gRPC SDKs are all named +like "`greptimedb-ingester-`". Please make sure your GreptimeDB SDK follow the same naming convention. ## `GreptimeDatabase` Service -[`database.proto`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto) defines two RPC methods: +GreptimeDB defines a custom gRPC service called `GreptimeDatabase`. All you need to do in your SDK are implement it. You +can find its Protobuf +definitions [here](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto). + +The service contains two RPC methods: ```protobuf service GreptimeDatabase { rpc Handle(GreptimeRequest) returns (GreptimeResponse); + rpc HandleRequests(stream GreptimeRequest) returns (GreptimeResponse); } ``` -`Handle` is a unary RPC. `HandleRequests` is a [client-streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc): the client sends a stream of requests, closes its send side, and receives one summarized response. A production SDK should apply bounded buffering and gRPC flow control rather than accumulating an unbounded batch in memory. +The `Handle` method is for unary call: when a `GreptimeRequest` is received and processed by a GreptimeDB +server, it responds with a `GreptimeResponse` immediately. -The protocol has no request-level idempotency key. An SDK must not promise exactly-once ingestion. If a transport failure leaves the server outcome unknown, an automatic retry can duplicate data for table configurations that retain duplicate rows; make retry behavior explicit to callers. +The `HandleRequests` acts in +a "[Client streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc)" style. It ingests a +stream of `GreptimeRequest`, and handles them on the fly. After all the requests have been handled, it returns a +summarized `GreptimeResponse`. Through `HandleRequests`, we can achieve a very high throughput of requests handling. ### `GreptimeRequest` +The `GreptimeRequest` is a Protobuf message defined like this: + ```protobuf message GreptimeRequest { RequestHeader header = 1; @@ -40,21 +51,27 @@ message GreptimeRequest { } ``` -For ingestion, prefer `RowInsertRequests`. Each `RowInsertRequest` names one table and carries a `Rows` schema plus row values. Validate column count, data type, semantic type, and null representation before sending so client-side construction errors do not become opaque server errors. The older column-oriented `InsertRequests` remains part of the protocol for compatibility. +A `RequestHeader` is needed, it includes some context, authentication and others. The "oneof" field contains the request +to the GreptimeDB server. -Every request includes a `RequestHeader`. Populate the target catalog and schema, authentication header, timezone, and W3C tracing context when the corresponding SDK option is set. Do not silently replace an explicitly selected catalog or schema with a client default. +Note that we have two types of insertions, one is in the form of "column" (the `InsertRequests`), and the other is " +row" (`RowInsertRequests`). It's generally recommended to use the "row" form, since it's more natural for insertions on +a table, and easier to use. However, if there's a need to insert a large number of columns at once, or there're plenty +of "null" values to insert, the "column" form is better to be used. ### `GreptimeResponse` +The `GreptimeResponse` is a Protobuf message defined like this: + ```protobuf message GreptimeResponse { ResponseHeader header = 1; - oneof response { - AffectedRows affected_rows = 2; - } + oneof response {AffectedRows affected_rows = 2;} } ``` -Successful gRPC transport does not by itself mean the database operation succeeded. Inspect `ResponseHeader.status`, map non-success status codes and `err_msg` into the SDK's error type, and return `affected_rows` only after that check. Preserve the underlying gRPC status separately from a GreptimeDB response status so callers can distinguish transport failures from server-side request errors. +The `ResponseHeader` contains the response's status code, and error message (if there's any). The "oneof" response only +contains the affected rows for now. -Protobuf clients must also tolerate unknown fields and an unset response variant. Add compatibility tests using serialized messages from the supported protocol versions, plus integration tests for unary writes, client streaming, authentication errors, partial stream failure, and server status propagation. +GreptimeDB has a lot of SDKs now, you can refer to +them [here](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=) for some examples. diff --git a/docs/contributor-guide/metasrv/admin-api.md b/docs/contributor-guide/metasrv/admin-api.md index 5e635cf6b3..ee8664b821 100644 --- a/docs/contributor-guide/metasrv/admin-api.md +++ b/docs/contributor-guide/metasrv/admin-api.md @@ -1,53 +1,155 @@ --- keywords: [admin api, health check, leader query, heartbeat, maintenance mode, RESTful API] -description: Maintainer reference for Metasrv's unauthenticated Admin API router and state-changing endpoints. +description: Details the Admin API for Metasrv, including endpoints for health checks, leader queries, heartbeat data, maintenance mode, and Procedure Manager controls. --- # Admin API -The Axum router is assembled in `src/meta-srv/src/service/admin.rs` and mounted under `/admin` on Metasrv's HTTP server. The default HTTP port is `4000`. +:::tip +Note that all Admin API endpoints in this document listen on Metasrv's `HTTP_PORT`, which defaults to `4000`. +::: -The router does not add authentication. Some endpoints change cluster behavior, so deployments must protect this port with network-level controls. When adding a route, define its HTTP method explicitly, keep read and mutation handlers separate, and add handler-level tests in `src/meta-srv/src/service/admin/`. +The Admin API exposes Metasrv health, leader, Datanode heartbeat, maintenance mode, and Procedure Manager information over HTTP. It does not provide authentication, and some endpoints change cluster behavior. Deployments must protect the HTTP port with network-level controls. +This page covers the following APIs: -## /health HTTP endpoint +- /health +- /leader +- /heartbeat +- /maintenance +- /procedure-manager -`GET /admin/health` returns `OK` when the HTTP service is running. It does not prove that this node is the current leader or that external dependencies are reachable. The handler is in `health.rs`. +All these APIs are under the parent resource `/admin`. + +In the following sections, we assume that your metasrv instance is running on localhost port 4000. + +## /health HTTP endpoint + +The `/health` endpoint accepts GET requests and returns `OK` when the HTTP service is running. It does not check whether this Metasrv is the leader or whether external dependencies are available. + +### Definition + +```bash +curl -X GET http://localhost:4000/admin/health +``` + +### Examples + +#### Request + +```bash +curl -X GET http://localhost:4000/admin/health +``` + +#### Response + +```json +OK +``` ## /leader HTTP endpoint -`GET /admin/leader` reads the elected Metasrv leader address through the configured election backend. The handler is in `leader.rs`. +The `/leader` endpoint accepts GET HTTP requests and you can use this endpoint to query the leader's addr of your metasrv instance. + +### Definition + +```bash +curl -X GET http://localhost:4000/admin/leader +``` + +### Examples + +#### Request + +```bash +curl -X GET http://localhost:4000/admin/leader +``` + +#### Response + +```json +127.0.0.1:4000 +``` ## /heartbeat HTTP endpoint -`GET /admin/heartbeat` returns Datanode heartbeat records. The optional `addr` query parameter filters by Datanode address, and `GET /admin/heartbeat/help` shows the supported query forms. The handler is in `heartbeat.rs` and reads through `MetaPeerClient`. +The `/heartbeat` endpoint accepts GET HTTP requests and you can use this endpoint to query the heartbeat of all datanodes. + +You can also query the heartbeat data of the datanode for a specified `addr`, however, specifying `addr` in the path is optional. + +### Definition + +```bash +curl -X GET http://localhost:4000/admin/heartbeat +``` + +| Query String Parameter | Type | Optional/Required | Definition | +|:-----------------------|:-------|:------------------|:--------------------------| +| addr | String | Optional | The addr of the datanode. | + +### Examples + +#### Request + +```bash +curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100' +``` + +#### Response + +```json +[ + [ + { + "timestamp_millis": 1677049348651, + "id": 1, + "addr": "127.0.0.1:4100", + "rcus": 0, + "wcus": 0, + "region_num": 2, + "region_stats": [], + "topic_stats": [], + "node_epoch": 0, + "datanode_workloads": { + "types": [] + }, + "gc_stat": null + } + ] +] +``` ## /maintenance HTTP endpoint -Maintenance mode disables selected automatic cluster-management work. Its user-facing behavior is documented under [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md). The router exposes: +Cluster Maintenance Mode is a safety feature in GreptimeDB that temporarily disables automatic cluster management operations. This mode is particularly useful during cluster upgrades, planned downtime, and any operation that might temporarily affect cluster stability. For more details, please refer to [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md). + +The `/maintenance` endpoint supports the following HTTP requests: - `GET /admin/maintenance` or `GET /admin/maintenance/status`: query the maintenance mode status. - `POST /admin/maintenance/enable`: enable maintenance mode. - `POST /admin/maintenance/disable`: disable maintenance mode. -The implementation is in `maintenance.rs` and updates `RuntimeSwitchManager`. +The response body uses the following format: + +```json +{ + "enabled": true +} +``` ## /procedure-manager HTTP endpoint -These routes pause or resume Procedure Manager scheduling. See [Prevent Metadata Changes](/user-guide/deployments-administration/maintenance/prevent-metadata-changes.md) for user-facing behavior. The router exposes: +This endpoint is used to manage the Procedure Manager status. For more details, please refer to [Prevent Metadata Changes](/user-guide/deployments-administration/maintenance/prevent-metadata-changes.md). + +The `/procedure-manager` endpoint supports the following HTTP requests: - `GET /admin/procedure-manager/status`: query the Procedure Manager status. - `POST /admin/procedure-manager/pause`: pause the Procedure Manager. - `POST /admin/procedure-manager/resume`: resume the Procedure Manager. -The implementation is in `procedure.rs` and also updates `RuntimeSwitchManager`. - -## Other internal endpoints - -The router also exposes these maintainer-facing endpoints: - -- `GET /admin/node-lease` returns the active Datanode lease records. -- `GET /admin/recovery/status` and `POST /admin/recovery/{enable,disable}` read or change recovery mode. -- `GET /admin/sequence/table/next-id` reads the next table ID without allocating it. -- `POST /admin/sequence/table/set-next-id` changes the allocator's next table ID. The handler rejects this operation unless recovery mode is enabled. +The response body uses the following format: -The recovery and sequence routes can change cluster state and are intended for controlled repair procedures. Read their handlers and tests before changing or invoking them; this page does not define a general recovery workflow. +```json +{ + "status": "running" +} +``` diff --git a/docs/contributor-guide/metasrv/overview.md b/docs/contributor-guide/metasrv/overview.md index e2f1efd930..b345204d0f 100644 --- a/docs/contributor-guide/metasrv/overview.md +++ b/docs/contributor-guide/metasrv/overview.md @@ -1,70 +1,60 @@ --- -keywords: [metasrv, metadata, routing, leader election, heartbeat, distributed procedures] -description: Overview of Metasrv's metadata, coordination, and cluster-management responsibilities. +keywords: [metasrv, metadata, routing, leader election, procedure, heartbeat] +description: Overview of the metadata and coordination mechanisms provided by Metasrv. --- # Metasrv - +## What's in Metasrv -## Responsibilities +Metasrv is the metadata and coordination service in a distributed GreptimeDB cluster. It does not sit on the data path. Its main responsibilities are: -Metasrv is the metadata and coordination service for distributed deployments. It: +- storing Catalog, Schema, Table, Region, route, and node metadata; +- choosing Datanodes for new Regions and maintaining table routes; +- electing one Metasrv leader to coordinate metadata changes; +- running recoverable procedures for DDL, Region migration, failover, and repartitioning; +- tracking node leases and Region statistics through heartbeats; +- notifying Frontends and Datanodes when cached metadata or Region state changes. -- persists Catalog, Schema, Table, Region, route, and node metadata through the KV backend; -- uses leader election so coordination and metadata-changing work runs on one leader; -- tracks node leases and Region statistics through heartbeat streams; -- selects Datanodes for Regions when tables are created; -- runs recoverable distributed procedures for DDL, Region migration, failover, repartitioning, and related maintenance work; -- publishes cache invalidations and other control messages to Frontend and Datanode. +## How the Frontend interacts with Metasrv -The data models, KV abstraction, election interfaces, key encoding, and DDL manager are implemented in `src/common/meta/`. The `src/meta-srv/` crate provides the server, state machine, heartbeat handlers, and control procedures. - - - -## Frontend interaction - -Frontend uses the `meta-client` crate to obtain table metadata and Region routes and to submit metadata-changing operations. It caches metadata locally; Metasrv sends invalidation messages when a procedure changes metadata. +Frontend obtains table metadata and Region routes from Metasrv and caches them locally. Metadata-changing statements are sent to the Metasrv leader, while reads and writes use the cached routes to reach Datanodes directly. ### Create Table 1. Frontend submits the DDL request to the Metasrv leader. -2. The DDL manager validates the request, derives the Regions from the partition rules, and selects Datanodes for those Regions. -3. A persisted procedure creates the Regions and records the table and route metadata. Persisted procedure state makes the operation recoverable after a restart or leader change. -4. Metasrv invalidates affected caches after the metadata change is committed. +2. Metasrv derives Regions from the partition rules and selects a Datanode for each Region. +3. A persisted procedure creates the Regions and records the table and route metadata. If leadership changes, the procedure can resume from its persisted state. +4. Metasrv notifies Frontends after the metadata change is committed so their caches can be refreshed. ### Insert -Frontend resolves the table route, splits rows by partition, and sends Region write requests to the corresponding Datanodes. Route metadata is cached, but cache invalidation or a stale-route error causes Frontend to refresh it from Metasrv. +Frontend resolves the table route, splits rows according to the partition rules, and sends each Region write to the corresponding Datanode. Route changes cause the cached metadata to be invalidated and fetched again from Metasrv. ### Select -Frontend uses table and Region metadata while planning a query. Partition predicates prune Regions, and the distributed query engine sends remote subplans to the Datanodes that own the selected Regions. See [Distributed Querying](../frontend/distributed-querying.md). - - - -## Source layout +Frontend uses table and Region metadata while planning the query. Predicates on partition columns prune Regions, and the distributed query engine sends work to the Datanodes that own the selected Regions. See [Distributed Querying](../frontend/distributed-querying.md). -The main implementation areas are: +## Metasrv Architecture -- `src/meta-srv/src/service/`: gRPC services and the HTTP Admin API. -- `src/meta-srv/src/handler/`: the heartbeat handler chain. -- `src/meta-srv/src/procedure/`: Region migration, repartition, WAL pruning, and other distributed procedures. -- `src/meta-srv/src/region/`: Region leases, supervision, and failover triggers. -- `src/meta-srv/src/selector/`: Datanode selection for Region placement. +Metasrv combines several coordination mechanisms: - +- A metadata layer stores cluster state through a key-value backend. +- Leader election ensures that one Metasrv coordinates metadata changes and cluster-management work. +- The Procedure Manager executes multi-step operations and persists enough state to resume them after failure. +- Heartbeat handlers update leases and Region statistics and deliver control messages. +- Region supervision uses lease state to detect unavailable Regions and start failover when appropriate. -## Leadership and persistence +These mechanisms share metadata, but they have different failure boundaries. A process restart may discard caches and leader-local state; metadata and procedure state required for recovery must be durable. -Metasrv separates leader election and durable metadata storage behind interfaces in `common-meta`. Coordination and metadata-changing operations run on the leader; a non-leader returns a not-leader response so the client can reconnect to the current leader. +## Distributed Consensus -Anything required after a leader change must be stored in the KV backend. In-memory caches and leader-local state are rebuilt or cleared during a transition. Distributed procedures persist their state and must keep each step idempotent so execution can resume safely. +Metasrv separates leader election from metadata storage. Only the elected Metasrv leader performs coordination and metadata-changing operations. Other Metasrv nodes direct clients to the current leader. - +The key-value backend stores table metadata, routes, procedure state, and other information that must survive a leader change. Metasrv does not use this election to create leader and follower replicas for Datanode Regions; Region availability is managed through leases, heartbeats, and failover procedures. -## Heartbeat invariants +## Heartbeat Management -Datanodes and Frontends maintain heartbeat streams to the Metasrv leader. Requests report node identity, leases, Region statistics, and other state. The handler chain under `src/meta-srv/src/handler/` checks leadership, updates leases and statistics, and handles mailbox messages. +Datanodes maintain heartbeat streams to the Metasrv leader. Heartbeat requests report node identity, lease information, Region statistics, and other state used for placement and supervision. Responses carry control messages such as Region lifecycle instructions and cache invalidations. -Heartbeat responses carry control messages such as Region lifecycle instructions and cache invalidations. Region supervision uses lease state to detect unavailable Regions and trigger failover procedures. Changes to heartbeat intervals must remain consistent with lease and supervisor timing in `common-meta` and `meta-srv`. +Metasrv treats a heartbeat as a lease renewal, not merely as a metrics sample. Lease expiration is therefore part of failure detection and can lead to a Region failover procedure. Changes to heartbeat timing must remain consistent with the lease and supervision intervals. diff --git a/docs/contributor-guide/metasrv/selector.md b/docs/contributor-guide/metasrv/selector.md index 5eb862af6e..1f00c2cd7a 100644 --- a/docs/contributor-guide/metasrv/selector.md +++ b/docs/contributor-guide/metasrv/selector.md @@ -1,40 +1,43 @@ --- -keywords: [selector, metasrv, datanode, lease based, load based, round robin] -description: Region placement selectors used by Metasrv and their configuration names. +keywords: [selector, metasrv, datanode, leasebased, loadbased, roundrobin] +description: Describes the different types of selectors in the Metasrv service, their characteristics, and how to configure them. --- # Selector ## Introduction -When a table is created, Metasrv must choose Datanodes for its Regions. The [`Selector` trait](https://github.com/GreptimeTeam/greptimedb/blob/main/src/meta-srv/src/selector.rs) receives the required number of peers and a selection context, then returns candidate Datanodes from the current lease and statistics data. +When a table is created, Metasrv uses a `Selector` to choose Datanodes for its Regions. Selection uses the current node leases and, depending on the selector, Region statistics. ## Selector Type -Metasrv provides three selector implementations: +The `Metasrv` service currently offers the following types of `Selectors`: ### LeaseBasedSelector -`LeaseBasedSelector` chooses randomly from Datanodes with valid leases. It does not use Region counts to rank candidates. +`LeaseBasedSelector` randomly selects from Datanodes with valid leases. ### LoadBasedSelector -`LoadBasedSelector` treats the number of Regions on a Datanode as its load and prefers nodes with fewer Regions. +The `LoadBasedSelector` load value is determined by the number of regions on each `Datanode`, fewer regions indicate lower load, and `LoadBasedSelector` prioritizes selecting low-load `Datanodes`. ### RoundRobinSelector [default] - -`RoundRobinSelector` rotates through available Datanodes. It is the default selector. +`RoundRobinSelector` selects `Datanode`s in a round-robin fashion. It is the default option. ## Configuration -Set the selector when starting Metasrv. The accepted names are: +You can configure the `Selector` by its name when starting the `Metasrv` service. -- `lease_based` or `LeaseBased` -- `load_based` or `LoadBased` -- `round_robin` or `RoundRobin` +- LeaseBasedSelector: `lease_based` or `LeaseBased` +- LoadBasedSelector: `load_based` or `LoadBased` +- RoundRobinSelector: `round_robin` or `RoundRobin` For example: ```shell cargo run -- metasrv start --selector round_robin ``` + +```shell +cargo run -- metasrv start --selector RoundRobin +``` diff --git a/docs/contributor-guide/overview.md b/docs/contributor-guide/overview.md index c9142d332c..cb9bcd7dea 100644 --- a/docs/contributor-guide/overview.md +++ b/docs/contributor-guide/overview.md @@ -1,19 +1,24 @@ --- -keywords: [contributor guide, architecture, frontend, datanode, metasrv, flownode] -description: Entry point for contributors who want to understand and develop GreptimeDB. +keywords: [architecture, key components, user requests, data processing, database components] +description: Overview of GreptimeDB's architecture, key components, and how they interact to process user requests. --- # Contributor Guide -This guide describes GreptimeDB's internal architecture and points contributors to the code that implements each subsystem. For build, test, and contribution requirements, start with the repository's [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md). +This guide explains the internal design of GreptimeDB for contributors. Build, test, and submission instructions are maintained in the source repository's [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md). ## Architecture -The [architecture overview](/user-guide/concepts/architecture.md) explains the components and request paths from a user's perspective. The contributor guides below cover their implementation boundaries: +For the architecture and components of GreptimeDB, please see the [Architecture](/user-guide/concepts/architecture.md) document in the user guide. -- [Frontend](./frontend/overview.md): protocol handling, request orchestration, routing, and distributed query planning. -- [Datanode](./datanode/overview.md): Region management, query execution, and storage engines. -- [Metasrv](./metasrv/overview.md): metadata, cluster coordination, and distributed procedures. -- [Flownode](./flownode/overview.md): continuous aggregation in standalone and distributed deployments. +For more details on each component, see the following guides: -To build GreptimeDB locally, continue with [Getting started](./getting-started.md). +- [frontend][1] +- [datanode][2] +- [metasrv][3] +- [flownode][4] + +[1]: /contributor-guide/frontend/overview.md +[2]: /contributor-guide/datanode/overview.md +[3]: /contributor-guide/metasrv/overview.md +[4]: /contributor-guide/flownode/overview.md diff --git a/docs/contributor-guide/tests/integration-test.md b/docs/contributor-guide/tests/integration-test.md index ad71f80563..5d5f6cb1a5 100644 --- a/docs/contributor-guide/tests/integration-test.md +++ b/docs/contributor-guide/tests/integration-test.md @@ -1,30 +1,13 @@ --- -keywords: [integration tests, Rust test harness, storage backend, Kafka, protocols] -description: Run multi-component and external-service tests from tests-integration. +keywords: [integration tests, Rust test harness, multiple components, HTTP testing, gRPC testing] +description: Guide on writing and running integration tests in GreptimeDB, covering scenarios involving multiple components. --- # Integration Test ## Introduction -The `tests-integration/` crate contains Rust test-harness cases that need several GreptimeDB components or an external service. Typical cases exercise HTTP or gRPC behavior, object-storage backends, Kafka WAL, and TLS-enabled dependencies. - -Use an integration test when the behavior cannot be established through a crate-local unit test or a sqlness query case. Keep protocol assertions at the public boundary and use the fixtures under `tests-integration/fixtures/` rather than introducing a second environment setup. - -The authoritative setup and command list is in [`tests-integration/README.md`](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md). Tests that require credentials or endpoints read them from a repository-root `.env` file created from `.env.example`; do not commit credentials. - -Run the general integration group from the repository root: - -```shell -cargo test integration -``` - -Backend-specific groups use their own filters, for example: - -```shell -cargo test s3 -cargo test oss -cargo test azblob -``` - -Kafka and TLS cases require the Docker Compose services documented in the integration README. Start only the dependencies required by the selected test, and clean up those services after the run. +Integration testing is written with Rust test harness (`#[test]`), unlike unit testing, they are placed separately +[here](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration). +It covers scenarios involving multiple components, in which one typical case is HTTP/gRPC-related features. You can check +its [documentation](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md) for more information. diff --git a/docs/contributor-guide/tests/overview.md b/docs/contributor-guide/tests/overview.md index 564eac9e97..feeefe2b6d 100644 --- a/docs/contributor-guide/tests/overview.md +++ b/docs/contributor-guide/tests/overview.md @@ -1,14 +1,8 @@ --- -keywords: [tests, unit tests, sqlness, integration tests, regression] -description: Choose and run the GreptimeDB test suite that matches a code change. +keywords: [testing methods, behavior testing, performance testing, test overview, GreptimeDB tests] +description: Overview of the testing methods used in GreptimeDB to ensure its behavior and performance. --- # Tests -GreptimeDB uses several test layers. Choose the narrowest layer that exercises the behavior being changed, then add a broader regression test when the behavior crosses component boundaries or is visible through a public interface. - -- [Unit tests](./unit-test.md) cover crate-local logic, invariants, and error paths. They live next to the Rust implementation and run with cargo-nextest. -- [Sqlness tests](./sqlness-test.md) cover user-visible SQL and query behavior against standalone or distributed test environments. Cases and expected results live under `tests/cases/`. -- [Integration tests](./integration-test.md) cover interactions that require multiple components or external services, including storage backends and protocol-level behavior. They live under `tests-integration/`. - -The repository also contains specialized suites such as `tests-fuzz/`, `tests/compatibility/`, and `tests/perf/`. Use their local README or `AGENTS.md` instructions when a change affects input robustness, persisted-format compatibility, or performance. Passing one layer does not replace a test at the layer where the regression would be observed. +Our team has conducted lots of tests to ensure the behaviours of `GreptimeDB` . This chapter will introduce several significant methods used to test `GreptimeDB`, and how to work with them. diff --git a/docs/contributor-guide/tests/sqlness-test.md b/docs/contributor-guide/tests/sqlness-test.md index 781335aba3..6712c3413b 100644 --- a/docs/contributor-guide/tests/sqlness-test.md +++ b/docs/contributor-guide/tests/sqlness-test.md @@ -1,45 +1,53 @@ --- -keywords: [SQL tests, sqlness, golden files, standalone, distributed] -description: Add and run sqlness regression cases for user-visible query behavior. +keywords: [SQL tests, sqlness, test suite, test cases, test output] +description: Instructions for running SQL tests in GreptimeDB using the `sqlness` test suite, including file types, case organization, and running tests. --- # Sqlness Test ## Introduction -Sqlness is GreptimeDB's golden-file test harness for SQL and query behavior. It builds and starts the requested GreptimeDB environment, executes case files, and compares the output with checked-in results. The harness and its current options are documented in [`tests/README.md`](https://github.com/GreptimeTeam/greptimedb/blob/main/tests/README.md). +SQL is an important user interface for `GreptimeDB`. We have a separate test suite for it (named `sqlness`). ## Sqlness manual ### Case file -Each case has two files: +Sqlness has two types of file -- `.sql` contains the statements and sqlness directives. -- `.result` contains the expected statements and output. +- `.sql`: test input, SQL only +- `.result`: expected test output, SQL and its results -Edit the `.sql` input first, run sqlness, and review the resulting `.result` diff. A changed result can be the intended new behavior or a regression; the harness cannot decide which one. Commit a result change only after checking every changed row and error message. +The `.result` file is the expected execution output. If you see `.result` files changed, +it means the test gets a different result and indicates it may fail. You should +check the change logs to solve the problem. + +You only need to write test SQL in the `.sql` file, and run the test. ### Case organization -Cases live under `tests/cases/`. The first directory level selects an environment, such as `standalone/`; directories below it organize related cases. Sqlness discovers case files recursively. +The root dir of input cases is `tests/cases`. It contains several sub-directories stand for different test +modes. E.g., `standalone/` contains all the tests to run under `greptimedb standalone start` mode. -Place a regression in the environment where the behavior is observable. Distributed planning, routing, and multi-node metadata behavior require a distributed case even when an equivalent standalone query also succeeds. +Under the first level of sub-directory (e.g. the `cases/standalone`), you can organize your cases as you like. +Sqlness walks through every file recursively and runs them. ## Run the test -The repository defines a cargo alias for the harness: +Unlike other tests, this harness is in a binary target form. You can run it with ```shell cargo sqlness bare ``` -This command builds GreptimeDB, starts the test environment, runs the cases, and updates or compares `.result` files. Inspect both the command result and `git diff`. +It automatically finishes the following procedures: compile `GreptimeDB`, start it, grab tests and feed it to +the server, then collect and compare the results. You only need to check whether any `.result` files changed. +If no unexpected result changes remain, the test passed. ### Run a specific test ```shell -cargo sqlness bare -t 'standalone:your_case' +cargo sqlness bare -t your_test ``` -`-t`/`--test-filter` accepts a regular expression and matches case names in `env:case` form. Use a narrow filter while iterating, then run the affected environment or full suite before submission. +The `-t` or `--test-filter` option accepts a regex string. Sqlness examines case names in the format of `env:case`. diff --git a/docs/contributor-guide/tests/unit-test.md b/docs/contributor-guide/tests/unit-test.md index 3d2831521c..82e30bf5fa 100644 --- a/docs/contributor-guide/tests/unit-test.md +++ b/docs/contributor-guide/tests/unit-test.md @@ -1,34 +1,32 @@ --- -keywords: [unit tests, Rust, cargo-nextest, package tests, coverage] -description: Write and run crate-local Rust tests with cargo-nextest. +keywords: [unit tests, Rust, cargo nextest, test runner, coverage] +description: Guide on writing and running unit tests in GreptimeDB using Rust's `#[test]` attribute and `cargo nextest`. --- # Unit Test ## Introduction -Rust unit tests normally live in the module they exercise or in a nearby `*_test.rs` file. Use them for local invariants, boundary conditions, error handling, and behavior that does not require a running GreptimeDB cluster. +Unit tests are embedded into the codebase, usually placed next to the logic being tested. +They are written using Rust's `#[test]` attribute and can run with `cargo nextest run`. -GreptimeDB's standard runner is [cargo-nextest](https://nexte.st/). Install it with: +The default test runner ships with `cargo` is not supported in GreptimeDB codebase. It's recommended +to use [`nextest`](https://nexte.st/) instead. You can install it with ```shell cargo install cargo-nextest --locked ``` -During development, run the affected package first: +And run the tests (here the `--workspace` is not necessary) ```shell -cargo nextest run -p +cargo nextest run ``` -Run the workspace configuration used by CI before submitting a change that can affect several crates: - -```shell -cargo nextest run --workspace --features pg_kvbackend,mysql_kvbackend -``` - -Feature-gated code requires the corresponding feature in the test command. Check the crate's `Cargo.toml`, local `AGENTS.md`, and CI workflow before assuming the default feature set covers the path. +Notes if your Rust is installed via `rustup`, be sure to install `nextest` with `cargo` rather +than the package manager like `homebrew`. Otherwise it will mess up your local environment. ## Coverage -CI records Rust test coverage. Add tests that protect the changed behavior and credible failure cases; do not add assertions solely to increase the percentage. Query-language behavior and cross-component flows usually need a sqlness or integration test in addition to a unit test. +Our continuous integration (CI) jobs have a "coverage checking" step. It will report how many +codes are covered by unit tests. Please add the necessary unit test to your patch. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md index 9505e72101..2c6ee25bde 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md @@ -5,65 +5,83 @@ description: 介绍了 GreptimeDB 的数据持久化和索引机制，包括 SST # 数据持久化与索引 -Mito 将 memtable 中的数据 flush 到本地文件系统或对象存储，SST 文件使用 [Apache Parquet][1] 作为数据格式。 +与所有类似 LSMT 的存储引擎一样，MemTables 中的数据被持久化到耐久性存储，例如本地磁盘文件系统或对象存储服务。GreptimeDB 采用 [Apache Parquet][1] 作为其持久文件格式。 ## SST 文件格式 -Parquet 是一种列式文件格式，其层级结构决定 Mito 扫描时可以读取、缓存或裁剪的单元。 +Parquet 是一种提供快速数据查询的开源列式存储格式，已经被许多项目采用，例如 Delta Lake。 Parquet 按 row group、column chunk 和 page 组织数据。每个 row group 为每一列保存一个 column chunk，每个 column chunk 再包含一个或多个 page。Page 是 column chunk 内最小的编码 I/O 单元。 -Column chunk 使投影扫描只读取查询需要的列。 +首先，数据按列聚集，这使得文件扫描更加高效，特别是当查询只涉及少数列时，这在分析系统中非常常见。 -同一列的 page 也适合使用字典编码、run-length encoding（RLE）等方式压缩。 +其次，相同列的数据往往是同质的（比如具备近似的值），这有助于在采用字典和 Run-Length Encoding（RLE）等技术进行压缩。 Parquet file format ## 数据持久化 -`region_engine.mito.global_write_buffer_size` 设置一个 Datanode 上所有 Mito memtable 共享的内存阈值。内存使用达到阈值后，write-buffer manager 选择 memtable，并通过 `src/mito2/src/flush.rs` 调度 SST flush。 +GreptimeDB 提供了 `region_engine.mito.global_write_buffer_size` 的配置项来设置全局的 Memtable 大小阈值。当数据库所有 MemTable 中的数据量之和达到阈值时将自动触发持久化操作，将 MemTable 的数据 flush 到 SST 文件中。 ## SST 文件中的索引数据 -Parquet 为 row group 和 page 保存列统计信息。Mito 将兼容的查询谓词转换为 Parquet pruning predicate，利用 min/max 和 null 统计信息跳过不可能匹配的 row group。 +Apache Parquet 文件格式在列块和数据页的头部提供了内置的统计信息，用于剪枝和跳过。 + +Column chunk header + +例如，在上述 Parquet 文件中，如果你想要过滤 `name` 等于 `Emily` 的行，你可以轻松跳过行组 0，因为 `name` 字段的最大值是 `Charlie`。这些统计信息减少了 IO 操作。 ## 索引文件 -Mito 将 SST 对应的索引 artifact 保存在带版本的 [Puffin][3] 文件中，Region manifest 记录当前生效的索引版本。发布或重建索引时，不能让 manifest 引用尚未完整写入的 artifact。 +对于每个 SST 文件，GreptimeDB 不但维护 SST 文件内部索引，还会单独生成一个文件用于存储针对该 SST 文件的索引结构。 + +索引文件采用 [Puffin][3] 格式，这种格式具有较大的灵活性，能够存储更多的元数据，并支持更多的索引结构。 + +![Puffin](/puffin.png) -`src/mito2/src/sst/index/` 负责将倒排索引、基于 bloom filter 的 skipping index、全文索引及 feature-gated vector index 接入 SST 读写。可复用的索引格式位于 `src/index/src/`，companion file 由 `puffin_manager.rs` 管理。 +GreptimeDB 会将多种索引结构作为 Blob 存储在 Puffin 文件中，包括倒排索引、跳数索引（基于 bloom filter）和全文索引。倒排索引是最早支持的索引结构，下面将详细介绍。 ## 倒排索引 -倒排索引按列把编码后的列值映射到包含该值的 SST 数据段。应用谓词后得到候选 segment ID；正常扫描仍会对候选数据段中的行执行完整谓词。 +在 v0.7 版本中，GreptimeDB 引入了倒排索引（Inverted Index）来加速查询。 + +倒排索引是全文搜索中常见的索引结构，它将文档中的每个单词映射到包含该单词的文档列表。GreptimeDB 将这项搜索引擎技术用于时序数据索引。 + +搜索引擎和时间序列数据库虽然运行在不同的领域，但是应用的倒排索引技术背后的原理是相似的。这种相似性需要一些概念上的调整： +1. 单词：在 GreptimeDB 中，指时间线的列值。 +2. 文档：在 GreptimeDB 中，指包含多个时间线的数据段。 + +倒排索引的引入，使得 GreptimeDB 可以跳过不符合查询条件的数据段，从而提高扫描效率。 ![Inverted index searching](/inverted-index-searching.png) -上图中的查询使用倒排索引找出 `job` 等于 `apiserver`、`handler` 匹配正则表达式 `.*users` 且 `status` 匹配正则表达式 `4...` 的候选数据段。Mito 扫描这些数据段，并对数据行应用完整查询谓词。 +例如，上述查询使用倒排索引来定位数据段，数据段满足条件：`job` 等于 `apiserver`，`handler` 符合正则匹配 `.*users` 及 `status` 符合正则匹配 `4..`，然后扫描这些数据段以产生满足所有条件的最终结果，从而显着减少 IO 操作的次数。 ### 倒排索引格式 ![Inverted index format](/inverted-index-format.png) -每个列索引包含一个 FST（Finite State Transducer）和多个 bitmap。FST 把编码后的列值映射到 bitmap 位置，并支持正则表达式匹配等查询。每个 bitmap 记录包含该值的数据段。 +GreptimeDB 按列构建倒排索引，每个倒排索引包含一个 FST 和多个 Bitmap。 + +FST（Finite State Transducer）允许 GreptimeDB 以紧凑的格式存储列值到 Bitmap 位置的映射，并且提供了优秀的搜索性能和支持复杂搜索（例如正则表达式匹配）；Bitmap 则维护了数据段 ID 列表，每个位表示一个数据段。 ### 索引数据段 -GreptimeDB 把 SST 文件分割成固定大小的索引数据段。匹配的 bitmap 会转换为 Parquet row selection，使 Mito 只读取候选行范围。 +GreptimeDB 把一个 SST 文件分割成多个索引数据段，每个数据段包含相同行数的数据。这种分段的目的是通过只扫描符合查询条件的数据段来优化查询性能。 -例如，每个数据段包含 1024 行且候选数据段 ID 为 `[0, 2]` 时，Mito 只扫描第 0–1023 行和第 2048–3071 行，不需要读取 SST 中的全部数据行。 +例如，当数据段的行数为 1024，如果查询条件应用倒排索引后，得到的数据段列表为 `[0, 2]`，那么只需扫描 SST 文件中的第 0 和第 2 个数据段（即第 0 行到第 1023 行和第 2048 行到第 3071 行）即可。 -引擎选项 `index.inverted_index.segment_row_count` 控制目标 segment 大小，默认值为 `1024`。较小的 segment 可以提高裁剪精度，但会增加索引大小和构建成本。 +数据段的行数由引擎选项 `index.inverted_index.segment_row_count` 控制，默认为 `1024`。较小的值意味着更精确的索引，往往会得到更好的查询性能，但会增加索引存储成本。通过调整该选项，可以在存储成本和查询性能之间进行权衡。 ## 统一数据访问层：OpenDAL -`object-store` crate 基于 [OpenDAL][2] 封装本地文件系统和对象存储。Mito 通过 `src/mito2/src/access_layer.rs` 执行 SST 与索引 I/O；存储引擎代码不应绕过该边界增加 backend-specific 路径。修改配置的 backend 不会迁移已有数据。 +GreptimeDB 使用 [OpenDAL][2] 为本地文件系统和对象存储提供统一访问层。修改配置的存储 backend 不会迁移已有数据。 [1]: https://parquet.apache.org -[2]: https://opendal.apache.org/ +[2]: https://github.com/datafuselabs/opendal [3]: https://iceberg.apache.org/puffin-spec diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md index e7d0547432..247354a3f0 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md @@ -1,39 +1,38 @@ --- -keywords: [Metric 引擎, 逻辑表, 物理表, Mito, Prometheus] -description: 介绍 Metric 引擎面向大量指标表的逻辑到物理存储模型。 +keywords: [Metric 引擎, 逻辑表, 物理表, DDL 操作] +description: 介绍了 Metric 引擎的概念、架构及设计，重点描述了逻辑表与物理表的区别和批量 DDL 操作的实现。 --- # Metric 引擎 ## 概述 -Metric 引擎是一种 `RegionEngine` 实现，面向包含大量小指标表的 Prometheus 类 workload。它把多个逻辑表复用到共享的 Mito 物理 Region 中，在保留表级读写接口的同时，降低每张表的元数据和存储开销。 +`Metric` 引擎是 GreptimeDB 的一个组件，属于存储引擎的一种实现，主要针对可观测 metrics 等存在大量小表的场景。 -Metric 引擎不实现另一套磁盘格式。它重写逻辑请求，再将物理存储、索引和扫描委托给 Mito。 +它的主要特点是利用合成的物理宽表来存储大量的小表数据，实现相同列复用和元数据复用等效果，从而达到减少小表的存储开销以及提高列式压缩效率等目标。表这一概念在 `Metric` 引擎下变得更更加轻量。 ## 概念 -### 逻辑表 +`Metric` 引擎引入了两个新的概念，分别是逻辑表与物理表。从用户视角看，逻辑表与普通表完全一样。从存储视角看，物理 Region 就是一个普通的 Region。 -逻辑表是对用户暴露的表，拥有独立的 Schema 和 Table ID。用户写入和查询都以逻辑表为目标；在内部，每个逻辑 Region 会记录实际存储数据的物理 Region。 +### 逻辑表 +逻辑表，即用户定义的表。与普通的表都完全一样，逻辑表的定义包括表的名称、列的定义、索引的定义等。用户的查询、写入等操作都是基于逻辑表进行的。用户在使用过程中不需要关心逻辑表和普通表的区别。 -写入时，Metric 引擎把逻辑表身份写入每一行，再将请求转发到物理数据 Region。读取时，它添加逻辑表过滤条件，只返回属于目标逻辑表的数据。 +从实现层面来说，逻辑表是一个虚拟的表，它并不直接读写物理的数据，而是通过将读写请求映射成对应物理表的请求来实现数据的存储与查询。 ### 物理表 +物理表是真实存储数据的表，它拥有若干个由分区规则定义的物理 Region。 -物理表持有共享 Region。每个物理 Region 由一对 Mito Region 表示： - -- 数据 Region，保存多个逻辑表的数据行； -- 元数据 Region，保存 Metric 引擎使用的逻辑表和逻辑列映射。 +## 架构及设计 -直接写入物理 Region 会绕过逻辑表映射，因此会被拒绝；查询物理表仍然受支持。 +`Metric` 引擎的主要设计架构如下： -## 架构及设计 +![Arch](/metric-engine-arch.png) -关联到同一物理表的逻辑表使用相同的分区布局。逻辑 Region ID 映射到对应的物理数据 Region 和元数据 Region，映射关系由 Metric 引擎及表路由元数据共同维护。 +`Metric` 引擎将物理存储和查询交给 `Mito` 引擎。每个物理 Region 由一个数据 Region 和一个元数据 Region 表示：数据 Region 保存多个逻辑表的数据，元数据 Region 保存逻辑表及逻辑列的映射。 -`row_modifier.rs` 和 `batch_modifier.rs` 将逻辑表身份与时间序列身份编码到 Mito 内部列中。根据物理 Region 的主键编码方式，具体表示为 `__table_id` 与 `__tsid` 列，或稀疏编码的 `__primary_key`。读取路径在委托 Mito 扫描前始终添加逻辑 Table ID 条件。 +关联到同一物理表的逻辑表使用相同的分区布局。写入时，Metric 引擎为每行数据记录逻辑表身份；读取时，它在扫描物理 Region 前增加逻辑表过滤条件。 -Metric 引擎为影响大量逻辑表的操作提供批量 DDL 路径，避免在 Prometheus Remote Write 自动建表或物理 Region 迁移时为每张表单独修改元数据。这里的 DDL 指数据定义语言操作；逻辑表的普通插入、删除和查询仍使用标准 Region 请求路径。 +逻辑表支持普通的 INSERT、DELETE 和 SELECT 操作。直接写入物理 Region 会绕过逻辑表映射，因此会被拒绝；物理表仍然可以查询。 -主要实现位于 `src/metric-engine/src/`。修改保留列、Region ID 转换或元数据编码会影响持久化数据，必须审查向后兼容性。 +批量 DDL 用于减少大量逻辑表同时创建或更新时的元数据操作，例如 Prometheus Remote Write 自动建表或物理 Region 迁移。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md index 6b97110cf9..4c4e53c9e7 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md @@ -1,25 +1,28 @@ --- -keywords: [Datanode, RegionServer, 存储引擎, 查询引擎, 心跳] -description: 介绍 Datanode 在 Region 级别的存储和查询职责。 +keywords: [Datanode, gRPC 服务, HTTP 服务, Heartbeat Task, Region Manager] +description: 介绍了 Datanode 的主要职责和组件，包括 gRPC 服务、HTTP 服务、Heartbeat Task 和 Region Manager。 --- # Datanode - +## Introduction -## 介绍 +`Datanode` 主要的职责是为 GreptimeDB 存储数据，我们知道在 GreptimeDB 中一个 `table` 可以有一个或者多个 `Region`, +而 `Datanode` 的职责便是管理这些 `Region` 的读写。`Datanode` 不感知 `table`，可以认为它是一个 `region server`。 +所以 `Frontend` 和 `Metasrv` 按照 `Region` 粒度来操作 `Datanode`。 -Datanode 存储表数据，并在本地 Region 上执行查询。一张表可以包含多个 Region，但 Datanode 不把表作为元数据对象管理。Frontend 和 Metasrv 通过 Region 级请求访问 Datanode，因此它的核心抽象是 Region server。 +![Datanode](/datanode.png) - +## Components -## 组件 +一个 datanode 包含了 region server 所需的全部组件。这里列出了比较重要的部分： -- `src/datanode/src/region_server.rs` 中的 `RegionServer` 将 Region 请求分发到已注册的存储引擎，并向查询层提供 Region 数据。 -- gRPC 服务接收 Frontend 和 Metasrv 发出的 Region 读写及生命周期操作。 -- 本地查询引擎规划并执行 Frontend 发送的逻辑子计划。Datanode 不解析客户端 SQL，也不负责协调分布式查询。 -- 心跳任务向 Metasrv 报告节点和 Region 状态，并接收 Region 打开、关闭、迁移及缓存失效等控制消息。 -- HTTP handler 提供指标、配置等运维端点。 -- Datanode 注册 Mito、Metric 和 File 三种 Region engine。Mito 是主要的时序存储引擎；Metric 面向大量指标表的场景，并将物理存储委托给 Mito；File 用于访问外部文件中的数据。 - -单机模式下，同一个 Region server 在进程内运行，不需要 Metasrv 协调。分布式模式下，Region 的可写状态和生命周期变更由 Metasrv 租约及心跳消息协调。 +- 一个 gRPC 服务来提供对 `Region` 数据的读写，`Frontend` 便是使用这个服务来从 `Datanode` 读写数据。 +- 一个 HTTP 服务，可以通过它来获得当前节点的 metrics、配置信息等 +- `Heartbeat Task` 用来向 `Metasrv` 发送心跳，心跳在 GreptimeDB 的分布式架构中发挥着至关重要的作用， + 是分布式协调和调度的基础通信通道，心跳的上行消息中包含了重要信息比如 `Region` 的负载，如果 `Metasrv` 做出了调度 + 决定（比如 Region 转移），它会通过心跳的下行消息发送指令到 `Datanode` +- `Datanode` 不负责解析用户 SQL 或进行分布式规划，用户对一个或多个 `Table` 的查询请求会在 `Frontend` 中被转换为 + `Region` 查询请求，`Datanode` 负责用本地 query engine 执行这些 `Region` 查询计划 +- Region server 管理 Datanode 上所有 Region 的生命周期，并把请求分发给相应的存储引擎。 +- GreptimeDB 支持多种 Region engine。`Mito` 是主要的时序存储引擎；`Metric` 将多个逻辑指标表存储在共享的 Mito Region 中；`File` 用于访问外部文件。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/query-engine.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/query-engine.md index b3aba35c70..ca406cc235 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/query-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/query-engine.md @@ -1,35 +1,39 @@ --- -keywords: [查询引擎, Apache DataFusion, 逻辑计划, 物理计划, Arrow, 索引] -description: 介绍 GreptimeDB 基于 DataFusion 的查询规划和执行链路。 +keywords: [查询引擎, DataFusion, 逻辑计划, 物理计划] +description: 介绍了 GreptimeDB 的查询引擎架构，基于 Apache DataFusion 构建，涵盖逻辑计划、物理计划、优化和执行过程。 --- # Query Engine ## 介绍 -GreptimeDB 查询引擎基于 [Apache DataFusion][1] 构建。`query` crate 负责 SQL、PromQL 和日志查询规划，以及 GreptimeDB optimizer rule、物理计划和执行。 +GreptimeDB 的查询引擎是基于[Apache DataFusion][1]（属于[Apache Arrow][2]的子项目）构建的，它是一个用 Rust 编写的出色的查询引擎。它提供了一整套功能齐全的组件，从逻辑计划、物理计划到执行运行时。下面将解释每个组件如何被整合在一起，以及在执行过程中它们的位置。 -![执行流程](/execution-procedure.png) +![Execution Procedure](/execution-procedure.png) -查询首先转换为 DataFusion 逻辑计划。SQL 及其他查询语言的 planner 会生成逻辑计划；分布式执行期间，Frontend 也会把序列化后的逻辑子计划发送给 Datanode。 +入口点是逻辑计划，它被用作查询或执行逻辑等的通用中间表示。逻辑计划的两个主要来源是：1. 用户查询，例如通过 SQL 解析器和规划器的 SQL；2. Frontend 的分布式查询，这将在下一节中详细解释。 -Analyzer 和 optimizer rule 会规范化计划、下推过滤与投影、裁剪 Region，并插入 `MergeScan` 等 GreptimeDB extension node。该阶段同时使用 DataFusion 原生规则和 `src/query/src/optimizer/` 下的自定义规则。 +接下来是物理计划，或称执行计划。与包含所有逻辑计划变体（除特殊扩展计划节点外）的大型枚举的逻辑计划不同，物理计划实际上是一个定义了在执行过程中调用的一组方法的特性。所有数据处理逻辑都包装在实现该特性的相应结构中。它们是对数据执行的实际操作，如聚合器 `MIN` 或 `AVG` ，以及表扫描 `SELECT ... FROM`。 -物理 planner 将优化后的逻辑计划转换为 DataFusion `ExecutionPlan` 实现。执行根计划会返回异步 Arrow `RecordBatch` stream。可以使用 `EXPLAIN` 或 `EXPLAIN VERBOSE` 查看 SQL 语句对应的计划。 +优化阶段通过转换逻辑计划和物理计划来提高执行性能，现在全部基于规则。它也被称为“基于规则的优化”。一些规则是 DataFusion 原生的，其他一些是在 GreptimeDB 中自定义的。在未来，我们计划添加更多规则，并利用数据统计进行基于成本的优化 (CBO)。 + +最后一个阶段"执行"是一个动词，代表从存储读取数据、进行计算并生成预期结果的过程。虽然它比之前提到的概念更抽象，但你可以简单地将它想象为执行一个 Rust 异步函数，并且它确实是一个异步流。 + +当你想知道 SQL 是如何通过逻辑计划或物理计划中表示时，`EXPLAIN [VERBOSE] ` 是非常有用的。 ## 数据表示 -GreptimeDB 使用 [Apache Arrow][2] array 和 `RecordBatch` 在内存中交换数据。存储扫描、查询算子、RPC stream 和结果编码器共享同一种列式表示，避免在执行链路中逐行转换。 +GreptimeDB 使用 [Apache Arrow][2]作为内存中的数据表示格式。它是面向列的，以跨平台格式，也包含许多高性能的基础操作。这些特性使得在许多不同的环境中共享数据和实现计算逻辑变得容易。 ## 索引 -索引构建和持久化格式属于存储引擎，而不是查询引擎。Mito 使用 Parquet 统计信息、倒排索引、跳数索引和全文索引裁剪 SST 文件、row group 及数据段；feature-gated vector index 为向量搜索提供候选行。参见[数据持久化与索引](./data-persistence-indexing.md)。 - -查询层向扫描提供谓词和投影。兼容的谓词可以通过索引减少读取的数据量，但查询计划仍需执行其余过滤算子。 +索引构建和持久化格式属于存储引擎。查询层向扫描提供谓词和投影，Mito 再利用时间范围、Parquet 统计信息和索引跳过不可能匹配的数据。参见[数据持久化与索引](./data-persistence-indexing.md)。 ## 分布式查询 -Frontend 将兼容的逻辑计划片段重写为远端 `MergeScan` 输入，使用 Substrait 序列化，再向 Datanode 发送 Region 级请求。参见[分布式查询](../frontend/distributed-querying.md)。 +参考 [Distributed Querying][6]. -[1]: https://datafusion.apache.org/ +[1]: https://github.com/apache/arrow-datafusion [2]: https://arrow.apache.org/ +[3]: https://parquet.apache.org +[6]: ../frontend/distributed-querying.md diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md index 22599cd49e..67f98216fe 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md @@ -1,24 +1,39 @@ --- keywords: [存储引擎, Mito, LSMT, 数据模型, Region] -description: 介绍 Mito 存储引擎的核心组件、Region 模型和 SST 数据布局。 +description: 详细介绍了 GreptimeDB 的存储引擎架构、数据模型和 region 的概念，重点描述了 Mito 存储引擎的优化和组件。 --- # 存储引擎 ## 概述 -Mito 是 GreptimeDB 主要的时序 Region engine，实现了 `RegionEngine` trait，并使用 [LSM tree][1] 写入链路：WAL 和 memtable 接收写入，不可变的 Parquet SST 文件保存持久化数据，后台 compaction 负责重组这些文件。 +`存储引擎` 负责存储数据库的数据。Mito 是我们默认使用的存储引擎，基于 [LSMT][1]（Log-structured Merge-tree）。我们针对处理时间序列数据的场景做了很多优化，因此 mito 这个存储引擎并不适用于通用用途。 ## 架构 - -实现代码位于 `src/mito2/src/`。`engine.rs` 分发 Region 请求，`worker/` 负责每个 Region 的写入循环，`read/` 构建扫描，`flush.rs`、`compaction/`、`manifest/` 和 `sst/` 共同实现持久化生命周期。 - -- **WAL** 记录尚未进入 SST 的写入，用于恢复 Region 的 memtable 状态。它通过 `LogStore` API 支持本地 raft-engine 和远端 Kafka provider。写入确认对应的持久性边界取决于 provider 配置；参见[预写日志](./wal.md)。 -- **Memtable** 通过可变的 active memtable 接收写入。Flush 会将其冻结为 immutable memtable；在数据写入 SST 前，immutable memtable 仍参与读取。 -- **SST 文件**是不可变的 Parquet 文件，其中的数据按照 primary key 和 time index 排序；详见 [SST 文件中的数据布局](#sst-文件中的数据布局)。 -- **Compaction** 合并 SST 文件并清理过期数据。默认策略为 [TWCS][3]，按时间窗口组织文件。详见 [Compaction](/user-guide/deployments-administration/manage-data/compaction.md)。 -- **Manifest** 保存带版本的 Region 元数据和 SST 文件变更，用于恢复。 -- **Cache** 保存文件元数据、数据页及其他可复用的扫描状态。 +下图展示了存储引擎的架构和处理数据的流程。 + +![Architecture](/storage-engine-arch.png) + +该架构与传统的 LSMT 引擎相同： + +- [WAL][2] + - 为尚未刷盘的数据提供高持久性保证。 + - 基于 `LogStore` API 实现，不关心底层存储介质。 + - WAL 的日志记录可以存储在本地磁盘上，也可以存储在实现了 `LogStore` API 的远程日志服务中，例如 Kafka（remote WAL）。 +- Memtable + - 数据首先写入 `active memtable`，又称 `mutable memtable`。 + - 当 `mutable memtable` 已满时，它将变为只读的 `immutable memtable`。 +- SST + - SST 的全名为有序字符串表（`Sorted String Table`）。 + - `immutable memtable` 刷到持久存储后形成一个 SST 文件。 + - SST 中的行按照主键和时间索引排序；详见 [SST 文件中的数据布局](#sst-文件中的数据布局)。 +- Compactor + - `Compactor` 通过 compaction 操作将小的 SST 合并为大的 SST。 + - 默认使用 [TWCS][3] 策略进行合并。Compaction 会按照时间窗口组织 SST 文件，并结合 TTL 清理过期数据。详见 [Compaction](/user-guide/deployments-administration/manage-data/compaction.md)。 +- Manifest + - `Manifest` 存储引擎的元数据，例如 SST 的元数据。 +- Cache + - 加速查询操作。 [1]: https://en.wikipedia.org/wiki/Log-structured_merge-tree [2]: https://en.wikipedia.org/wiki/Write-ahead_logging @@ -26,11 +41,26 @@ Mito 是 GreptimeDB 主要的时序 Region engine，实现了 `RegionEngine` tra ## 数据模型 -Mito 接收由 `RegionMetadata` 描述的 schema，其中包含 primary-key column list、一个非空 time-index column 和 field columns。SQL 层把 primary-key column 暴露为 tag，Mito 本身依据 column ID 和 semantic type 工作，不解析 SQL 表定义。 +存储引擎提供的数据模型介于 `key-value` 模型和表模型之间 + +```txt +tag-1, ..., tag-m, timestamp -> field-1, ..., field-n +``` + +每一行数据包含多个 tag 列，一个 timestamp 列和多个 field 列 +- `0 ~ m` 个 tag 列 + - tag 列是可空的 + - 在建表时通过 `PRIMARY KEY` 指定 +- 必须包含一个 timestamp 列 + - timestamp 列非空 + - 在建表时通过 `TIME INDEX` 指定 +- `0 ~ n` 个 field 列 + - field 列是可空的 +- 数据按照 tag 列和 timestamp 列有序存储 -### Region +## Region -Region 是 Mito 的隔离、恢复和请求单元，其中每一行都遵循该 Region 的元数据。一张表可以跨多个 Region，但表路由和放置不属于存储引擎职责。 +数据在存储引擎中以 `region` 的形式存储，`region` 是引擎中的一个逻辑隔离存储单元。`region` 中的行必须具有相同的 `schema`（模式），该 `schema` 定义了 `region` 中的 tag 列，timestamp 列和 field 列。数据库中表的数据存储在一到多个 `region` 中。 ## SST 文件中的数据布局 @@ -38,6 +68,28 @@ Region 是 Mito 的隔离、恢复和请求单元，其中每一行都遵循该 在一个 SST 文件内，行按照 `(primary key, time index)` 排序。具有相同 primary key（tag 列）的行属于同一条时间序列，会连续存储并按时间戳排序。这种局部性使得扫描单条时间序列的成本更低，也有助于提升压缩效果。对于没有 primary key 的 append-only 表，行仅按 time index 排序。 +例如，考虑一个存储主机指标的表： + +```sql +CREATE TABLE host_metrics ( + host STRING, + region STRING, + ts TIMESTAMP TIME INDEX, + cpu DOUBLE, + memory DOUBLE, + PRIMARY KEY (host, region) +); +``` + +Mito 会按 primary key 对行分组，并按时间排序，因此 SST 中的数据在概念上类似于： + +| host | region | ts | cpu | memory | +| --- | --- | --- | --- | --- | +| host-a | us-east | 10:00 | 0.42 | 7.1 | +| host-a | us-east | 10:01 | 0.47 | 7.4 | +| host-a | us-west | 10:00 | 0.31 | 6.8 | +| host-b | us-east | 10:00 | 0.80 | 8.6 | + 除了表中的列，Mito 还会在每个 SST 文件中存储三个内部列，以便在从多个 memtable 和 SST 文件读取时正确地合并、去重并应用删除操作： - `__primary_key`：行的编码后 primary key（tags）。 @@ -56,6 +108,6 @@ Mito 会组合多个从粗到细的裁剪步骤，避免读取不可能匹配查 1. **时间范围裁剪。** 如果文件和 memtable 的时间范围与查询时间范围不相交，就会在打开 reader 之前被跳过。对于时间序列查询，这通常是成本最低且最有效的步骤。 2. **Row group 统计信息。** 如果 row group 的 min-max 统计信息能够证明没有任何行匹配谓词，则会跳过整个 row group。 -3. **索引。** 倒排索引、跳数索引和全文索引可以针对统计信息无法处理的谓词提供更精细的裁剪；feature-gated vector index 为向量搜索选择候选行。详见[数据持久化和索引](data-persistence-indexing.md)。 +3. **索引。** 倒排索引、跳数索引和全文索引可以针对统计信息无法处理的谓词提供更精细的裁剪。详见[数据持久化和索引](data-persistence-indexing.md)。 Scan pruning pipeline diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/wal.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/wal.md index 0cc48c0778..937cc9c3d7 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/wal.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/wal.md @@ -1,6 +1,6 @@ --- -keywords: [预写日志, WAL, 恢复, raft-engine, Kafka] -description: 介绍 Mito 的 WAL 抽象、恢复路径和持久性配置。 +keywords: [预写日志, WAL, 数据持久化, 同步刷盘, 异步刷盘] +description: 介绍了 GreptimeDB 的预写日志（WAL）机制，包括其命名空间、同步/异步刷盘策略和在数据节点重启时的重放功能。 --- # 预写日志 @@ -9,18 +9,16 @@ description: 介绍 Mito 的 WAL 抽象、恢复路径和持久性配置。 ## 介绍 -Mito 在把数据刷写为 SST 文件前，先将写入应用到内存中的 memtable。为了恢复尚未进入 SST 的数据，每个 Region 的写操作会先追加到预写日志（WAL），再写入 memtable。 +Mito 在数据 flush 到 SST 文件前，先把写入应用到内存中的 MemTable。每个 Region 的写操作会先追加到预写日志（WAL），从而恢复尚未进入 SST 的数据。 -打开 Region 或重启 Datanode 时，Mito 从已持久化的最后一个 sequence 之后开始重放 WAL，重建内存状态。Sequence number 在 Region 内分配，同时用于去重和 snapshot read。 +Datanode 重启并重新打开 Region 时，Mito 会重放最后一个已持久化 sequence 之后的 WAL 条目，重建内存状态。WAL 通过统一的 log-store 抽象访问，可以使用本地 raft-engine 或远端 Kafka。 -存储引擎通过 `LogStore` 抽象访问 WAL。Datanode 支持本地 `raft_engine` provider 和远端 Kafka provider，因此 WAL 并不等同于本地文件。Provider 在 `src/datanode/src/datanode.rs` 中构建，Mito 的 WAL 接入位于 `src/mito2/src/wal.rs` 及写入 worker。 +![WAL in Datanode](/wal.png) ## 命名空间 -WAL 按 Region 隔离。追加和读取操作使用 Region ID 作为 namespace，使恢复过程只重放当前 Region 的日志。一张表可以包含多个 Region，因此 WAL namespace 不是 Table ID。 +WAL 的命名空间用于区分来自不同 region 的条目。追加和读取操作必须提供一个命名空间。目前，region ID 被用作命名空间，因为每个 region 都有一个在数据节点重新启动时需要重构的 MemTable。 ## 同步/异步刷盘 -对于本地 `raft_engine` provider，`sync_write` 控制追加操作是否等待日志同步到持久化存储，默认值为 `false`。异步写入延迟较低，但主机或存储在日志同步前发生故障时，最近已确认的 entry 可能丢失。设置 `sync_write = true` 可以加强这一持久性边界，同时会增加写入延迟。 - -Kafka WAL 的持久性取决于 Kafka producer 和集群配置，而不是本地 `sync_write` 选项。无论使用哪一种 provider，确认写入的代码都必须保持先追加 WAL、再修改 memtable 的顺序。 +对于本地 raft-engine，`sync_write` 控制追加写是否等待日志同步到持久化存储，默认值为 `false`。异步写入延迟较低，但主机在缓冲数据同步前故障时，可能丢失最近确认的日志。Kafka WAL 的持久性由 producer 和集群配置决定，不受这个本地选项控制。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/arrangement.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/arrangement.md index b7aae65d6d..7b35b50259 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/arrangement.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/arrangement.md @@ -1,14 +1,20 @@ --- -keywords: [旧流处理模式, Arrangement, 状态, 差分更新, watermark] -description: 介绍 Flownode 旧 streaming 路径使用的内存 Arrangement 状态。 +keywords: [Arrangement, 状态存储, 键值对] +description: 描述了 Arrangement 在数据流进程中的状态存储功能，包括键值对存储、查询和删除操作的实现。 --- # Arrangement -`Arrangement` 是 Flownode 旧 streaming 路径使用的内存状态索引，实现在 `src/flow/src/utils.rs` 中；batching 模式不使用它。 +本页介绍 Flownode 旧 streaming 模式使用的状态结构；batching 模式不使用 Arrangement。 -Arrangement 以 `((key row, value row), timestamp, diff)` 保存更新。`timestamp` 按 dataflow 时间排列变更，差分值 `diff` 用于添加或删除 value。`get(now: Timestamp, key: &Row)` 返回指定时间对该 key 可见的 value。 +Arrangement 存储数据流进程中的状态，存储 flow 的更新流（stream）以供进一步查询和更新。 -Low watermark 表示仍可能需要保留历史状态的最早时间。早于该 watermark 的状态被视为已经写入 sink，可以进行压缩。过早推进 watermark 会使后续差分更新无法正确合并。 +Arrangement 本质上存储的是带有时间戳的键值对。 +在内部，Arrangement 接收类似 `((Key Row, Value Row), timestamp, diff)` 的 tuple，并将其存储在内存中。 +你可以使用 `get(now: Timestamp, key: Row)` 查询某个时间的键值对。 +Arrangement 假定早于某个时间（也称为 Low Watermark）的所有内容都已被写入到 sink 表中，不会为其保留历史记录。 -在当前实现中，`diff` 为 `-1` 时删除 key；以不同 value 再次插入同一个 key 时，会替换原 value。这些语义属于旧 streaming 状态模型，不能套用到 batching 模式的 sink 写入。 +:::tip 注意 +Arrangement 允许通过将传入 tuple 的 `diff` 设置为 -1 来删除键。 +此外，如果已将行数据添加到 Arrangement 并且使用不同的值插入相同的键，则原始值将被新值覆盖。 +::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md index 9e2610c0dd..bec092b0af 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md @@ -1,50 +1,74 @@ --- -keywords: [批处理模式, BatchingEngine, 脏时间窗口, checkpoint, 持续聚合] -description: 介绍批处理模式的任务生命周期、脏窗口处理和恢复不变量。 +keywords: [批处理模式, flow 管理, Flownode 组件, Flownode 限制, 持续聚合] +description: Flownode 批处理模式概述，这是持续数据聚合当前使用的执行模式，包括其架构和查询执行流程。 --- # Flownode 批处理模式开发者指南 -批处理模式通过重新执行可能受源数据变更影响的 Flow 查询来维护 sink 表，是当前持续开发的 Flownode 执行路径。模式选择属于内部行为，参见 [Flownode 概览](./overview.md)。 +本指南简要概述了 `flownode` 中的批处理模式。它旨在帮助希望了解此模式内部工作原理的开发人员。 ## 概述 -对于按时间窗口计算的 Flow，源表写入会把对应窗口标记为脏。后台任务消费这些窗口；如果查询形态允许，任务会给 Flow 查询添加时间谓词，然后把 insert plan 发送到 Frontend。Frontend 执行查询并将结果写入 sink 表。 +`flownode` 中的批处理模式专为持续数据聚合而设计。它在离散的、微小的时间窗口上周期性地执行用户定义的 SQL 查询。这与原始的流处理模式形成对比；流处理模式现在已经废弃，在该模式下数据会在到达时即被处理。 -按固定间隔执行的 Flow 和 TQL Flow 可能需要执行完整查询，不能使用脏窗口过滤。因此，batching 路径会根据 Flow 定义，把脏窗口视为需要重算的精确范围，或视为需要执行完整查询的信号。 +其核心思想是： +1. 定义一个带有 SQL 查询的 `flow`，该查询将数据从源表聚合到目标表。 +2. 查询通常在时间戳列上包含一个时间窗口函数（例如 `date_bin`）。 +3. 当新数据插入源表时，系统会将相应的时间窗口标记为“脏”(dirty)。 +4. 一个后台任务会周期性地唤醒，识别这些脏窗口，并为那些特定的时间范围重新运行聚合查询。 +5. 然后将结果插入到目标表中，从而有效地更新聚合视图。 ## 架构 +批处理模式由几个协同工作的关键组件组成，以实现这种持续聚合。如下图所示： + +![batching mode architecture](/batching_mode_arch.png) + ### `BatchingEngine` -`src/flow/src/batching_mode/engine.rs` 中的 `BatchingEngine` 持有 `FlowId` 到 `BatchingTask` 的映射。它负责创建和删除任务、处理 flush 请求，并将脏窗口通知分发给所有读取相关源表的 Flow。 +`BatchingEngine` 是批处理模式的核心。它是一个管理所有活动 flow 的中心组件。其主要职责是： -创建任务时会解析 Flow 查询，记录源表和 sink 表，在需要时创建 sink 表，并初始化执行状态。Flow 自身的元数据由 `common-meta` 持久化。 +- **任务管理**: 维护一个从 `FlowId` 到 `BatchingTask` 的映射。它处理这些任务的创建、删除和检索。 +- **事件分发**: 当新数据到达（通过 `handle_inserts_inner`）或当时间窗口被显式标记为脏（`handle_mark_dirty_time_window`）时，`BatchingEngine` 会识别受影响的 flow，并将信息转发给相应的 `BatchingTask`。 ### `BatchingTask` -每个 `BatchingTask` 对应一个 Flow。`TaskConfig` 保存不可变的查询、表、窗口、过期时间及调度配置，`TaskState` 保存可变的执行状态。 +`BatchingTask` 代表一个独立的、单个的数据流。每个任务都与一个 `flow` 定义相关联，并在其自己的异步循环中运行。 -后台循环等待调度时间或通知，生成下一次 insert plan，通过 `FrontendClient` 执行并记录结果。Execution lock 会串行化后台执行、手动 flush、计划生成和 checkpoint 更新，避免两个执行过程同时消费同一份状态。 +- **配置 (`TaskConfig`)**: 此结构体持有 flow 的不可变配置，例如 SQL 查询、源表和目标表名以及时间窗口表达式。 +- **状态 (`TaskState`)**: 包含任务的动态、可变状态，最重要的是 `DirtyTimeWindows`。 +- **执行循环**: 任务运行一个无限循环 (`start_executing_loop`)，该循环： + 1. 检查关闭信号。 + 2. 等待一个预定的时间间隔或直到被唤醒。 + 3. 基于当前的脏时间窗口集合生成一个新的查询计划 (`gen_insert_plan`)。 + 4. 对数据库执行查询 (`execute_logical_plan`)。 + 5. 清理已处理的脏窗口。 ### `TaskState` 和 `DirtyTimeWindows` -`DirtyTimeWindows` 保存需要重新计算且互不重叠的时间范围。生成计划时会从队列中取出数量受限的一组范围。如果计划生成或执行失败，这些范围会重新放回队列；任务不会因为一次执行已经开始就直接丢弃它们。 +- **`TaskState`**: 此结构体跟踪 `BatchingTask` 的运行时状态。它包括 `dirty_time_windows`，这对于确定需要完成哪些操作至关重要。 +- **`DirtyTimeWindows`**: 这是一个关键的数据结构，用于跟踪自上次查询执行以来哪些时间窗口接收到了新数据。它存储一组不重叠的时间范围。当任务的执行循环运行时，它会参考此结构来构建一个 `WHERE` 子句，该子句仅过滤源表中的脏时间窗口。 -`TaskState` 还为实验性的增量读取路径保存每个 Region 的 checkpoint。只有执行结果为参与查询的 Region 提供完整 watermark 证明时，增量模式才会推进 checkpoint。按范围执行 full-snapshot repair 时，任务会冻结 high watermark 并逐步处理脏窗口；期间的新写入仍保留在 live queue。Repair 失败或 watermark 证明不完整时，尚未完成的窗口会返回队列。 +### `TimeWindowExpr` -增量读取通过 `experimental_enable_incremental_read` 控制，默认关闭。关闭该选项或查询形态不兼容时，任务使用 full-snapshot 执行。 +`TimeWindowExpr` 是一个用于处理像 `date_bin` 这样的时间窗口表达式的辅助工具。 -### `TimeWindowExpr` +- **求值**: 它可以接受一个时间戳并对时间窗口表达式求值，以确定该时间戳所属窗口的开始和结束。 +- **窗口大小**: 它还可以从表达式中确定时间窗口的大小（持续时间）。 -`src/flow/src/batching_mode/time_window.rs` 中的 `TimeWindowExpr` 负责计算 `date_bin` 等窗口表达式。它把输入时间戳映射到窗口边界，并提供合并范围及生成谓词所需的窗口大小。 +这对于标记窗口为脏以及在查询源表时生成正确的过滤条件都至关重要。 ## 查询执行流程 -1. 源表写入或显式 mark-dirty 请求确定受影响的 Flow 和时间范围。 -2. `BatchingEngine` 将范围加入任务的 `DirtyTimeWindows`，并在需要时唤醒任务。 -3. `BatchingTask` 取出数量受限的一组窗口，构建带过滤条件的 insert plan；无法安全限定范围的 Flow 则执行完整查询。 -4. `FrontendClient` 将序列化逻辑计划发送到 Frontend。Frontend 执行查询并把结果写入 sink 表。 -5. 执行成功后，任务提交执行状态，并且只推进有返回 watermark 证明的 checkpoint；执行失败时，先恢复已取出的窗口，再等待下一次重试。 - -修改 plan coverage、checkpoint 推进或脏窗口恢复逻辑会影响正确性。尚未反映到 sink 表的工作不能被清除，checkpoint 也不能越过结果已证明包含的数据范围。 +以下是批处理模式下查询执行的简化分步演练： + +1. **数据摄取**: 新数据被写入源表。 +2. **标记为脏**: `BatchingEngine` 收到有关新数据的通知。它使用与每个相关 flow 关联的 `TimeWindowExpr` 来确定哪些时间窗口受到新数据点的影响。然后将这些窗口添加到相应 `TaskState` 中的 `DirtyTimeWindows` 集合中。 +3. **任务唤醒**: `BatchingTask` 的执行循环被唤醒，原因可能是其周期性调度，也可能是因为它被通知有大量积压的脏窗口。 +4. **计划生成**: 任务调用 `gen_insert_plan`。此方法： + - 检查 `DirtyTimeWindows`。 + - 生成一系列 `OR` 连接的 `WHERE` 子句（例如 `(ts >= 't1' AND ts < 't2') OR (ts >= 't3' AND ts < 't4') ...`），覆盖所有脏窗口。 + - 重写原始 SQL 查询以包含此新过滤器，确保只处理必要的数据。 +5. **执行**: 修改后的查询计划被发送到 `Frontend` 执行。数据库处理已过滤数据的聚合。 +6. **Upsert**: 结果被插入到目标表中。目标表通常定义了一个包含时间窗口列的主键，因此现有窗口的新结果将覆盖（upsert）旧结果。 +7. **状态更新**: `DirtyTimeWindows` 集合中刚刚处理过的窗口被清除。然后任务返回睡眠状态，直到下一个时间间隔。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md index edb39ba7f1..24f9c04146 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md @@ -1,14 +1,20 @@ --- -keywords: [旧流处理模式, Dataflow, DFIR, 差分数据, Flow] -description: 介绍 Flownode 旧 streaming 执行路径使用的内部计算图。 +keywords: [Dataflow, SQL 查询, 执行计划, 数据流, map, reduce] +description: 解释了 Dataflow 模块的核心计算功能，包括 SQL 查询转换、内部执行计划、数据流的触发运行和支持的操作。 --- # 数据流 -本文说明 Flownode 旧 streaming 模式使用的计算图。新的持续聚合工作使用[批处理模式](./batching_mode.md)，不能根据本页推断 batching 行为。 +本页介绍 Flownode 旧 streaming 模式使用的计算图。新的持续聚合功能使用[批处理模式](./batching_mode.md)。 -Streaming 路径通过 `src/flow/src/transform.rs` 将 Flow 定义转换为 `plan.rs` 中的 typed plan。`src/flow/src/compute/render.rs` 把受支持的 plan node 渲染为 DFIR 风格的 dataflow graph，`src/flow/src/adapter/` 下的 worker 持有并执行这些 graph。 +Dataflow 模块（参见 `flow::compute` 模块）是 `flow` 的核心计算模块。 +它接收 SQL 查询并将其转换为 `flow` 的内部执行计划。 +然后，该执行计划被转化为实际的数据流，而数据流本质上是一个由带有输入和输出端口的函数组成的有向无环图（DAG）。 +数据流会在需要时被触发运行。 -内部记录使用差分行 `(row, timestamp, diff)`。`row` 保存值，`timestamp` 跟踪 dataflow 进度，`diff` 表示插入（`+1`）、删除（`-1`）等 multiplicity 变更。算子沿执行图传递这些变更，从而增量更新聚合状态和 sink 输出。 +目前该数据流只支持 `map`和 `reduce` 操作，未来将添加对 `join` 等操作的支持。 -Typed plan 可以表示 map/filter/project、reduce、join 和 union 节点。当前 streaming renderer 可以执行 map/filter/project 和 reduce；join 与 union 的渲染仍返回 not-implemented 错误。添加算子时必须同时检查 `plan.rs` 和 `compute/render.rs`，因为能出现在计划中并不等于已经可以执行。 +在内部，数据流使用 `tuple(row, time, diff)` 以行格式处理数据。 +这里 `row` 表示实际传递的数据，可能包含多个 `value` 对象。 +`time` 是系统时间，用于跟踪数据流的进度，`diff` 通常表示行的插入或删除（+1 或 -1）。 +因此，`tuple` 表示给定系统时间的 `row` 的插入/删除操作。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/overview.md index e2ff865515..a093e564b4 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/overview.md @@ -1,28 +1,25 @@ --- -keywords: [Flownode, 持续聚合, 批处理模式, 流处理模式, Flow] -description: 介绍 Flownode 的执行模式、路由边界和实现目录。 +keywords: [持续聚合, flow 管理, 单机模式, Flownode 组件, Flownode 限制] +description: Flownode 概览，一个为数据库提供 Flow 计算能力的组件，包括 batching mode、已废弃的 streaming mode 和核心组件。 --- # Flownode ## 简介 -Flownode 是 GreptimeDB Flow 的执行组件，负责根据源表持续计算结果并写入 sink 表。单机模式下它运行在 GreptimeDB 进程内，分布式模式下则作为独立服务运行。 +`Flownode` 为数据库提供 Flow 计算能力。 +`Flownode` 管理 `flow`，这些 `flow` 是从 `source` 接收数据并将数据发送到 `sink` 的任务。 -Flownode 包含两条执行路径： +`Flownode` 支持 `standalone`（单机）和 `distributed`（分布式）两种模式。在 `standalone` 模式下，`Flownode` 与数据库运行在同一进程中。在 `distributed` 模式下，`Flownode` 运行在单独的进程中，并通过网络与数据库通信。 -- **批处理模式**是当前持续开发的路径。它跟踪受影响的时间窗口，并定期通过 Frontend 执行聚合查询。参见[批处理模式开发者指南](./batching_mode.md)。 -- **流处理模式**是旧的增量 dataflow 路径。它在 worker 持有的计算图中处理行级变更，目前仅为兼容性保留。 - -用户不能直接选择执行模式。`flow_type` 是保留的内部元数据。创建 Flow 时，`src/operator/src/statement/ddl.rs` 中的 `StatementExecutor::determine_flow_type` 决定执行模式，Flownode 内部再由 `FlowDualEngine` 完成兼容路由。 +一个 flow 有两种执行模式： +- **批处理模式 (Batching Mode)**: 持续数据聚合当前使用的模式。它在离散的、微小的时间窗口上周期性地执行用户定义的 SQL 查询。聚合和 TQL 查询使用此模式。更多详情，请参阅[批处理模式开发者指南](./batching_mode.md)。 +- **流处理模式 (Streaming Mode，已废弃)**: 原始的模式，数据在到达时即被处理。该模式保留用于兼容旧 workload，不推荐新 workload 使用。 ## 组件 -- `src/flow/src/engine.rs` 中的 `FlowEngine` 定义两条路径共用的创建、删除、flush 和 insert 生命周期。 -- `src/flow/src/adapter/flownode_impl.rs` 中的 `FlowDualEngine` 把每个 Flow 路由到 batching engine 或 streaming engine。 -- `src/flow/src/batching_mode/` 包含时间窗口跟踪、任务调度、Frontend RPC、sink 表创建和 checkpoint 逻辑。 -- `src/flow/src/adapter/`、`compute/`、`expr/` 和 `plan.rs` 实现旧的 streaming 路径。 -- `src/flow/src/server.rs` 提供 Flownode gRPC 服务；`heartbeat.rs` 向 Metasrv 报告 Flownode 状态。 -- 持久化 Flow 元数据和 DDL Procedure 位于 `src/common/meta/`，不属于 `flow` crate。 +`Flownode` 包含了执行一个 flow 所需的所有组件。所涉及的具体组件取决于执行模式。在较高的层面上，关键部分包括： -修改某一种执行模式时，必须同时检查 `FlowDualEngine` 和共享元数据契约，不能默认一条路径的修复也适用于另一条路径。 +- **Flow Manager**: 一个负责管理所有 flow生命周期的中心组件。 +- **Task Executor**: flow 逻辑执行的运行时环境。在批处理模式下，它是一个 `BatchingTask`；在已废弃的流处理模式下，这通常是一个 `FlowWorker`。 +- **Flow Task**: 代表一个独立的、单个的数据流，包含将数据从 source 转换为 sink 的逻辑。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/distributed-querying.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/distributed-querying.md index e5734b5945..e6d0c35160 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/distributed-querying.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/distributed-querying.md @@ -1,6 +1,6 @@ --- -keywords: [分布式查询, DistPlannerAnalyzer, MergeScan, Substrait, Region 裁剪] -description: GreptimeDB 如何将逻辑查询计划划分为本地和远端执行阶段。 +keywords: [分布式查询, 逻辑计划, MergeScan, Substrait, Region 裁剪] +description: 介绍 GreptimeDB 如何把逻辑查询计划划分为 Frontend 和 Datanode 上的执行任务。 --- # 分布式查询 @@ -9,16 +9,12 @@ Frontend 和 Datanode 使用同一套基于 DataFusion 的查询引擎。在分 ![Frontend query](/frontend-query.png) -## 分布式规划器 +## 分布式规划 -`src/query/src/dist_plan/analyzer.rs` 中的 `DistPlannerAnalyzer` 会重写 DataFusion 逻辑计划。它将可下推的算子移向表扫描，并用 `MergeScan` 节点包装远端子计划。规划器根据算子的交换律和计划形态判断哪些工作可以安全地在各 Datanode 执行；不支持的计划形态保留在 Frontend，或使用配置允许的 fallback 路径。 +分布式规划器重写逻辑计划，把可以下推的算子移向表扫描，并用 `MergeScan` 节点包装远端子计划。分区列上的谓词还会在任务调度前用于裁剪 Region。 -分区列上的过滤条件同时用于裁剪 Region。执行前，Frontend 通过 `FrontendRegionQueryHandler` 将每个入选 Region 解析到对应 Datanode。 - -初始设计及交换律规则参见[分布式规划器 RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md)。 +算子能否下推取决于计划形态和算子本身的性质。不支持的部分会保留在 Frontend。初始设计及交换律规则参见[分布式规划器 RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md)。 ## 分布式计划 -`MergeScan` 的远端输入是一个完整的逻辑子计划。Frontend 使用 Substrait 对子计划进行序列化，并向选定的 Datanode 发送 Region 级查询请求。Datanode 针对本地 Region 规划并执行该子计划，再以 Arrow RecordBatch stream 返回结果。 - -Frontend 合并远端数据流，并执行无法下推的算子。这个边界并不局限于逻辑计划中的 `TableScan` 节点：过滤、投影、部分聚合以及其他兼容算子都可能进入远端子计划。 +远端输入是完整的逻辑子计划，并不局限于表扫描。Frontend 使用 [Substrait](https://substrait.io) 序列化子计划，再向持有相应数据的 Datanode 发送 Region 级请求。Datanode 在本地规划并执行子计划，将结果流返回 Frontend。Frontend 合并远端数据流，并执行没有下推的算子。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/overview.md index adbf0e5b13..ee48da3ce3 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/overview.md @@ -1,46 +1,43 @@ --- -keywords: [Frontend, 协议, 请求路由, 分布式查询, 权限校验] -description: GreptimeDB 无状态请求入口和查询协调组件 Frontend 的实现概览。 +keywords: [frontend, proxy, protocol, routing, distributed query, tenant management, authorization, flow control, cloud deployment, endpoints] +description: GreptimeDB Frontend 组件概述 - 为客户端请求提供服务的无状态代理服务。 --- # Frontend -Frontend 是 GreptimeDB 的无状态请求入口和编排层。它实现协议服务背后的业务逻辑，负责查询规划、写入与 Region 读取路由，以及分布式查询协调。 +**Frontend** 是一个无状态服务，作为 GreptimeDB 中客户端请求的入口点。它为多种数据库协议提供统一接口，并充当代理，将读写请求转发到分布式系统中的相应 Datanode。 -网络监听和 wire format 属于 `servers` crate。`frontend` crate 为 SQL、gRPC、MySQL、PostgreSQL、InfluxDB、OpenTelemetry、Prometheus、OpenTSDB、Jaeger 等接口实现对应的 handler trait。 +## 核心功能 - - -## 职责 - -- 解析和规划 SQL、PromQL 及日志查询。 -- 校验权限，并在请求处理链路中传递 session context。 -- 使用 Catalog 和路由元数据分发插入、删除及 Region 查询。 -- 将分布式查询片段发送到 Datanode，并合并执行结果。 - -面向用户的接口参见[协议概览](/user-guide/protocols/overview.md)。 +- **协议支持**：支持多种数据库协议，包括 SQL、PromQL、MySQL 和 PostgreSQL。详见[协议][1] +- **请求路由**：基于元数据将请求路由到相应的 Datanode +- **查询分发**：将分布式查询拆分到多个节点 +- **响应聚合**：合并来自多个 Datanode 的结果 +- **认证授权**：安全和访问控制验证 ## 架构 ### 关键组件 - -- `src/frontend/src/instance.rs` 中的 `Instance` 是主要业务逻辑容器，实现各类 server handler trait。 -- `src/frontend/src/instance/` 下的模块处理不同请求类型和协议。 -- `operator` crate 中的 `StatementExecutor` 负责语句及写入侧操作。 -- `query` crate 负责逻辑计划、优化和分布式计划。 -- `instance/region_query.rs` 中的 `FrontendRegionQueryHandler` 解析 Region 目标并向 Datanode 发送查询请求。 +- **协议处理器**：处理不同的数据库协议 +- **目录管理器**：缓存来自 Metasrv 的元数据以实现高效的请求路由和 Schema 校验 +- **分布式规划器**：将逻辑计划转换为分布式执行计划 +- **请求路由器**：为每个请求确定目标 Datanodes ### 请求流程 -单机模式下，Frontend 通过本地 `RegionServer` adapter 访问内嵌的 Datanode。分布式模式下，Frontend 使用 Metasrv 提供的元数据和 RPC client 访问远端 Datanode。 +![request flow](/request_flow.png) ### 部署 -Frontend 不持有表数据。多个 Frontend 实例可以共同服务同一组 Metasrv 和 Datanode。 +下图是 GreptimeDB 在云上的一个典型的部署。`Frontend` 实例组成了一个集群处理来自客户端的请求： + +![frontend](/frontend.png) - +## 详细信息 -## 实现指南 +- [表分片][2] +- [分布式查询][3] -- [表分片](./table-sharding.md) -- [分布式查询](./distributed-querying.md) +[1]: /user-guide/protocols/overview.md +[2]: ./table-sharding.md +[3]: ./distributed-querying.md diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/table-sharding.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/table-sharding.md index b24605d670..63a8ac10a6 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/table-sharding.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/table-sharding.md @@ -5,7 +5,7 @@ description: 介绍 GreptimeDB 中表数据的分片方法，包括分区和 Reg # 表分片 -GreptimeDB 将表拆分为分区，并把每个分区存储在一个 Region 中。本文说明这两个对象在实现上的关系。 +对于任何分布式数据库来说，数据的分片都是必不可少的。本文将描述 GreptimeDB 中的表数据如何进行分片。 @@ -15,7 +15,10 @@ GreptimeDB 将表拆分为分区，并把每个分区存储在一个 Region 中 ## Region -每个分区对应一个 Region。Region 是由 Datanode 管理的存储和调度单元，Metasrv 保存 Region 到 Datanode 的路由信息。如果建表后需要调整分区布局， +在创建分区后，表中的数据被逻辑上分割。你可能会问："在 GreptimeDB 中，被逻辑上分区的数据是如何存储的？" 答案是保存在 `Region` 当中。 + +每个 `Region` 对应一个分区，并保存分区的数据。所有的 `Region` 分布在各个 `Datanode` 之中。 +`Metasrv` 管理 `Region` 到 `Datanode` 的路由信息。如果建表后需要调整分区布局， GreptimeDB 支持通过显式的 [repartition](/user-guide/deployments-administration/manage-data/repartition.md) 操作拆分或合并分区。 分区和 Region 的关系参见下图： @@ -38,7 +41,7 @@ GreptimeDB 支持通过显式的 [repartition](/user-guide/deployments-administr │ P0 │ │ P1 │ │ Px │ └─────┬─────┘ └─────┬─────┘ └─────┬─────┘ │ │ │ - │ │ │ + │ │ │ ┌───────┼──────────────────┼───────┐ │ Partition 和 Region 是一一对应的 │ │ │ │ │ │ ┌─────▼─────┐ ┌─────▼─────┐ │ ┌─────▼─────┐ diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/getting-started.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/getting-started.md index 6335f80fbb..d6828a8a17 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/getting-started.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/getting-started.md @@ -1,28 +1,33 @@ --- -keywords: [开发环境, 源码构建, Rust 工具链, 单元测试] -description: 配置开发环境，并从源码构建、运行和测试 GreptimeDB。 +keywords: [编译, 运行, 源代码, 系统要求, 依赖项, Docker] +description: 介绍如何在本地环境中从源代码编译和运行 GreptimeDB，包括系统要求和依赖项。 --- # 立即开始 -本页说明从源码构建和运行 GreptimeDB 所需的基本环境。 +本页面介绍如何在本地环境中从源代码运行 GreptimeDB。 ## 先决条件 ### 系统和架构 -GreptimeDB 支持 x86-64 和 Arm64 架构的 Linux 与 macOS，也支持 Windows。 +目前，GreptimeDB 支持 Linux（amd64 和 arm64）、macOS（amd64 和 Apple Silicon）和 Windows。 ### 构建依赖项 -- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line)。 -- C/C++ 构建工具链，例如 Ubuntu 上的 `build-essential` 或 macOS 上的 Xcode Command Line Tools。 -- [Rustup](https://rustup.rs/)。仓库中的 `rust-toolchain.toml` 会自动选择项目要求的 nightly 工具链。 -- 3.15 或更高版本的 [Protocol Buffers 编译器](https://grpc.io/docs/protoc-installation/)。使用 `protoc --version` 检查版本。 +- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) +- C/C++ 工具链：提供编译和链接的基本工具。在 Ubuntu 上，这可用作 `build-essential`。在其他平台上，也有类似的命令。 +- [Rustup][1]。仓库通过 `rust-toolchain.toml` 指定所需的 nightly 工具链。 +- Protobuf（[指南][2]） + - 编译 proto 文件 + - 请注意，版本需要 >= 3.15。你可以使用 `protoc --version` 检查它。 + +[1]: +[2]: ## 编译和运行 -克隆仓库并启动单机实例： +只需几个命令即可使用以 Standalone 模式启动 GreptimeDB 实例： ```shell git clone https://github.com/GreptimeTeam/greptimedb.git @@ -30,30 +35,34 @@ cd greptimedb cargo run -- standalone start ``` -只构建、不启动服务时运行： +接下来，你可以选择与 GreptimeDB 交互的协议。 + +如果你只想构建服务器而不运行它： ```shell -cargo build +cargo build # --release ``` -优化构建请添加 `--release`。构建产物位于 `target/debug` 或 `target/release`。 +根据构建的模式（是否传递了 `--release` 选项），构建后的文件可以在 `$REPO/target/debug` 或 `$REPO/target/release` 目录下找到。 ## 单元测试 -GreptimeDB 使用 [cargo-nextest](https://nexte.st/) 作为标准 Rust 测试运行器。安装命令如下： +GreptimeDB 经过了充分的测试，整个单元测试套件都随源代码一起提供。要测试它们，请使用 [nextest](https://nexte.st/index.html)。 + +要使用 cargo 安装 nextest，请运行： ```shell cargo install cargo-nextest --locked ``` -使用 CI 对应的 feature 运行 workspace 测试： +或者，你可以查看他们的[文档](https://nexte.st/docs/installation/pre-built-binaries/)以了解其他安装方式。 + +安装好 nextest 后，你可以使用以下命令运行测试套件： ```shell cargo nextest run --workspace --features pg_kvbackend,mysql_kvbackend ``` -按 crate 运行测试以及其他测试类型参见[测试指南](./tests/overview.md)。 - ## Docker -预构建镜像发布在 [Docker Hub](https://hub.docker.com/r/greptime/greptimedb)。镜像适合直接运行 GreptimeDB；开发和验证代码改动时仍应使用源码构建。 +我们还通过 Docker 提供预构建二进制文件，可以在 [Docker Hub 上获取](https://hub.docker.com/r/greptime/greptimedb)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-trace-greptimedb.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-trace-greptimedb.md index 27c85cc9b2..6c1d922ac6 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-trace-greptimedb.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-trace-greptimedb.md @@ -1,31 +1,30 @@ --- -keywords: [tracing, W3C Trace Context, RPC, instrument, runtime] -description: 介绍 GreptimeDB 代码中的分布式 trace 传递和埋点方法。 +keywords: [tracing, 分布式追踪, tracing 上下文, RPC 调用, 代码埋点] +description: 介绍如何在 GreptimeDB 中使用 Rust 的 tracing 框架进行代码埋点，包括在 RPC 中定义和传递 tracing 上下文的方法。 --- # How to trace GreptimeDB -GreptimeDB 使用 Rust [`tracing`](https://docs.rs/tracing/latest/tracing/) 生态和 OpenTelemetry context propagation。只有 tracing context 沿同一条异步执行路径传递时，本地 span 才会自动建立父子关系；跨 RPC 或 runtime 时必须显式传递。 +GreptimeDB 使用 Rust 的 [tracing](https://docs.rs/tracing/latest/tracing/) 框架进行代码埋点，tracing 的具体原理和使用方法参见 tracing 的官方文档。 -公共实现在 `common-telemetry` 的 [`TracingContext`](https://github.com/GreptimeTeam/greptimedb/blob/main/src/common/telemetry/src/tracing_context.rs) 中，负责在当前 span context 和 W3C Trace Context 字段之间转换。 +通过将 `trace_id` 等信息在整个分布式数据链路上透传，使得我们能够记录整个分布式链路的函数调用链，知道每个被追踪函数的调用时间等相关信息，从而对整个系统进行诊断。 - +## 在 RPC 中定义 tracing 上下文 -## RPC 中的 context 字段 +因为 tracing 框架并没有原生支持分布式追踪，我们需要手动将 `trace_id` 等信息在 RPC 消息中传递，从而正确的识别函数的调用关系。我们使用基于 [w3c 的标准](https://www.w3.org/TR/trace-context/#traceparent-header-field-values) 将相关信息编码为 `tracing_context` ，将消息附在 RPC 的 header 中。主要定义在： -GreptimeDB 的 protobuf header 使用 `map tracing_context` 保存 W3C trace 字段： +- `frontend` 与 `datanode` 交互：`tracing_context` 定义在 [`RegionRequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/region/server.proto) 中 +- `frontend` 与 `metasrv` 交互：`tracing_context` 定义在 [`RequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/meta/common.proto) 中 +- Client 与 `frontend` 交互：`tracing_context` 定义在 [`RequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/common.proto) 中 -- Frontend 到 Datanode：[`RegionRequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/region/server.proto) -- Meta client 和 service：[`RequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/meta/common.proto) -- Client 到 Frontend database RPC：[`RequestHeader`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/common.proto) +## 在 RPC 调用中传递 tracing 上下文 -增加内部 RPC 时应尽量复用已有 header type。另建 tracing 字段或采用不同编码，会产生公共 helper 无法处理的传递路径。 +我们构建了一个 `TracingContext` 结构体，封装了与 tracing 上下文有关的操作。[相关代码](https://github.com/GreptimeTeam/greptimedb/blob/main/src/common/telemetry/src/tracing_context.rs) - +GreptimeDB 在使用 `TracingContext::from_current_span()` 获取当前 tracing 上下文，使用 `to_w3c()` 方法将 tracing 上下文编码为符合 w3c 的格式，并将其附在 RPC 消息中，从而使 tracing 上下文正确的在分布式组件之中传递。 -## 跨 RPC 传递 context +下面的例子说明了如何获取当前 tracing 上下文，并在构造 RPC 消息时正确传递参数，从而使 tracing 上下文正确的在分布式组件之中传递。 -构造出站请求时获取当前 context： ```rust let request = RegionRequest { @@ -37,52 +36,45 @@ let request = RegionRequest { }; ``` -接收端解析 header，并把新的本地 span 挂到该 context 下： +在 RPC 消息的接收方，需要将 tracing 上下文正确解码，并且使用该上下文构建第一个 `span` 对函数调用进行追踪。比如下面的代码就将接收到的 RPC 消息中的 `tracing_context` 使用 `TracingContext::from_w3c` 方法正确解码。并使用 `attach` 方法将新建的 `info_span!("RegionServer::handle_read")`  附上了上下文消息，从而能够跨分布式组件对调用进行追踪。 ```rust +... let tracing_context = request .header .as_ref() - .map(|header| TracingContext::from_w3c(&header.tracing_context)) + .map(|h| TracingContext::from_w3c(&h.tracing_context)) .unwrap_or_default(); - let result = self .handle_read(request) .trace(tracing_context.attach(info_span!("RegionServer::handle_read"))) .await?; +... ``` -Header 缺失或 context 无效时会得到空 context，请求仍可在没有 parent trace 的情况下执行。不能把一个请求的 context 复用于无关工作。 - - - -## 使用 `tracing::instrument` 创建 span +## 使用 `tracing::instrument` 对监测代码进行埋点 -在异步边界或开销较大的操作上使用 `#[tracing::instrument]`，便于关联延迟和错误。该宏默认通过 `Debug` 记录参数。凭据、token、大 batch、查询 payload 以及不适合进入 telemetry 的完整参数必须跳过。 +我们使用 tracing 提供的 `instrument` 宏对代码进行埋点，只要将 `instrument` 宏标记在需要进行埋点的函数即可。 `instrument` 宏会每次将函数调用的参数以 `Debug` 的形式打印到 span 中。对于没有实现 `Debug` trait 的参数，或者结构体过大、参数过多，最后导致 span 过大，希望避免这些情况就需要使用 `skip_all`，跳过所有的参数打印。 ```rust -#[tracing::instrument(skip_all, fields(region_id = %region_id))] -async fn handle_region(region_id: RegionId, request: RegionRequest) { - region_server.handle(request).await; +#[tracing::instrument(skip_all)] +async fn instrument_function(....) { + ... } ``` -`fields(...)` 中应记录少量稳定标识符，不要记录完整请求。为每个 helper 都添加 span 会增加大量 trace 数据，却不能改善请求级调用链。 - - - -## 跨 runtime 传递 context +## 跨越 runtime 的代码埋点 -把 future 移到另一个 runtime，或在当前 instrumented future 之外 spawn 任务时，可能丢失当前 parent。跨越边界前先获取 context，再在新 future 中挂载 span： +Rust 的 tracing 库会自动处理埋点函数间的嵌套关系，但如果某个函数的调用跨越 runtime 的话，tracing 不能自动对这类调用进行追踪，我们需要手动跨越 runtime 去传递上下文。 ```rust let tracing_context = TracingContext::from_current_span(); let handle = runtime.spawn(async move { handler .handle(query) - .trace(tracing_context.attach(info_span!("background_query"))) - .await + .trace(tracing_context.attach(info_span!("xxxxx"))) + ... }); ``` -Context 必须在 spawn 前获取。挂载的 span 只应覆盖该异步操作，避免无关任务继承同一个 parent。 +比如上面这段代码需要跨越 runtime 去进行 tracing，我们先通过 `TracingContext::from_current_span()` 获取当前 tracing 上下文，通过在另外一个 runtime 里新建一个 span，并将 span 附着在当前上下文中，我们就完成了跨越 runtime 的代码埋点，正确追踪到了调用链。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-use-tokio-console.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-use-tokio-console.md index c66257fc70..e32c4f3a55 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-use-tokio-console.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-use-tokio-console.md @@ -1,30 +1,34 @@ --- -keywords: [tokio-console, tokio_unstable, 异步任务, 诊断] -description: 构建启用 tokio-console 的 GreptimeDB 并检查 Tokio runtime。 +keywords: [tokio-console, GreptimeDB, 构建配置, 启动配置, 调试工具] +description: 介绍如何在 GreptimeDB 中启用 tokio-console，包括构建和启动时的配置方法。 --- # 如何在 GreptimeDB 中启用 tokio-console -[`tokio-console`](https://github.com/tokio-rs/console) 用于查看实时 Tokio task 和 resource。GreptimeDB 通过 `cmd/tokio-console` feature 编译 subscriber，同时要求启用 Tokio 的 unstable instrumentation cfg。 +本文介绍了如何在 GreptimeDB 中启用 [tokio-console](https://github.com/tokio-rs/console)。 -使用以下命令构建： +首先，在构建 GreptimeDB 时带上 feature `cmd/tokio-console`。同时 `tokio_unstable` cfg 也必须开启： ```bash RUSTFLAGS="--cfg tokio_unstable" cargo build -F cmd/tokio-console ``` -启动组件时为 console subscriber 指定完整 socket address： +启动 GreptimeDB，可设置 tokio console 绑定的地址，配置是 `--tokio-console-addr`。例如： ```bash -./target/debug/greptime --tokio-console-addr="127.0.0.1:6669" standalone start +greptime --tokio-console-addr="127.0.0.1:6669" standalone start ``` -该参数是全局参数，也可以用于以相同 feature 构建的 `frontend`、`datanode`、`metasrv` 或 `flownode` 命令。 - -按照 [tokio-console 仓库](https://github.com/tokio-rs/console#installing-the-console)的说明安装 client，再连接到配置地址： +这样就可以使用 `tokio-console` 命令去连接 GreptimeDB 的 tokio console 服务了： ```bash -tokio-console http://127.0.0.1:6669 +tokio-console [TARGET_ADDR] ``` -Subscriber 应绑定到 loopback 或其他受保护的地址。它是诊断端点，不是公开的 GreptimeDB 协议。该 feature 和 `tokio_unstable` instrumentation 会增加 runtime 诊断信息，只应在排查 task 阻塞、唤醒或资源争用时按需启用。 +"`TARGET_ADDR`" 默认是 "\"。 + +:::tip Note + +`tokio-console` 命令的安装方法参见 [tokio-console](https://github.com/tokio-rs/console)。 + +::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-write-sdk.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-write-sdk.md index abb783a704..28ff757763 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-write-sdk.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-write-sdk.md @@ -1,31 +1,42 @@ --- -keywords: [gRPC ingester SDK, GreptimeDatabase, RowInsertRequests, streaming RPC] -description: 介绍 GreptimeDB gRPC ingester SDK 的协议和可靠性要求。 +keywords: [gRPC SDK, GreptimeDatabase, GreptimeRequest, GreptimeResponse, 插入请求] +description: 介绍如何为 GreptimeDB 开发一个 gRPC SDK，包括 GreptimeDatabase 服务的定义、GreptimeRequest 和 GreptimeResponse 的结构。 --- # 如何为 GreptimeDB 开发一个 gRPC SDK -本文面向基于 GreptimeDB 原生 gRPC database service 的 **ingester SDK**，不涵盖查询 driver 和 client。官方写入库统一采用 `greptimedb-ingester-` 命名。 - -消息和 client 代码应由版本化的 [greptime-proto](https://github.com/GreptimeTeam/greptime-proto) 定义生成，不要在 SDK 中复制 message layout。生成的协议 package 应与提供给应用代码的 row 和 batch API 分离。 +GreptimeDB 的 gRPC SDK 只需要处理写请求即可。读请求是标准 SQL 或 PromQL，可以由任何 JDBC 客户端或 Prometheus +客户端处理。这也是为什么所有的 GreptimeDB SDK 都命名为 "`greptimedb-ingester-`"。请确保你的 GreptimeDB SDK +遵循相同的命名约定。 ## `GreptimeDatabase` 服务 -[`database.proto`](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto) 定义了两个 RPC 方法： +GreptimeDB 自定义了一个 gRPC 服务：`GreptimeDatabase` +。你只需要实现这个服务即可。你可以在[这里](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto) +找到它的 Protobuf 定义。 + +`GreptimeDatabase` 有 2 个 RPC 方法： ```protobuf service GreptimeDatabase { rpc Handle(GreptimeRequest) returns (GreptimeResponse); + rpc HandleRequests(stream GreptimeRequest) returns (GreptimeResponse); } ``` -`Handle` 是 unary RPC。`HandleRequests` 是 [client-streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc)：client 发送一组 request stream，关闭发送端，再接收一个汇总 response。生产 SDK 应使用有界缓冲和 gRPC flow control，不能在内存中无限累积 batch。 +`Handle` 方法是一个 unary 调用：当 GreptimeDB 服务接收到一个 `GreptimeRequest` 请求后，它立刻处理该请求并返回一个相应的 +`GreptimeResponse`。 -协议没有 request 级 idempotency key，因此 SDK 不能承诺 exactly-once ingestion。传输故障导致服务端结果未知时，自动重试可能在保留重复行的表配置中写入重复数据；重试行为必须显式暴露给调用方。 +`HandleRequests` 方法则是一个 "[Client Streaming RPC][3]" 方式的调用。 +它可以接受一个连续的 `GreptimeRequest` 请求流，持续地发给 GreptimeDB 服务。 +GreptimeDB 服务会在收到流中的每个请求时立刻进行处理，并最终（流结束时）返回一个总结性的 `GreptimeResponse`。 +通过 `HandleRequests`，我们可以获得一个非常高的请求吞吐量。 ### `GreptimeRequest` +`GreptimeRequest` 是一个 Protobuf 消息，定义如下： + ```protobuf message GreptimeRequest { RequestHeader header = 1; @@ -40,21 +51,23 @@ message GreptimeRequest { } ``` -写入优先使用 `RowInsertRequests`。每个 `RowInsertRequest` 指定一张表，并携带一个 `Rows` schema 和对应数据行。发送前应校验列数、数据类型、semantic type 及 null 表示，避免 client 构造错误变成难以定位的服务端错误。较早的列式 `InsertRequests` 仍作为兼容协议保留。 +`RequestHeader` 是必需，它包含了一些上下文，鉴权和其他信息。"oneof" 的字段包含了发往 GreptimeDB 服务的请求。 -每个 request 都包含 `RequestHeader`。SDK 配置指定相关值时，应填写目标 Catalog、Schema、认证 header、时区和 W3C tracing context。调用方显式选择 Catalog 或 Schema 后，不能静默替换为 client 默认值。 +注意我们有两种类型的插入请求，一种是以 "列" 的形式（`InsertRequests`），另一种是以 "行" 的形式（`RowInsertRequests` +）。通常我们建议使用 "行" 的形式，因为它对于表的插入更自然，更容易使用。但是，如果需要一次插入大量列，或者有大量的 "null" +值需要插入，那么最好使用 "列" 的形式。 ### `GreptimeResponse` +`GreptimeResponse` 是一个 Protobuf 消息，定义如下： + ```protobuf message GreptimeResponse { ResponseHeader header = 1; - oneof response { - AffectedRows affected_rows = 2; - } + oneof response {AffectedRows affected_rows = 2;} } ``` -gRPC 传输成功不代表数据库操作成功。SDK 必须检查 `ResponseHeader.status`，把非成功 status code 和 `err_msg` 转换为 SDK error，随后才能返回 `affected_rows`。底层 gRPC status 应与 GreptimeDB response status 分开保留，使调用方能够区分传输故障和服务端请求错误。 +`ResponseHeader` 包含了返回值的状态码，以及错误信息（如果有的话）。"oneof" 的字段目前只有 "affected rows"。 -Protobuf client 还必须容忍未知字段及未设置的 response variant。兼容性测试应覆盖支持版本的序列化消息；集成测试应覆盖 unary 写入、client streaming、认证错误、stream 中途失败和服务端 status 传递。 +GreptimeDB 现在有很多 SDK，你可以参考[这里](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=)获取一些示例。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/admin-api.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/admin-api.md index 640d58d5a7..1fedfc09e7 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/admin-api.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/admin-api.md @@ -1,53 +1,151 @@ --- keywords: [Admin API, 健康检查, leader 查询, 心跳检测, 维护模式] -description: 面向维护者的 Metasrv Admin API router 及状态修改端点参考。 +description: 介绍 Metasrv 的 Admin API，包括健康检查、leader 查询、心跳检测、维护模式和 Procedure Manager 控制等功能。 --- # Admin API -Axum router 在 `src/meta-srv/src/service/admin.rs` 中组装，并挂载到 Metasrv HTTP server 的 `/admin` 路径下。默认 HTTP 端口为 `4000`。 +Admin API 通过 HTTP 提供 Metasrv health、leader、Datanode 心跳、维护模式和 Procedure Manager 信息。该 API 不提供认证，且部分端点会改变集群行为，部署时必须通过网络策略保护 HTTP 端口。 +本页介绍以下 API： -Router 本身不增加认证层。部分端点会改变集群行为，部署时必须通过网络策略保护该端口。增加路由时应显式指定 HTTP method，分离读取与修改 handler，并在 `src/meta-srv/src/service/admin/` 中添加 handler-level 测试。 +- /health +- /leader +- /heartbeat +- /maintenance +- /procedure-manager + +所有这些 API 都在父资源 `/admin` 下。 + +在以下部分中，我们假设你的 metasrv 实例运行在本地主机的 4000 端口。 ## /health HTTP 端点 -`GET /admin/health` 在 HTTP service 正常运行时返回 `OK`，但不能证明当前节点是 Leader，也不能证明外部依赖可用。Handler 位于 `health.rs`。 +`/health` 端点接受 GET 请求。HTTP 服务运行时返回 `OK`，但不会检查当前 Metasrv 是否为 leader，也不会检查外部依赖是否可用。 + +### 定义 + +```bash +curl -X GET http://localhost:4000/admin/health +``` + +### 示例 + +#### 请求 + +```bash +curl -X GET http://localhost:4000/admin/health +``` + +#### 响应 + +```json +OK +``` ## /leader HTTP 端点 -`GET /admin/leader` 通过已配置的 election backend 读取当前 Metasrv Leader 地址。Handler 位于 `leader.rs`。 +`/leader` 端点接受 GET HTTP 请求，你可以使用此端点查询你的 metasrv 实例的 leader 地址。 + +### 定义 + +```bash +curl -X GET http://localhost:4000/admin/leader +``` + +### 示例 + +#### 请求 + +```bash +curl -X GET http://localhost:4000/admin/leader +``` + +#### 响应 + +```json +127.0.0.1:4000 +``` ## /heartbeat HTTP 端点 -`GET /admin/heartbeat` 返回 Datanode 心跳记录，可通过 `addr` query parameter 按 Datanode 地址过滤。`GET /admin/heartbeat/help` 展示支持的查询形式。Handler 位于 `heartbeat.rs`，并通过 `MetaPeerClient` 读取数据。 +`/heartbeat` 端点接受 GET HTTP 请求，你可以使用此端点查询所有数据节点的心跳。 + +你还可以查询指定 `addr` 的数据节点的心跳数据，但在路径中指定 `addr` 是可选的。 + +### 定义 + +```bash +curl -X GET http://localhost:4000/admin/heartbeat +``` + +| 查询字符串参数 | 类型 | 可选/必选 | 定义 | +|:---------------|:-------|:----------|:--------------------| +| addr | String | 可选 | 数据节点的地址。 | + +### 示例 + +#### 请求 + +```bash +curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100' +``` + +#### 响应 + +```json +[ + [ + { + "timestamp_millis": 1677049348651, + "id": 1, + "addr": "127.0.0.1:4100", + "rcus": 0, + "wcus": 0, + "region_num": 2, + "region_stats": [], + "topic_stats": [], + "node_epoch": 0, + "datanode_workloads": { + "types": [] + }, + "gc_stat": null + } + ] +] +``` ## /maintenance HTTP 端点 -维护模式会禁用部分自动集群管理操作，面向用户的行为参见[集群维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。Router 提供： +集群维护模式是 GreptimeDB 中的一项安全功能，它可以临时禁用自动集群管理操作。此模式在集群升级、计划停机以及任何可能暂时影响集群稳定性的操作期间特别有用。有关更多详细信息，请参阅[集群维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。 + +`/maintenance` 端点支持以下 HTTP 请求： - `GET /admin/maintenance` 或 `GET /admin/maintenance/status`：查询维护模式状态。 - `POST /admin/maintenance/enable`：启用维护模式。 - `POST /admin/maintenance/disable`：禁用维护模式。 -实现位于 `maintenance.rs`，通过 `RuntimeSwitchManager` 修改状态。 +响应体使用以下格式： + +```json +{ + "enabled": true +} +``` ## /procedure-manager HTTP 端点 -这些路由用于暂停或恢复 Procedure Manager 调度，面向用户的行为参见[防止元数据变更](/user-guide/deployments-administration/maintenance/prevent-metadata-changes.md)。Router 提供： +该端点用于管理 Procedure Manager 状态。有关更多详细信息，请参阅[防止元数据变更](/user-guide/deployments-administration/maintenance/prevent-metadata-changes.md)。 + +`/procedure-manager` 端点支持以下 HTTP 请求： - `GET /admin/procedure-manager/status`：查询 Procedure Manager 状态。 - `POST /admin/procedure-manager/pause`：暂停 Procedure Manager。 - `POST /admin/procedure-manager/resume`：恢复 Procedure Manager。 -实现位于 `procedure.rs`，同样通过 `RuntimeSwitchManager` 修改状态。 - -## 其他内部端点 - -Router 还提供以下维护端点： - -- `GET /admin/node-lease` 返回当前 Datanode 租约记录。 -- `GET /admin/recovery/status` 和 `POST /admin/recovery/{enable,disable}` 查询或修改 recovery mode。 -- `GET /admin/sequence/table/next-id` 读取下一个 Table ID，但不执行分配。 -- `POST /admin/sequence/table/set-next-id` 修改 allocator 的下一个 Table ID。未启用 recovery mode 时，handler 会拒绝该操作。 +响应体使用以下格式： -Recovery 和 sequence 路由会改变集群状态，只能用于受控的修复流程。修改或调用前必须阅读对应 handler 和测试；本文不提供通用恢复流程。 +```json +{ + "status": "running" +} +``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md index d60ea36d24..dccf818e83 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md @@ -1,70 +1,60 @@ --- -keywords: [Metasrv, 元数据, 路由, Leader 选举, 心跳, 分布式 Procedure] -description: 介绍 Metasrv 的元数据、协调和集群管理职责。 +keywords: [Metasrv, 元数据, 路由, Leader 选举, Procedure, 心跳] +description: 介绍 Metasrv 提供的元数据及集群协调机制。 --- # Metasrv - +## Metasrv 包含什么 -## 职责 +Metasrv 是 GreptimeDB 分布式集群中的元数据和协调服务，不参与数据读写链路。它主要负责： -Metasrv 是分布式部署中的元数据和协调服务，负责： +- 存储 Catalog、Schema、Table、Region、路由和节点元数据； +- 为新 Region 选择 Datanode，并维护表路由； +- 选举一个 Metasrv leader 负责协调元数据变更； +- 通过可恢复的 Procedure 执行 DDL、Region 迁移、故障转移和 repartition； +- 通过心跳维护节点租约和 Region 统计信息； +- 在缓存元数据或 Region 状态变化时通知 Frontend 和 Datanode。 -- 通过 KV backend 持久化 Catalog、Schema、Table、Region、路由和节点元数据； -- 通过 Leader 选举保证协调操作和元数据修改只在一个 Leader 上执行； -- 通过心跳流跟踪节点租约和 Region 统计信息； -- 建表时为 Region 选择 Datanode； -- 执行可恢复的 DDL、Region 迁移、故障转移、repartition 等分布式 Procedure； -- 向 Frontend 和 Datanode 发布缓存失效及其他控制消息。 +## 前端如何与 Metasrv 交互 -数据模型、KV 抽象、选举接口、key 编码和 DDL manager 位于 `src/common/meta/`。`src/meta-srv/` crate 实现服务端、状态机、心跳 handler 和控制 Procedure。 - - - -## Frontend 与 Metasrv 的交互 - -Frontend 通过 `meta-client` crate 获取表元数据和 Region 路由，并提交修改元数据的操作。Frontend 在本地缓存元数据；Procedure 修改元数据后，Metasrv 会发送缓存失效消息。 +Frontend 从 Metasrv 获取表元数据和 Region 路由，并缓存在本地。修改元数据的语句会发送给 Metasrv leader；普通读写则使用缓存的路由直接访问 Datanode。 ### 创建表 -1. Frontend 向 Metasrv Leader 提交 DDL 请求。 -2. DDL manager 校验请求，根据分区规则生成 Region，并为 Region 选择 Datanode。 -3. 持久化的 Procedure 创建 Region，随后记录表和路由元数据。Procedure 状态持久化后，可以在服务重启或 Leader 切换后恢复执行。 -4. 元数据提交后，Metasrv 使相关缓存失效。 +1. Frontend 向 Metasrv leader 提交 DDL 请求。 +2. Metasrv 根据分区规则确定 Region，并为每个 Region 选择 Datanode。 +3. 持久化的 Procedure 创建 Region，并写入表元数据和路由。发生 leader 切换后，Procedure 可以从已保存的状态继续执行。 +4. 元数据提交后，Metasrv 通知 Frontend 刷新相关缓存。 ### `Insert` -Frontend 获取表路由，按分区拆分数据行，并把 Region 写请求发送到对应 Datanode。路由元数据保存在本地缓存中；收到缓存失效消息或 stale-route 错误时，Frontend 会从 Metasrv 刷新路由。 +Frontend 解析表路由，按照分区规则拆分数据行，再把各 Region 的写入发送到对应 Datanode。路由发生变化时，相关缓存会失效，Frontend 随后从 Metasrv 重新获取元数据。 ### `Select` -Frontend 在查询规划期间使用表和 Region 元数据。分区谓词用于裁剪 Region，分布式查询引擎再将远端子计划发送到持有这些 Region 的 Datanode。参见[分布式查询](../frontend/distributed-querying.md)。 - - - -## 源码结构 +Frontend 在查询规划期间使用表和 Region 元数据。分区列上的谓词用于裁剪 Region，分布式查询引擎再把任务发送给持有这些 Region 的 Datanode。参见[分布式查询](../frontend/distributed-querying.md)。 -主要实现目录如下： +## Metasrv 架构 -- `src/meta-srv/src/service/`：gRPC 服务和 HTTP Admin API。 -- `src/meta-srv/src/handler/`：心跳 handler chain。 -- `src/meta-srv/src/procedure/`：Region 迁移、repartition、WAL 清理等分布式 Procedure。 -- `src/meta-srv/src/region/`：Region 租约、监控和故障转移触发逻辑。 -- `src/meta-srv/src/selector/`：为 Region 选择 Datanode。 +Metasrv 由几类协调机制组成： - +- 元数据层通过 key-value backend 保存集群状态。 +- Leader 选举保证同一时间只有一个 Metasrv 负责元数据变更和集群管理。 +- Procedure Manager 执行多步骤操作，并持久化恢复执行所需的状态。 +- 心跳处理链更新租约和 Region 统计信息，并传递控制消息。 +- Region 监控根据租约判断 Region 是否不可用，并在需要时启动故障转移。 -## Leader 与持久化 +这些机制共享元数据，但故障边界不同。进程重启可以丢弃缓存和 leader 本地状态；恢复所需的元数据和 Procedure 状态必须持久化。 -Metasrv 通过 `common-meta` 中的接口隔离 Leader 选举与持久化元数据存储。协调操作和元数据修改在 Leader 上执行；非 Leader 节点返回 not-leader 响应，client 随后连接到当前 Leader。 +## 分布式共识 -Leader 切换后仍需保留的数据必须写入 KV backend。进程内缓存和 Leader 本地状态会在切换时重建或清空。分布式 Procedure 会持久化状态，其每个执行步骤必须保持幂等，才能安全恢复。 +Metasrv 将 leader 选举与元数据存储分开。只有选出的 Metasrv leader 执行协调和元数据变更操作，其他 Metasrv 节点会把 client 引导到当前 leader。 - +Key-value backend 保存表元数据、路由、Procedure 状态以及其他必须跨 leader 切换保留的信息。Metasrv 不使用这套选举为 Datanode Region 创建读写副本；Region 可用性由租约、心跳和故障转移 Procedure 管理。 -## 心跳不变量 +## 心跳管理 -Datanode 和 Frontend 与 Metasrv Leader 保持心跳流。请求携带节点身份、租约、Region 统计信息及其他状态。`src/meta-srv/src/handler/` 下的 handler chain 负责检查 Leader、更新租约与统计信息，并处理 mailbox 消息。 +Datanode 与 Metasrv leader 保持心跳流。心跳请求报告节点身份、租约、Region 统计信息以及放置和监控所需的其他状态；响应则携带 Region 生命周期指令、缓存失效等控制消息。 -心跳响应携带 Region 生命周期指令、缓存失效等控制消息。Region supervisor 根据租约状态发现不可用 Region，并触发故障转移 Procedure。修改心跳间隔时，必须同步检查 `common-meta` 和 `meta-srv` 中的租约与 supervisor 时序。 +对 Metasrv 而言，心跳不仅是指标上报，也是租约续期。租约过期会参与故障检测，并可能触发 Region 故障转移。因此，修改心跳周期时必须同时考虑租约和监控周期。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/selector.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/selector.md index ade57eefa5..dbdf68e43f 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/selector.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/selector.md @@ -1,42 +1,45 @@ --- -keywords: [Selector, Metasrv, Datanode, Region 放置, 负载均衡] -description: 介绍 Metasrv 的 Region 放置 Selector 及其配置名称。 +keywords: [Selector, Metasrv, Datanode, 路由表, 负载均衡] +description: 介绍 Metasrv 中的 Selector，包括其类型和配置方法。 --- # Selector ## 介绍 -建表时，Metasrv 需要为各 Region 选择 Datanode。[`Selector` trait](https://github.com/GreptimeTeam/greptimedb/blob/main/src/meta-srv/src/selector.rs) 接收所需 peer 数量和 selection context，再根据当前租约及统计信息返回候选 Datanode。 +建表时，Metasrv 使用 `Selector` 为各 Region 选择 Datanode。Selector 根据当前节点租约进行选择；部分实现还会使用 Region 统计信息。 ## Selector 类型 -Metasrv 提供三种 Selector 实现： +`Metasrv` 目前提供以下几种类型的 `Selectors`: ### LeaseBasedSelector -`LeaseBasedSelector` 从租约有效的 Datanode 中随机选择，不使用 Region 数量为候选节点排序。 +`LeaseBasedSelector` 从租约有效的 Datanode 中随机选择。 ### LoadBasedSelector -`LoadBasedSelector` 使用 Datanode 上的 Region 数量表示负载，优先选择 Region 较少的节点。 +`LoadBasedSelector` 按照负载来选择，负载值则由每个 `Datanode` 上的 region 数量决定，较少的 region 表示较低的负载，`LoadBasedSelector` 优先选择低负载的 `Datanode`。 ### RoundRobinSelector [默认选项] - -`RoundRobinSelector` 依次轮询可用 Datanode，是默认的 Selector。 +`RoundRobinSelector` 以轮询方式选择 Datanode，是默认选项。 ## 配置 -启动 Metasrv 时可以指定 Selector。可用名称如下： +您可以在启动 `Metasrv` 服务时通过名称配置 `Selector`。 -- `lease_based` 或 `LeaseBased` -- `load_based` 或 `LoadBased` -- `round_robin` 或 `RoundRobin` +- LeaseBasedSelector: `lease_based` 或 `LeaseBased` +- LoadBasedSelector: `load_based` 或 `LoadBased` +- RoundRobinSelector: `round_robin` 或 `RoundRobin` 例如： ```shell cargo run -- metasrv start --selector round_robin ``` + +```shell +cargo run -- metasrv start --selector RoundRobin +``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/overview.md index 72d35c86fa..839930989b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/overview.md @@ -1,19 +1,24 @@ --- -keywords: [贡献者指南, 架构, Frontend, Datanode, Metasrv, Flownode] -description: 介绍 GreptimeDB 内部架构及各子系统源码入口的贡献者文档。 +keywords: [架构, 关键概念, 数据处理, 组件交互, 数据库] +description: 介绍 GreptimeDB 的架构、关键概念和工作原理，包括各组件的交互方式和数据处理流程。 --- # 贡献者指南 -本指南介绍 GreptimeDB 的内部架构，并提供各子系统的源码入口。构建、测试及贡献要求以源码仓库的 [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md) 为准。 +本指南面向 GreptimeDB 贡献者，介绍理解内部实现所需的设计机制。构建、测试和提交要求以源码仓库的 [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md) 为准。 ## 架构 -[架构概览](/user-guide/concepts/architecture.md) 从用户视角说明系统组件和请求链路。以下贡献者文档进一步说明各组件的实现边界： +有关 GreptimeDB 的架构和组件，请参阅用户指南中的 [架构](/user-guide/concepts/architecture.md) 文档。 -- [Frontend](./frontend/overview.md)：协议处理、请求编排、路由和分布式查询规划。 -- [Datanode](./datanode/overview.md)：Region 管理、查询执行和存储引擎。 -- [Metasrv](./metasrv/overview.md)：元数据、集群协调和分布式 Procedure。 -- [Flownode](./flownode/overview.md)：单机及分布式部署中的持续聚合。 +有关每个组件的更多详细信息，请参阅以下指南： -本地构建 GreptimeDB 请继续阅读[快速开始](./getting-started.md)。 +- [frontend][1] +- [datanode][2] +- [metasrv][3] +- [flownode][4] + +[1]: /contributor-guide/frontend/overview.md +[2]: /contributor-guide/datanode/overview.md +[3]: /contributor-guide/metasrv/overview.md +[4]: /contributor-guide/flownode/overview.md diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/integration-test.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/integration-test.md index 074296e671..63ffa56bc0 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/integration-test.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/integration-test.md @@ -1,30 +1,14 @@ --- -keywords: [集成测试, Rust test harness, 存储 backend, Kafka, 协议] -description: 运行 tests-integration 中的多组件和外部服务测试。 +keywords: [集成测试, Rust, HTTP, gRPC, 测试工具] +description: 介绍 GreptimeDB 的集成测试，包括测试范围和如何运行这些测试。 --- # 集成测试 ## 介绍 -`tests-integration/` crate 包含需要多个 GreptimeDB 组件或外部服务的 Rust test-harness case。常见场景包括 HTTP 或 gRPC 行为、对象存储 backend、Kafka WAL，以及启用 TLS 的依赖服务。 +集成测试使用 Rust 测试工具（`#[test]`）编写，与单元测试不同，它们被单独放置在 +[这里](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration)。 +它涵盖了涉及多个组件的场景，其中一个典型案例是与 HTTP/gRPC 相关的功能。你可以查看 +其[文档](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md)以获取更多信息。 -如果某项行为无法通过 crate 内单元测试或 sqlness 查询 case 验证，应使用集成测试。协议断言应放在公共接口边界，并复用 `tests-integration/fixtures/` 中的 fixture，不要另建一套环境配置。 - -环境准备和命令以 [`tests-integration/README.md`](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md) 为准。需要凭据或 endpoint 的测试会读取仓库根目录下由 `.env.example` 创建的 `.env` 文件；不要提交凭据。 - -在仓库根目录运行通用集成测试组： - -```shell -cargo test integration -``` - -特定 backend 使用对应的 filter，例如： - -```shell -cargo test s3 -cargo test oss -cargo test azblob -``` - -Kafka 和 TLS case 需要集成测试 README 中记录的 Docker Compose 服务。只启动当前测试需要的依赖，并在测试结束后清理这些服务。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/overview.md index 7fb89ac1fb..81b6defa45 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/overview.md @@ -1,14 +1,9 @@ --- -keywords: [测试, 单元测试, sqlness, 集成测试, 回归] -description: 根据代码改动选择并运行相应的 GreptimeDB 测试套件。 +keywords: [测试] +description: GreptimeDB 的测试 --- # 测试 -GreptimeDB 使用多层测试。首先选择能够覆盖当前改动的最窄测试；如果行为跨组件或通过公共接口暴露，再增加对应层级的回归测试。 +我们的团队进行了大量测试，以确保 GreptimeDB 的行为。本章将介绍几种用于测试 GreptimeDB 的重要方法，以及如何使用它们。 -- [单元测试](./unit-test.md)覆盖 crate 内部逻辑、不变量和错误路径。测试与 Rust 实现放在一起，通过 cargo-nextest 运行。 -- [Sqlness 测试](./sqlness-test.md)覆盖单机或分布式测试环境下用户可见的 SQL 和查询行为。测试输入及预期结果位于 `tests/cases/`。 -- [集成测试](./integration-test.md)覆盖需要多个组件或外部服务的交互，包括存储 backend 和协议级行为。测试位于 `tests-integration/`。 - -仓库还包含 `tests-fuzz/`、`tests/compatibility/` 和 `tests/perf/` 等专项测试。改动涉及输入健壮性、持久化格式兼容性或性能时，应遵循相应目录中的 README 或 `AGENTS.md`。通过某一层测试，不能替代最接近回归可观察位置的测试。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/sqlness-test.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/sqlness-test.md index b9985eda89..af260a14de 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/sqlness-test.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/sqlness-test.md @@ -1,45 +1,47 @@ --- -keywords: [SQL 测试, sqlness, golden file, 单机, 分布式] -description: 为用户可见的查询行为添加并运行 sqlness 回归测试。 +keywords: [Sqlness 测试, SQL, 测试套件, 测试文件, 测试案例] +description: 介绍 GreptimeDB 的 Sqlness 测试，包括测试文件类型、组织测试案例和运行测试的方法。 --- # Sqlness 测试 ## 介绍 -Sqlness 是 GreptimeDB 用于验证 SQL 和查询行为的 golden-file 测试框架。它会构建并启动指定的 GreptimeDB 环境，执行测试文件，再将输出与仓库中的预期结果比较。测试框架及当前参数参见 [`tests/README.md`](https://github.com/GreptimeTeam/greptimedb/blob/main/tests/README.md)。 +SQL 是 `GreptimeDB` 的一个重要用户接口。我们为它提供了一个单独的测试套件（名为 `sqlness`）。 ## Sqlness 手册 ### 测试文件 -每个 case 包含两类文件： +Sqlness 有两种类型的文件 -- `.sql` 保存测试语句和 sqlness directive。 -- `.result` 保存预期语句和输出。 +- `.sql`：测试输入，仅包含 SQL +- `.result`：预期的测试输出，包含 SQL 和其结果 -先修改 `.sql` 输入，再运行 sqlness 并审查生成的 `.result` diff。结果变化可能是预期的新行为，也可能是回归，测试框架无法替你判断。只有逐项确认变更的数据行和错误信息后，才能提交 `.result` 改动。 +`.result` 文件是预期的执行输出。如果 `.result` 文件发生变化，意味着测试结果不同，测试可能失败。你应该检查变更日志来解决问题。 + +你只需要在 `.sql` 文件中编写测试 SQL，然后运行测试。 ### 组织测试案例 -测试位于 `tests/cases/`。第一层目录选择运行环境，例如 `standalone/`；后续目录用于组织相关 case。Sqlness 会递归发现测试文件。 +输入案例的根目录是 `tests/cases`。它包含几个子目录，代表不同的测试模式。例如，`standalone/` 包含所有在 `greptimedb standalone start` 模式下运行的测试。 -回归测试应放在能够观察到该行为的环境中。分布式规划、路由和多节点元数据行为需要分布式 case，即使同一查询在单机模式下也能成功。 +在第一级子目录下（例如 `cases/standalone`），你可以随意组织你的测试案例。Sqlness 会递归地遍历每个文件并运行它们。 ## 运行测试 -仓库为测试框架定义了 cargo alias： +与其他测试不同，这个测试工具是以二进制目标形式存在的。你可以用以下命令运行它 ```shell cargo sqlness bare ``` -该命令会构建 GreptimeDB、启动测试环境、执行 case，并更新或比较 `.result` 文件。需要同时检查命令结果和 `git diff`。 +它会自动完成以下步骤：编译 `GreptimeDB`、启动测试环境、执行测试，再收集和比较结果。确认没有非预期的 `.result` 变化后，测试才算通过。 ### 运行特定测试 ```shell -cargo sqlness bare -t 'standalone:your_case' +cargo sqlness bare -t your_test ``` -`-t`/`--test-filter` 接收正则表达式，并匹配 `env:case` 格式的 case 名称。开发时可以使用窄过滤器，提交前仍应运行受影响环境或完整测试套件。 +`-t` 或 `--test-filter` 选项接受正则表达式字符串。Sqlness 会检查格式为 `env:case` 的案例名称。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/unit-test.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/unit-test.md index e56d0cafb6..79c73775b4 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/unit-test.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/unit-test.md @@ -1,34 +1,28 @@ --- -keywords: [单元测试, Rust, cargo-nextest, crate 测试, 覆盖率] -description: 使用 cargo-nextest 编写并运行 crate 内的 Rust 测试。 +keywords: [单元测试, Rust, nextest, 测试覆盖率, CI] +description: 介绍 GreptimeDB 的单元测试，包括如何编写、运行和检查测试覆盖率。 --- # 单元测试 ## 介绍 -Rust 单元测试通常位于被测模块内或相邻的 `*_test.rs` 文件中。适合覆盖 crate 内部不变量、边界条件、错误处理，以及不需要启动 GreptimeDB 集群的行为。 +单元测试嵌入在代码库中，通常放置在被测试逻辑的旁边。它们使用 Rust 的 `#[test]` 属性编写，并可以使用 `cargo nextest run` 运行。 -GreptimeDB 的标准测试运行器是 [cargo-nextest](https://nexte.st/)。安装命令如下： +GreptimeDB 代码库不支持默认的 `cargo` 测试运行器。推荐使用 [`nextest`](https://nexte.st/)。你可以通过以下命令安装它： ```shell cargo install cargo-nextest --locked ``` -开发期间先运行受影响 crate 的测试： +然后运行测试（这里 `--workspace` 不是必须的） ```shell -cargo nextest run -p +cargo nextest run ``` -可能影响多个 crate 的改动，在提交前应运行 CI 对应的 workspace 配置： - -```shell -cargo nextest run --workspace --features pg_kvbackend,mysql_kvbackend -``` - -Feature-gated 代码需要在测试命令中启用对应 feature。不要假设默认 feature set 已覆盖相关路径，应检查 crate 的 `Cargo.toml`、本地 `AGENTS.md` 和 CI workflow。 +注意，如果你的 Rust 是通过 `rustup` 安装的，请确保使用 `cargo` 安装 `nextest`，而不是像 `homebrew` 这样的包管理器，否则会弄乱你的本地环境。 ## 覆盖率 -CI 会记录 Rust 测试覆盖率。测试应保护实际改动的行为和可信的失败场景，不要只为提高百分比增加断言。查询语言行为和跨组件流程通常还需要 sqlness 或集成测试。 +我们的持续集成（CI）作业有一个“覆盖率检查”步骤。它会报告有多少代码被单元测试覆盖。请在你的补丁中添加必要的单元测试。 From ff6e6ae959eaf03f9608c2b3077fa040b80c9b85 Mon Sep 17 00:00:00 2001 From: Dennis Zhuang Date: Tue, 25 Aug 2026 11:59:23 +0800 Subject: [PATCH 03/14] docs: clarify contributor architecture guides --- .../datanode/data-persistence-indexing.md | 12 ++++--- .../flownode/batching_mode.md | 2 +- docs/contributor-guide/flownode/dataflow.md | 31 ++++++++++++++---- docs/contributor-guide/metasrv/overview.md | 26 +++++++++++++++ docs/contributor-guide/metasrv/selector.md | 2 +- docs/contributor-guide/overview.md | 4 +++ .../datanode/data-persistence-indexing.md | 12 ++++--- .../flownode/batching_mode.md | 2 +- .../contributor-guide/flownode/dataflow.md | 31 ++++++++++++++---- .../contributor-guide/metasrv/overview.md | 26 +++++++++++++++ .../contributor-guide/metasrv/selector.md | 2 +- .../current/contributor-guide/overview.md | 4 +++ static/parquet-file-layout.gif | Bin 0 -> 43589 bytes 13 files changed, 130 insertions(+), 24 deletions(-) create mode 100644 static/parquet-file-layout.gif diff --git a/docs/contributor-guide/datanode/data-persistence-indexing.md b/docs/contributor-guide/datanode/data-persistence-indexing.md index 86ddb7b667..bcf055fbad 100644 --- a/docs/contributor-guide/datanode/data-persistence-indexing.md +++ b/docs/contributor-guide/datanode/data-persistence-indexing.md @@ -5,19 +5,23 @@ description: Explanation of data persistence and indexing in GreptimeDB, includi # Data Persistence and Indexing -Similar to all LSMT-like storage engines, data in MemTables is persisted to durable storage, for example, the local disk file system or object storage service. GreptimeDB adopts [Apache Parquet][1] as its persistent file format. +Like other LSM-tree storage engines, GreptimeDB persists data from memtables to durable storage such as a local filesystem or object storage. It uses [Apache Parquet][1] as the persistent file format. ## SST File Format Parquet is an open source columnar format that provides fast data querying and has already been adopted by many projects, such as Delta Lake. -Parquet organizes data as row groups, column chunks, and pages. A row group contains one column chunk for each column, and each column chunk contains one or more pages. Pages are the smallest encoded I/O units within a column chunk. +Parquet organizes data as row groups, column chunks, and pages. A row group contains one column chunk for each column, and each column chunk contains one or more pages. Pages are the units of encoding and compression; column chunks are the I/O units for reading selected columns. First, clustering data by column makes file scanning more efficient, especially when only a few columns are queried, which is very common in analytical systems. -Second, data of the same column tends to be homogeneous which helps with compression when apply techniques like dictionary and Run-Length Encoding (RLE). +Second, values within a column tend to be similar, which improves compression with techniques such as dictionary encoding and run-length encoding (RLE). -Parquet file format +The following diagram from the Apache Parquet specification also shows the physical file layout: column chunks are stored by row group, while file metadata and its length are written in the footer. + +Apache Parquet file layout + +*Source: [Apache Parquet file-format specification](https://parquet.apache.org/docs/file-format/).* ## Data Persistence diff --git a/docs/contributor-guide/flownode/batching_mode.md b/docs/contributor-guide/flownode/batching_mode.md index 37a695ce9c..43b00ff4bd 100644 --- a/docs/contributor-guide/flownode/batching_mode.md +++ b/docs/contributor-guide/flownode/batching_mode.md @@ -9,7 +9,7 @@ This guide provides a brief overview of the batching mode in `flownode`. It's in ## Overview -The batching mode in `flownode` is designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. This is in contrast to the original streaming mode, now deprecated, where data was processed as it arrived. +The batching mode in `flownode` is designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. This is in contrast to the legacy streaming path, which processes data as it arrives and is retained for compatibility but deprecated for new workloads. The core idea is to: 1. Define a `flow` with a SQL query that aggregates data from a source table into a sink table. diff --git a/docs/contributor-guide/flownode/dataflow.md b/docs/contributor-guide/flownode/dataflow.md index 090ac870e9..e335d152a9 100644 --- a/docs/contributor-guide/flownode/dataflow.md +++ b/docs/contributor-guide/flownode/dataflow.md @@ -1,19 +1,38 @@ --- -keywords: [dataflow module, SQL query transformation, execution plan, DAG, map and reduce operations] -description: Explanation of the dataflow module in Flownode, its operations, internal data handling, and future enhancements. +keywords: [Flownode, batching mode, streaming mode, dataflow, dirty time windows] +description: How Flownode selects and runs its batching and legacy streaming execution paths. --- # Dataflow -This page describes the compute graph used by Flownode's legacy streaming mode. New continuous-aggregation work uses [batching mode](./batching_mode.md). +Flownode has two internal execution paths: + +- **Batching mode** is the primary path for aggregation and TQL workloads. It evaluates queries over persisted source data and writes materialized results to a sink table. +- **Streaming mode** is the legacy path retained for compatibility and deprecated for new workloads. It incrementally processes rows mirrored from Frontend as they arrive. + +Users do not select the mode directly. When a Flow is created, GreptimeDB chooses the path from the query and source-table properties. Aggregation, `DISTINCT`, and TQL queries use batching mode. Simple non-aggregation queries, and any Flow whose source table has `ttl = 'instant'`, currently use streaming mode. A Flow deferred because its source table does not yet exist starts as a pending batching Flow. + +## Batching mode + +Batching mode reuses GreptimeDB's query engine instead of maintaining an operator graph for every incoming row. For a time-windowed Flow, its main loop is: + +1. A source-table write marks the affected time windows as dirty. +2. A `BatchingTask` wakes on its schedule or after a notification and collects pending dirty windows. +3. The task adds time predicates for those windows to the Flow query and asks Frontend to execute it against the source tables. +4. The query result is inserted into the sink table, updating the materialized result for windows that were evaluated. +5. Successfully processed windows are removed from the dirty set. Failed work remains available for a later evaluation. + +Flows with an evaluation interval but without a time-window expression run the complete query on each scheduled evaluation. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Flownode Batching Mode Developer Guide](./batching_mode.md) for the task and dirty-window components. + +## Streaming mode The `dataflow` module (see `flow::compute` module) is the core computing module of `flow`. It takes a SQL query and transforms it into flow's internal execution plan. This execution plan is then rendered into an actual dataflow, which is essentially a directed acyclic graph (DAG) of functions with input and output ports. -The dataflow is triggered to run when needed. +New row changes drive the graph incrementally. -Currently, this dataflow only supports `map` and `reduce` operations. Support for `join` operations will be added in the future. +The renderer supports map/filter/project and reduce operations. Join and union plan nodes exist, but their streaming renderers are not implemented. Internally, the dataflow handles data in row format, using a tuple `(row, time, diff)`. Here, `row` represents the actual data being passed, which may contain multiple `Value` objects. `time` is the system time which tracks the progress of the dataflow, and `diff` typically represents the insertion or deletion of the row (+1 or -1). -Therefore, the tuple represents the insert/delete operation of the `row` at a given system `time`. +Therefore, the tuple represents the insert/delete operation of the `row` at a given system `time`. Stateful operators keep indexed traces of these changes in an [Arrangement](./arrangement.md). diff --git a/docs/contributor-guide/metasrv/overview.md b/docs/contributor-guide/metasrv/overview.md index b345204d0f..91a85ef219 100644 --- a/docs/contributor-guide/metasrv/overview.md +++ b/docs/contributor-guide/metasrv/overview.md @@ -20,6 +20,32 @@ Metasrv is the metadata and coordination service in a distributed GreptimeDB clu Frontend obtains table metadata and Region routes from Metasrv and caches them locally. Metadata-changing statements are sent to the Metasrv leader, while reads and writes use the cached routes to reach Datanodes directly. +The control and data paths are separate: + +```text +Frontend + |-- metadata lookup and DDL ------------> Metasrv leader + `-- Region reads and writes ------------> Datanode + +Metasrv leader + |-- Region lifecycle procedures --------> Datanode + `-- cache and Region-state notifications -> Frontend / Datanode + +Datanode + `-- heartbeat, lease renewal, Region stats -> Metasrv leader +``` + +A table route maps each Region to its current Datanode peer. It does not contain a separate list of read replicas: + +```text +Table route + |-- Region 0 -> Datanode A + |-- Region 1 -> Datanode B + `-- Region 2 -> Datanode C +``` + +Region migration or failover changes this mapping. Frontend refreshes its cached route before sending subsequent reads or writes to the new peer. + ### Create Table 1. Frontend submits the DDL request to the Metasrv leader. diff --git a/docs/contributor-guide/metasrv/selector.md b/docs/contributor-guide/metasrv/selector.md index 1f00c2cd7a..23190cd6a9 100644 --- a/docs/contributor-guide/metasrv/selector.md +++ b/docs/contributor-guide/metasrv/selector.md @@ -22,7 +22,7 @@ The `Metasrv` service currently offers the following types of `Selectors`: The `LoadBasedSelector` load value is determined by the number of regions on each `Datanode`, fewer regions indicate lower load, and `LoadBasedSelector` prioritizes selecting low-load `Datanodes`. ### RoundRobinSelector [default] -`RoundRobinSelector` selects `Datanode`s in a round-robin fashion. It is the default option. +`RoundRobinSelector` selects `Datanode`s in a round-robin fashion. It is the default and recommended choice for most deployments. ## Configuration diff --git a/docs/contributor-guide/overview.md b/docs/contributor-guide/overview.md index cb9bcd7dea..944c541019 100644 --- a/docs/contributor-guide/overview.md +++ b/docs/contributor-guide/overview.md @@ -22,3 +22,7 @@ For more details on each component, see the following guides: [2]: /contributor-guide/datanode/overview.md [3]: /contributor-guide/metasrv/overview.md [4]: /contributor-guide/flownode/overview.md + +## Additional reference + +[DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb) provides an automatically generated walkthrough of the GreptimeDB repository. It can help when exploring an unfamiliar area, but it is a secondary reference: verify version-sensitive behavior against the source code. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md index 2c6ee25bde..21ac66cf5a 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md @@ -5,19 +5,23 @@ description: 介绍了 GreptimeDB 的数据持久化和索引机制，包括 SST # 数据持久化与索引 -与所有类似 LSMT 的存储引擎一样，MemTables 中的数据被持久化到耐久性存储，例如本地磁盘文件系统或对象存储服务。GreptimeDB 采用 [Apache Parquet][1] 作为其持久文件格式。 +与其他 LSM-tree 存储引擎类似，GreptimeDB 将 memtable 中的数据持久化到本地文件系统或对象存储，并使用 [Apache Parquet][1] 作为持久化文件格式。 ## SST 文件格式 Parquet 是一种提供快速数据查询的开源列式存储格式，已经被许多项目采用，例如 Delta Lake。 -Parquet 按 row group、column chunk 和 page 组织数据。每个 row group 为每一列保存一个 column chunk，每个 column chunk 再包含一个或多个 page。Page 是 column chunk 内最小的编码 I/O 单元。 +Parquet 按 row group、column chunk 和 page 组织数据。每个 row group 为每一列保存一个 column chunk，每个 column chunk 再包含一个或多个 page。Page 是编码和压缩单元，读取指定列时则以 column chunk 为 I/O 单元。 首先，数据按列聚集，这使得文件扫描更加高效，特别是当查询只涉及少数列时，这在分析系统中非常常见。 -其次，相同列的数据往往是同质的（比如具备近似的值），这有助于在采用字典和 Run-Length Encoding（RLE）等技术进行压缩。 +其次，同一列中的值通常比较相似，有利于字典编码和 Run-Length Encoding（RLE）等压缩技术发挥作用。 -Parquet file format +下面这张来自 Apache Parquet 规范的图进一步展示了物理文件布局：column chunk 按 row group 写入，文件元数据及其长度则保存在 footer 中。 + +Apache Parquet 文件布局 + +*来源：[Apache Parquet 文件格式规范](https://parquet.apache.org/docs/file-format/)。* ## 数据持久化 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md index bec092b0af..0cf9c2edd0 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md @@ -9,7 +9,7 @@ description: Flownode 批处理模式概述，这是持续数据聚合当前使 ## 概述 -`flownode` 中的批处理模式专为持续数据聚合而设计。它在离散的、微小的时间窗口上周期性地执行用户定义的 SQL 查询。这与原始的流处理模式形成对比；流处理模式现在已经废弃，在该模式下数据会在到达时即被处理。 +`flownode` 中的批处理模式专为持续数据聚合而设计。它在离散的小时间窗口上周期性执行用户定义的 SQL 查询。旧 streaming 路径则在数据到达时进行处理，目前仅为兼容已有 workload 而保留，不推荐新 workload 使用。 其核心思想是： 1. 定义一个带有 SQL 查询的 `flow`，该查询将数据从源表聚合到目标表。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md index 24f9c04146..7d5ad54780 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md @@ -1,20 +1,39 @@ --- -keywords: [Dataflow, SQL 查询, 执行计划, 数据流, map, reduce] -description: 解释了 Dataflow 模块的核心计算功能，包括 SQL 查询转换、内部执行计划、数据流的触发运行和支持的操作。 +keywords: [Flownode, batching mode, streaming mode, Dataflow, 脏时间窗口] +description: 介绍 Flownode 如何选择并运行 batching 和旧 streaming 两条执行路径。 --- # 数据流 -本页介绍 Flownode 旧 streaming 模式使用的计算图。新的持续聚合功能使用[批处理模式](./batching_mode.md)。 +Flownode 内部有两条执行路径： + +- **Batching mode** 是聚合和 TQL workload 的主要执行路径。它查询已经持久化的 source 数据，并将物化结果写入 sink table。 +- **Streaming mode** 是为兼容已有 workload 而保留的旧执行路径，不推荐新 workload 使用。Frontend 会把新到达的行同步给它进行增量处理。 + +用户不能直接选择执行模式。创建 Flow 时，GreptimeDB 根据查询和 source table 的属性选择执行路径。聚合、`DISTINCT` 和 TQL 查询使用 batching mode；简单的非聚合查询，以及任何 source table 使用 `ttl = 'instant'` 的 Flow，目前仍使用 streaming mode。如果 source table 尚不存在并选择延迟创建，Flow 会先成为 pending batching Flow。 + +## Batching mode + +Batching mode 复用 GreptimeDB 的查询引擎，不需要为每一行输入维护一张算子图。对于基于时间窗口的 Flow，主循环如下： + +1. Source table 收到写入后，把受影响的时间窗口标记为 dirty。 +2. `BatchingTask` 按调度周期或通知唤醒，并收集待处理的 dirty window。 +3. 任务把这些窗口转换成时间谓词，加入 Flow 查询，再请求 Frontend 查询 source table。 +4. 查询结果写入 sink table，更新已重新计算窗口对应的物化结果。 +5. 成功处理的窗口从 dirty set 中移除；执行失败的工作仍可在后续调度中处理。 + +设置了 evaluation interval、但查询中没有时间窗口表达式的 Flow，会在每次调度时执行完整查询。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见 [Flownode 批处理模式开发者指南](./batching_mode.md)。 + +## Streaming mode Dataflow 模块（参见 `flow::compute` 模块）是 `flow` 的核心计算模块。 它接收 SQL 查询并将其转换为 `flow` 的内部执行计划。 然后，该执行计划被转化为实际的数据流，而数据流本质上是一个由带有输入和输出端口的函数组成的有向无环图（DAG）。 -数据流会在需要时被触发运行。 +新到达的行变更会增量驱动这张图执行。 -目前该数据流只支持 `map`和 `reduce` 操作，未来将添加对 `join` 等操作的支持。 +Renderer 支持 map/filter/project 和 reduce 操作。执行计划中已经有 join 和 union 节点，但 streaming renderer 尚未实现它们。 在内部，数据流使用 `tuple(row, time, diff)` 以行格式处理数据。 这里 `row` 表示实际传递的数据，可能包含多个 `value` 对象。 `time` 是系统时间，用于跟踪数据流的进度，`diff` 通常表示行的插入或删除（+1 或 -1）。 -因此，`tuple` 表示给定系统时间的 `row` 的插入/删除操作。 +因此，`tuple` 表示给定系统时间的 `row` 的插入/删除操作。有状态算子通过 [Arrangement](./arrangement.md) 保存这些变更的索引 trace。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md index dccf818e83..96ad0e69ad 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md @@ -20,6 +20,32 @@ Metasrv 是 GreptimeDB 分布式集群中的元数据和协调服务，不参与 Frontend 从 Metasrv 获取表元数据和 Region 路由，并缓存在本地。修改元数据的语句会发送给 Metasrv leader；普通读写则使用缓存的路由直接访问 Datanode。 +控制链路和数据链路相互分离： + +```text +Frontend + |-- 元数据查询和 DDL -------------------> Metasrv leader + `-- Region 读写 ------------------------> Datanode + +Metasrv leader + |-- Region 生命周期 Procedure ----------> Datanode + `-- 缓存和 Region 状态通知 -------------> Frontend / Datanode + +Datanode + `-- 心跳、租约续期和 Region 统计信息 ----> Metasrv leader +``` + +表路由把每个 Region 映射到当前 Datanode peer，其中没有单独的只读副本列表： + +```text +Table route + |-- Region 0 -> Datanode A + |-- Region 1 -> Datanode B + `-- Region 2 -> Datanode C +``` + +Region 迁移或故障转移会修改这项映射。Frontend 刷新缓存路由后，再把后续读写发送给新的 peer。 + ### 创建表 1. Frontend 向 Metasrv leader 提交 DDL 请求。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/selector.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/selector.md index dbdf68e43f..29b521212b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/selector.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/selector.md @@ -24,7 +24,7 @@ description: 介绍 Metasrv 中的 Selector，包括其类型和配置方法。 `LoadBasedSelector` 按照负载来选择，负载值则由每个 `Datanode` 上的 region 数量决定，较少的 region 表示较低的负载，`LoadBasedSelector` 优先选择低负载的 `Datanode`。 ### RoundRobinSelector [默认选项] -`RoundRobinSelector` 以轮询方式选择 Datanode，是默认选项。 +`RoundRobinSelector` 以轮询方式选择 Datanode，是默认选项，也适用于大多数部署。 ## 配置 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/overview.md index 839930989b..71c62609ee 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/overview.md @@ -22,3 +22,7 @@ description: 介绍 GreptimeDB 的架构、关键概念和工作原理，包括 [2]: /contributor-guide/datanode/overview.md [3]: /contributor-guide/metasrv/overview.md [4]: /contributor-guide/flownode/overview.md + +## 补充参考 + +[DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb) 提供了自动生成的 GreptimeDB 源码导读，可用于了解不熟悉的模块。它属于辅助资料；涉及具体版本的行为时，仍应以对应源码为准。 diff --git a/static/parquet-file-layout.gif b/static/parquet-file-layout.gif new file mode 100644 index 0000000000000000000000000000000000000000..b54641d175c47232e2d10ea571ec69560df14961 GIT binary patch literal 43589 zcmaHyS5y;S{Aed_AkptD{Fi-PzX6DRVd+(pVp`NOmlMVDGWC!?f!RY7_j7*G-jC8Ch zBs&KuCnqB}4?8yxGao+(A0IzIKkFGZN)XK_h-MQO;XnJ|;t-SI6_Y@VONdBHosyE~ zmy-UsxMk#c&&x~7%JIl6p%s;76qOVe75SCbgf6I^R=apsO;b@#T|-0Tj3x$sNn7nQ zMo8<*MQt4k9X%Z#owHXACG-qM4NWBtO?3?ojg5`POt2DH&5f^NCCw~N&CJZr&ClJi zv#_$3v39Vuwvn^BX=i75-riZx(N!Mj?uc`8adA;}^>*{{RQB}u@bpyi4)pf+R`m~7 z!-oY01*zYPxOn@)rHIHockW>B$7vDbwWE`xqoc3H6ZI0)larGTQgaL+<{4)eW@cuZ z=8~`Gmyt*$Y+)s~sQP+oeOXzVMP-v^ZChPkopnQ}bz`?}b8l;Ft9?7AtEI|Ni}-4gUXKgnq)201yI5|98`W&jg@5A%sLl$HWH5 zCnP2%r=+GmB&KI%W@YE(=H-(L3Lg~}lS@j=$}1|fV{2+_su~&}H#N7kwzYS3c6C3j z>#OYUgGvIg7NULB&+A+>7S=GSAt93qC;y8xyy7XNob=&{Una>bWw5OsOkU zRW;8mZmtEvgdia58~H*Y-S>jN`AfF=jTlZ#&#hP<_iCZIk&v#~0Q9fQ^@Q$KcVWMl3i<*c z>6Xg1WVs?S7q7BKo;45Eh>q2?0*9ULhnWb;H+I;c9wK(vN7KQRQ>r)|4ld0EK?Is) z0n30+eR|GW#r#wHG5m3oJ9(}-UOV}ol{Gsg*MU>|>G~Pp3b{?1nke?*U4jm~m33Dv z>q39*&?(QriR%^~_mG4bUzLNsGNPW;emQ^7Pq7DqrCD$Itd3|yk@zdvC|2_{B!ZFl z6HOPLGTr7~in}iLuD)%_yRJOPktAvGEK$gT9LP(EWYWPl?{Fyq=;+4eE>d-A%1f#D zZ3~t@@9Y9NWA__TkgidbDB;%l~plWd%icfB`Kj*w}GwBE|7UB|AZUFq<b?|kAA-B^e#K+3^1aXY)d>ra?7-0zl&VCRs02`CvS}~<(MGe~!dl>m zov8hTtxpq=O@Dn+*MIiwL&}{;-$4V4LSR79OBeVNS@9Cr*YoE^(F+!Gp9)2P5^#aM zl;`Z0GFw~dTP3ADfSg>{Zy|{Xnd^eHzdQu?wVZ%4z9$g2P`%9Qu#M>q@Lw21d|I>7u6HA*l=rmWRt#x7I(v;!hIX&RIRey?G=tC0to~r zaGB##)({;{p1`WUd*$tnbc&JGi?~LLmP$m>3Bg~JT_86H&buccUAks2t{BtZrAt8i zB;-D>bRj!ISZ|t1uh8Aa#40F#&h;{gnncAH8Xd4^x*V4)x|oO3Zc`}yW)@3 z*gXfBVb9BpsuMy?4UpFgpjWJg>O}SF4l;5FEUI+*_siLb&5JCy`4yCFo~5l3i+Liq z-{w)v36_~;#`aU3!(NXABWeq_orgB-Rp&_8IRUP3+L|_U*HV{S!v(%oW3`ZkHk%9~ z;Rh}Ue*Pg%qW09hjT5ByxH&4*?U^coUqHLpw z%}Hz5#@^nav?j*LP&w;v`-z0StT9hH6Y?N{S=uJ~*OmL4yUaI#qfH(^SrRHVZsN;0 zE2EeIN!a<;aR8V(t}>5? z+G}T-8t0=NLFjnSwkcwUB(vo(@#kcev+;cD2E<7SBUu|)vE*eLC@0vMm zB+!{`Livv<^jB#?gUN+0W7;g~S&$RX`OT+(TF|=zv}{&9RMW~PURR!+mCEPoNfa=-gmv!3ZQ0~qOHJE$tzlS zjuLhHp&WOyZu-b;B=g44*{4kdWb8hwX8iR@Nw32(a#EHiOtl~wQwJ6NuWnNm1vXoF zaHG?o z4#6kK&%%Y)0mmtRG_Fvt;Ewz*Nu^m8Y(ic1xN>5LOs?=!#u1itxGJnU2aTdRsWR5~^%Edv0 zBLR=1p{dMd5XghD=#EDM8!S*|0>FGRfaX1_v8zT1+2iSfT4Wo>ifoeMrVp;W<9dMn z#*-Js3UCbb^2_ovLj%5+2uUiG{2iu70L>_Xn>b)jMz9hAJlY#i zgs9=c%V@~d+b}0Iu(^y!$bl1A09PskAi^AI0H-ISKrH$)3KYjdlyXl1YZSk{B*xQK~GC}%YSNq~DadEserJ_?A&!F_QE_3`LSM}RE`l-vx&qX7qi z&H)3Px&pe8Yl zcBsn=G>Ea@#SUGBGr`$W4lXPrk$6kslZ$wePKXu7^AC5@{?`YiLr{PKbLxO<&piP5$afGp;=Br#}y5voQ8y-^X;5)gK|97T_K|0Z`?LgwYYQ1zW#^&v#v zIXyl{O#&UX=m$@MC#j3m{7=0+cJH(sr%d@m7vROt;Cw_xqnNaF;@r$q8;=G?^5OeDEM0pj{HG$J$i0~j3*P{Bbw4d6&TphSu1qfbQ={6tCiH~^^q4hNcUBPpf*`=ypR2$BdO z$Q8bu0lr~{({7~xrLs16sObcxg!|U#JbbZX?GZhl3JMU|j4kC(HOB(fkcx?l*r^Ic z!A?af9iSss>7-YDnVZprkIu_BZ*_#2dXxa_NM~4At@Shv23O@tWHoA#*v|-_&nJ8sDSagx{b6N9rvLA)o-! z;p)E=E6*XWs^v>}lm0@JE<|IVK`CCc`Oy4)YGSjtMw@0y^V6YbiE?@o!vv*X`!y5|eG>fC!w`_cP)?sgVydYV*C*tmh+O;G<;J zTV*oU9(t$qwnpV^t74oTgU2t>#ZK0j;eiQ4l+xPaSJ1_qd~(eZ)%jkbvp}*XT%$Y7 zu6u2?K2puYxxk=9yCH4dV&wXTFO^;8c0FI`y8vo)&^zugiIbZ}1dwZUI8_;DNgc}U9o1FW8j^Ri{d$f5_kjMdB8+>WO zyYx$AEUJsnet2T4>)rYQ^C9Xxe2C+Cxbyl@39S=LI2jU;Mqiv8N>I8`>N9-lc*Htd zKysRrwL?kZ8Imh}k|sQ|cUD$e0CT(S>S_BvjQuD^da$yLB4z(XnrYCw{t0eF!sJlk zs_zJ+%cy7J*kH`4&T-d=^-?>-F{z{DK2um1M9W-s7TKNeLu)>x+u z5aA^weF45Gnd8yh$KynSiHy5~E9V#=Ou+#fWb~7owA^@+{b=kV;%d~m?ncPy4bD$K z9-B?s)(4l(fKfZQs*pg)_c)re_!!AJm-;amT{eG-o&XX37u8&P|UmX->b{ zm<^en$^Y&5s&HcKaQ3@>{`tECefE=zPMkH*{>Yu49 z&m!Wg+2Eb>U?%$!x{-d$(Fiu&Bo z=Xp+{L;&H!>Hg;yFBX-*J$FPc2;Z8QcH%^OEd+vqLBoO%k?5s^V3Y$y(V#O5VkrXC zcm(p`t-Lm2Z!+Lb7EoL(!kH$?CbwuuJh_njeBu%jUKI_K&UnWS+Ppd&M6n;#A+HzBG~~XY$?rd3F}}L^;)UP6HOwEX z%f$tD2TpRs{G<%G;(JN8bfOzdw9*B5_+;_i=@$_#AR`g(c|}Q0OMC1QhZS+7Fr6cy zl>^pTCc|)iE=7*%LURuv3)Y$F&LQS`}>?*hR?O*B!zO3$__dPx6Opy1AR#0P8aoa=$ z&2Z!ottFh@@!6cW`LN_ggFyY4hfH71_b)p(0dvf|S(3Z%NsA%=qM3aM^)C;I)4NwM zt-YDq+F^d_)xV^^bwSoIDKT-=hoWJ85iorDQc~xW9wtS?%yMpPYq1wy?Z? zjefhA>SjWKV9_RYH_e0FX*mwPe0~Ymr>~)8KS@0lBJS+kn=G_{*!un2^Dt&vKF!Kp z__OvpO-xo+V(>HO+2^J0&y~JxJeT*>S?4t$42nON);jshIsf^~-_L|IbW!&gBt9J2 z4t}%0DQI(ZV&(mps~_7PZhm)v@cnkAfS}=53BQlrX&)s9ID;<#2)_9v^udqY#Xlkj zf82fbn6SCr^I0vwmkuTkN@((0?*ld6KxM@XoC9m%0D()T=)) zPyTEcbIx2o`gBO^evmYO`45-FA20kLt_RF|b9!}y92*Zf1E_zuicfwGo@8JCdxweM z*(WeC=vMImVYW%Pf?@&`09D64N5JR*VYXG(HMMp1|ClWq$j^fTR$eiI@kt^51OJ%q zKzASsY|R4-P96bI0^Qgc*#N`YZa2oI{|#+7w_b1m2W?+X`Am+JfT)m@E7$!4l@R z_7g1?#sLQ_LgwIO>XQi7cQ2B^h`kyx`9qfW9DSoR@*lKyAQ>&y+uGsTdJLbI-oWud z1PWFkpDhs*O-NiFZhby0@e*>iG!IrA%hwCuE&t?b$h)rR|inZePlKB2Uv|V!1~(;-e6Eid%&)W$9!7 zLEG?=BxRK}edU%rx|#QEKfp(Utl!xA1YS%<3R9UO`Rypwq+A~E;Kn2u!=peS6Yb@= zAhs35c|N#+8*5m>7%*UiD2y0+`#JLFZ28tA-5<9v0zz}97xH*c|CEbKj^V+^S;u5; zErx4;-#e?iA9y2E^W!OGu}EOOkRUb}BDZYsc7$8tlhYo7iyQLX;Phd`m%8w&n%Hdp z$80BKc|VL9$8*2UV6zfp7mj9nQr;ZPz0DQ{K)v1y$Jq!T z9&MKqfANU8=yZ6vUe#zYg919!Pm%6L;{d;S3lnOK&my?{Z3wLG7YmDOp5(jwmeTx! zIE&_0v1(u#_5g2w-?@Z1TJ}Qa7~X! zY8jkWo6@!RVWRkwSD6@Ogc2g)6#hW0nM?;dJ|(SV0&iU2nI1V+Q}s7%9qAR-)iKuX zRd0vl6re`pglZo9^P#hrZU&x8Dd{y=3xsh%H5Je_Z@=$G(_dZO_?!6{AkEeyf1~n88-= zKH4DL1GSpTc-i~q(<6U0ug;l|eExRW`3(F0ukg%0S*RV$?x*eD$0ts2%RU_cOg6a5 zzO*|40DL%~ALyMJ5DPi(3&_M?28c^(>A8WQCe*Zj(3_M!-8)xt&m`LR1jwm>}nxLIt zO*Pk+F6FFRNp0Mu#+`e1OSieQD2<<+3b7twFe zGU?&)1Mm0 zeHS-Xb2FYvpDV8Ft5EN}VNloyVQ?R!_~6knm@b*Aj$=ZzRZ<-3=yZhb5?bMqdH!)wW$2Vy&|9OFJ#=uQ2 zP3W4wD{Xmz=aYcBTU4N4$2D2pV)0>c1!i>r#x^8S-wsumNY6&5=cy7OI5neq2?waN zxyZ(>JhNRTUF{spVGHVlg2J1*5q)E4KEr{VhJ1AM>M#aTHecQZAA7MTjCIiPkg*0A zAusO^qXiZ>xNsOCW;T-(c?h&Swk?g|B<56iE%|r&ep6lWO)p0uk zuBMe^-%z}OOH?>A7~!=cf#kU-bxmE0N848jdM&#tEIgT;>rj+AK#tM&HF}GuSkz3M zOTI|%xcy=In_8QD=T&7SjZNc1*qu%yz_7a!Wy?o&zy$?LwlGtDo2Zw|6vEZ60!h_P zQC9Lvm=-&eUW4%LeEh;~=+u%XPnKTz0UBdc4jYmdmz(-?!YMhZ>MmDQ%gb)GmoNXrt`rOi`E01I0d^vL%D z`wG8wR`1vJ?!L@jJhh=>5Vvl80gvj@JF%AIr!tKD-NDs|&_>#zetq(Ez2cY9cc8AC zpT~KXkk-!dd8RLp5Lr9?rxRm-##pXg_`0T(6*;lFsp^x8bcX4Nkw=klq36D}sLZoH zmFthK&G^K8N?ebff$?**ZTIZ8skmx-t=KkEBmKZwUtVp6HF~a`;lLFcA*`<6BlB#A z#)-L8dVk-$o-1sh&Kvc4rT(VzDt{;A?dXeI?=LX8kWhlBqb$o+{JCuM*WR2d>t4>+ z)4cK2ZadPk@7eT=Q@g`6wz(vTij$*TST4hOs;!q}t##2R`9Rc5+Wke8GZUt-$wZ>h zjoG#=`GnIf>UGys4d}b=D6e*y^+=M~-Lgz+)$VN@i;4m^|D-8N&F`GjMM0mc-0K~$ znMhoGCx2P7y#-x6_C(;lEBitR7~7+K7q`(Rk&BvY{4J6&gwuYbb{OP)KKHGB z{Kkh1{w9Q|wJXl?9G7*<7x&Z&uxCYqnd`oJZb# z5|L`+AiV_paE7lSMtOJB*JSz5rzcG;U1-=%x_eGa zp`5Ymb8djw7tH_~WIb2p#gBVjs+O;7EZtiGKSJ4f-`&>i8kH2!TFo=ATesZEs8>CDJEt~-+o$f z6ShjY85Wor4F%@VD*Cb3mn1^x!~A2gf>&-|x+Mn~s{TIHykDTM{=WgxyM6d zuA5t106uoOq-uI@K*%Q(tHOqozM~TpfJGBk+nRZ$1)jbe zcs6`b`eg3H8GRM^XcZS1Riay3yE}8#4P_W!S)}x#Wwi2-E`T_i7M1Ng7R6rsB(X@< zZotaU^p@uRtdr;|ahbhQj%oMwR)b`el7aShFep1&JS#MXD0yr5-o`1G86g(sR^~Jz z90`#rhp>r(xIBnVT&IPb@3KvWvG9j8KS7}-iJ$6b7Hwh` zSWKUR$B2GG?`5bzyT#Y{CRxfbAcb3Bl$^QzIsE5Y)a9Hk+fT&dZ8p=N!8>>}Rl*(1 zL?de(ebFVi+A{N-`5BcKI#FtQF*bQ|k~&YQM8EDV6mRYuSJyD!{DlrC^%fpGqPORh z+?6oI124u%BpsQTRH{ZQw;@%AldAJcwcVupRZ`asmKP6v-S7+*~6eP+)U6%mGXC#zDl; zBuhfI8V26rg+S8ai>q+QNr*i|6^Bt}=UmzPY94yBK=`1hnpEpTDi!=`*pWwm33a?V zA$u=^^iQ;u{DjKVNFLOhMm10vN3x~?W#P~~^_qsC&@7`z{s3^(8$MNA`=h(;ryhuKo%>pQK1soZlFuCB)@@c-X$ThpNd9QR2B0&g0?bWtvKQ1G z2XP`J&=^1xUm=D8@k9hG8W2ZS8&KhrSU?h6-OYy}BTwK6ERbacGNS=S6v-3~AaMva zB1{Ym$WSW$l z{GV~*r6QC7Km&CGvJz@86Jeex1QS4aE4lig(mH3?VPMq2M?&r;aIMsyTbry*R3lje z0JN!J9Me5#1Gvx%xtjXwYO1;9{L}`8lVOrl7OoKoj(>J zO@fgT)w-y9Rwkg#7)H*AVdO}`jC6zigJ^)xlGa%2RgWja_+hHzriB)>f`dm6Uj$)X+#U zF7%@Vje_lU5=DxHkb7(*FmQvfdW2!v>==&3J^kk!i|mSRnCN8WM)4F7Nru;`jRTsa zR9i}YZAar|hj`5SfO8cn_3{!!_#}C}G!Z^z3nY^OD7G2eG=>B~vR4h128G|LRzx*G zG2l1@9fmMe;svrx0pmUhcHD@n07<izE(8WOKM1de^P#g7U-s_p&51xJBb_c4F5EO||EpEY zZqSkf{9Xs8Ux2X!BrRI?ugS+hq=87|HobMw?Bb$|?}8ewLg7&p9!0?u+w6TPKKwDy z%hhj;8jH2a<~^USN(BEkVIQSOu55r-Bp~4c4B-d=d0yZ&Xha?H!~z<9^_|ir{Jjb; z%u^#AjACE^w5NVZqce6AMD@;f^{kLrSMY@85R=hc_hy7rYWy~5%ru`^qGy-)M?2a2 zolW{(efvdX`fsIzO0-5C<*^I?q#93zssW(vKmC-_Q+p2hV+U;ihR;rda01*N1A9nX zcWPQsV1PW@{-vR7h+%2s_cJ_ZDbsg}-dALmyu@+kBI}!_stE|nZ$_938vPkAY5)=V3Ulm8 zyb(x;2frRHx8GeUl?1YU;4-ud0W9L73`vqu*DlcKkDm9y0=0+J290k_KCA+#2Ns#= zmg_+&lgBPBC)MO|0A9D&YB-y>x%PZY?97(DW$$eN{?aNVims{J71QL49TJ+Zz%YRr z=_k6}%`8c#!*%Z(o~qR{wdLCsE^NeJm(J%xod?D&qH z_)h$gD#i5Lj%g(QF6Hfk3_y(Wxya!+Lox+`@%7zP>+c?=_WdXSFvcC^6F=t+;Zvtk%;38h#Z2*2Q~|kTJ5Lk$deA36;WoA zEv9~LdPYhU7RU71?C^k}wRzsRsoNWFnNvql zf(7iwaHr#))9kwkhmVHB`ibhxrTH+3ahtF#fMo@<{&eTQ7?TM&fp`3+XdM3%rorQ< zDn6|tdIU+42VbV|ZwRLTcEElmhkyO3KF+ndLQ|iLd{2#$`DPLDO%F9#XbY4#3>!4n zjnyG~{(dXjU-d-)Cg-=7&;G>|Ten$Bu4KgW2_=U4wE%IA>UedYh5lCXztA==J|Q}g zC-GKLV4xQOIA&)9KKa1EaO3}mwnb>ys{f#Ev4H2}|3X`Q0WmNy-zO`_5a4n4AqGXo zhD=UP&&*Es1XuuUIXRjA13le^C6>ri^fkOI^qMc!9DfbK`6Es79ta#6;ZlTjaDaN? zuWvp{SESq3;#4&U4&VJ?-FPTi3cuTYlKb{~|6KX}i=ktIG^zg~Z&znlB(&5`yM z3*G51`EzX#?%6P^)LR$2#1(nH6!5cc?^s##e}0N*;eweZgY`J^Y@{%G_fU3gVjoqViEpnPF+pQ8Wak-Ufi8)A*31tnt$ zOzxU90M4nygrpMKu}LHYCv=~bSTviQB0&LuKm<2@W3zoO%A7nMhXq(UQP0A*OozzP zMS#TSjd$(br)>P`@ryR)fnTg{-tTwgbq}eyoxr2|27$Z@BZ%30r7e1Hr>gE$pFJ<+ z{>4}{YbYe&KHAsaRq>J8^BnFHFNHOghPEwM0ne_XjT?x&dwM+HZKZlVg2O@;;w_Y+ zIGf_TygL$><#h9IK*J1iZVIln; zTm6jU?7i*G7Yoqg6Bg^~O|km=@~A_p@benh(wTAAHL|%qZs*bqmk@|kzj9*FTja5^3ewsZyA-9zp`_GY+l%Ma!%LX+gvy;u zO4pVY_qj3UTB+mSTf^!Y{9C4Gi{w4YBx`?453&$j`e9h2YFes~WUZ*%AnyDx+9ev4 ziFl?b@{~?${xG49^y(JMj&|Cba5MUNY91(8G!8?eY!{+IdT1f)#=lSb2aZR8lNN3X?>1UhCUmh189)EvD*rxxQzLVNTK+ z6+Z?N81qh^R-hKOIq$ZFj;p6cas=n->uM1X}6WdUB>TMqq zp1BbOQZ$Mf-*^%rV}-?@pR^%B9$pN1-7aqqvA)gT%fWRUYQe8&5b;9mUrfboF$6NE zM2FJ*X1})w2^*u*ihZ~AGUR|hUgR}<nNX_3OZx&p#D?f!D?-65%1et;k*LfB+FqdlS3RK|KXCCx;B z!W0$2p2 zzgtHoJa__!CTL<#U&V8r=6=mec*Ax{J0==4$!1`qL2A6WyUYqzXmQHIovt^)yMDg~?IZh`Ed--yM`f2US%`sm5gHT(73>wVKzh|(!`NZ3SW(pyWZ^gIY+W>gu26(Y z7NZRnL2Fz)YD4BN9&+f}U{u<+ngwty*B1W`N*EB-GJ}0YWBII1PA17rdhO?~3r3js ztB2{|O+Bzk&bd~%4L0+NM2^WJAWcII0D73$6UYp}2{ADxK>0B^ zxXMcKz_)fu0VP`WHa)=X0AN_g@llKSqN+Z`25|OC6-mVC_PtS-y_QIXDJyM3T4CN> z32_bXIh)KA{lh`|OU=6^>&UB%Z}lsFFK6h%dLFM%eL>JmTmTKIyrC+AHM2Pv>D4#k%+qJLeaa z%$U4@_#G{%fqxzVp<#ia8bLNsot{hMRybtkw7D-`RFH@+o6nsUWOJv2kZK4Mkp*DC zt*#8x&c^uoc4p@6e%K&vKRbd8MR zrRl!LlOs7b$nXlzUA$y+gTrCQ89tV{2ugv83JDBs-Lw6OH20e51wsRS$t=CXBG?Qb8C5GWCfX zkaLTHuEAo|sN7Qe9i{nuN9X4I5UU>=ujN_4AmGcF^qo3SvGrv$2x+{H#$ zIZ@|j`4@;Rt?fb+W`-D8+i|o2S%~2bZ#V}JwPL!!3JG#)X;u>Mv_tTV)MWO{X(^8_(Yt@5n_1)DfyF z`u1t!+AbpDSr>d=oiFXRx~Yevu3-gFnfdl@!HcMf*KDAXJDU-nQ`Iuk1Gwym0$ONL zaVIzYQ#L^!&YL@TQbpZ#RPL&6BIS1@^|?8xJK0az-1RTqGqbv99(K<%%lTQRWAQ^JEPq1i5-*Q=+HK3j z6agJufa47o_!GzeOhQb7p5?NL+s`Gnkz5a*A;>I!Y6bo2t3r$?!T=)7`OLSZRdL0Z zBT&xLxv2XD zX3+Hz<$B`K%Hwhm77MOsUFpg?)8>(4~lR($67E=H%)~*+y*c6V17mnu_P9iV%bvZF4 z=93b$3&X=Dbii6sa}{V(=1+#C)q)g5PZNp4jrK>NO};>!do&iDn{bc*26@sY=Ylzb zH|C#3>J)9%6l7BhnyT;pSt)wG>i1pk-pp6Uw!m@;$iB3F(Bz; zG)ktbkaKOwbd&DLgQDphC0qwtEC&TQ5JeliCGqaoCcLFbD~0ZQ4tcKR;gu4`$x_pO zGid;L$(x*NSjNGST~woM=ULW3C>g9S5mO3Jep7~0)13NQj^(Sku3mvnDt086snW>$ zUZnw)a{GACa|eoM)awv5APf|ciLj6yfFZN;!VPn}6^0qHDwVJ1=acYqe0u*xl+iD6 z2n$s3-$JQsYL|qaNLV zN++VH zN28IR7~9H_;F@+@m+!XC?JRA|x4KAneapdvv6vE4m9Pj%eNv@-t5%Ryo0(V!5Gxwg zJt;Pd$qf0aeB}~4b;UH8|LX`}eAG=E3l4>YS>n`UZ+?$GfBg0)Acljx;~EhxL!G#Y zggal=olb>1twdCr*EGD?M(AMmUh&Z&*9Wxy3BKyg*Ht1=UFwXj54<+#Z((rJ7@antVdbrox7kf;@QZRY7U z*jy|BFxeb&rE%xiBjNQ5k>3?k2i49I{-?CzapO8I-&^h^JgBg2ReS4IjPmxZdr;-> z^@yI1xaaM{@6`rEJK}uYs@n#}+g@a1ul)vPN1*)1?a$ttiPD;Pd+ymLm)iTZv@&$e zdLou3&HzB`Pf}R$u0L%D`T2WCr6meik#@Wj5OF&NYW47kC;ol`k=I+viUNxxhA z>Lff$_NYjH2m2jPpVr&hZcx%%8(1t3{UDL?71tqut|y2)Y|lB&QmE%gWvH)yW@H#@ z(&u)|*UXN6#nm+@CtU;<+q)drt3G`F{Hd$*ql&J?6PPgCwF3j*onrWcVNgX3^f3Yh z1@|cH4~GjBpJ-^fs3Q?_Ft_=f>E7;daU8#tAM9O!z^UC&_iyBL?txT6k6$7#p0|qr zs3#}9T+iHstlhQC-+1`h&*9q*splXQ5r#uU>9rZnftc@gVPu@;FXQTLYVTlgg$+OD z+Fm;{nDkh#{L)hFV_Ix2dt8ZJT+h1VR&r4=1#TtR_06?QqVK#^pLJ?TeMJbnEcUWF z5mH2i3GX7U(GXr5{6b~0j2s7sqSHMDac5)9xMCxKvyqOEZ~r%K=K)6&q4u``?J;G? z5zqzerIvtjcn;O_^5>F^Dk7^mtJQb8pG4Pjb^LoIfln)`r#)q@-8{KBU1@Z0 z0&Yhb6_;)oUzbW;4D|&Ze=pvA8w>tJv66Iv)rfe53c{8Kux#CRrNX)JpbG`qCY*+2 z2KU- zYqR>jJ~&1je9X{YA>Ca)ZBd)ze(UX|0%1Vyy33PWE{34ETJpc&7!aOeT4@$|brXf* z@i*^apEVh@|K>u*!RUweiYcrfclEy#F8_jER+bhM-vV`UBlKIRuMs$`tTEd-OGQ5n zG_^6AD(fs?5}zQ!kav6s-`Kjy32;+eX{;>+l*NA>o^8_81PYDdV+ zoFK~4tXLUyf9#d4yM0=+$%`A_#a8=ka7X)27{dKbX23r^R($ zeT(PY7+t-Xf6l!^<070a&~o`=%P;ALWBUb2$ihrkg>nDr?EEO=6Z|7rfcO@N5*b8N zq2KC4|DM(POy84S0zc2Kp{h5y>pm0J0)NQ{&Oc=wf0g^(hVz??!8!f^kM=zyqj30Km?PZ&<>-B3TjSlQw||C2S1n2dESrHbrXDd)_5CZEURh^4+P8JORF;j2``?POl!=cm{KqHVT=}t z7Y#B}4GlP6s!uLlh+5t5M1PN;sl~uv&S~rG!gxZHKFk2Voh%Rqm9KYH4t@liyL;i0 z3`e<}t8)Uo7IKMwwoEwqrs`eDbzQ*{zV}Z++S99J@(9&FX`xH6e;ht9Eo_!NPdWGU zb^6Q;xVMs^*tRNWyQxK@fdHPm%Kuc`%mrt5pc*R_Ss5L*S4A)jjFNbemP*EfF_%Q$ zT4MQ>Pm0n0E5c`9?DeeNuWfEtFABPEEPF;TDS9{JgQ6!MxQa#mFP`4|p~*L3-@cc` z28>R1ASoat1rZ$5rK3|E(j_BA5XBMF!svz}oereS5fXxm3W9(*O2Qf_f*5@Fd7kHe z|AYNP&^X}ms0eysqB-lzdBD0e15*1|MKhntAAhW%)W^E zD}GS!S}NVyVz0Ml1k{&Tliz>DA$BApzwQp`_bzKe&;zmLFM;&0>rL;O%jm36=p^p1 z`vV^((#}aWf9pKC`M2~d^`Tk#8T$Lj1PQZjfen)w4s6{6q1NPOG-=d$;G;IkCR?!k@YRY62;IXn6M2V22oTT{~(17#)%IzlDz z`?rZz!zbYvzAbUmzT3^YwqM?zzOifbL|Q3H$=x@je-EPW zzxcZUk^QG%`VU;)F8AcF0XI{|K^t=7iB*}>wbwnFJWt)<{FfKh$tzBPbk%UiljK za_?c(uT#Z8hW9(oKKx2WAEpQ&sh^T%`hEki%E7_K8nSa%X#U9TA4_P!{%+iUdf@NO zM|<7Ze)ZnF{_;_ZTbX41XM-7>$sT@_c~NcvCOqJWdfUQ9mVOS#;WPzK{!u zXa1A$l^=5#t&@a)XgbWHW(6geGmXI~a*b#_jTspQ7hR5JZP9x@Ldqy%AiOaZ>pC&{iMlEA z%5xvO8F8CXYAo)dUqYZsSG6~1#pulD+q)w23d5PsR__E>>2X!2x@yr)fdk86f;ub*@K4BkK5@JKmxa6_jO3ZHurkrLfp2s0D-DEH60JMEhR|`*IfVQSqz(cR|T;2 zskAgI!^8b5o-K_#ruCa-$0Mn58M&5m0ol2Ugf8?l6TbuszS~V4ZvI&eB8Px4}Iv@_M<1i?b2mU z=o&=bLQ3WR@3WNEb)x-65yKtaGGNT!N<6aIexu)>umJIaRh5?JTpDupn8(@KXArf~0GzpnK;WBtzh z8o>2fU5Kq&&hCWB5)&`irG!)hn}GDn=V04jx0c;>A~OH)A2qZGPaS zLQXd5xqb?gRDR+oDxTKkDye8ZG2O+9COAJ}+MFkyS6feuaF1bWNo>5_vzK!uu+DYd zfrJ1+#ue*zvEEnXhMs3K??+c*5|YxdtR}}k%swr%jW>xWzZPsL7XOfF{M?uIv;9Fu zm6Ctbq>;RL-aGGn1%4*8B!}LSP&&Nxpbh-IRBX*-tx+mnTqSZenBL zh;iM@X~lxG>h$4hG>FgdHp2A#?&~aAC}vIC`z%K??*6~b^1--T-0?M;SGa*U4`&Z^ zn4{a`9~gYE_@X`t(^-Y)^yt>hY`}yoEA{P*UCWN<>$QDZaT2U;79D+;gyKQ^aguF$ zhz&R01e3YMx4yY!ckD$n>Pho`FY11}OK28iTcz!>fy%XD&O%DR$h53-?)ZI{aZl#6 z>v*OdoDE8V4rHR#K(^yL>}F(EijWxV>uo^^G2wX6@|Wjr53Sh6I3+;EK>S4`BC{TX ziJMg3ZWP`mN+c0^C&vC!s-uqY$YmfkD zp4ncFc~i3vt=IVUY*~(PwxnTARu#Qx2hrla-_Q7HPMto^FG;N722WoS5Ta;u>%({g zXqlpo03cQz4L1ktUKyoG{@pvr1Wp{vorkkF6NjkMJ76Ff8_$iq>|!er3nQTwh+-<=E*dZWq>sV)NMyT@>$8C-pp# z_&&e=YSkr2nXZ4<%{l>(@(+Hx^h;U|e1D3)TTZxkrx}V=AF1cMj4wlGGxHrk)Tw_4 zVU?XHznsh*X|>MRMSO-fXyiwW1_Mys9X6gwmz?-b-uBz5L1_Hp5KkgjL!IoZSy2x& z8=i)lczuHP#zL5*P|cEED~f5hbkVaV+vdl&j^6k)t4ao=d~9C{u@{!MIt2Bk%4XS3 z&d`GM)`;qcqdAyQCmy%KBJ9FGsuCmhLm2e&N44KUZ#QsYY04Reb8Xlq`C6n%8=aUw z!J=Y!{j!m!bu)%M$jPEkEx(l+i+1LJ>Hd%=x^zNGJT$t&)&4Eg(l1dk-weoj%+6_W zm!K=e%&6WT~i=Kq63fSH~S>HniA3SVD^mc1EQR{uOC4ZLKa72qS zm7j-hrr8JW%OP06Qn7s5WbZ2qh~w?}aL`lEfkue?ib-R&zJTR)r7ls6-L21r6RY6(1x+8w`Z1GPdA zgI-XmlAPtRVp8*iZI_4qT_B%}I9*Vja;>h&DRo$Q)Q_5Lch#LQ{!XjvpQqCPbUkQ( zApfY9zq#-JMCa_DNaR%t|JJFtyH8=2&RbB=> zWD^7z+0qX23Yv_8D==_40$AJ?0-}PjVa%@B*e-G`3pJKNi^Ybk=t|%E^fUT&Yi#v~ z@vj>5-%P+?{*atGE6`y?!L1OuKtlXVL>>3-JsWT!ov6bVUCm_kW%{f}oEYy6w3y~y zIKpl_EqZYVKC|f)4@#F>sN_zBg21Of>gM{-|w;KFtg_6C&{}f zs#_=XU-IxAzxU6}rFWf8o92HF9sre2=wapxHL`ftW~PpV-|m7hiAUGk#mWPO?Ocu| zY3nQ0W037Ml!a>iWg{74!|u~Xbb=G7Y*W_6i90zdz3OQ{r&GwctoL9rYfS3X`Gms_ zb}RhJ1PtsNiNg&Ov$YPl!ph6rJn#sA;GOTmo8#|nkkVlv(v{D?K_MaEk=TEcP=Xm; z8W}te8GI2L0tFdD-5DYd?95#{p7CjSB;eZAv`+Wb_593@!}JF+vfy9${O2Dqv81Un zq17U?)C;oCcV}IAm!n)RO^vi&*O1h&8JPS4Sxu0429fI`A@sB&s z;aGz?$%H%(S6@E#|CF8kA88wrpO}>E_wKT2-Gf_Q@GCx?A=_z(Bg$b(dBn;*&`6$X zoxh7r{)4Ri|0Qh`y7T9LM!bvn48_9CPz5^I{X^d&{`&`kjl`3{1#iWTh)@~2pcO7= zKb%<}bKZ*X)kBoptjif^En`!SSvM+C&nMo97{|e|Q$7q9iJ{w5dCb(MRU;vg|4SQdY7%7Pp^nC^X z7gk^cVS-}e{#d{W4_6`jaM!@isWx*-CC8wM$ai^u>LE(Ma=7-6wxJxW|{i8x0~jB zvMLwyZ^&HJr5Ao5v7Vo=@1fUUrKzlgz>is(+IMQY(0A85v4oIwc|M?=U2b=r?~D{| z9{~5RDQDF-ik26+&>EMvb4yuDXH2sGkeb+kQ_6apOx~5=kzj(;h=LN$LFdc#?8+uW zSPH0s3Z?nW?S^ab;F(W-e9JD~NX?%K3Nv$j8WY2?F z&+1%QTMQ!GHe_1reG(dtnQ%nNiZAEdvleb?4sT5S0bbAIAn|z>+!e4DiEY=^q)a(d z@)t^2CaYXn0myX6jo%#v;m%;q&QPb$u*l=i@S@J!y`6U!JEMMgh9Uq?QZ?&*leaAx zM{2Ruv^cq+=GKr2MYK(`K78FnS)DNSn`zy4ZUw8`_&!#O(q}mOct6KKtnK>o=$dIZFlVgiIfC^=Q#st(T<6fD>?P% zQ=j^x-sL2m?$6bHXkghtzxaf`S4gw2-?+O?p0!UK1@go8jV^ZO{9@O@0p8T0!!CsM zEP_F9kC?<<0id_)5WK~R>$=B-r}w)a3HLrJ>Ifu0nQ7?$w>VUG1a~4qB^Y!r22i5F z&dX5!Cz`4k+B_(L9L09c?=*6%hYkcWU{Twr2!k<%J+VCKmcjEJgn24Lv=L!*p+jP+ zPpYp<=0bnRV!uH0I--I2vj+^5IB{49q4Cl%H z7;)VRozn~?8yel=P|}XALNYb_+oa6e{{Rb?p%7do&=oYn9GG~bW9Rf4cD1qYjL4{* z2vgophaB!0?j@M&zVyU1b(A3t@Vf&8oW~fwZ8#Bf9fJ(>@B3UZcH2x?o8YmYf(*pL z1q7ffbg&E|RWh3EOi=VSGTdw&&U?FDC+q(_LVSD*CUCbK^ve7Z_`%fR@@v4HVxyzx z4!Xk3lC_lEGIgB7i)H}AD)CD*iM4JeCjeyxWV>#AApYF$;lNY?H6QPK9=A z0gT>GVrae(b?V89o-2Q=hk)_trp*dac_}Amuac3uFFxbR*i5vD%~V)9MQ-iHt&K18 zHiNo5gSUR%WlqV9nJ&L6M!o4EC1Ur_)vOaMLO+`~4A-0vbHs66`l^E=OY*yu89T*Ofyhz?;g0`=Q*f(=6Z3|oQ-JIy4|Uo7vX1D z;l^8$sr+{yS_&qSKAoNiKjM$nA!8ktVQ)4Wj-X?Xa_ya}(vQMBkKt+WhDDkPQQce9 zv_K;Izz;xB0BbZnRu;*LgXdiu?g)K8gx`$2Q5v7N1(3I_fi;PptpcsJQ1o_7%32o& z*8g2I*!8_m*0_d~cMurteHu8(yPwo|zp?Z!fwnWg8)so3eM(I`4W)8sJm#!@v`klw z{P;GluCjkj$R6Y$1)W(tnvBJ9j@I>-JVv-X#4)9)qyPsWf7Y7^gEx;G&NSPb-M zRX6}q*U;2*b9?90=Px@+f$D&Ie^td3YN8(^{jY?!3t@cAkV0nI5$3`(ez+2&1g;`58CX4%k)pv}();Mz%0$K3RuPpH%QkvzDA@Owpsa>0I-8V|L!Jj?80+N>c zUi|!}T9DX+Qgdl@;7}&0B)0+(l4U8?d(8E*KnNj88jJ$3k0~#V_h5*g<&!gO;~92e z;$K$TzPRcdUGn1c=L>HRaAP_RUA+gf!P`QI37s@P5xAImfj{G~uH8Yd2iAJob z(hagsanM9n?W*eJRk)e$8rF_1J2PvaaMQitJ=bPwfAh5Siie7e`}OslTmfA@Jb-=B z81xi%zXe^Eaq;ZlD4#KHZVqKfx2w9B_dtxC%9JZ(8iP=iHt`+D)*_FE#F*(sK0+I# zd11su!t(e|c@-01LURpsC2G7s|bO%vC@MELQfYcs?|W z1Qup!4$K7PsnL4eGCZUq3xXsaLMFrw>VOm|EAh-0)S!5n7tgRhyCx4cT5tr)095&g zKvEFBbxe6vSA5rMyAH1#FVX%!_vz@?d7~2-l$`BsL-0{`Ur+n_zse^a<1*~;`Cwj= zkWLpIz<1=tS714xI*J%hU?dKlM{FwkdIglLUtNHD6Yc49L8DekMd%N8lm|>^$bhcZ z27x*K^%@MgMTX+_1(14!aApJh$auiYH`KV@vdDn9Jr zL?EoAsXUIKS(un*LKPvv)&FH;NsBztAH}gZ08|j1tuAI|TL3nbdx;(22LJ+&AzxHG z!aK*Cq^mKzO&Tl72)^!*JBeSzj%EIM^nN?f^Q|xWiH<(Y-y0xG&t-NUdbqKF24yw* z$9z^kd#rFv0k^XKZsEnN&PX9VvD`q>9YBdOwSq9e5rqr*q0jBn4Diu>UG2J}ULqDL z)ZsL!lTZjQA#&^UPR9xR=Z9ZVG~2{N6g7ZX2EgOT(Kla=)PkrMV)i&8;XJqcV0SD*B@AxWZu9Kspk89L>-pRB+w9q7S=?s{K0j`yL=3A0Pprk$@MzAq7ESYV#&c=t1oL14KH~6C8Yb3rQ0LN%*pniaOz`Ve13)2vnyYtC4!w>=42zCJdgu^9palS! zZTSJb?)wYr0s*?QY(GMYQlZhhBi?|(v+z+BzG-5>^~>PDd?4O|p=ei~5#eXNI+35f zT_yZq!j!voe(EV@@1F6DH7T_aieU)C< z^*T|{#<9gKvOvz3I1_Do{G#w0PsU)c%#kFBaoW%=N%ivVQcPE;2B#8N;h@k#=5B|R zik5UD6g8d9WncOw7;{EFk(%~MJRYDDpU?f}BrR}>e=yM*w|EW8;II}y4O1B$xoOGi zL$x#-wV1f)(u(AtU>03>!_mcWgqF`dB?5*fkOHop`ZiYCGSO(6!(>3Oz(Y}oTzgdK z$|UBkE9?kB)hn@9ZJJn0A*tT=Bj)y4;{%~(rUV%xwF- zATe~&16}}XqLFQw5~PYiD@O(P)hWi>3aigEx@q}Y zca&xD;jD^Z6H?2?A#83Wg6b3|p4D$&&_5rgG=qxgkUgC&R(*|9ya8WBRo&Z`Z3#Q> z6n^=&zMTXas0sg2{&L88x^IWe?5h4luCwh2WvtOL93~W^PGeqda=!E3PD17AXMU!L z{lHs*h($gwo2sgai8KDC<*>>-xFh^De;(cbJYyz2UAOJ~$v4$1Nq+!)Y=3P3X2i z+gF0DrN~;m=qI)Mum1U6Yev7+3*TwI^i78@N8}!)JIpapOrW*21;A0Me+jW?ena2_ zamcj4pNXO(tDrSC>2uGi$!~_T1^89RbX*=NPo)cp7%9GK8eH^j`;snx?^WZKw9W8O zLd|mD3V*E3dhB>s z*3G;w;SEjO?$eRa!$wzt&C~BRS{E<>N<^G!;m-TLit2DC3y`^TEQ7DTP@AIKOSi^A<;Zb0hAT{XqO!;Usoti2 z$L595Ulx;7Qtzsx_M7f$wWQu}G;1gSj6IVL{MBpF{wsr9xmL>TuHSH-?jj?J9_A6p z-eWN;x9xnEW6c_vEJ`R@?aK9w{OFcq82E42&f5?NKVZb9L>sXnut8AKT*N8Dt;&zc zmp9>$!AydW6;C`i&>$FI(Y5i%+Aw6GrpJGx$P1CYevbDr!jws~Raf)VV^ckxW@qeWT9!EsfbZ#E{QjK*C8-X(NlQ{M zGUDgTL@jT;_BfbH1O>CA8{}H@nDfRl8m_vBTILXFdk9YeCaHsgInJx(*Ct)DyC`#* zG^Bf>dqtDF=J>=)mRy^~D9f(d?4)kWhw3}Us zYk?zjo5TQ~Uh(TC@9CZwRN+X{!&@WWDusD9MD03d>M~;lk_=?u@~mv#JB3XP@JPew zB^VP0ezYW{p97-iH9-97c$xG(e*6#V=nyZHWbf$mIpiB_9t)ZYr4n`vn#J-CHY9b~ zf0}f?(zLJAtbNEVN)SHPmU7_@+z6eyt9(%d2Rm(yyn=~)HIsXTp1TcYYEj8!a>%>2 zE!R+r!4h`VfSQ9r(wR0XA&pHw?wgjs6 z?^{L8*1@=mqC_YMgRI?D50vYapY2eH*MVIVWQt7GaW>|9kX!Lwu##=Rm}@`awU0@d zEtnxgoW{cv_keaqhny-I*81au@ha!hm4aVxdgKj9iiFc6UZ>E9Ffdx?`E@B4jG-&} z!U!4+FoH^eY^9=hIoa95#39$FO35eBuE+V~gmUR@m9+<`D90i0nS;}j!{-90Uj2C2 zGfn|+;0g?@=CsojeW?xyx2|n2S;vH=8Y2av3fa7b?R;iFQOK`Bu-QlV@cpz)NA(K^ zl`hp03F;-u*_Ea{lFx8rydUxAM;9#fkUX|Xk$B`&-v_Q3&5<7EK&9{zYQxiMJjW(x24ScKW91D00kKSSjpbVd2B>ZAkBS-9{}(xns_vWqm3^!U?FPI6iWe zYzXp7Y)`HP$d##y$G67IJ)^wXz?u{*&a^F|oSpMIpIIxu9e{aVP(~h9u9U-QjHzPYk5SY`vn#uh%TKcWMnHMjM^526Hzva%CCEk5rC<1-`oUpfay+s zOujtB%y0w*P-RME;PP~bd`IMi6sR(u=gAmoR0nb}XQf6LWurs8@-MzqeuBnIP4=D{ z!9!bv2L2Qwo94|c?!!7?gQ;0(=)YykEib=tVq1~su$=6c-|6P)?U!C6zAuV6Vjb{} zZeD9h_wal?iH}v62K>qJE6T8CYt4316}|8Oqe({@{Ot8bVl>r?V@;^lVq_UB62uL4 zoMb`2aLH_BxTQ~RT22jf!8CO^fa|7+E)RoM-J5hUgN*1a$0{BUo=@p!5g)ShJ8cs= zBy(pB+d0HJ-s`p9iz)7U!KL}^NEDzXH!<6?-~b04PlmeIQV7pe*hLtlL-#Nl#}7MI zVF|*EDPW~IsQC)24@v%-j#E#K@Dq)-rlUkLF)}OwD|G`4y_hpP6Y^m6)95s6XjOAe z)pATHWsLgy+HL3DJ8~saNI(e~R~a#QO33i4%org6#v|i~8Gwy-%71DEddkm8J)89@ zL5&N1IG8O-Z`YVd4+9B61CJ|U@A{rEE?7I6AytS_nlmunD6a`ICySm<^ciAme_C8V(LOZo?lGo5ASR zC8XgsEYx`Bl|O;!P!pn72{ZncVN1U7$L;m5eFw*5}i2Rwga9>I(bc6~BL1`u83A#KJ?Zh03c7)pw55fk_`fp@?;o2Aid(dXI5wjJ^9ET+z zj`~0RdH>7U!FIJi%=jg|MYPQq(bP-V&JV0zSYFfq zx`z0=@WJnuujnFI)D3qcl!ewfrf4eHEVd$Wy!OMq{pa2Gi%}c)r5la|8_s9DkSK4m4@1!Nflx`OeY!|cB;#^uB?j|_1-QM`L zRsHWW>GEdSzy9!Mj$D@gL3}Wwa{gPc14eGmTa{mDYob2G zvUf6H^ftL{Gk6aO6?#YW%Z$sHw^3i_OTR1(d|6ul;@u0B#zr_1FxOtN`o6NmQK9a) zVEW+C9(%MqYC9Z53qM@G{fzo{So-bg#qqmu0jvu3J6QwGFQT^Rg+WB4S?1%@oT#tY z!7z<6L@klXWFuaK3I80N6z(Bwe|K2(UXV5D%0hM^yi*i^F2qek|zkDcsm=SjJ{I@4bAn$8@}B zdCaGnE?A@mR1jeHClJ9j#vle@jAHuswf9mWhobur@3PN(*+mM;5UH0tGEeqkr}nL` z9q@2|-+J~=nPaC7#bk{F2@qhxfRN6&UC(PDeaa5uZ9mVx`Wg802l(3$=*4>Zz=PaJ z0*c?>Y1nPmeD8%|H_JsI`{91Nvmfsz-#apYarjT0BmEn%%GqBw4!=@J@X3f@9c9dl zW$+3Nl(Euqs8)C9l+Y7TK$o&>D0QT8{#QcaQSFC6m+OwIXosfXj?5JfvOl=|kYJXe z6eF+)Ornseo4*4RdZ8BkWdr;0hv++;oPIcTJmR;|1Nh#>;~&?Kf8IMj{2!(5pAX0X zIGDZvn;~P^5&n0E%%6}!0D^KD>}&ypoh^|7)I9=o7{=S0+MEBQv~9@sE7L7$cLDMX zIX!ZNB$RTMB>MXieIty_R>}I=|ILswGTXOKN|00V=Px;(3c5NlHz*K-)%eKs^-8B_anPrJcudqA~HP69&abEi1#x0#YthU7T)X$){9< ze|Wd752XoPlv*>BABdaeLT}XX%kW&(_~x>`BMHiZa2!ht%&b`s^$g02fLO$`4dSUx9gn@=)L)&n748(@>bS$aDYvA483ZC*F+F

}isa4s+fqbFhy?Lcyi79{0%!dQ|mFF4U0uUmXoHpJb59CENp8{yQT!7giOwixD z<9!TLZ40diBl5ABK_?6feQMilg!~%2;UxK%?O-zz+#US#xgR6DPK06J1YHTZv*;4| ztnF&7$6%jPP7=q@tu=@DLBr7Sm5E#fE_{pEM8x`fyItg`5vdK$?>4Ol#8dTBqkPr? z0E?D#ClMJV2Zw8n8L|mmkarwk)_oP0e9_Ba3IE}KFjX=45iI8z;A7ni3zD^6g#|&* zKQ$FpkrWz3dBVQ45qxb6JxQmK4(rtga_5KzL70M4joU4o%h|Q9RvyO}L6V>%$xJ~# zXl*WIhpY+Srcv%qxGlD6F1%33T^ccA;B@h}gOaz%sc;JxZcknzCf-D~s?~=N91%VW zO5&vuN1YFEj5n^`s0g*)|jV45$M!X@6&z6zHk#CG>M zh&Hw}&Q{7z*2%0KZC=^y3hH+-&-|YwZ6}zVoO?%s?QC4rLraea>&iErn<=S~&eZ3^ z>+JZiJzKt+aNcD>lAciEIB#0*_d=nneyi2RorbkTSr$&9_w$c8FA_mb78BBh_`$p% zJNJh|;GYPxBQdpJ3V{@W3O=?M9K1K|ES&I;lZI(sOy#SQKYkW8PUk&`J7_{qZCn}* zd2Vp~DH~5j>a&;xRwcQOO89+rNiY()B*MBr!N1!>}S6tw3|F9BrAI5dVzW_h;dc= z=`Iktu(*6TojI}G-TD`t zyU+sc>xxBcV*tb%ix?fnGyzGa4CJU~?gPaJew%I++m2x5q8{}MPTg!Q6BW%o_00l- zkLH-#g9|PVJNP(L|J#VleUD^9p@?uI26k0&1np)N4O;d{f*szl3%q16r8G_nBTR>% zyu-%q_4oqUZ6!^=u9E;$&5#nN-1%%JoTqXNhA0TsIyWMECv;plN`fsa1Gw}?n-;5@ zdf!1ee3%zbkMGfV#$>EWWf~zuH1;Lg;Z*7cY4KY1(A4v9jZMbRd$O#d4K=1(K7#wG zMY1SG84r@D5R%fjOYi|aqq8xAA(!iFkF!eTBZv9YurVNM0B#NpLpY3au|!cQE8al_ zjfqE?1p6INlO+`1>DYSQ$P)WgX;3u=MdH(0E0;=TERy)Z?R#f+QmW*RFHN32BcBJD zLhgqdlyl$)^WR>QR4I{w=P!BJV0PToN}rx~Kg1d;7G@~)ZRB*Sx}7r|#BRIOo4N8g z5-p>RLFI){0LOYHjx-2{PJ__)I+tMPvO4Vq5OXLJ>`K^Dk2wcTYK?u!``qrgZ*AX$ zqnq-6TU$Urr{z$=UYs1x+YxRxkbTLg>T%_ji=VAb58ekWbyt#a-d?#+8v!xtD!uc{ z60Cj6HmC%*e-EZ4r~YLbdwe`X$W|mJ3fxB$)nRua>_`f6>j$`uVbHbt#7Q<;59{pT z-%0pu^n7e#uuDMPi@WiuTJIivrew-~la&XUuH_UfnzTun4*Ob?P>O-Gc zXMXk%+p=_K&yOk3Jm%yj4z5HA07*5U#{d!k1!hf;p(RTPmNx$iq+5f!)l`v3>t;#~ zJ2})C7!`iD7yr=Av>}RXcg#7pMAdna>1Q;Nt&~14Qq#_M)jNjmEE<5-VwoZ8AifLc z<}^WvYew4lGxLfkBq19U7BBX*gK!?QDWFLunx|KnoPxqdzNx$0kMf?$(N2;h5DA>D zZ4;|CJxw~I0(ek~57gD$BwR33w%+1z>d3j!wUh0FRK5c4>nYXn8+Q!#5>*#i^aU0_&DHV9V zJ3d(OIG%dSWtji!lb_xcem#GoOxC;O@d>TvRR7q|!1EkpDK6wi!S>bDYz?~bNEd<7 zQsI(&hI3QTy}f!x%m$H6QnkWVdYlLW&)8}jL@k>>380b@K2N)Dxt>@SPoG;8cpout zsJ^$nigp<|If=jjJXnx+EBNR^ME-1tztfG&R1NB{g^Jxva8b%hRy-L}aRa654PrSP z4ghEr&xeos{oO^?-=*XHeuM7-w|J9vzN!;!(;fer2%mKN z_Jzv7;QjU1DFZR4R6VWl&nPfyqz?Qf)(OOeiFofrZfKKBfdUU8G27ZQ4IR4C$5>G#Z)V;&rY z_ohD?W{!pM6V&8L&{gsY92No>4coL6C?DhaG#2#5@U{>Cy*ZT0vQEH1K7CFaM9RbC zs8j7uGAqFTIW0I`I@TwyKfndSe zl5BVhTXzpzzD*xZKByF#A#W7mJV4{{5Y_ps=el$STVpG{EIRm(#j2U&UZc%^Na`1g z@u$oP;(nYmO1$SsgK1OLw259mdvW0h9uTAZw%#aj016>MUhtp&91NQt0t*lU00;0B zz#2rTB_UpFJUn5Bu{{Nbpdg_KjJK)t^agZ2&y6hUTUiZA6QGYmByl82w2^yRLX_^q zSeFpvuVZ%fvBYuu?UVbUHi}HsiDw6o5aO&o%dVrADEBRdky#xq^nr2*k z$%zK}qX?Dqk@I+0z>#18z)A+mQXnQ2k~fXS zA;{v5g4@%fdL)Pp0mQ`&I*Ecn=r^JIa6K}>*l@{>_sQz?DY3o!*6!vB;!SZh)R`Y+ zMuE(FN6pP=E_CVe@Md+YWz|ZVg|z|J`pULpuv65m8FE^uM|OS}>_-Q1b3?^ALfh0Q z{V+4VikZnD2uoOpo0Ea(;PWn6m@yF|fdyTyV|*`yvC~j1I&9`vE`bL3nuakjXB{l$ zGy!zg0pP@fRWVQ;0d{5R_(H(}U=Cz5YgfS@oDIh!z1Z&@<{`k+9S z16Htekoi=%AbH$)w?k36pRfA8a_n&P>TAE!6=dd>of?1E4BsT6Te6SShzC=W(q(tC@^X|2aW?t z6CuU`%n3L-M}sZvKqj_)ZeZZ^=fI0|cRv_v8Dq<29T47G4}QrMEN#Vp>r52AWq+>j zJhVp(UhmAp%6}-O`AOjR40m}4|NV-s?AexdPrh(744{IlAYfqJ04Uk!VP6D9`aA%h z2IO{*AG+ca`o`gIWS9jJs!0Q%m@75P{s`5{B43ycWyudSO^`>j>#Lh zJ^|`RJK;Dqrg0e*H$759WFq8lt&1@h+E}mJyP4_8n)TS%WqGSgST*}kc>r~xp%4eR zCPMyr0nZc8+7Ur(0=Io`!@|}Lc6ZKAEfiWNDkO0>@h1~Au}!wY5EvQi$#Tr(h=M4OE0HpocLAmAXHR`M=n34(n zDh6gwtT;`nV6gy60F{?aD&Zp~GpSg45XABtq9gC4 z1?%~H#W1vvM)kSJYywnh-I4JP&2kA@PbuM|b9u<&UBCg0ORa}l0(F^3z?o!FUSXXJ zrH)qXa%s8?_o+Tcx!u#So2e+#4WgP>i&_{hsP2X75-yd?9<%v~L*AOIpHq&j7Y=Sh z)DRH8B<`XY@ni%(W#>#Zg4XPyb9vXH5BCvbLhIUO>Ykd6vWdLt9@Z^iaqpxAxhk?> z7pK>Fi!Bf@!jBl}-qC6(a@pvOz0Ke0L>`EllhP4x{pr+qVH$cA`QUF$U#4mM3%~Z` z`|TZP&Yr%RrNPpYP}A_Sk}uhSz0krJtX$AtP%?Sewj4Eu8T4{J@q9>Yc}43&qYGog>`-i^#b?B5X>^td zF-01u84H;&J-KlBBwO`q%>i0hkUV?uPkS%rmM&J00WVw$ah?}x3 z85brxmK<}ZMhg7L5}2v7&IJyW=&t(iUXy3Ga9bW(W!>P(*WS0T@+CJ5*XW2$W(=nH zj7-h@vnY}~dytlWB2Nbz$JkzzNr#mFBSTQn)VrmrTZZZ4vd^p6ftO%mvdvqqg^gv`th!5zDP^gG-*I}9 z2kotEkcVU{Xr<5C zhn6Kr%$cUdDJhe$X%2gpjQeOs`dy8DF7$fW>~&twbne}E`6ciE3vEl@#e~lWYrcW? zmT_Hez!yRHN4=iWhko)6L(dN*sD7#M=P!a^)6ICY`!(A77e}V+Bma)5e8H&o&lW3j z{B9YvZS_oueu@$vcQzaE`=ZowcOVDL1EJ2$iX6XR__A0Yz*6*i?w`kkek*MGmFctA zr1@d!M>DeJY1`Q1XA1$cZ%Srn0|e&(YVYI{?zW~`%D`8 z;n!W+?|ytvFz~2P}p=Hd-i?$2_oNTS`KdAWAu^hjY$EFn2q+@ zpuAiQX+sXln(%my*8A!y>qo!lLO1sN9@^_1rN=0P0S({f%^OP}-x>-1`P*@P_svZg z;rFP>6Ns4#nVH0a#(PmS(d?djm$!V<(hAaUUuoM?b>C1Ks8VC!rZz5CbL-ccKJ&r7 zz1U5ySRO9=`XbQuLl6N0$8BL+8vb8f=M~jd!++_M3L%uxq^hA;CDM$5h=NErA{|7k zbO;~{h!`M1=%H63AW}o`)zGVoSg0?Cq5>j<1rbmt@Av&@)~q#idoFWsPWIlv=V=Qc ze5yI3qGOVr{DHbu^rlq)C(1jIGD|(bFtGjU3pJz5_hS$H=3UvXw}tlYn{OvvHW?(! zS79S`9|HEDt)nP$&K=%{TLbUeU9W6!WYIs<>=8?vU&TtVBV0HZ2oP?K?P2?Eyym>h z^ZVVO#8 zT^Ny7T+XFs3Pf8q3A%lwYZ=|^`I z4Ba^dR&{>F2mSLl31plq)b&azC=hAvzdjr!s0aYAIQZg}!Ay)8RSW>f9>pn-$g|NjncBXNpeX^d%YR(6JyS0p2>Eus{cJSlxz6c}X-AhTC^lt{WbL?HI4iMr$`C=7}xP=O>88b z(S`a;EcoN({pEYTM*aAEqZD!PHT{=XG`Pi)6iP%gm#p}j1Of?x8;I{VdM%fn*M`%j z8(gR4tTMYRs)+I?Ql>$s|Iv_*U-l9O2-D+S5>^fJn~TR|jPY!dRus!a9uHGJ%YRgJ zf4Edt-ZZU%-m*4-Ejeo40%v1xAEuEVB-$Ku?mr!DKfeW!i^E7qHZ^1G5nT6-QOQ12 zjbjqZFL3{0k>kuRZ4SU7$a&+&fTKen_~ANqME~1$qd3rMmJ`j19Sl8^6PougW-WgD zu}3YdOao!H_`H5z$e=(#X?B{_wT{K-LS;sIDt`8VjJn%*S4Y#iVIe#>O-+o74DJj5 zeEx>nyeDi@AU3OpkRI`nnJ;$ilC6Wja#{yKnLzICdBdGt1xzcfDm!aVU|()TF@3F5@p0WN*+AnWw#q) zc}leLmVB0x3e9J#cyJ*}?zieq^L}H~JM%>SAa4l%g z#}b8v!Xc-4PiJI12%o;25d-=reoiZ^@6DB~O0K1u3{-@c=_!mCD}hhdgB2}sdkyuC zkKkh?cK!bChNJan{i4Q?VH ztd2A$cdl=bu_r#R{h|K&vLxOlGR&vqh~*FHBISG zd`4Xznl3Qb`&hOncNL z!llDp7NRT=0pc`*%&N}o%aHiw8tb(LBj%rq5N-(&K*-=sR3GqQcVp&JBk+Iu=NX-Ov+cQ2^T)5xY5C`lrnP-W0qs(3~ie zuVhWiDEurA=Vs@?f$5bBV8;Gee4M63*JN|vjjS{E`MFf$o+M_341O$U^8X-h{r>!% zRllykWZ-f5Mi}6)A8w_XwMpL7U>~^>Rg_7O;`_LnVM+EP`A3H+Qb~|kyjdU^yO8r7 z@0I(U3P&6eh`X@|Ge3_4tHY(moIF0KAQd5m-}ynM!obFn|Iv^YGhcHrt=sUicrV2F z`--C`o4E&8NcM%$U#7n3yWY}%(gPs5fM z%!c$TjI1=dPw>!k>bER~bG;uMe?Vz>@N2iwu=61k-Ho0I&WsML_`W3mGhlPG`Zc(e>Li7I7pC?ysfyH**+oy^8GT`rRHc6cTZ4jeyQj* zpd33ianBz6oZO7fKWkF&c{^iOYLIkU(Gz9sllWBLO@G3sms3Ux`b0((qp4o#BoO`D z`Cwm~)O&^!Lul4xzmFv*d3*C-7Vbw~pG{X5FTSOX#TuAdA8^|OnaRSeT2r@8#d01Pxr@A`Ly`*KV7;b9 zcq%>&Ky|rUbX+#BKtrNyPDGWEG9r}g=&*C6jDxnFH+qOHqA@?4w&2Q?Ia8`&EDr~S zvbb)9xN-AdKXW;JN~R`w{x!CHD^qJI+C|E6ZroRcm}w;G&jpbOEY=!zEdHrU-ON1m zL(u5BJ|_Lu4At98CIT+*AbGuhfN3IKWY#Rw%fK7PH!-XB=D0~#K(r=wW@2ft^Ho#%B*7Uw5t^S&c`}TzHEd|F-@;QRkYfc0`~p0H;^zC z-S=ioUjUCG{Cxa^QGI0|mWO>GcO!;A5DJJJpBEVz}`$@I=PB2YL$U7 zSEMButX1q;DteF^4a8;Ua~edO@M@brJm!vWe{S9J-oj>PP*)||a}GI{6j)_(HM%H{ zq%-$VWbW!>y4z{?MB1Mi7<)`cz{F73Z) z>;3ioYxv>edw`>z#$>V+DRe~N|0{z_mpysn$8>U_Lvz!)T^m-THdd-NSu|k%2DG8N z|APTI=49e_T7Eamuq$ZUYS%a9lb^a;H$Iqo9Eb3SocWrE7C6Vx+%-F5ve%gZXUqM1 z*Zj-XTgx+VRF7`1}{_5jSq%_8H;x6l6mp86DbU+^^7{O89r zJ!KkzydPrn@b4l0s|LFu-@yi9t?U8k#9N)}5w2pn0KEu0U$$K;tfz63}-lXHdEB<^H0-z!2 zeI83PR9ufuvYoIsf}AGeP4e0N$B3_uRRrimbTTYP6&n-eB)Eq@Tkr{HZhpplG@^1Dq8NdkeQGNh zDLM`zUfEZ{5kRA~z!D4-%N-YH=3`>$Xqf3xB#k-InqMKS{bnMJJ1^%!E)&uXjGS@d znED{?p=Dyt3o7#$@mW95?1Y%h3;}&q%cI{)LmPV$KxCd+tiOnaX}r+-C8Wx2O`3)1 z4f70ueJq`*a*iuwila4{#dhcX@0J9qc4D6u8HG)>m5yr5j#H^f#QIB8nqZ8adjJJ; zqQj7Q^_3<`9uHA$v4gFHf=m?uxu?>uK8TXrTo>vH=Psq`Watw|N619f! zq;t}sX=LbWvee6=V{rFiKpIB!`ZrdY*Q{nkhCg4k%I2nNqmYIBATzRtl74zun@*bQ zZJe32>+V@?dQ?LLLH4G#)(=pkhhCe*xz==eKz&B}R7P@4U`}psUPrEq)Mc|V{La0^ z+p}bW_Cyz~R-_An#Yq)6!x=*rFB}79B^fHJS&xU|MG;7{V#kGSjAFL@MwX0z>S$AT zn6(*Osz5tkO{&DXDFHtTmaj;U*gi@cCjjY@r}H?`#{{sw1v!F78E6$a}DwNDG{O z(9j%~RgZ&nG{g0>iq3ihd34nFg{&j%Z1QS}Wkoh@v)HRyExVzhR3zmQ{8oE?*sLh` ztV1#1c*X>!*t|d2rt_)Y{8Nf6q9Fn#+8Lx2T$nu)_gosGU=9B~#qTVGuwTy8Pffm- zp(L7`KU6LPCy3euoG?f!k8T0fv(#%)pSvnv8{(~Qof496XBJw#*uftdEN*X8nH5@T z!%`+=8gR1wEWr}0WxgFLs;67&xc zF5~Pwv4XmJ-KBu^<@9t-UD!L`u#0!xp;k}%L!YjFs(x)#w-;KspI3L#S$A|${kF4` zMJBJ%vqXF?MslTU8ltjVr5ipLpRuFUoad}sNnyF~yq#CQ5JxDIBJ{ngJ9XHA7HSlg zYdj9E0ZrT-Wd@8oYmn--z7+*uzp2?yx$WX&R3Hj}z%FwokIg%)g={6ngqp9YH|y9o zOQ@H%O4Td3H)2L>&P-%8Z&ES{`n=W7T(T`B5lAvb>uV%LDc_lIqKRB}-0*$A;p&4{ z(a&&SJk`6tb$GM&*CyPX(E8$_&W%(fUS(?}7h`hR>}3T%GL5llZo4=P#T-%+zQ;1J zPy`UI>d3k>dK2YZ-Q5JL1;$#Yo@#-DPcjgs=$d>a6GIT^hXmsfp%>6p#t=4ZqGN0x zE)29~CQ$DsG~b%7G&HOa?}~}>tep)~e^ge>F2q^z@s_@WpUTfOJN>dJ4%^8{US@ZN zt$0L}R|f*9k)m}@Y<5iN^^_QP4tYXN0ICZEicIO8_o6OUQ+0vfw-2Z-32g?|ZJk|R z!t2TG>eU3*hOb~NP5Q0FMmaMK?Rg|Ni-@$hb#^@Lp6WVgokli)wWV6%d;0VKV~8Wb zpRxQag)3cvg2{lz|2E1!^8_=Da)N#lRGHsu>UlJygG#UBA>(wG9AJ>q`Oc?|-yBvS=b5JX0Bh;+ zYI2`*Mo!Y&_-PGEK8G zDPZ~;Lb;;PJr^*LXL%4U6xD(TdNvO`L5I&%I~guF12PEj0J_QW-)PFerW(Fh^#L=i zbH?a4!K1eu>eGKiU+QPQZ0~yc^YhE`CN)Kesk9bN_q7=9%F?Ij+?^^Ziq72}KBGQC zkFRq{7&~!zt6d&V!(!whVTsjF)zur><=Q|Wo{foHo~lIe1><^a8J;(-cg^y^NqOLe z)`SC^TF6knP4&T0fZi00W2f2?2jFoGvtIS(`Zv^k$Yt_GC6cN)1%9UX+(8Q}Fx8K* z?{w;3n0P!WxYdK8&BN&Pl%9VSwBXP3zy>>WwE5w{GtZ%w;bXVkFl|vyw;vwFNG=_LKF(wW{OWz z4d`$11bD9x$a<%mp;@c?1H4&;2j&CQU-~a;!GZcgh6T@uve5lyfoW%%F%2I4(=&RK zs)u^*iw0@+{v(JQQZeo~`7>rge=BCc;!_P*DF9EhF$ojid`w!B(3(-2U0UCGS-_*5 z7R`U(e{S1F;ii}A45a-l2`O}?{h@{aA=^iSFb#J06;n7T`3zTr5i7O*RsXMAvfQ%v z+??peX+D!)hVx8(a*+Y-HJk*Qqu&_=O#cHyK)sbdyKspPN1!&`aDW4PRb^}8%nrEF z2q?-2r)qYpvs3Bv)SLCspUs20(8q1UsCT#j03X;k(o8n8*rA>{AW$84j*W>4z4Vcd z$s+lKc-05_&uw01GInE%<_j#gH2$wEkV^s=j}p>phVD3W>CFjS(#6qu<4-sz=79R` zt-=d?4zCir1>5U2j|ZbUd*>qTmx;d5X=;oqBPcf+&V~b6wKl}Jo_lKzG2D?-g)58I zFz^%r*6J)-S`hjRhLHiOoxY{X&9eoI&*tF(x~rkK^Qt_f)m_m803Wvgo8+C7lRJ2a ze*s+EtW5gUVX>aA&66WrHI4UuN^iQRG zD7c9|l>y%JGK91*!PV+I+moq2gms!AObSJfJg&ZXntBJd_)HBZShyu}F5GhII~B_P zVyXGc@ukCy+fo0x>AJlCB7+$F{ees5W2D7wJ5H1k)@|WO_ags%jPl1tgl&wdpv1Ub zqp^m|Aw|@8LuNgm1G00==K#T?2&Q#(YBa3welT|sr3J1UET@SmQ8Gp*Svw7q7P%_A& zLB7eXs{V=f^Lu`BvkUD5_Pe{q@&M#?m{zeNBeaeDbAkul%eSKQj8BPO^61shofnS< z>h#5izhr&2S9D4a?Fsv2l&>{n`?9y^@Yl}jMA7wUy?+k(zkhyz>9&ejMeGHx1wznd zjJeO!S{XfZIpcCP2S4N8BeZe;1)7$uyO$9s?AMns+KO`)Tl0_BTk3F|^E}qg2{OB- z_hH%0^qO>LaTvKX9OQ9727G13ewlM=}SUIy?7cjje0 zIn^A^;R0VH2g>}-bWS)G!BauWkk(xlqII^*)fMD6>k?z&W9yBRstFZOj(Ja1mcG%6 zLtEx5jwY*@pD=a0SQGw@PP%gy2WDSineq^kuNU<7xzC%Ho!?U1&RKiU=Gs<}(mTX@ z{o$1>?u|vv8(V$MQW`wXM%xBo%;52AAD+%FNxnn53HHbuuR_S^kHAcGklBqQ}@#x{p`XC+JX8M=~Nvbj)-pn*8PtoX|2#z zv_|Q@&8GzHp+<$1`g@suy-#O!>ZRbPT|t2Hz&RM8*UtFoRiz9foOqHmMLcUkz$6Nx zrT^%#KW*L_l(itfG7u7$zQw|+h1!J(p8?QX0BmjQ2UmWIV$?R686Ck}Iz?9#<{<6@ z{7=rYZZO-!Ky&o^hOb6+G7hczqVim(W z^0=S~%%(FmvjPgS8UyCU6+^|P`50o!y@&Z%0n3T`mUbz2`6Q2vw+^^|FZS02^HX%Z)2*+{_bX<#LnXC?ukDB7 zi!Fx?e9oooz4zQZe~Wh~IXFXS1<+D;mDz!b5w(R_$dTj4#V{BE`a~zd5H56t0%3ql zlLSXG)1edGSGkR75zL-si2fQN*gpjp6Gy`w=?wm%tQ(Adk^oTNxIhH!p*M0ISWb+V z3PAxKYfS+q<13c?t0 zW4*=+<)%5Z!q&kcvMS_MJeUo@K!izT2uPj_>Ir}ZOJ)!!07BgBs6p-;KBg^?6sNg) zp7Aw_D4fqLZ^q46$E@Tv#xE-t-gx7FD_(bh`6Zx@X6NpQ6GC>M+ABMA_5#@>S)I*X zOl!{IOs5EGW;CC<-esihum4DsSW%@w86}!Y9AM9m(uN1%V@~T4c=B1JfW68-aN=bi zC|a8jAV;0maHy~#*jYU~M#XSjp-BkZ6@Gpc3@c6n8Iyzsl4V&$(IPDUJXeu1RnbuN zGAlW1fLoacNAX9nS?GPXSvUaBkVRl}#3&&26--XQrJaji#KAY6KM)N{H(-5n%c6>5%kX+eTfSJ@{Kv0y?U%K{OiAt%sh-1#%A z)cvV+rvvEtJY6Md%kQnI9o5N26#@Od5z!f3&b^9Ct?krxc@(9==~AWtq2 zK|-E5d?Dv3E=bPr#aW6G-?b7hW%`K7JMSh@r@Dru%YA&tA1242tulS|SX@G=#&_7D zv;DfbxSzwCP{GmOrH7fno`W<}0xA~ch6P-EZek-6=1{lK;I#l@!lJwmXUq{?1bJ5_ zI(m*F;6_>TfTuhmIt>tkP3+w#Xsu!xjsRw+QxSwyGGw$vn=Xh|XH-kle}-1lAMvJZ z(Jh{v=5f79r2!m#nd_#*fO#{T$rGoo&?XB)ObL=ogCMW%$?k1uzbWhdEl-81;iT=4 zuaEuKW6l+wS_mQ-U+LSJ(pGoWRD3#Vr%72q_?u>mpPihT&-FE2Di;YkvmNfbG5z;l z;;Ri3pDvy~xFAX8Q?8_Ik2inS`n}DSw=JLm^_&RlgZMY-0+IexF|tubI{;@`El}$W z0h+69ux2(q*_xkO+kO*TSrkXA!&}xDao)~IEl4RzK{yY+n_%lWh}ZLW3?=N50|4t=vm?Vm-S0yUCc_XWqnE>+RX8mcXtQ<;yB*w?Kf z)KET>Z1Q~$Q9(vbn#X$p~?%*50Kd5v*1{V&4de*cR?_9oFAN4rK z`M)34NgoV3RbenP%bXNT+*KAE-~#=VA%F{8ixqqo6ApSB-llSIFq*>*4e33{!LH(4 zgqHkx@5ES;Y0I?T^s=8%Yw&EJ()3kE<4ibMy+t32oy6Fq+DU@}r*<$A6oRxn8R<)b zGhl{`E}^A#`06h;oND<*Ebq^Np(;qDC z8LPQpeh1(Gh?FA-!n~wDn$PET3OW}Bk%Uf+q zSpP|TbP;0|yN8gp3UM8s&6G>a>WEC|x}YUJ{ut3&qV6bf3N6M6K2h`(&ci^r5SbI! zDhUo;;n~GedS_4C91DMl(G*w8NsD;=fSJ-+QX)TJDj#}ELG1~{LuPh>Pzik!KrejD zouqOl>5>iO2rh2!-PH=Ne5J}q$kD{}GFc|`Wxjd*!~H4dYUOtiD3)r)U07Tq`Q!|p zpL~ww!c}%yFXZKR_FaeFiK}D-&l11+3ct`(0XF4nYLx>a8x(is@h@3t^{{YLSjZb$kFK#~=yz5zRZ`bB zE9y6M*Yj#OJCAF(=4-#FLZ=0p2GUYO@79F%R~LE`kRahZvbvY_l{{^(dE3-NyQsrg zOt*)^-v#gWIMo&8)nHz!D&;2H&-o>&)x(4uua7_3)%V-D*C?Ie zDBIO2ztDL0un{BFq^RDcY}=&Dpd;irofT5zX@DsgCxjg6C^|Ll3ktBumHic{^$Mkk zE;P%{3nX1m`ue27%(lhiL5o#>i%nOH-9n4~VT*%MtCM=Gi*4(ze9=8ropEH$6?N#1{q0Ur9i^Ip0njs84!vBYk(E6Z@fwZk%Xz+ZH z|6^W^5pu8M@*gsHmOSY887oin$a0(Oe!+-Vg?gsdd*0agFp##h`91SpJ&OxHONKoK zpZ&aSJKl~f^i}7>SQUw(gYWKL;IDuj{4BT{h}KE;&%Or zBT?*)z6-e}MqPbpC;Lt&_9v^gGP&kCP#xf;UQCvAv3!ni?AW9f+5DDRJUNz{%FWSco_d~$jNT_PS~(}w}4B* zu=nDyukf(f(Qtsq$X)qPf4h<3f|1b0iu>Io;YTBc$*hOMqeQ#W7(4r@u+jML(Zor) yxW!Sj@K~Br&5`<8R@hjM5mQRRSi$00Q1@7&@OVksFhyg$ENuLm5pM(lsQ)i0FqVb@ literal 0 HcmV?d00001 From 3bca74cd1445445047c28f9af3283423e583da4b Mon Sep 17 00:00:00 2001 From: Dennis Zhuang Date: Tue, 25 Aug 2026 12:38:06 +0800 Subject: [PATCH 04/14] docs: clarify contributor mechanisms and tests --- .../datanode/data-persistence-indexing.md | 8 +-- .../datanode/metric-engine.md | 6 +- docs/contributor-guide/datanode/overview.md | 37 ++++++------ .../datanode/query-engine.md | 39 ++++-------- .../datanode/storage-engine.md | 2 +- docs/contributor-guide/datanode/wal.md | 20 +++++-- .../flownode/batching_mode.md | 6 +- docs/contributor-guide/frontend/overview.md | 42 +++++++++---- .../frontend/table-sharding.md | 23 +++++--- .../how-to/how-to-write-sdk.md | 50 ++++++++-------- docs/contributor-guide/metasrv/admin-api.md | 55 +++++++++++++++-- .../tests/integration-test.md | 15 +++-- docs/contributor-guide/tests/overview.md | 11 +++- docs/contributor-guide/tests/sqlness-test.md | 22 +++---- docs/contributor-guide/tests/unit-test.md | 19 +++--- .../datanode/data-persistence-indexing.md | 8 +-- .../datanode/metric-engine.md | 6 +- .../contributor-guide/datanode/overview.md | 32 +++++----- .../datanode/query-engine.md | 30 +++++----- .../datanode/storage-engine.md | 2 +- .../current/contributor-guide/datanode/wal.md | 20 +++++-- .../flownode/batching_mode.md | 6 +- .../contributor-guide/frontend/overview.md | 44 ++++++++++---- .../frontend/table-sharding.md | 20 ++++--- .../how-to/how-to-write-sdk.md | 46 ++++++++------- .../contributor-guide/metasrv/admin-api.md | 59 +++++++++++++++++-- .../tests/integration-test.md | 14 +++-- .../contributor-guide/tests/overview.md | 10 +++- .../contributor-guide/tests/sqlness-test.md | 16 +++-- .../contributor-guide/tests/unit-test.md | 16 +++-- 30 files changed, 433 insertions(+), 251 deletions(-) diff --git a/docs/contributor-guide/datanode/data-persistence-indexing.md b/docs/contributor-guide/datanode/data-persistence-indexing.md index bcf055fbad..64efb659ee 100644 --- a/docs/contributor-guide/datanode/data-persistence-indexing.md +++ b/docs/contributor-guide/datanode/data-persistence-indexing.md @@ -35,13 +35,13 @@ Apache Parquet file format provides inherent statistics in headers of column chu Column chunk header -For example, in the above Parquet file, if you want to filter rows where `name` = `Emily`, you can easily skip row group 0 because the max value for `name` field is `Charlie`. This statistical information reduces IO operations. +For example, a query filtering for `name` = `Emily` can skip row group 0 because the maximum `name` value is `Charlie`. This avoids reading that row group. ## Index Files -For each SST file, GreptimeDB not only maintains an internal index but also generates a separate file to store the index structures specific to that SST file. +When an SST has one or more configured index outputs, GreptimeDB writes them to a Puffin file associated with that SST. An SST with no applicable index does not need a Puffin file. -The index files utilize the [Puffin][3] format, which offers significant flexibility, allowing for the storage of additional metadata and supporting a broader range of index structures. +Puffin provides a container for index blobs and their metadata, allowing different index structures to share one file. ![Puffin](/puffin.png) @@ -61,7 +61,7 @@ The inverted index enables GreptimeDB to skip data segments that do not meet que ![Inverted index searching](/inverted-index-searching.png) -For instance, the query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches the regex `.*users`, and `status` matches the regex `4...`. It then scans these data segments to produce the final results that meet all conditions, significantly reducing the number of IO operations. +The query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches `.*users`, and `status` matches `4...`. It scans only those segments before applying the remaining filters. ### Inverted Index Format diff --git a/docs/contributor-guide/datanode/metric-engine.md b/docs/contributor-guide/datanode/metric-engine.md index 59d536392e..f74c29e8f2 100644 --- a/docs/contributor-guide/datanode/metric-engine.md +++ b/docs/contributor-guide/datanode/metric-engine.md @@ -7,9 +7,9 @@ description: Overview of the Metric engine in GreptimeDB, its concepts, architec ## Overview -The `Metric` engine is a component of GreptimeDB, and it's an implementation of the storage engine. It mainly targets scenarios with a large number of small tables for observable metrics. +The `Metric` engine stores workloads with many small metric tables. -Its main feature is to use synthetic physical wide tables to store a large amount of small table data, achieving effects such as reuse of the same column and metadata. This reduces storage overhead for small tables and improves columnar compression efficiency. The concept of a table becomes even more lightweight under the `Metric` engine. +It maps those logical tables onto shared physical wide tables so they can reuse columns and metadata. This reduces per-table storage overhead and improves columnar compression. ## Concepts @@ -19,7 +19,7 @@ The `Metric` engine introduces two new concepts: "logical table" and "physical t A logical table refers to user-defined tables. Just like any other ordinary table, its definition includes the name of the table, column definitions, index definitions etc. All operations such as queries or write-ins by users are based on these logical tables. Users don't need to worry about differences between logical and ordinary tables during usage. -From an implementation standpoint, a logical table is virtual; it doesn't directly read or write physical data but maps read/write requests into corresponding requests for physical tables in order to implement data storage and querying. +A logical table is virtual. The engine maps its read and write requests to the corresponding physical table instead of storing data for it directly. ### Physical Table diff --git a/docs/contributor-guide/datanode/overview.md b/docs/contributor-guide/datanode/overview.md index 83e9d3b889..4d3a1d26a0 100644 --- a/docs/contributor-guide/datanode/overview.md +++ b/docs/contributor-guide/datanode/overview.md @@ -7,27 +7,24 @@ description: Overview of Datanode in GreptimeDB, its responsibilities, component ## Introduction -`Datanode` is mainly responsible for storing the actual data for GreptimeDB. As we know, in GreptimeDB, -a `table` can have one or more `Region`s, and `Datanode` is responsible for managing the reading and writing -of these `Region`s. `Datanode` is not aware of `table` and can be considered as a `region server`. Therefore, -`Frontend` and `Metasrv` operate `Datanode` at the granularity of `Region`. +A Datanode stores and processes Region data. A table can contain multiple Regions, but the Datanode does not own table-level routing. Frontend sends data requests by Region, while Metasrv controls Region placement and lifecycle. -![Datanode](/datanode.png) +This boundary lets the same Region server host different storage engines without exposing their implementation to Frontend or Metasrv. ## Components -A `Datanode` contains all the components needed for a `region server`. Here we list some of the vital parts: - -- A gRPC service is provided for reading and writing region data, and `Frontend` uses this service - to read and write data from `Datanode`s. -- An HTTP service, through which you can obtain metrics, configuration information, etc., of the current node. -- `Heartbeat Task` is used to send heartbeat to the `Metasrv`. The heartbeat plays a crucial role in the - distributed architecture of GreptimeDB and serves as a basic communication channel for distributed coordination. - The upstream heartbeat messages contain important information such as the workload of a `Region`. If the - `Metasrv `has made scheduling(such as `Region` migration) decisions, it will send instructions to the - `Datanode` via downstream heartbeat messages. -- The `Datanode` does not parse user SQL or perform distributed planning. The user's query requests for one or - more `Table`s will be transformed into `Region` query requests in the `Frontend`. The `Datanode` is responsible - for executing these `Region` query plans with its local query engine. -- A Region server manages the lifecycle of all `Region`s on a `Datanode` and dispatches requests to the appropriate storage engine. -- GreptimeDB supports multiple Region engines. `Mito` is the primary time-series storage engine, `Metric` stores many logical metric tables in shared Mito Regions, and `File` provides access to external files. +The main components are: + +- The Region server tracks open Regions and dispatches reads, writes, and lifecycle requests to the engine registered for each Region. +- `Mito` is the primary time-series Region engine. `Metric` maps many logical metric Regions onto shared Mito Regions, and `File` exposes external files through the Region interface. +- The local query engine executes Region query plans. It does not parse client SQL or perform cluster-wide planning. +- The heartbeat task reports node and Region state to Metasrv and receives instructions such as open, close, upgrade, downgrade, and migration steps. +- gRPC carries Region requests to the Datanode. HTTP exposes node diagnostics such as metrics and configuration. + +## Region Request Lifecycle + +For a Mito write, the Region server selects Mito from the Region metadata. Mito appends the mutation to the WAL, applies it to a memtable, and later flushes the memtable to SST files. A Metric write is first rewritten with the logical-table identity and then delegated to its physical Mito Region. + +For a read, the local query engine executes the Region plan against a table provider backed by the Region engine. A Mito scan takes an immutable Region version, reads the relevant memtables and SST files, merges and deduplicates rows, and returns a stream of Arrow record batches. + +Region ownership can change without restarting the Datanode. Metasrv sends lifecycle instructions over the heartbeat stream; the Region server applies them to the engine and reports the new Region role and statistics in subsequent heartbeats. diff --git a/docs/contributor-guide/datanode/query-engine.md b/docs/contributor-guide/datanode/query-engine.md index cceec7a766..91f307aa59 100644 --- a/docs/contributor-guide/datanode/query-engine.md +++ b/docs/contributor-guide/datanode/query-engine.md @@ -7,37 +7,22 @@ description: Overview of GreptimeDB's query engine, its architecture, data repre ## Introduction -GreptimeDB's query engine is built on [Apache DataFusion][1] (subproject under [Apache -Arrow][2]), a brilliant query engine written in Rust. It provides a set of well functional components from -logical plan, physical plan and the execution runtime. Below explains how each component is orchestrated and their positions during execution. +GreptimeDB's query engine is built on [Apache DataFusion][1]. DataFusion supplies the logical and physical plan interfaces, optimizer framework, and execution runtime. GreptimeDB adds planners for its query languages, storage-aware optimizer rules, custom plan nodes, and distributed execution. -![Execution Procedure](/execution-procedure.png) +DDL and other control-plane operations are dispatched by the statement executor. The query engine receives plans for data processing, including the input side of operations such as `INSERT ... SELECT`. -The entry point is the logical plan, which is used as the general intermediate representation of a -query or execution logic etc. Two noticeable sources of logical plan are from: 1. the user query, like -SQL through SQL parser and planner; 2. the Frontend's distributed query, which is explained in details in the following section. +## Query Lifecycle -Next is the physical plan, or the execution plan. Unlike the logical plan which is a big -enumeration containing all the logical plan variants (except the special extension plan node), the -physical plan is in fact a trait that defines a group of methods invoked during -execution. All data processing logics are packed in corresponding structures that -implement the trait. They are the actual operations performed on the data, like -aggregator `MIN` or `AVG`, and table scan `SELECT ... FROM`. +1. The SQL, PromQL, or log-query planner resolves tables through the catalog and produces a DataFusion logical plan. GreptimeDB plan extensions represent operations that DataFusion does not provide directly. +2. DataFusion analyzer and optimizer rules run together with GreptimeDB rules. These rules normalize expressions and types, rewrite time-range operations, push projections and filters toward scans, and introduce distributed plan nodes when required. +3. The physical planner converts the optimized logical plan into streaming operators. GreptimeDB then applies physical rules for scan parallelism, ordering, and distributed execution. +4. Execution pulls Arrow record batches through the physical plan. Storage scans receive the projection and predicates, and downstream operators consume the resulting stream without materializing the complete result first. -The optimization phase which improves execution performance by transforming both logical and physical plans, is now all based on rules. It is also called, "Rule Based Optimization". Some of the rules are DataFusion native and others are customized in Greptime DB. In the future, we plan to add more -rules and leverage the data statistics for Cost Based Optimization/CBO. - -The last phase "execute" is a verb, stands for the procedure that reads data from storage, performs -calculations and generates the expected results. Although it's more abstract than previously mentioned concepts, you can just -simply imagine it as executing a Rust async function. And it's indeed a future (stream). - -`EXPLAIN [VERBOSE] ` is very useful if you want to see how your SQL is represented in the logical or physical plan. +Use [`EXPLAIN`](/reference/sql/explain.md) to inspect the logical and physical plans. `EXPLAIN ANALYZE` also executes the plan and reports runtime metrics. ## Data Representation -GreptimeDB uses [Apache Arrow][2] as the in-memory data representation. It's column-oriented, in -cross-platform format, and also contains many high-performance data operators. These features -make it easy to share data in many different environments and implement calculation logic. +GreptimeDB uses [Apache Arrow][2] record batches as its in-memory data representation. A record batch contains equal-length column arrays and a schema. Query operators exchange streams of these batches, which keeps the execution path columnar from Region scans through result encoding. ## Indexing @@ -45,9 +30,7 @@ Index construction and persistent index formats belong to the storage engine. Th ## Distributed Execution -Covered in [Distributed Querying][6]. +In distributed mode, the Frontend plans the cluster-wide query and Datanodes execute Region-local subplans. [`MergeScan`](../frontend/distributed-querying.md) is the boundary between those stages. -[1]: https://github.com/apache/arrow-datafusion +[1]: https://datafusion.apache.org/ [2]: https://arrow.apache.org/ -[3]: https://parquet.apache.org -[6]: ../frontend/distributed-querying.md diff --git a/docs/contributor-guide/datanode/storage-engine.md b/docs/contributor-guide/datanode/storage-engine.md index c220c72208..3ebb0d8af2 100644 --- a/docs/contributor-guide/datanode/storage-engine.md +++ b/docs/contributor-guide/datanode/storage-engine.md @@ -7,7 +7,7 @@ description: Overview of the storage engine in GreptimeDB, its architecture, com ## Introduction -The `storage engine` is responsible for storing the data of the database. Mito, based on [LSMT][1] (Log-structured Merge-tree), is the storage engine we use by default. We have made significant optimizations for handling time-series data scenarios, so mito engine is not suitable for general purposes. +Mito is GreptimeDB's default storage engine. It uses an [LSM tree][1] and is designed for time-series workloads rather than as a general-purpose embedded storage engine. ## Architecture diff --git a/docs/contributor-guide/datanode/wal.md b/docs/contributor-guide/datanode/wal.md index 92523d756c..bb6e733bae 100644 --- a/docs/contributor-guide/datanode/wal.md +++ b/docs/contributor-guide/datanode/wal.md @@ -7,17 +7,25 @@ description: Introduction to Write-Ahead Logging (WAL) in GreptimeDB, its purpos ## Introduction -Mito applies writes to an in-memory MemTable before the data is flushed to SST files. It first appends each Region's write operations to the write-ahead log (WAL), so data that has not reached an SST can be recovered. +Mito buffers writes in memtables before flushing them to SST files. It first appends each Region's mutations to the write-ahead log (WAL), so data that has not reached an SST can be recovered. -When a Region is reopened after a Datanode restart, Mito replays WAL entries after the last persisted sequence to rebuild its in-memory state. The WAL is accessed through a common log-store abstraction and can use local raft-engine storage or a remote Kafka cluster. +The WAL uses a common log-store abstraction with local raft-engine and remote Kafka providers. -![WAL in Datanode](/wal.png) +## Write and Recovery Cycle + +The order of a normal write is: + +1. The Region worker assigns sequence numbers and a WAL entry ID. +2. It appends the mutations to the WAL. If the append fails, the mutations are not applied to the memtable. +3. After the append succeeds, Mito writes the mutations to the memtable and publishes the new committed sequence. +4. A flush writes immutable SST files and persists a manifest edit containing the new files and `flushed_entry_id`. +5. After the manifest edit is durable, WAL entries through `flushed_entry_id` are marked obsolete. The log store may reclaim them later. + +The manifest is the recovery boundary. On a normal reopen, Mito rebuilds the Region from the manifest and replays WAL entries starting at `flushed_entry_id + 1`. Region transitions may supply a later replay checkpoint, but they never replay entries before the persisted flush boundary. ## Namespace -Namespace of WAL is used to separate entries from different tables (different regions). Append and -read operations must provide a Namespace. Currently, region ID is used as the Namespace, because -each region has a MemTable that needs to be reconstructed when Datanode restarts. +WAL entries are isolated by Region, not by table. Each append and read identifies a Region namespace so one Region can be replayed or truncated independently. The local raft-engine provider uses the Region ID as its namespace ID. Kafka keeps Region identity within the provider's topic-backed log. ## Synchronous/Asynchronous flush diff --git a/docs/contributor-guide/flownode/batching_mode.md b/docs/contributor-guide/flownode/batching_mode.md index 43b00ff4bd..4dc0c42195 100644 --- a/docs/contributor-guide/flownode/batching_mode.md +++ b/docs/contributor-guide/flownode/batching_mode.md @@ -46,8 +46,8 @@ A `BatchingTask` represents a single, independent data flow. Each task is associ ### `TaskState` and `DirtyTimeWindows` -- **`TaskState`**: This struct tracks the runtime state of a `BatchingTask`. It includes `dirty_time_windows`, which is crucial for determining what work needs to be done. -- **`DirtyTimeWindows`**: This is a key data structure that keeps track of which time windows have received new data since the last query execution. It stores a set of non-overlapping time ranges. When a task's execution loop runs, it consults this structure to build a `WHERE` clause that filters the source table for only the dirty time windows. +- **`TaskState`**: This struct tracks the runtime state of a `BatchingTask`, including the `dirty_time_windows` that determine its pending work. +- **`DirtyTimeWindows`**: This data structure tracks which time windows have received new data since the last query execution. It stores a set of non-overlapping time ranges. The execution loop uses it to build a `WHERE` clause that selects only the dirty windows from the source table. ### `TimeWindowExpr` @@ -56,7 +56,7 @@ The `TimeWindowExpr` is a helper utility for dealing with time window expression - **Evaluation**: It can take a timestamp and evaluate the time window expression to determine the start and end of the window that the timestamp falls into. - **Window Size**: It can also determine the size (duration) of the time window from the expression. -This is essential for both marking windows as dirty and for generating the correct filter conditions when querying the source table. +The same calculation is used to mark dirty windows and generate the source-table filters. ## Query Execution Flow diff --git a/docs/contributor-guide/frontend/overview.md b/docs/contributor-guide/frontend/overview.md index 3a2f63e7ae..18cabfc2b8 100644 --- a/docs/contributor-guide/frontend/overview.md +++ b/docs/contributor-guide/frontend/overview.md @@ -5,27 +5,47 @@ description: Overview of GreptimeDB's Frontend component - a stateless proxy ser # Frontend -The **Frontend** is a stateless service that serves as the entry point for client requests in GreptimeDB. It provides a unified interface for multiple database protocols and acts as a proxy that forwards read/write requests to appropriate Datanodes in the distributed system. +Frontend is GreptimeDB's stateless request-orchestration service. The server layer terminates protocols and converts wire messages; Frontend supplies the database behavior behind those handlers, including permission checks, statement execution, routing, and distributed query planning. + +Frontend does not store table data. It caches catalog and route metadata obtained from Metasrv, and Metasrv invalidates those caches through heartbeat responses when metadata changes. ## Core Functions -- **Protocol Support**: Multiple database protocols including SQL, PromQL, MySQL, and PostgreSQL. See [Protocols][1] for details -- **Request Routing**: Routes requests to appropriate Datanodes based on metadata -- **Query Distribution**: Splits distributed queries across multiple nodes -- **Response Aggregation**: Combines results from multiple Datanodes -- **Authorization**: Security and access control validation +- Provide query and ingestion behavior for the supported [protocols][1]. +- Resolve catalogs, schemas, tables, and Region routes. +- Validate permissions before executing a request. +- Plan distributed queries and merge results from Datanodes. +- Convert table-level writes and deletes into Region requests. ## Architecture ### Key Components -- **Protocol Handlers**: Handle different database protocols -- **Catalog Manager**: Caches metadata from Metasrv to enable efficient request routing and schema validation -- **Dist Planner**: Converts logical plans to distributed execution plans -- **Request Router**: Determines target Datanodes for each request + +- Protocol handlers adapt SQL, PromQL, gRPC ingestion, and observability protocols to Frontend's internal request interfaces. +- The catalog and partition managers provide table metadata, partition rules, and Region routes. +- The statement executor dispatches queries, DML, and DDL to their respective execution paths. +- The distributed planner replaces table scans with `MergeScan` plans that can run across Datanodes. ### Request Flow -![request flow](/request_flow.png) +The request path depends on the operation. + +#### Queries + +1. A protocol handler creates the query context and performs authentication and permission checks. +2. The language-specific planner produces a logical plan. In distributed mode, the planner uses partition metadata to select Regions and constructs a distributed plan. +3. Frontend sends Region subplans to the owning Datanodes. Datanodes execute them against local Region engines and return streams of Arrow record batches. +4. Frontend runs the remaining operators, merges the streams, and formats the result for the client protocol. + +#### Writes and deletes + +1. Frontend validates the request against the table schema. Protocols that support schema-on-write may create a missing table or add columns before retrying the write. +2. The partition rule assigns rows to Regions. Frontend builds one Region request per target and routes it to the current Region leader. +3. The Datanode's Region server dispatches each request to the Region engine. In standalone mode, the same request is sent to an embedded Region server instead of over RPC. + +#### DDL + +The statement executor converts DDL into a task. In distributed mode, Metasrv runs that task as a persisted procedure, updates metadata, and coordinates Region operations on Datanodes. Standalone mode uses the same statement boundary with local implementations of the metadata and procedure services. ### Deployment diff --git a/docs/contributor-guide/frontend/table-sharding.md b/docs/contributor-guide/frontend/table-sharding.md index a60276d14e..beaece40c0 100644 --- a/docs/contributor-guide/frontend/table-sharding.md +++ b/docs/contributor-guide/frontend/table-sharding.md @@ -5,21 +5,15 @@ description: Explains how table data in GreptimeDB is sharded and distributed, i # Table Sharding -The sharding of stored data is essential to any distributed database. This document will describe how table's data in GreptimeDB is being sharded, and distributed. +GreptimeDB shards a table into Regions. Partition expressions define which rows belong to each Region, while Region routes define which Datanode currently owns each Region. ## Partition -For the syntax of creating a partitioned table, please refer to the [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) section in the User Guide. +A partition is a logical row set described by an expression over one or more columns. The partition layout must cover the table's input domain so each row has one target Region. See [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) for the SQL syntax and supported expressions. ## Region -The data within a table is logically split after creating partitions. You may ask the question " -how are the data, after being logically partitioned, stored in the GreptimeDB? The answer is in "`Region`"s. - -Each region is corresponding to a partition, and stores the data in the partition. The regions are distributed among -`Datanode`s. `Metasrv` manages the route information that maps regions to Datanodes. -If the partition layout needs to change after table creation, GreptimeDB supports explicit -[repartitioning](/user-guide/deployments-administration/manage-data/repartition.md) through split and merge operations. +Each partition maps to one Region. Region IDs remain the storage and routing identity used by Frontend, Datanode, and Metasrv. Multiple Regions from the same table may be placed on one Datanode. The relationship between partition and region can be viewed as the following diagram: @@ -53,3 +47,14 @@ The relationship between partition and region can be viewed as the following dia │ │ └──────────────────────────────────┘ Could be placed in one Datanode +``` + +## Routing and Pruning + +For writes, Frontend evaluates the partition rule for each row, groups rows by Region, and sends Region requests to the current leaders from the route table. + +For queries, the distributed planner compares query predicates with the partition expressions. It scans only Regions that can satisfy the predicates. If partition metadata is missing or cannot be interpreted safely, the planner falls back to all Regions rather than risk omitting data. + +## Changing the Partition Layout + +[Repartitioning](/user-guide/deployments-administration/manage-data/repartition.md) changes an existing layout through explicit split and merge operations. Metasrv runs the change as a persisted procedure, updates the Region routes and partition expressions, and invalidates stale table-route caches. New requests use the published layout after their Frontend refreshes that metadata. diff --git a/docs/contributor-guide/how-to/how-to-write-sdk.md b/docs/contributor-guide/how-to/how-to-write-sdk.md index 40d0bc3713..0fd47f94e4 100644 --- a/docs/contributor-guide/how-to/how-to-write-sdk.md +++ b/docs/contributor-guide/how-to/how-to-write-sdk.md @@ -1,21 +1,17 @@ --- keywords: [gRPC SDK, GreptimeDatabase, Handle, HandleRequests, GreptimeRequest, GreptimeResponse] -description: Explains how to write a gRPC SDK for GreptimeDB, focusing on the GreptimeDatabase service, its methods, and the structure of requests and responses. +description: Protocol contracts and error-handling requirements for a GreptimeDB gRPC ingestion SDK. --- # How to write a gRPC SDK for GreptimeDB -A GreptimeDB gRPC SDK only needs to handle the writes. The reads are standard SQL and PromQL, can be handled by any JDBC -client or Prometheus client. This is also why GreptimeDB gRPC SDKs are all named -like "`greptimedb-ingester-`". Please make sure your GreptimeDB SDK follow the same naming convention. +GreptimeDB's public gRPC SDKs are ingestion clients. Queries normally use SQL or PromQL through their standard clients. A new SDK should therefore focus on writes and deletes unless it has a separate requirement, and follow the `greptimedb-ingester-` naming convention. See the [gRPC SDK overview](/user-guide/ingest-data/for-iot/grpc-sdks/overview.md) for the user-facing API. ## `GreptimeDatabase` Service -GreptimeDB defines a custom gRPC service called `GreptimeDatabase`. All you need to do in your SDK are implement it. You -can find its Protobuf -definitions [here](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto). +Generate client stubs from the [`GreptimeDatabase` Protobuf definition](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto). Do not maintain a handwritten copy of the messages or service definition. -The service contains two RPC methods: +The service provides a unary method and a client-streaming method: ```protobuf service GreptimeDatabase { @@ -25,13 +21,9 @@ service GreptimeDatabase { } ``` -The `Handle` method is for unary call: when a `GreptimeRequest` is received and processed by a GreptimeDB -server, it responds with a `GreptimeResponse` immediately. +`Handle` returns one response for one request. It is the usual choice for an SDK's insert and delete APIs. -The `HandleRequests` acts in -a "[Client streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc)" style. It ingests a -stream of `GreptimeRequest`, and handles them on the fly. After all the requests have been handled, it returns a -summarized `GreptimeResponse`. Through `HandleRequests`, we can achieve a very high throughput of requests handling. +`HandleRequests` is a [client-streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc). The server returns a cumulative response only after the client closes the request stream. An SDK that exposes streaming must document this acknowledgement boundary and bind the stream to one endpoint. ### `GreptimeRequest` @@ -51,13 +43,13 @@ message GreptimeRequest { } ``` -A `RequestHeader` is needed, it includes some context, authentication and others. The "oneof" field contains the request -to the GreptimeDB server. +A client must populate `RequestHeader` with the database context and authentication expected by the server. Set exactly one request variant. -Note that we have two types of insertions, one is in the form of "column" (the `InsertRequests`), and the other is " -row" (`RowInsertRequests`). It's generally recommended to use the "row" form, since it's more natural for insertions on -a table, and easier to use. However, if there's a need to insert a large number of columns at once, or there're plenty -of "null" values to insert, the "column" form is better to be used. +The message also contains query and DDL variants used by internal callers. The public ingester API should not expose them: `GreptimeDatabase` does not return query result streams. + +GreptimeDB accepts row-oriented `RowInsertRequests` and column-oriented `InsertRequests`. Row-oriented requests are the default for public ingestion APIs. A column-native client may use the column form, but it must keep column lengths consistent and preserve null values, timestamp precision, data types, and column semantic types during conversion. + +Deletes have the same row-oriented and column-oriented distinction. Expose only the forms that the SDK can map without losing type information. ### `GreptimeResponse` @@ -70,8 +62,18 @@ message GreptimeResponse { } ``` -The `ResponseHeader` contains the response's status code, and error message (if there's any). The "oneof" response only -contains the affected rows for now. +On success, the response contains a successful header and `affected_rows`. Treat that value as the number acknowledged by the server, including the cumulative value returned when a request stream closes. + +Request failures are returned as a gRPC status. When present, the trailing metadata keys `x-greptime-err-code` and `x-greptime-err-retry-hint` carry GreptimeDB's error code and retry classification. Preserve the gRPC status and expose the GreptimeDB metadata rather than replacing them with a generic SDK error. + +## Retry and Delivery Semantics + +Retries must be bounded and observable. A unary request may be retried only when the failure is classified as retryable and the deadline still permits it. Do not retry cancellation or deadline-expiration errors. + +A lost response does not prove that the server rejected a write. Retrying such a request can insert duplicate rows unless the caller's data model makes the operation idempotent. Document this possibility and return the final error when delivery is ambiguous. + +Do not transparently retry a partially sent `HandleRequests` stream. The server may already have accepted some requests even though the client has not received the cumulative response. Close the failed stream and report the ambiguity to the caller. + +Keep Arrow Flight bulk ingestion separate from the `GreptimeDatabase` RPCs. Its batching and partial-acceptance behavior needs its own API contract. -GreptimeDB has a lot of SDKs now, you can refer to -them [here](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=) for some examples. +Use the existing [GreptimeDB ingester repositories](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=) to compare public API conventions, but derive wire behavior from the current Protobuf definition and server contract. diff --git a/docs/contributor-guide/metasrv/admin-api.md b/docs/contributor-guide/metasrv/admin-api.md index ee8664b821..0b50d6733f 100644 --- a/docs/contributor-guide/metasrv/admin-api.md +++ b/docs/contributor-guide/metasrv/admin-api.md @@ -1,6 +1,6 @@ --- -keywords: [admin api, health check, leader query, heartbeat, maintenance mode, RESTful API] -description: Details the Admin API for Metasrv, including endpoints for health checks, leader queries, heartbeat data, maintenance mode, and Procedure Manager controls. +keywords: [admin api, health check, leader query, heartbeat, maintenance mode, recovery mode, table id sequence] +description: Details the Metasrv Admin API for status inspection, cluster controls, and metadata recovery. --- # Admin API @@ -9,14 +9,17 @@ description: Details the Admin API for Metasrv, including endpoints for health c Note that all Admin API endpoints in this document listen on Metasrv's `HTTP_PORT`, which defaults to `4000`. ::: -The Admin API exposes Metasrv health, leader, Datanode heartbeat, maintenance mode, and Procedure Manager information over HTTP. It does not provide authentication, and some endpoints change cluster behavior. Deployments must protect the HTTP port with network-level controls. +The Admin API exposes Metasrv status, cluster controls, and metadata recovery operations over HTTP. It does not provide authentication, and some endpoints change cluster behavior or metadata allocation. Deployments must protect the HTTP port with network-level controls. This page covers the following APIs: - /health - /leader - /heartbeat +- /node-lease - /maintenance - /procedure-manager +- /recovery +- /sequence/table All these APIs are under the parent resource `/admin`. @@ -118,9 +121,17 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100' ] ``` +## /node-lease HTTP endpoint + +The `/node-lease` endpoint returns the current leases recorded for Datanodes. Use it when diagnosing whether Metasrv still considers a Datanode active. + +```bash +curl -X GET http://localhost:4000/admin/node-lease +``` + ## /maintenance HTTP endpoint -Cluster Maintenance Mode is a safety feature in GreptimeDB that temporarily disables automatic cluster management operations. This mode is particularly useful during cluster upgrades, planned downtime, and any operation that might temporarily affect cluster stability. For more details, please refer to [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md). +Maintenance mode temporarily disables automatic cluster management operations during upgrades, planned downtime, or similar work. See [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md) for its effect on the cluster. The `/maintenance` endpoint supports the following HTTP requests: @@ -153,3 +164,39 @@ The response body uses the following format: "status": "running" } ``` + +## /recovery HTTP endpoints + +Recovery mode gates metadata repair endpoints such as manual table ID sequence changes. It is intended for recovery work, not routine maintenance. + +- `GET /admin/recovery/status`: query whether recovery mode is enabled. +- `POST /admin/recovery/enable`: enable recovery mode. +- `POST /admin/recovery/disable`: disable recovery mode. + +The response body uses the following format: + +```json +{ + "enabled": true +} +``` + +Disable recovery mode after the repair is complete. Use [maintenance mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md) instead when the goal is to suspend automatic cluster operations during planned maintenance. + +## /sequence/table HTTP endpoints + +These endpoints inspect or repair the table ID sequence: + +- `GET /admin/sequence/table/next-id`: return the next table ID without allocating it. +- `POST /admin/sequence/table/set-next-id`: advance the next table ID. + +Setting the sequence requires recovery mode. The new value must be greater than the current value; the API cannot move the sequence backwards. + +```bash +curl -X POST \ + -H 'Content-Type: application/json' \ + -d '{"next_table_id": 2048}' \ + http://localhost:4000/admin/sequence/table/set-next-id +``` + +Changing this value affects IDs allocated to future tables. Use the endpoint only when repairing metadata after confirming the required next ID. diff --git a/docs/contributor-guide/tests/integration-test.md b/docs/contributor-guide/tests/integration-test.md index 5d5f6cb1a5..bc52ee4854 100644 --- a/docs/contributor-guide/tests/integration-test.md +++ b/docs/contributor-guide/tests/integration-test.md @@ -7,7 +7,14 @@ description: Guide on writing and running integration tests in GreptimeDB, cover ## Introduction -Integration testing is written with Rust test harness (`#[test]`), unlike unit testing, they are placed separately -[here](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration). -It covers scenarios involving multiple components, in which one typical case is HTTP/gRPC-related features. You can check -its [documentation](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md) for more information. +Integration tests cover behavior that crosses crate or service boundaries, such as HTTP and gRPC handling, distributed components, or external storage. They use Rust's test harness and live in the [`tests-integration`](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration) package. + +Run the package with: + +```shell +cargo nextest run -p tests-integration +``` + +Some cases require environment variables or fixtures for external services. Follow the package's [setup instructions](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md) before running those cases. + +Use an integration test when a crate-level test or a Sqlness case cannot exercise the required boundary. Keep isolated logic in unit tests so that failures remain fast to reproduce. diff --git a/docs/contributor-guide/tests/overview.md b/docs/contributor-guide/tests/overview.md index feeefe2b6d..b241f2f078 100644 --- a/docs/contributor-guide/tests/overview.md +++ b/docs/contributor-guide/tests/overview.md @@ -5,4 +5,13 @@ description: Overview of the testing methods used in GreptimeDB to ensure its be # Tests -Our team has conducted lots of tests to ensure the behaviours of `GreptimeDB` . This chapter will introduce several significant methods used to test `GreptimeDB`, and how to work with them. +Choose the narrowest test that exercises the behavior you changed: + +| Test type | Use it for | Typical command | +| --- | --- | --- | +| [Unit test](unit-test.md) | Logic contained within one crate or component | `cargo nextest run -p ` | +| [Sqlness test](sqlness-test.md) | SQL, protocol, planner, execution, and end-to-end regressions | `cargo sqlness bare -t ` | +| [Integration test](integration-test.md) | Behavior that crosses components or requires external services | `cargo nextest run -p tests-integration` | +| [Compatibility test](https://github.com/GreptimeTeam/greptimedb/blob/main/tests/compatibility/README.md) | Reading data or metadata written by an older release | `cargo run -p sqlness-runner -- compat --from-version ` | + +Run `make test` when a change needs the full Rust workspace test suite. Changes to persisted metadata, WAL records, SST files, or wire formats should also include a compatibility test when an older release may have produced the input. diff --git a/docs/contributor-guide/tests/sqlness-test.md b/docs/contributor-guide/tests/sqlness-test.md index 6712c3413b..0262ff6851 100644 --- a/docs/contributor-guide/tests/sqlness-test.md +++ b/docs/contributor-guide/tests/sqlness-test.md @@ -7,42 +7,34 @@ description: Instructions for running SQL tests in GreptimeDB using the `sqlness ## Introduction -SQL is an important user interface for `GreptimeDB`. We have a separate test suite for it (named `sqlness`). +Sqlness is GreptimeDB's end-to-end regression suite for SQL and protocol behavior. A case sends statements to a running GreptimeDB instance and compares the output with a checked-in result file. ## Sqlness manual ### Case file -Sqlness has two types of file +Each case uses two files: - `.sql`: test input, SQL only - `.result`: expected test output, SQL and its results -The `.result` file is the expected execution output. If you see `.result` files changed, -it means the test gets a different result and indicates it may fail. You should -check the change logs to solve the problem. - -You only need to write test SQL in the `.sql` file, and run the test. +Write the input in the `.sql` file and run the test to generate or update `.result`. Review every result diff: accept it only when the behavior change is intended. ### Case organization -The root dir of input cases is `tests/cases`. It contains several sub-directories stand for different test -modes. E.g., `standalone/` contains all the tests to run under `greptimedb standalone start` mode. +Input cases live under `tests/cases`. The first directory level selects an environment. For example, `standalone/` runs against a standalone GreptimeDB instance. -Under the first level of sub-directory (e.g. the `cases/standalone`), you can organize your cases as you like. -Sqlness walks through every file recursively and runs them. +Within an environment, group a new case with the feature it exercises. Sqlness discovers case files recursively. ## Run the test -Unlike other tests, this harness is in a binary target form. You can run it with +Run the suite with: ```shell cargo sqlness bare ``` -It automatically finishes the following procedures: compile `GreptimeDB`, start it, grab tests and feed it to -the server, then collect and compare the results. You only need to check whether any `.result` files changed. -If no unexpected result changes remain, the test passed. +The command builds and starts GreptimeDB, runs the selected cases, and compares their output. A changed `.result` file is part of the review, not proof that the new output is correct. ### Run a specific test diff --git a/docs/contributor-guide/tests/unit-test.md b/docs/contributor-guide/tests/unit-test.md index 82e30bf5fa..183a72aab4 100644 --- a/docs/contributor-guide/tests/unit-test.md +++ b/docs/contributor-guide/tests/unit-test.md @@ -8,25 +8,26 @@ description: Guide on writing and running unit tests in GreptimeDB using Rust's ## Introduction Unit tests are embedded into the codebase, usually placed next to the logic being tested. -They are written using Rust's `#[test]` attribute and can run with `cargo nextest run`. +They are written using Rust's `#[test]` attribute. GreptimeDB uses [`cargo-nextest`](https://nexte.st/) as its primary Rust test runner. -The default test runner ships with `cargo` is not supported in GreptimeDB codebase. It's recommended -to use [`nextest`](https://nexte.st/) instead. You can install it with +Install it with: ```shell cargo install cargo-nextest --locked ``` -And run the tests (here the `--workspace` is not necessary) +Run the package you changed first: ```shell -cargo nextest run +cargo nextest run -p ``` -Notes if your Rust is installed via `rustup`, be sure to install `nextest` with `cargo` rather -than the package manager like `homebrew`. Otherwise it will mess up your local environment. +Use a test name or nextest filter to narrow the run further while developing. Before submitting a change with broad effects, run the full workspace suite: + +```shell +make test +``` ## Coverage -Our continuous integration (CI) jobs have a "coverage checking" step. It will report how many -codes are covered by unit tests. Please add the necessary unit test to your patch. +CI reports unit-test coverage. Add tests for changed behavior and failure cases that could otherwise regress; coverage percentage alone is not the goal. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md index 21ac66cf5a..09df007a18 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md @@ -34,14 +34,14 @@ Apache Parquet 文件格式在列块和数据页的头部提供了内置的统 Column chunk header -例如，在上述 Parquet 文件中，如果你想要过滤 `name` 等于 `Emily` 的行，你可以轻松跳过行组 0，因为 `name` 字段的最大值是 `Charlie`。这些统计信息减少了 IO 操作。 +例如，查询 `name` 等于 `Emily` 的行时，可以跳过 row group 0，因为其中 `name` 的最大值是 `Charlie`，无需读取该 row group。 ## 索引文件 -对于每个 SST 文件，GreptimeDB 不但维护 SST 文件内部索引，还会单独生成一个文件用于存储针对该 SST 文件的索引结构。 +当一个 SST 存在已配置且适用的索引输出时，GreptimeDB 将这些索引写入与该 SST 关联的 Puffin 文件。没有适用索引的 SST 不需要生成 Puffin 文件。 -索引文件采用 [Puffin][3] 格式，这种格式具有较大的灵活性，能够存储更多的元数据，并支持更多的索引结构。 +Puffin 是索引 Blob 及其元数据的容器，使不同索引结构可以共用一个文件。 ![Puffin](/puffin.png) @@ -62,7 +62,7 @@ GreptimeDB 会将多种索引结构作为 Blob 存储在 Puffin 文件中，包 ![Inverted index searching](/inverted-index-searching.png) -例如，上述查询使用倒排索引来定位数据段，数据段满足条件：`job` 等于 `apiserver`，`handler` 符合正则匹配 `.*users` 及 `status` 符合正则匹配 `4..`，然后扫描这些数据段以产生满足所有条件的最终结果，从而显着减少 IO 操作的次数。 +上述查询使用倒排索引定位 `job` 等于 `apiserver`、`handler` 匹配 `.*users` 且 `status` 匹配 `4..` 的数据段。Mito 只扫描这些数据段，再应用剩余过滤条件。 ### 倒排索引格式 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md index 247354a3f0..7a99edd478 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md @@ -7,9 +7,9 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述 ## 概述 -`Metric` 引擎是 GreptimeDB 的一个组件，属于存储引擎的一种实现，主要针对可观测 metrics 等存在大量小表的场景。 +`Metric` 引擎用于存储包含大量小型指标表的负载。 -它的主要特点是利用合成的物理宽表来存储大量的小表数据，实现相同列复用和元数据复用等效果，从而达到减少小表的存储开销以及提高列式压缩效率等目标。表这一概念在 `Metric` 引擎下变得更更加轻量。 +它将这些逻辑表映射到共享的物理宽表，使其复用列和元数据，从而降低每张表的存储开销并改善列式压缩。 ## 概念 @@ -18,7 +18,7 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述 ### 逻辑表 逻辑表，即用户定义的表。与普通的表都完全一样，逻辑表的定义包括表的名称、列的定义、索引的定义等。用户的查询、写入等操作都是基于逻辑表进行的。用户在使用过程中不需要关心逻辑表和普通表的区别。 -从实现层面来说，逻辑表是一个虚拟的表，它并不直接读写物理的数据，而是通过将读写请求映射成对应物理表的请求来实现数据的存储与查询。 +逻辑表是虚拟表，本身不直接存储数据。Metric 引擎将其读写请求映射为对应物理表的请求。 ### 物理表 物理表是真实存储数据的表，它拥有若干个由分区规则定义的物理 Region。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md index 4c4e53c9e7..75c9e005d0 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md @@ -7,22 +7,24 @@ description: 介绍了 Datanode 的主要职责和组件，包括 gRPC 服务、 ## Introduction -`Datanode` 主要的职责是为 GreptimeDB 存储数据，我们知道在 GreptimeDB 中一个 `table` 可以有一个或者多个 `Region`, -而 `Datanode` 的职责便是管理这些 `Region` 的读写。`Datanode` 不感知 `table`，可以认为它是一个 `region server`。 -所以 `Frontend` 和 `Metasrv` 按照 `Region` 粒度来操作 `Datanode`。 +Datanode 存储并处理 Region 数据。一张表可以包含多个 Region，但 Datanode 不负责表级路由。Frontend 按 Region 发送数据请求，Metasrv 则控制 Region 的放置和生命周期。 -![Datanode](/datanode.png) +这个边界使同一个 Region server 可以承载不同的存储引擎，而不向 Frontend 或 Metasrv 暴露引擎实现。 ## Components -一个 datanode 包含了 region server 所需的全部组件。这里列出了比较重要的部分： - -- 一个 gRPC 服务来提供对 `Region` 数据的读写，`Frontend` 便是使用这个服务来从 `Datanode` 读写数据。 -- 一个 HTTP 服务，可以通过它来获得当前节点的 metrics、配置信息等 -- `Heartbeat Task` 用来向 `Metasrv` 发送心跳，心跳在 GreptimeDB 的分布式架构中发挥着至关重要的作用， - 是分布式协调和调度的基础通信通道，心跳的上行消息中包含了重要信息比如 `Region` 的负载，如果 `Metasrv` 做出了调度 - 决定（比如 Region 转移），它会通过心跳的下行消息发送指令到 `Datanode` -- `Datanode` 不负责解析用户 SQL 或进行分布式规划，用户对一个或多个 `Table` 的查询请求会在 `Frontend` 中被转换为 - `Region` 查询请求，`Datanode` 负责用本地 query engine 执行这些 `Region` 查询计划 -- Region server 管理 Datanode 上所有 Region 的生命周期，并把请求分发给相应的存储引擎。 -- GreptimeDB 支持多种 Region engine。`Mito` 是主要的时序存储引擎；`Metric` 将多个逻辑指标表存储在共享的 Mito Region 中；`File` 用于访问外部文件。 +Datanode 包含以下主要组件： + +- Region server 记录已打开的 Region，并把读写和生命周期请求分发给该 Region 注册的 engine。 +- `Mito` 是主要的时序 Region engine。`Metric` 将多个逻辑指标 Region 映射到共享的 Mito Region，`File` 通过 Region 接口访问外部文件。 +- 本地 query engine 执行 Region 查询计划。它不解析客户端 SQL，也不进行集群级规划。 +- Heartbeat task 向 Metasrv 上报节点和 Region 状态，并接收 open、close、upgrade、downgrade 和迁移步骤等指令。 +- gRPC 承载发往 Datanode 的 Region 请求；HTTP 提供 metrics 和配置等节点诊断信息。 + +## Region 请求生命周期 + +Mito 写入到达 Region server 后，Region server 根据 Region 元数据选择 Mito。Mito 将 mutation 追加到 WAL，写入 memtable，并在之后把 memtable flush 为 SST 文件。Metric 写入会先补充逻辑表标识，再委托给对应的物理 Mito Region。 + +读取时，本地 query engine 在 Region engine 提供的 table provider 上执行 Region 计划。Mito scan 获取不可变的 Region version，读取相关 memtable 和 SST 文件，合并并去重数据，最后返回 Arrow record batch 流。 + +Region 所有权可以在不重启 Datanode 的情况下改变。Metasrv 通过心跳流下发生命周期指令；Region server 将指令应用到对应 engine，并在后续心跳中上报新的 Region role 和统计信息。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/query-engine.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/query-engine.md index ca406cc235..84445759d4 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/query-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/query-engine.md @@ -7,33 +7,33 @@ description: 介绍了 GreptimeDB 的查询引擎架构，基于 Apache DataFusi ## 介绍 -GreptimeDB 的查询引擎是基于[Apache DataFusion][1]（属于[Apache Arrow][2]的子项目）构建的，它是一个用 Rust 编写的出色的查询引擎。它提供了一整套功能齐全的组件，从逻辑计划、物理计划到执行运行时。下面将解释每个组件如何被整合在一起，以及在执行过程中它们的位置。 +GreptimeDB 的查询引擎基于 [Apache DataFusion][1]。DataFusion 提供逻辑计划、物理计划、优化器框架和执行运行时；GreptimeDB 在此基础上增加各查询语言的 planner、存储相关优化规则、自定义计划节点和分布式执行。 -![Execution Procedure](/execution-procedure.png) +DDL 和其他控制面操作由 statement executor 分发。Query engine 接收数据处理计划，包括 `INSERT ... SELECT` 等操作中读取输入数据的部分。 -入口点是逻辑计划，它被用作查询或执行逻辑等的通用中间表示。逻辑计划的两个主要来源是：1. 用户查询，例如通过 SQL 解析器和规划器的 SQL；2. Frontend 的分布式查询，这将在下一节中详细解释。 +## 查询生命周期 -接下来是物理计划，或称执行计划。与包含所有逻辑计划变体（除特殊扩展计划节点外）的大型枚举的逻辑计划不同，物理计划实际上是一个定义了在执行过程中调用的一组方法的特性。所有数据处理逻辑都包装在实现该特性的相应结构中。它们是对数据执行的实际操作，如聚合器 `MIN` 或 `AVG` ，以及表扫描 `SELECT ... FROM`。 +1. SQL、PromQL 或日志查询 planner 通过 catalog 解析表，并生成 DataFusion logical plan。DataFusion 不直接支持的操作由 GreptimeDB plan extension 表示。 +2. DataFusion 的 analyzer 和 optimizer rule 与 GreptimeDB rule 共同运行。这些规则规范化表达式和类型、改写时间范围操作、将 projection 和 filter 下推到 scan，并在需要时引入分布式计划节点。 +3. Physical planner 将优化后的 logical plan 转换为流式 operator。GreptimeDB 随后应用 scan 并行度、排序和分布式执行相关的 physical rule。 +4. 执行阶段通过 physical plan 拉取 Arrow record batch。存储 scan 接收 projection 和 predicate，下游 operator 消费数据流，无需先物化完整结果。 -优化阶段通过转换逻辑计划和物理计划来提高执行性能，现在全部基于规则。它也被称为“基于规则的优化”。一些规则是 DataFusion 原生的，其他一些是在 GreptimeDB 中自定义的。在未来，我们计划添加更多规则，并利用数据统计进行基于成本的优化 (CBO)。 - -最后一个阶段"执行"是一个动词，代表从存储读取数据、进行计算并生成预期结果的过程。虽然它比之前提到的概念更抽象，但你可以简单地将它想象为执行一个 Rust 异步函数，并且它确实是一个异步流。 - -当你想知道 SQL 是如何通过逻辑计划或物理计划中表示时，`EXPLAIN [VERBOSE] ` 是非常有用的。 +使用 [`EXPLAIN`](/reference/sql/explain.md) 查看逻辑和物理计划。`EXPLAIN ANALYZE` 还会执行计划并报告运行时指标。 ## 数据表示 -GreptimeDB 使用 [Apache Arrow][2]作为内存中的数据表示格式。它是面向列的，以跨平台格式，也包含许多高性能的基础操作。这些特性使得在许多不同的环境中共享数据和实现计算逻辑变得容易。 +GreptimeDB 使用 [Apache Arrow][2] record batch 作为内存数据表示。一个 record batch 包含等长的列数组和 schema。查询 operator 交换这些 batch 组成的数据流，使 Region scan 到结果编码的执行路径保持列式处理。 ## 索引 -索引构建和持久化格式属于存储引擎。查询层向扫描提供谓词和投影，Mito 再利用时间范围、Parquet 统计信息和索引跳过不可能匹配的数据。参见[数据持久化与索引](./data-persistence-indexing.md)。 +索引构建和持久化格式属于存储引擎。查询层向 scan 提供 predicate 和 projection，Mito 再利用时间范围、Parquet 统计信息和索引跳过不可能匹配的数据。参见[数据持久化与索引](./data-persistence-indexing.md)。 + + -## 分布式查询 +## 分布式执行 -参考 [Distributed Querying][6]. +分布式模式下，Frontend 规划集群级查询，Datanode 执行 Region 本地子计划。[`MergeScan`][6] 是两个阶段之间的边界。 -[1]: https://github.com/apache/arrow-datafusion +[1]: https://datafusion.apache.org/ [2]: https://arrow.apache.org/ -[3]: https://parquet.apache.org [6]: ../frontend/distributed-querying.md diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md index 67f98216fe..c63e3bea1b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md @@ -7,7 +7,7 @@ description: 详细介绍了 GreptimeDB 的存储引擎架构、数据模型和 ## 概述 -`存储引擎` 负责存储数据库的数据。Mito 是我们默认使用的存储引擎，基于 [LSMT][1]（Log-structured Merge-tree）。我们针对处理时间序列数据的场景做了很多优化，因此 mito 这个存储引擎并不适用于通用用途。 +Mito 是 GreptimeDB 的默认存储引擎，基于 [LSM tree][1]，面向时间序列负载设计，而不是通用的嵌入式存储引擎。 ## 架构 下图展示了存储引擎的架构和处理数据的流程。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/wal.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/wal.md index 937cc9c3d7..805b294091 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/wal.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/wal.md @@ -9,16 +9,26 @@ description: 介绍了 GreptimeDB 的预写日志（WAL）机制，包括其命 ## 介绍 -Mito 在数据 flush 到 SST 文件前，先把写入应用到内存中的 MemTable。每个 Region 的写操作会先追加到预写日志（WAL），从而恢复尚未进入 SST 的数据。 +Mito 在将数据 flush 为 SST 文件前，先在 memtable 中缓冲写入。每个 Region 的 mutation 会先追加到预写日志（WAL），从而恢复尚未进入 SST 的数据。 -Datanode 重启并重新打开 Region 时，Mito 会重放最后一个已持久化 sequence 之后的 WAL 条目，重建内存状态。WAL 通过统一的 log-store 抽象访问，可以使用本地 raft-engine 或远端 Kafka。 +WAL 通过统一的 log-store 抽象访问，可以使用本地 raft-engine 或远端 Kafka。 -![WAL in Datanode](/wal.png) +## 写入与恢复流程 + +正常写入遵循以下顺序： + +1. Region worker 分配 sequence number 和 WAL entry ID。 +2. 将 mutation 追加到 WAL。追加失败时，不会把 mutation 写入 memtable。 +3. WAL 追加成功后，Mito 将 mutation 写入 memtable，并发布新的 committed sequence。 +4. Flush 将不可变 memtable 写为 SST 文件，并持久化包含新文件和 `flushed_entry_id` 的 manifest edit。 +5. Manifest edit 持久化后，`flushed_entry_id` 及以前的 WAL entry 被标记为 obsolete；log store 可以稍后再回收物理空间。 + +Manifest 是恢复边界。正常重新打开 Region 时，Mito 根据 manifest 重建 Region，并从 `flushed_entry_id + 1` 开始重放 WAL。Region 状态切换可以指定更晚的 replay checkpoint，但不会重放早于已持久化 flush 边界的 entry。 ## 命名空间 -WAL 的命名空间用于区分来自不同 region 的条目。追加和读取操作必须提供一个命名空间。目前，region ID 被用作命名空间，因为每个 region 都有一个在数据节点重新启动时需要重构的 MemTable。 +WAL entry 按 Region 隔离，而不是按表隔离。追加和读取都需要指定 Region namespace，使单个 Region 可以独立重放或截断。本地 raft-engine 使用 Region ID 作为 namespace ID；Kafka provider 则在基于 topic 的日志中保留 Region 标识。 ## 同步/异步刷盘 -对于本地 raft-engine，`sync_write` 控制追加写是否等待日志同步到持久化存储，默认值为 `false`。异步写入延迟较低，但主机在缓冲数据同步前故障时，可能丢失最近确认的日志。Kafka WAL 的持久性由 producer 和集群配置决定，不受这个本地选项控制。 +对于本地 raft-engine，`sync_write` 控制追加写是否等待日志同步到持久化存储，默认值为 `false`。异步写入延迟较低，但主机在缓冲数据同步前故障时，可能丢失最近确认的 entry。Kafka WAL 的持久性由 producer 和集群配置决定，不受这个本地选项控制。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md index 0cf9c2edd0..f725aaba6b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md @@ -46,8 +46,8 @@ description: Flownode 批处理模式概述，这是持续数据聚合当前使 ### `TaskState` 和 `DirtyTimeWindows` -- **`TaskState`**: 此结构体跟踪 `BatchingTask` 的运行时状态。它包括 `dirty_time_windows`，这对于确定需要完成哪些操作至关重要。 -- **`DirtyTimeWindows`**: 这是一个关键的数据结构，用于跟踪自上次查询执行以来哪些时间窗口接收到了新数据。它存储一组不重叠的时间范围。当任务的执行循环运行时，它会参考此结构来构建一个 `WHERE` 子句，该子句仅过滤源表中的脏时间窗口。 +- **`TaskState`**: 此结构体跟踪 `BatchingTask` 的运行时状态，包括用于确定待处理工作的 `dirty_time_windows`。 +- **`DirtyTimeWindows`**: 此数据结构跟踪上次查询执行后接收到新数据的时间窗口，并保存一组不重叠的时间范围。执行循环根据它构造 `WHERE` 子句，只从源表选择脏窗口。 ### `TimeWindowExpr` @@ -56,7 +56,7 @@ description: Flownode 批处理模式概述，这是持续数据聚合当前使 - **求值**: 它可以接受一个时间戳并对时间窗口表达式求值，以确定该时间戳所属窗口的开始和结束。 - **窗口大小**: 它还可以从表达式中确定时间窗口的大小（持续时间）。 -这对于标记窗口为脏以及在查询源表时生成正确的过滤条件都至关重要。 +标记脏窗口和生成源表过滤条件使用同一套计算。 ## 查询执行流程 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/overview.md index ee48da3ce3..6befdc55a9 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/overview.md @@ -5,31 +5,51 @@ description: GreptimeDB Frontend 组件概述 - 为客户端请求提供服务 # Frontend -**Frontend** 是一个无状态服务，作为 GreptimeDB 中客户端请求的入口点。它为多种数据库协议提供统一接口，并充当代理，将读写请求转发到分布式系统中的相应 Datanode。 +Frontend 是 GreptimeDB 中负责请求编排的无状态服务。Server 层负责终止协议并转换线上消息；Frontend 为这些协议处理器提供数据库行为，包括权限检查、语句执行、路由和分布式查询规划。 + +Frontend 不存储表数据。它缓存从 Metasrv 获取的 catalog 和路由元数据；元数据发生变化时，Metasrv 通过心跳响应通知 Frontend 失效相应缓存。 ## 核心功能 -- **协议支持**：支持多种数据库协议，包括 SQL、PromQL、MySQL 和 PostgreSQL。详见[协议][1] -- **请求路由**：基于元数据将请求路由到相应的 Datanode -- **查询分发**：将分布式查询拆分到多个节点 -- **响应聚合**：合并来自多个 Datanode 的结果 -- **认证授权**：安全和访问控制验证 +- 为支持的[协议][1]提供查询和写入行为。 +- 解析 catalog、schema、table 和 Region 路由。 +- 在执行请求前完成权限检查。 +- 规划分布式查询并合并 Datanode 返回的结果。 +- 将表级写入和删除转换为 Region 请求。 ## 架构 ### 关键组件 -- **协议处理器**：处理不同的数据库协议 -- **目录管理器**：缓存来自 Metasrv 的元数据以实现高效的请求路由和 Schema 校验 -- **分布式规划器**：将逻辑计划转换为分布式执行计划 -- **请求路由器**：为每个请求确定目标 Datanodes + +- 协议处理器将 SQL、PromQL、gRPC 写入和可观测性协议转换为 Frontend 的内部请求接口。 +- Catalog manager 和 partition manager 提供表元数据、分区规则和 Region 路由。 +- Statement executor 将查询、DML 和 DDL 分发到各自的执行路径。 +- 分布式规划器把表扫描替换为可跨 Datanode 执行的 `MergeScan` 计划。 ### 请求流程 -![request flow](/request_flow.png) +不同操作会走不同的请求路径。 + +#### 查询 + +1. 协议处理器创建查询上下文，并完成认证和权限检查。 +2. 对应查询语言的 planner 生成逻辑计划。分布式模式下，planner 根据分区元数据选择 Region 并生成分布式计划。 +3. Frontend 将 Region 子计划发送到对应 Datanode。Datanode 在本地 Region engine 上执行，并返回 Arrow record batch 流。 +4. Frontend 执行剩余算子、合并数据流，再按客户端协议编码结果。 + +#### 写入和删除 + +1. Frontend 根据表 schema 校验请求。支持 schema-on-write 的协议可以先创建缺失的表或新增列，再重试写入。 +2. 分区规则把每一行分配给 Region。Frontend 为各目标 Region 构造请求，并路由到当前 Region leader。 +3. Datanode 的 Region server 将请求分发到对应的 Region engine。单机模式下，请求直接发送给内嵌的 Region server。 + +#### DDL + +Statement executor 将 DDL 转换为 task。分布式模式下，Metasrv 以持久化 procedure 执行 task、更新元数据，并协调 Datanode 上的 Region 操作。单机模式复用相同的语句边界，但使用本地元数据和 procedure 实现。 ### 部署 -下图是 GreptimeDB 在云上的一个典型的部署。`Frontend` 实例组成了一个集群处理来自客户端的请求： +下图展示了 GreptimeDB 的一种云上部署。多个 Frontend 实例共同处理客户端请求： ![frontend](/frontend.png) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/table-sharding.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/table-sharding.md index 63a8ac10a6..cf39afe27b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/table-sharding.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/frontend/table-sharding.md @@ -5,21 +5,17 @@ description: 介绍 GreptimeDB 中表数据的分片方法，包括分区和 Reg # 表分片 -对于任何分布式数据库来说，数据的分片都是必不可少的。本文将描述 GreptimeDB 中的表数据如何进行分片。 +GreptimeDB 将一张表分为多个 Region。分区表达式定义每行数据属于哪个 Region，Region 路由则定义当前由哪个 Datanode 持有该 Region。 ## 分区 -有关创建分区表的语法，请参阅用户指南中的[表分片](/user-guide/deployments-administration/manage-data/table-sharding.md)部分。 +分区是由一个或多个列上的表达式描述的逻辑行集合。分区布局需要覆盖表的输入域，使每一行都能找到唯一的目标 Region。SQL 语法和支持的表达式参见[表分片](/user-guide/deployments-administration/manage-data/table-sharding.md)。 ## Region -在创建分区后，表中的数据被逻辑上分割。你可能会问："在 GreptimeDB 中，被逻辑上分区的数据是如何存储的？" 答案是保存在 `Region` 当中。 - -每个 `Region` 对应一个分区，并保存分区的数据。所有的 `Region` 分布在各个 `Datanode` 之中。 -`Metasrv` 管理 `Region` 到 `Datanode` 的路由信息。如果建表后需要调整分区布局， -GreptimeDB 支持通过显式的 [repartition](/user-guide/deployments-administration/manage-data/repartition.md) 操作拆分或合并分区。 +每个分区对应一个 Region。Region ID 是 Frontend、Datanode 和 Metasrv 用于存储和路由的标识。同一张表的多个 Region 可以放在同一个 Datanode 上。 分区和 Region 的关系参见下图： @@ -54,3 +50,13 @@ GreptimeDB 支持通过显式的 [repartition](/user-guide/deployments-administr └──────────────────────────────────┘ 可以放在同一个 Datanode 之中 ``` + +## 路由与剪枝 + +写入时，Frontend 对每行数据计算分区规则，按 Region 分组，再根据路由表把 Region 请求发送到当前 leader。 + +查询时，分布式 planner 将查询谓词与分区表达式比较，只扫描可能满足谓词的 Region。如果分区元数据缺失或无法安全解释，planner 会退化为扫描所有 Region，避免漏掉数据。 + +## 调整分区布局 + +[Repartition](/user-guide/deployments-administration/manage-data/repartition.md) 通过显式的 split 或 merge 调整已有布局。Metasrv 以持久化 procedure 执行变更，更新 Region 路由和分区表达式，并使旧的表路由缓存失效。Frontend 刷新到新元数据后，后续请求使用新的布局。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-write-sdk.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-write-sdk.md index 28ff757763..916aec36f3 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-write-sdk.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/how-to/how-to-write-sdk.md @@ -1,21 +1,17 @@ --- keywords: [gRPC SDK, GreptimeDatabase, GreptimeRequest, GreptimeResponse, 插入请求] -description: 介绍如何为 GreptimeDB 开发一个 gRPC SDK，包括 GreptimeDatabase 服务的定义、GreptimeRequest 和 GreptimeResponse 的结构。 +description: 介绍 GreptimeDB gRPC 写入 SDK 需要遵守的协议契约和错误处理要求。 --- # 如何为 GreptimeDB 开发一个 gRPC SDK -GreptimeDB 的 gRPC SDK 只需要处理写请求即可。读请求是标准 SQL 或 PromQL，可以由任何 JDBC 客户端或 Prometheus -客户端处理。这也是为什么所有的 GreptimeDB SDK 都命名为 "`greptimedb-ingester-`"。请确保你的 GreptimeDB SDK -遵循相同的命名约定。 +GreptimeDB 的公开 gRPC SDK 是写入客户端。查询通常通过标准 SQL 或 PromQL 客户端完成。除非有单独需求，新 SDK 应聚焦写入和删除，并遵循 `greptimedb-ingester-` 命名约定。面向用户的 API 参见 [gRPC SDK 概述](/user-guide/ingest-data/for-iot/grpc-sdks/overview.md)。 ## `GreptimeDatabase` 服务 -GreptimeDB 自定义了一个 gRPC 服务：`GreptimeDatabase` -。你只需要实现这个服务即可。你可以在[这里](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto) -找到它的 Protobuf 定义。 +从 [`GreptimeDatabase` Protobuf 定义](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto)生成客户端 stub，不要在 SDK 中手写一份 message 或 service 定义。 -`GreptimeDatabase` 有 2 个 RPC 方法： +该 service 提供一个 unary method 和一个 client-streaming method： ```protobuf service GreptimeDatabase { @@ -25,13 +21,9 @@ service GreptimeDatabase { } ``` -`Handle` 方法是一个 unary 调用：当 GreptimeDB 服务接收到一个 `GreptimeRequest` 请求后，它立刻处理该请求并返回一个相应的 -`GreptimeResponse`。 +`Handle` 对一个请求返回一个响应，是 SDK insert 和 delete API 通常使用的方法。 -`HandleRequests` 方法则是一个 "[Client Streaming RPC][3]" 方式的调用。 -它可以接受一个连续的 `GreptimeRequest` 请求流，持续地发给 GreptimeDB 服务。 -GreptimeDB 服务会在收到流中的每个请求时立刻进行处理，并最终（流结束时）返回一个总结性的 `GreptimeResponse`。 -通过 `HandleRequests`，我们可以获得一个非常高的请求吞吐量。 +`HandleRequests` 是 [client-streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc)。客户端关闭请求流后，服务端才返回累计响应。SDK 如果暴露 streaming API，需要明确这个确认边界，并将一个 stream 绑定到一个 endpoint。 ### `GreptimeRequest` @@ -51,11 +43,13 @@ message GreptimeRequest { } ``` -`RequestHeader` 是必需，它包含了一些上下文，鉴权和其他信息。"oneof" 的字段包含了发往 GreptimeDB 服务的请求。 +客户端需要在 `RequestHeader` 中填写服务端要求的 database context 和认证信息，并且只能设置一个 request variant。 -注意我们有两种类型的插入请求，一种是以 "列" 的形式（`InsertRequests`），另一种是以 "行" 的形式（`RowInsertRequests` -）。通常我们建议使用 "行" 的形式，因为它对于表的插入更自然，更容易使用。但是，如果需要一次插入大量列，或者有大量的 "null" -值需要插入，那么最好使用 "列" 的形式。 +该 message 还包含供内部调用者使用的 query 和 DDL variant。公开 ingester API 不应暴露它们，因为 `GreptimeDatabase` 不返回 query result stream。 + +GreptimeDB 同时接受行式 `RowInsertRequests` 和列式 `InsertRequests`。公开写入 API 默认使用行式请求。面向列的客户端可以使用列式请求，但转换过程中必须保持列长度一致，并保留 null、时间戳精度、数据类型和列 semantic type。 + +删除同样区分行式和列式。SDK 只应暴露能够在不丢失类型信息的前提下完成映射的形式。 ### `GreptimeResponse` @@ -68,6 +62,18 @@ message GreptimeResponse { } ``` -`ResponseHeader` 包含了返回值的状态码，以及错误信息（如果有的话）。"oneof" 的字段目前只有 "affected rows"。 +成功响应包含 success header 和 `affected_rows`。该值表示服务端确认的行数；关闭请求流时返回的是累计值。 + +请求失败通过 gRPC status 返回。Trailing metadata 中的 `x-greptime-err-code` 和 `x-greptime-err-retry-hint` 在存在时分别提供 GreptimeDB error code 和 retry classification。SDK 应保留 gRPC status 并暴露这些 GreptimeDB metadata，不能用一个通用 SDK error 将其覆盖。 + +## 重试与交付语义 + +重试次数必须有上限，并且对调用者可见。只有错误被标记为 retryable 且 deadline 仍允许时，才能重试 unary request。Cancellation 和 deadline expiration 不应重试。 + +响应丢失不代表服务端拒绝了写入。除非调用者的数据模型保证操作幂等，重试这类请求可能插入重复行。SDK 需要说明这一点，并在交付结果不确定时返回最终错误。 + +不要自动重试只发送了一部分的 `HandleRequests` stream。即使客户端尚未收到累计响应，服务端也可能已经接受了部分请求。此时应关闭失败的 stream，并将不确定状态返回给调用者。 + +Arrow Flight bulk ingestion 与 `GreptimeDatabase` RPC 应使用不同的 API。它的 batching 和 partial acceptance 需要独立的契约。 -GreptimeDB 现在有很多 SDK，你可以参考[这里](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=)获取一些示例。 +可以参考现有 [GreptimeDB ingester 仓库](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=)的公开 API 约定，但线上行为应以当前 Protobuf 定义和服务端契约为准。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/admin-api.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/admin-api.md index 1fedfc09e7..bd8b2551ca 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/admin-api.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/admin-api.md @@ -1,18 +1,25 @@ --- -keywords: [Admin API, 健康检查, leader 查询, 心跳检测, 维护模式] -description: 介绍 Metasrv 的 Admin API，包括健康检查、leader 查询、心跳检测、维护模式和 Procedure Manager 控制等功能。 +keywords: [Admin API, 健康检查, leader 查询, 心跳检测, 维护模式, 恢复模式, table id sequence] +description: 介绍 Metasrv 用于状态检查、集群控制和元数据恢复的 Admin API。 --- # Admin API -Admin API 通过 HTTP 提供 Metasrv health、leader、Datanode 心跳、维护模式和 Procedure Manager 信息。该 API 不提供认证，且部分端点会改变集群行为，部署时必须通过网络策略保护 HTTP 端口。 +:::tip +本页所有 Admin API 都监听 Metasrv 的 `HTTP_PORT`，默认值为 `4000`。 +::: + +Admin API 通过 HTTP 提供 Metasrv 状态、集群控制和元数据恢复操作。该 API 不提供认证，且部分端点会改变集群行为或元数据分配，部署时必须通过网络策略保护 HTTP 端口。 本页介绍以下 API： - /health - /leader - /heartbeat +- /node-lease - /maintenance - /procedure-manager +- /recovery +- /sequence/table 所有这些 API 都在父资源 `/admin` 下。 @@ -114,9 +121,17 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100' ] ``` +## /node-lease HTTP 端点 + +`/node-lease` 返回 Metasrv 当前记录的 Datanode lease，可用于判断 Metasrv 是否仍将某个 Datanode 视为存活。 + +```bash +curl -X GET http://localhost:4000/admin/node-lease +``` + ## /maintenance HTTP 端点 -集群维护模式是 GreptimeDB 中的一项安全功能，它可以临时禁用自动集群管理操作。此模式在集群升级、计划停机以及任何可能暂时影响集群稳定性的操作期间特别有用。有关更多详细信息，请参阅[集群维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。 +维护模式在升级、计划停机等操作期间临时禁用自动集群管理。它对集群的具体影响参见[集群维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。 `/maintenance` 端点支持以下 HTTP 请求： @@ -149,3 +164,39 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100' "status": "running" } ``` + +## /recovery HTTP 端点 + +Recovery mode 控制手动修改 table ID sequence 等元数据修复端点。它只用于恢复工作，不用于常规维护。 + +- `GET /admin/recovery/status`：查询 recovery mode 是否开启。 +- `POST /admin/recovery/enable`：开启 recovery mode。 +- `POST /admin/recovery/disable`：关闭 recovery mode。 + +响应体格式如下： + +```json +{ + "enabled": true +} +``` + +修复完成后应关闭 recovery mode。如果只是计划暂停自动集群操作，应使用[维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。 + +## /sequence/table HTTP 端点 + +这些端点用于检查或修复 table ID sequence： + +- `GET /admin/sequence/table/next-id`：返回下一个 table ID，但不执行分配。 +- `POST /admin/sequence/table/set-next-id`：推进下一个 table ID。 + +设置 sequence 前必须开启 recovery mode。新值必须大于当前值，不能通过该 API 回退 sequence。 + +```bash +curl -X POST \ + -H 'Content-Type: application/json' \ + -d '{"next_table_id": 2048}' \ + http://localhost:4000/admin/sequence/table/set-next-id +``` + +该操作会影响后续新表分配到的 ID。只有在确认所需 next ID 后，才能用它修复元数据。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/integration-test.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/integration-test.md index 63ffa56bc0..767109f773 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/integration-test.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/integration-test.md @@ -7,8 +7,14 @@ description: 介绍 GreptimeDB 的集成测试，包括测试范围和如何运 ## 介绍 -集成测试使用 Rust 测试工具（`#[test]`）编写，与单元测试不同，它们被单独放置在 -[这里](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration)。 -它涵盖了涉及多个组件的场景，其中一个典型案例是与 HTTP/gRPC 相关的功能。你可以查看 -其[文档](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md)以获取更多信息。 +集成测试覆盖跨 crate 或服务边界的行为，例如 HTTP 和 gRPC 处理、分布式组件或外部存储。测试使用 Rust test harness，位于 [`tests-integration`](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration) package。 +运行命令如下： + +```shell +cargo nextest run -p tests-integration +``` + +部分 case 依赖外部服务的环境变量或 fixture。运行前按照 package 的[准备说明](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md)配置环境。 + +只有 crate 级测试或 Sqlness case 无法覆盖所需边界时才使用集成测试。隔离的逻辑仍放在单元测试中，便于快速复现失败。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/overview.md index 81b6defa45..a130f8aeb4 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/overview.md @@ -5,5 +5,13 @@ description: GreptimeDB 的测试 # 测试 -我们的团队进行了大量测试，以确保 GreptimeDB 的行为。本章将介绍几种用于测试 GreptimeDB 的重要方法，以及如何使用它们。 +选择能够覆盖本次改动的最小测试范围： +| 测试类型 | 适用场景 | 常用命令 | +| --- | --- | --- | +| [单元测试](unit-test.md) | 单个 crate 或组件内的逻辑 | `cargo nextest run -p ` | +| [Sqlness 测试](sqlness-test.md) | SQL、协议、planner、执行和端到端回归 | `cargo sqlness bare -t ` | +| [集成测试](integration-test.md) | 跨组件或依赖外部服务的行为 | `cargo nextest run -p tests-integration` | +| [兼容性测试](https://github.com/GreptimeTeam/greptimedb/blob/main/tests/compatibility/README.md) | 读取旧版本写入的数据或元数据 | `cargo run -p sqlness-runner -- compat --from-version ` | + +需要运行完整 Rust workspace 测试时使用 `make test`。如果改动涉及持久化元数据、WAL record、SST 文件或线上协议，并且输入可能由旧版本产生，还需要补充兼容性测试。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/sqlness-test.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/sqlness-test.md index af260a14de..06503a434c 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/sqlness-test.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/sqlness-test.md @@ -7,36 +7,34 @@ description: 介绍 GreptimeDB 的 Sqlness 测试，包括测试文件类型、 ## 介绍 -SQL 是 `GreptimeDB` 的一个重要用户接口。我们为它提供了一个单独的测试套件（名为 `sqlness`）。 +Sqlness 是 GreptimeDB 针对 SQL 和协议行为的端到端回归测试。每个 case 向运行中的 GreptimeDB 发送语句，并将输出与仓库中的结果文件比较。 ## Sqlness 手册 ### 测试文件 -Sqlness 有两种类型的文件 +每个 case 使用两类文件： - `.sql`：测试输入，仅包含 SQL - `.result`：预期的测试输出，包含 SQL 和其结果 -`.result` 文件是预期的执行输出。如果 `.result` 文件发生变化，意味着测试结果不同，测试可能失败。你应该检查变更日志来解决问题。 - -你只需要在 `.sql` 文件中编写测试 SQL，然后运行测试。 +在 `.sql` 文件中编写输入，运行测试后生成或更新 `.result`。必须检查每一处结果差异，只有行为变化符合预期时才能接受。 ### 组织测试案例 -输入案例的根目录是 `tests/cases`。它包含几个子目录，代表不同的测试模式。例如，`standalone/` 包含所有在 `greptimedb standalone start` 模式下运行的测试。 +输入 case 位于 `tests/cases`。第一级目录选择运行环境，例如 `standalone/` 表示使用单机 GreptimeDB。 -在第一级子目录下（例如 `cases/standalone`），你可以随意组织你的测试案例。Sqlness 会递归地遍历每个文件并运行它们。 +在环境目录内，新 case 应与它覆盖的功能放在一起。Sqlness 会递归发现 case 文件。 ## 运行测试 -与其他测试不同，这个测试工具是以二进制目标形式存在的。你可以用以下命令运行它 +运行命令如下： ```shell cargo sqlness bare ``` -它会自动完成以下步骤：编译 `GreptimeDB`、启动测试环境、执行测试，再收集和比较结果。确认没有非预期的 `.result` 变化后，测试才算通过。 +该命令会构建并启动 GreptimeDB、执行选中的 case，再比较输出。`.result` 发生变化只是待审查的结果，不代表新输出一定正确。 ### 运行特定测试 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/unit-test.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/unit-test.md index 79c73775b4..b63b29df94 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/unit-test.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/unit-test.md @@ -7,22 +7,26 @@ description: 介绍 GreptimeDB 的单元测试，包括如何编写、运行和 ## 介绍 -单元测试嵌入在代码库中，通常放置在被测试逻辑的旁边。它们使用 Rust 的 `#[test]` 属性编写，并可以使用 `cargo nextest run` 运行。 +单元测试通常放在被测逻辑旁边，使用 Rust 的 `#[test]` 属性编写。GreptimeDB 主要使用 [`cargo-nextest`](https://nexte.st/) 运行 Rust 测试。 -GreptimeDB 代码库不支持默认的 `cargo` 测试运行器。推荐使用 [`nextest`](https://nexte.st/)。你可以通过以下命令安装它： +安装命令如下： ```shell cargo install cargo-nextest --locked ``` -然后运行测试（这里 `--workspace` 不是必须的） +开发时先运行本次修改的 package： ```shell -cargo nextest run +cargo nextest run -p ``` -注意，如果你的 Rust 是通过 `rustup` 安装的，请确保使用 `cargo` 安装 `nextest`，而不是像 `homebrew` 这样的包管理器，否则会弄乱你的本地环境。 +可以继续使用测试名称或 nextest filter 缩小范围。影响范围较广的改动在提交前运行完整 workspace 测试： + +```shell +make test +``` ## 覆盖率 -我们的持续集成（CI）作业有一个“覆盖率检查”步骤。它会报告有多少代码被单元测试覆盖。请在你的补丁中添加必要的单元测试。 +CI 会报告单元测试覆盖率。测试应覆盖本次改变的行为和可能回归的失败路径，而不是只追求覆盖率数字。 From 51e8580cffc0d1c6a58d3d606940af3a21ef47c8 Mon Sep 17 00:00:00 2001 From: Dennis Zhuang Date: Tue, 25 Aug 2026 16:03:28 +0800 Subject: [PATCH 05/14] docs: improve contributor guide diagrams --- .../datanode/data-persistence-indexing.md | 10 +-- .../datanode/metric-engine.md | 4 +- .../datanode/storage-engine.md | 2 +- .../datanode/data-persistence-indexing.md | 8 +- .../datanode/metric-engine.md | 4 +- .../datanode/storage-engine.md | 2 +- static/inverted-index-blob-layout.svg | 69 +++++++++++++++++ static/inverted-index-blob-layout.zh.svg | 69 +++++++++++++++++ static/metric-engine-architecture.svg | 75 +++++++++++++++++++ static/metric-engine-architecture.zh.svg | 75 +++++++++++++++++++ static/mito-sst-layout.svg | 62 +++++++++++++++ static/mito-sst-layout.zh.svg | 62 +++++++++++++++ static/parquet-row-group-statistics.svg | 61 +++++++++++++++ static/parquet-row-group-statistics.zh.svg | 61 +++++++++++++++ 14 files changed, 549 insertions(+), 15 deletions(-) create mode 100644 static/inverted-index-blob-layout.svg create mode 100644 static/inverted-index-blob-layout.zh.svg create mode 100644 static/metric-engine-architecture.svg create mode 100644 static/metric-engine-architecture.zh.svg create mode 100644 static/mito-sst-layout.svg create mode 100644 static/mito-sst-layout.zh.svg create mode 100644 static/parquet-row-group-statistics.svg create mode 100644 static/parquet-row-group-statistics.zh.svg diff --git a/docs/contributor-guide/datanode/data-persistence-indexing.md b/docs/contributor-guide/datanode/data-persistence-indexing.md index 64efb659ee..b280c9237e 100644 --- a/docs/contributor-guide/datanode/data-persistence-indexing.md +++ b/docs/contributor-guide/datanode/data-persistence-indexing.md @@ -31,9 +31,9 @@ When the size of data buffered in MemTables reaches that threshold, GreptimeDB w ## Indexing Data in SST Files -Apache Parquet file format provides inherent statistics in headers of column chunks and data pages, which are used for pruning and skipping. +Parquet stores row-group column statistics such as minimum, maximum, and null count in each column chunk's metadata. Page metadata and optional column indexes can provide finer-grained statistics. -Column chunk header +![A name predicate uses Parquet column statistics to skip one row group while retaining another as a read candidate.](/parquet-row-group-statistics.svg) For example, a query filtering for `name` = `Emily` can skip row group 0 because the maximum `name` value is `Charlie`. This avoids reading that row group. @@ -61,13 +61,13 @@ The inverted index enables GreptimeDB to skip data segments that do not meet que ![Inverted index searching](/inverted-index-searching.png) -The query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches `.*users`, and `status` matches `4...`. It scans only those segments before applying the remaining filters. +The query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches `.*users`, and `status` matches `4..`. It scans only those segments before applying the remaining filters. ### Inverted Index Format -![Inverted index format](/inverted-index-format.png) +![An inverted-index blob contains one index per column followed by footer metadata; each column index contains a null bitmap, posting bitmaps, and an FST.](/inverted-index-blob-layout.svg) -GreptimeDB builds inverted indexes by column, with each inverted index consisting of an FST and multiple Bitmaps. +GreptimeDB builds inverted indexes by column. Each column index contains a null bitmap, multiple posting bitmaps, and an FST. The blob footer records the offsets, sizes, and metadata needed to locate and decode the column indexes. The FST (Finite State Transducer) enables GreptimeDB to store mappings from column values to Bitmap positions in a compact format and provides excellent search performance and supports complex search capabilities (such as regular expression matching). The Bitmaps maintain a list of data segment IDs, with each bit representing a data segment. diff --git a/docs/contributor-guide/datanode/metric-engine.md b/docs/contributor-guide/datanode/metric-engine.md index f74c29e8f2..0073de4497 100644 --- a/docs/contributor-guide/datanode/metric-engine.md +++ b/docs/contributor-guide/datanode/metric-engine.md @@ -29,9 +29,9 @@ A physical table is a table that actually stores data, possessing several physic The main design architecture of the `Metric` engine is as follows: -![Arch](/metric-engine-arch.png) +![Multiple logical tables map through the Metric engine to shared data and metadata Regions managed by Mito.](/metric-engine-architecture.svg) -The `Metric` engine delegates physical storage and queries to the `Mito` engine. Each physical Region is represented by a data Region, which stores rows from many logical tables, and a metadata Region, which stores the logical-table and logical-column mappings. +The `Metric` engine delegates physical storage and queries to the `Mito` engine. Each physical Region group contains a data Region, which stores rows from its mapped logical tables, and a metadata Region, which stores the logical-table and logical-column mappings. Logical tables associated with the same physical table share its partition layout. During writes, the engine records the logical table identity with each row. During reads, it adds a logical-table filter before scanning the physical Region. diff --git a/docs/contributor-guide/datanode/storage-engine.md b/docs/contributor-guide/datanode/storage-engine.md index 3ebb0d8af2..10637138f6 100644 --- a/docs/contributor-guide/datanode/storage-engine.md +++ b/docs/contributor-guide/datanode/storage-engine.md @@ -103,7 +103,7 @@ Each Parquet SST is split into row groups, the unit that Parquet can read or ski Mito supports two SST formats: `flat` and `primary_key`. `flat` is the default for new tables and works well across primary-key cardinalities, including high-cardinality keys. `primary_key` is the legacy format kept for compatibility with older tables. See [SST format](/reference/sql/create.md#create-a-table-with-sst-format) and the [table design guide](/user-guide/deployments-administration/performance-tuning/design-table.md#sst-format) for more details. -SST layout +![The default flat Mito SST layout combines file-level metadata with Parquet row groups containing data columns and merge metadata.](/mito-sst-layout.svg) ## Scan Pruning diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md index 09df007a18..fdc07dec4b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md @@ -30,9 +30,9 @@ GreptimeDB 提供了 `region_engine.mito.global_write_buffer_size` 的配置项 ## SST 文件中的索引数据 -Apache Parquet 文件格式在列块和数据页的头部提供了内置的统计信息，用于剪枝和跳过。 +Parquet 在每个 column chunk 的元数据中保存 row group 级列统计信息，例如最小值、最大值和 null 数量。Page 元数据和可选的 column index 可以提供粒度更细的统计信息。 -Column chunk header +![查询 name 列时，Parquet 列统计信息排除了一个 row group，并将另一个保留为待读取对象。](/parquet-row-group-statistics.zh.svg) 例如，查询 `name` 等于 `Emily` 的行时，可以跳过 row group 0，因为其中 `name` 的最大值是 `Charlie`，无需读取该 row group。 @@ -66,9 +66,9 @@ GreptimeDB 会将多种索引结构作为 Blob 存储在 Puffin 文件中，包 ### 倒排索引格式 -![Inverted index format](/inverted-index-format.png) +![倒排索引 Blob 先保存各列索引，再保存 footer 元数据；每个列索引包含 null bitmap、posting bitmap 和 FST。](/inverted-index-blob-layout.zh.svg) -GreptimeDB 按列构建倒排索引，每个倒排索引包含一个 FST 和多个 Bitmap。 +GreptimeDB 按列构建倒排索引。每个列索引包含一个 null bitmap、多个 posting bitmap 和一个 FST。Blob footer 记录定位和解码各列索引所需的 offset、size 和元数据。 FST（Finite State Transducer）允许 GreptimeDB 以紧凑的格式存储列值到 Bitmap 位置的映射，并且提供了优秀的搜索性能和支持复杂搜索（例如正则表达式匹配）；Bitmap 则维护了数据段 ID 列表，每个位表示一个数据段。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md index 7a99edd478..325002f7c5 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md @@ -27,9 +27,9 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述 `Metric` 引擎的主要设计架构如下： -![Arch](/metric-engine-arch.png) +![多个逻辑表通过 Metric 引擎映射到由 Mito 管理的共享数据 Region 和元数据 Region。](/metric-engine-architecture.zh.svg) -`Metric` 引擎将物理存储和查询交给 `Mito` 引擎。每个物理 Region 由一个数据 Region 和一个元数据 Region 表示：数据 Region 保存多个逻辑表的数据，元数据 Region 保存逻辑表及逻辑列的映射。 +`Metric` 引擎将物理存储和查询交给 `Mito` 引擎。每个物理 Region 组包含一个数据 Region 和一个元数据 Region：数据 Region 保存映射到该 Region 组的逻辑表数据，元数据 Region 保存逻辑表及逻辑列的映射。 关联到同一物理表的逻辑表使用相同的分区布局。写入时，Metric 引擎为每行数据记录逻辑表身份；读取时，它在扫描物理 Region 前增加逻辑表过滤条件。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md index c63e3bea1b..7e57e3c03d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md @@ -100,7 +100,7 @@ Mito 会按 primary key 对行分组，并按时间排序，因此 SST 中的数 Mito 支持两种 SST 格式：`flat` 和 `primary_key`。`flat` 是新表的默认格式，适用于各种 primary key 基数，包括高基数 key。`primary_key` 是为了兼容旧表而保留的遗留格式。更多详情请参考 [SST format](/reference/sql/create.md#创建指定-sst-格式的表) 和[表设计指南](/user-guide/deployments-administration/performance-tuning/design-table.md#sst-格式)。 -SST layout +![Mito 默认的 flat SST 布局将文件级元数据与包含数据列和合并元数据的 Parquet row group 组合在一起。](/mito-sst-layout.zh.svg) ## 扫描裁剪 diff --git a/static/inverted-index-blob-layout.svg b/static/inverted-index-blob-layout.svg new file mode 100644 index 0000000000..2fc2efbda0 --- /dev/null +++ b/static/inverted-index-blob-layout.svg @@ -0,0 +1,69 @@ + + GreptimeDB inverted-index blob layout + An inverted-index blob stored in a Puffin file contains one index for each indexed column followed by a footer payload and its four-byte size. Each column index contains a null bitmap, posting bitmaps, and an FST that maps encoded column values to bitmap byte ranges. + + + + + + + Puffin index blob + One blob contains column indexes followed by footer metadata + + + + Inverted-index blob + stored as a typed blob in the SST's optional Puffin sidecar + + + Column index + job + + Column index + handler + + Column index + status + … + + Footer payload + offsets · sizes · index metadata + + Footer size + 4 bytes + + + expand one column + + + Column index layout + values are encoded in byte order; the FST points to their posting bitmaps + + + Null bitmap + + Posting bitmap 0 + segments for value 0 + + Posting bitmap 1 + segments for value 1 + … + + Posting bitmap n + + FST + value → bitmap range + + + There is no separate inverted-index header; the footer describes how to locate and decode each column index. + diff --git a/static/inverted-index-blob-layout.zh.svg b/static/inverted-index-blob-layout.zh.svg new file mode 100644 index 0000000000..e99394f8de --- /dev/null +++ b/static/inverted-index-blob-layout.zh.svg @@ -0,0 +1,69 @@ + + GreptimeDB 倒排索引 Blob 布局 + Puffin 文件中的倒排索引 Blob 先保存各索引列的倒排索引，随后保存 footer payload 及其四字节长度。每个列索引包含 null bitmap、posting bitmap 和把编码列值映射到 bitmap 字节范围的 FST。 + + + + + + + PUFFIN 索引 BLOB + 一个 Blob 先保存各列索引，再保存 footer 元数据 + + + + 倒排索引 Blob + 作为 typed blob 存储在 SST 可选的 Puffin sidecar 中 + + + 列索引 + job + + 列索引 + handler + + 列索引 + status + … + + Footer payload + offset · size · 索引元数据 + + Footer 长度 + 4 bytes + + + 展开一个列索引 + + + 列索引布局 + 列值按字节序编码，FST 指向对应的 posting bitmap + + + Null bitmap + + Posting bitmap 0 + 值 0 对应的数据段 + + Posting bitmap 1 + 值 1 对应的数据段 + … + + Posting bitmap n + + FST + 列值 → bitmap 范围 + + + 倒排索引没有单独的 header；footer 记录定位和解码各列索引所需的元数据。 + diff --git a/static/metric-engine-architecture.svg b/static/metric-engine-architecture.svg new file mode 100644 index 0000000000..69d9fb4e07 --- /dev/null +++ b/static/metric-engine-architecture.svg @@ -0,0 +1,75 @@ + + Metric engine logical-to-physical mapping + Multiple logical tables share one physical Region group. The Metric engine maps logical tables and columns to a pair of Mito Regions: a metadata Region for mappings and a data Region for shared rows carrying logical table identity. + + + + + + + + Logical-to-physical mapping + Many logical tables share one physical Region group + + + Logical tables + + Logical table A + independent schema and identity + + Logical table B + independent schema and identity + + Logical table C + independent schema and identity + + + + + + + Metric engine + maps logical requests to shared storage + + Resolve logical table + + Map logical columns + + shared partition layout + + + mappings + + rows + + + Mito engine + a physical Region group contains two Mito Regions + + + Metadata Region + logical table mappings + logical columns → physical columns + + + Data Region + rows from mapped logical tables + + logical table identity per row + + + Write: + attach logical table identity before writing the shared data Region. + Read: + add a logical-table filter before scanning the data Region. + diff --git a/static/metric-engine-architecture.zh.svg b/static/metric-engine-architecture.zh.svg new file mode 100644 index 0000000000..dd5821f428 --- /dev/null +++ b/static/metric-engine-architecture.zh.svg @@ -0,0 +1,75 @@ + + Metric 引擎的逻辑表到物理表映射 + 多个逻辑表共享一个物理 Region 组。Metric 引擎把逻辑表和逻辑列映射到两个 Mito Region：元数据 Region 保存映射，数据 Region 保存带有逻辑表身份的共享数据。 + + + + + + + + 逻辑表到物理表的映射 + 多个逻辑表共享一个物理 Region 组 + + + 逻辑表 + + 逻辑表 A + 独立的 schema 和表身份 + + 逻辑表 B + 独立的 schema 和表身份 + + 逻辑表 C + 独立的 schema 和表身份 + + + + + + + Metric 引擎 + 将逻辑请求映射到共享存储 + + 解析逻辑表 + + 映射逻辑列 + + 共享分区布局 + + + 映射 + + 数据行 + + + Mito 引擎 + 一个物理 Region 组包含两个 Mito Region + + + 元数据 Region + 逻辑表映射 + 逻辑列 → 物理列 + + + 数据 Region + 保存所映射逻辑表的数据行 + + 每行记录逻辑表身份 + + + 写入： + 附加逻辑表身份后写入共享数据 Region。 + 读取： + 扫描数据 Region 前增加逻辑表过滤条件。 + diff --git a/static/mito-sst-layout.svg b/static/mito-sst-layout.svg new file mode 100644 index 0000000000..675c4c42a8 --- /dev/null +++ b/static/mito-sst-layout.svg @@ -0,0 +1,62 @@ + + Mito Parquet SST layout + Mito records file-level metadata for a Parquet SST and divides the file into independently readable row groups. Row groups store field columns, the time index, the encoded primary key, sequence, and operation type, plus raw primary-key columns when the selected encoding includes them. An SST is not shown as belonging to exactly one compaction time window because a file can span windows. + + + + + + Default flat SST layout + A Parquet SST stores table columns with Mito merge metadata + + + + Mito SST metadata + time range · primary-key range · row count · row-group count · available indexes + + + Parquet SST + row groups are the independently readable and skippable units + + + Row group 0 + + Raw primary-key columns + when present + + Field columns + cpu · memory + + Time index + ts + + Encoded key + __primary_key + + Merge metadata + __sequence · __op_type + + + Row group 1 + the same physical columns, containing the next independently readable group of rows + + column min / max / null count + + + Ordering: + for tables with primary keys, rows are ordered by primary key and then by time index. + Compaction: + an SST may span more than one compaction time window. + diff --git a/static/mito-sst-layout.zh.svg b/static/mito-sst-layout.zh.svg new file mode 100644 index 0000000000..315d01ba4a --- /dev/null +++ b/static/mito-sst-layout.zh.svg @@ -0,0 +1,62 @@ + + Mito Parquet SST 布局 + Mito 为 Parquet SST 记录文件级元数据，并把文件切分为可以独立读取的 row group。Row group 保存 field 列、time index、编码后的 primary key、sequence 和操作类型；所选编码需要时还会保存原始 primary-key 列。由于一个 SST 可以跨越多个 compaction time window，图中不把 SST 限定在单个 time window 内。 + + + + + + 默认 FLAT SST 布局 + Parquet SST 同时保存表数据列和 Mito 合并元数据 + + + + Mito SST 元数据 + 时间范围 · primary-key 范围 · 行数 · row-group 数量 · 可用索引 + + + Parquet SST + row group 是可以独立读取和跳过的单位 + + + Row group 0 + + 原始 primary-key 列 + 所选编码需要时保存 + + Field 列 + cpu · memory + + Time index + ts + + 编码后的 key + __primary_key + + 合并元数据 + __sequence · __op_type + + + Row group 1 + 使用相同物理列，保存下一组可以独立读取的数据行 + + 列 min / max / null 数量 + + + 排序： + 对于带 primary key 的表，先按 primary key、再按 time index 排序。 + Compaction： + 一个 SST 可以跨越多个 compaction time window。 + diff --git a/static/parquet-row-group-statistics.svg b/static/parquet-row-group-statistics.svg new file mode 100644 index 0000000000..83d400b5d1 --- /dev/null +++ b/static/parquet-row-group-statistics.svg @@ -0,0 +1,61 @@ + + Parquet row-group statistics pruning + A query for name equals Emily compares the predicate with column statistics stored in row-group column metadata. Row group zero is skipped because its maximum name is Charlie, while row group one remains a candidate because Emily falls between Doug and John. + + + + + + + Parquet row-group pruning + Column statistics eliminate row groups before data is read + + + + Query predicate + name = "Emily" + + compare + + + Parquet file metadata + ColumnMetaData for each row-group column chunk + + + Row group 0 + + name + min Alice + max Charlie + + age + 18…23 + + Skip + Emily is greater than the maximum value Charlie. + + + Row group 1 + + name + min Doug + max John + + age + 20…30 + + Keep + Emily falls within the recorded min-max range. + + + Statistics prove when a row group cannot match; they do not prove that every remaining row matches. + diff --git a/static/parquet-row-group-statistics.zh.svg b/static/parquet-row-group-statistics.zh.svg new file mode 100644 index 0000000000..2fc32ba804 --- /dev/null +++ b/static/parquet-row-group-statistics.zh.svg @@ -0,0 +1,61 @@ + + Parquet row group 统计信息裁剪 + 查询 name 等于 Emily 时，系统将谓词与 row group 的列元数据统计信息比较。Row group 0 的 name 最大值为 Charlie，因此可以跳过；Emily 位于 Doug 和 John 之间，因此 row group 1 仍需读取。 + + + + + + + PARQUET ROW GROUP 裁剪 + 读取数据前，先用列统计信息排除不匹配的 row group + + + + 查询谓词 + name = "Emily" + + 比较 + + + Parquet 文件元数据 + 每个 row group column chunk 的 ColumnMetaData + + + Row group 0 + + name + min Alice + max Charlie + + age + 18…23 + + 跳过 + Emily 大于最大值 Charlie。 + + + Row group 1 + + name + min Doug + max John + + age + 20…30 + + 保留 + Emily 位于记录的 min-max 范围内。 + + + 统计信息只能证明某个 row group 不可能匹配，不能证明保留下来的每一行都匹配。 + From 35fa11a7bbf6ed77c5a96ddef2ac9d88c0e75f75 Mon Sep 17 00:00:00 2001 From: Dennis Zhuang Date: Tue, 25 Aug 2026 17:59:22 +0800 Subject: [PATCH 06/14] docs: address contributor guide review feedback --- .../datanode/data-persistence-indexing.md | 2 +- docs/contributor-guide/datanode/overview.md | 2 + .../datanode/storage-engine.md | 2 + docs/contributor-guide/tests/overview.md | 2 +- .../datanode/data-persistence-indexing.md | 2 +- .../contributor-guide/datanode/overview.md | 2 + .../datanode/storage-engine.md | 2 + .../contributor-guide/tests/overview.md | 2 +- static/datanode-architecture.svg | 61 ++++++++++ static/datanode-architecture.zh.svg | 61 ++++++++++ static/inverted-index-blob-layout.svg | 94 ++++++--------- static/inverted-index-blob-layout.zh.svg | 94 ++++++--------- static/metric-engine-architecture.svg | 110 +++++++---------- static/metric-engine-architecture.zh.svg | 112 +++++++----------- static/mito-sst-layout.svg | 102 +++++++--------- static/mito-sst-layout.zh.svg | 102 +++++++--------- static/parquet-row-group-statistics.svg | 80 +++++-------- static/parquet-row-group-statistics.zh.svg | 80 +++++-------- 18 files changed, 455 insertions(+), 457 deletions(-) create mode 100644 static/datanode-architecture.svg create mode 100644 static/datanode-architecture.zh.svg diff --git a/docs/contributor-guide/datanode/data-persistence-indexing.md b/docs/contributor-guide/datanode/data-persistence-indexing.md index b280c9237e..1fa5d6d258 100644 --- a/docs/contributor-guide/datanode/data-persistence-indexing.md +++ b/docs/contributor-guide/datanode/data-persistence-indexing.md @@ -21,7 +21,7 @@ The following diagram from the Apache Parquet specification also shows the physi Apache Parquet file layout -*Source: [Apache Parquet file-format specification](https://parquet.apache.org/docs/file-format/).* +*Source: Apache Parquet [FileLayout.gif](https://github.com/apache/parquet-format/blob/master/doc/images/FileLayout.gif). Copyright 2014 The Apache Software Foundation, licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).* ## Data Persistence diff --git a/docs/contributor-guide/datanode/overview.md b/docs/contributor-guide/datanode/overview.md index 4d3a1d26a0..a21c112faa 100644 --- a/docs/contributor-guide/datanode/overview.md +++ b/docs/contributor-guide/datanode/overview.md @@ -11,6 +11,8 @@ A Datanode stores and processes Region data. A table can contain multiple Region This boundary lets the same Region server host different storage engines without exposing their implementation to Frontend or Metasrv. +![Frontend sends Region requests to the Datanode Region server, while Metasrv exchanges lifecycle instructions through the heartbeat task. The Region server uses the local query engine and dispatches requests to the Mito, Metric, or File Region engine.](/datanode-architecture.svg) + ## Components The main components are: diff --git a/docs/contributor-guide/datanode/storage-engine.md b/docs/contributor-guide/datanode/storage-engine.md index 10637138f6..b800c218eb 100644 --- a/docs/contributor-guide/datanode/storage-engine.md +++ b/docs/contributor-guide/datanode/storage-engine.md @@ -105,6 +105,8 @@ Mito supports two SST formats: `flat` and `primary_key`. `flat` is the default f ![The default flat Mito SST layout combines file-level metadata with Parquet row groups containing data columns and merge metadata.](/mito-sst-layout.svg) +An SST may span more than one compaction time window. + ## Scan Pruning Mito avoids reading data that cannot match a query by combining several pruning steps, from coarse to fine: diff --git a/docs/contributor-guide/tests/overview.md b/docs/contributor-guide/tests/overview.md index b241f2f078..6fe48b2c32 100644 --- a/docs/contributor-guide/tests/overview.md +++ b/docs/contributor-guide/tests/overview.md @@ -12,6 +12,6 @@ Choose the narrowest test that exercises the behavior you changed: | [Unit test](unit-test.md) | Logic contained within one crate or component | `cargo nextest run -p ` | | [Sqlness test](sqlness-test.md) | SQL, protocol, planner, execution, and end-to-end regressions | `cargo sqlness bare -t ` | | [Integration test](integration-test.md) | Behavior that crosses components or requires external services | `cargo nextest run -p tests-integration` | -| [Compatibility test](https://github.com/GreptimeTeam/greptimedb/blob/main/tests/compatibility/README.md) | Reading data or metadata written by an older release | `cargo run -p sqlness-runner -- compat --from-version ` | +| [Compatibility test](https://github.com/GreptimeTeam/greptimedb/blob/main/tests/compatibility/README.md) | Reading data or metadata written by an older release | `cargo sqlness compat --from-version ` | Run `make test` when a change needs the full Rust workspace test suite. Changes to persisted metadata, WAL records, SST files, or wire formats should also include a compatibility test when an older release may have produced the input. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md index fdc07dec4b..e0ce16721d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/data-persistence-indexing.md @@ -21,7 +21,7 @@ Parquet 按 row group、column chunk 和 page 组织数据。每个 row group Apache Parquet 文件布局 -*来源：[Apache Parquet 文件格式规范](https://parquet.apache.org/docs/file-format/)。* +*来源：Apache Parquet [FileLayout.gif](https://github.com/apache/parquet-format/blob/master/doc/images/FileLayout.gif)。Copyright 2014 The Apache Software Foundation，依据 [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) 使用。* ## 数据持久化 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md index 75c9e005d0..c7684eb165 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/overview.md @@ -11,6 +11,8 @@ Datanode 存储并处理 Region 数据。一张表可以包含多个 Region， 这个边界使同一个 Region server 可以承载不同的存储引擎，而不向 Frontend 或 Metasrv 暴露引擎实现。 +![Frontend 向 Datanode Region server 发送 Region 请求，Metasrv 通过 heartbeat task 与 Datanode 交换生命周期指令。Region server 使用本地 query engine，并将请求分发给 Mito、Metric 或 File Region engine。](/datanode-architecture.zh.svg) + ## Components Datanode 包含以下主要组件： diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md index 7e57e3c03d..261a38bcff 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md @@ -102,6 +102,8 @@ Mito 支持两种 SST 格式：`flat` 和 `primary_key`。`flat` 是新表的默 ![Mito 默认的 flat SST 布局将文件级元数据与包含数据列和合并元数据的 Parquet row group 组合在一起。](/mito-sst-layout.zh.svg) +一个 SST 可能跨越多个 compaction time window。 + ## 扫描裁剪 Mito 会组合多个从粗到细的裁剪步骤，避免读取不可能匹配查询的数据： diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/overview.md index a130f8aeb4..9ac8eb2ec1 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/tests/overview.md @@ -12,6 +12,6 @@ description: GreptimeDB 的测试 | [单元测试](unit-test.md) | 单个 crate 或组件内的逻辑 | `cargo nextest run -p ` | | [Sqlness 测试](sqlness-test.md) | SQL、协议、planner、执行和端到端回归 | `cargo sqlness bare -t ` | | [集成测试](integration-test.md) | 跨组件或依赖外部服务的行为 | `cargo nextest run -p tests-integration` | -| [兼容性测试](https://github.com/GreptimeTeam/greptimedb/blob/main/tests/compatibility/README.md) | 读取旧版本写入的数据或元数据 | `cargo run -p sqlness-runner -- compat --from-version ` | +| [兼容性测试](https://github.com/GreptimeTeam/greptimedb/blob/main/tests/compatibility/README.md) | 读取旧版本写入的数据或元数据 | `cargo sqlness compat --from-version ` | 需要运行完整 Rust workspace 测试时使用 `make test`。如果改动涉及持久化元数据、WAL record、SST 文件或线上协议，并且输入可能由旧版本产生，还需要补充兼容性测试。 diff --git a/static/datanode-architecture.svg b/static/datanode-architecture.svg new file mode 100644 index 0000000000..ef62bc07d1 --- /dev/null +++ b/static/datanode-architecture.svg @@ -0,0 +1,61 @@ + + Datanode component boundary + Frontend sends Region requests to the Datanode over gRPC. Metasrv exchanges Region lifecycle instructions and status with the heartbeat task. The Region server uses the local query engine and dispatches requests to the Mito, Metric, or File Region engine. + + + + + + + Frontend + + Metasrv + + + Datanode + + + + Region requests + + + + status + instructions + + + gRPC services + Region · Flight + + + + + Region server + + Local query engine + + + Heartbeat task + + + lifecycle + + + + + Region engines + + Mito + + Metric + + File + diff --git a/static/datanode-architecture.zh.svg b/static/datanode-architecture.zh.svg new file mode 100644 index 0000000000..07419628ea --- /dev/null +++ b/static/datanode-architecture.zh.svg @@ -0,0 +1,61 @@ + + Datanode 组件边界 + Frontend 通过 gRPC 向 Datanode 发送 Region 请求。Metasrv 与 heartbeat task 交换 Region 生命周期指令和状态。Region server 使用本地 query engine，并将请求分发给 Mito、Metric 或 File Region engine。 + + + + + + + Frontend + + Metasrv + + + Datanode + + + + Region 请求 + + + + 状态 + 指令 + + + gRPC 服务 + Region · Flight + + + + + Region server + + 本地 query engine + + + Heartbeat task + + + 生命周期 + + + + + Region engines + + Mito + + Metric + + File + diff --git a/static/inverted-index-blob-layout.svg b/static/inverted-index-blob-layout.svg index 2fc2efbda0..282ccb8bb8 100644 --- a/static/inverted-index-blob-layout.svg +++ b/static/inverted-index-blob-layout.svg @@ -1,69 +1,53 @@ - + GreptimeDB inverted-index blob layout - An inverted-index blob stored in a Puffin file contains one index for each indexed column followed by a footer payload and its four-byte size. Each column index contains a null bitmap, posting bitmaps, and an FST that maps encoded column values to bitmap byte ranges. + An inverted-index blob stores one index for each indexed column, followed by a footer payload and its four-byte length. Each column index contains a null bitmap, posting bitmaps, and an FST that maps encoded values to bitmap ranges. - - - Puffin index blob - One blob contains column indexes followed by footer metadata - + - - Inverted-index blob - stored as a typed blob in the SST's optional Puffin sidecar + + Inverted-index blob - - Column index - job - - Column index - handler - - Column index - status - … - - Footer payload - offsets · sizes · index metadata - - Footer size - 4 bytes + + job + + handler + + status + … + + Footer + offsets · sizes · metadata + + Footer size + 4 bytes - - expand one column + + - - Column index layout - values are encoded in byte order; the FST points to their posting bitmaps + + Column index - - Null bitmap - - Posting bitmap 0 - segments for value 0 - - Posting bitmap 1 - segments for value 1 - … - - Posting bitmap n - - FST - value → bitmap range - - - There is no separate inverted-index header; the footer describes how to locate and decode each column index. + + Null bitmap + + Posting bitmap 0 + + Posting bitmap 1 + … + + Posting bitmap n + + FST + value → range diff --git a/static/inverted-index-blob-layout.zh.svg b/static/inverted-index-blob-layout.zh.svg index e99394f8de..55b461a778 100644 --- a/static/inverted-index-blob-layout.zh.svg +++ b/static/inverted-index-blob-layout.zh.svg @@ -1,69 +1,53 @@ - + GreptimeDB 倒排索引 Blob 布局 - Puffin 文件中的倒排索引 Blob 先保存各索引列的倒排索引，随后保存 footer payload 及其四字节长度。每个列索引包含 null bitmap、posting bitmap 和把编码列值映射到 bitmap 字节范围的 FST。 + 倒排索引 Blob 为每个索引列保存一个列索引，随后保存 footer payload 及其四字节长度。每个列索引包含 null bitmap、posting bitmap 和将编码列值映射到 bitmap 范围的 FST。 - - - PUFFIN 索引 BLOB - 一个 Blob 先保存各列索引，再保存 footer 元数据 - + - - 倒排索引 Blob - 作为 typed blob 存储在 SST 可选的 Puffin sidecar 中 + + 倒排索引 Blob - - 列索引 - job - - 列索引 - handler - - 列索引 - status - … - - Footer payload - offset · size · 索引元数据 - - Footer 长度 - 4 bytes + + job + + handler + + status + … + + Footer + offset · size · 元数据 + + Footer 长度 + 4 bytes - - 展开一个列索引 + + - - 列索引布局 - 列值按字节序编码，FST 指向对应的 posting bitmap + + 列索引 - - Null bitmap - - Posting bitmap 0 - 值 0 对应的数据段 - - Posting bitmap 1 - 值 1 对应的数据段 - … - - Posting bitmap n - - FST - 列值 → bitmap 范围 - - - 倒排索引没有单独的 header；footer 记录定位和解码各列索引所需的元数据。 + + Null bitmap + + Posting bitmap 0 + + Posting bitmap 1 + … + + Posting bitmap n + + FST + 列值 → 范围 diff --git a/static/metric-engine-architecture.svg b/static/metric-engine-architecture.svg index 69d9fb4e07..1025b1f26b 100644 --- a/static/metric-engine-architecture.svg +++ b/static/metric-engine-architecture.svg @@ -1,75 +1,51 @@ - + Metric engine logical-to-physical mapping - Multiple logical tables share one physical Region group. The Metric engine maps logical tables and columns to a pair of Mito Regions: a metadata Region for mappings and a data Region for shared rows carrying logical table identity. + Multiple logical tables share a physical Region group. The Metric engine stores table and column mappings in the metadata Region and rows from the logical tables in the data Region. Both Regions are managed by Mito. - - - - Logical-to-physical mapping - Many logical tables share one physical Region group - - - Logical tables - - Logical table A - independent schema and identity - - Logical table B - independent schema and identity - - Logical table C - independent schema and identity - - - - - - - Metric engine - maps logical requests to shared storage - - Resolve logical table - - Map logical columns - - shared partition layout - - - mappings - - rows - - - Mito engine - a physical Region group contains two Mito Regions - - - Metadata Region - logical table mappings - logical columns → physical columns - - - Data Region - rows from mapped logical tables - - logical table identity per row - - - Write: - attach logical table identity before writing the shared data Region. - Read: - add a logical-table filter before scanning the data Region. + + + Logical tables + + Table A + + Table B + + Table C + + + + + + Metric engine + logical → physical + + + + mappings + + + rows + + + Mito + physical Region group + + + Metadata Region + table · column mappings + + + Data Region + shared rows · __table_id diff --git a/static/metric-engine-architecture.zh.svg b/static/metric-engine-architecture.zh.svg index dd5821f428..1b3e60ee63 100644 --- a/static/metric-engine-architecture.zh.svg +++ b/static/metric-engine-architecture.zh.svg @@ -1,75 +1,51 @@ - - Metric 引擎的逻辑表到物理表映射 - 多个逻辑表共享一个物理 Region 组。Metric 引擎把逻辑表和逻辑列映射到两个 Mito Region：元数据 Region 保存映射，数据 Region 保存带有逻辑表身份的共享数据。 + + Metric engine 的逻辑表到物理 Region 映射 + 多个逻辑表共享一个物理 Region 组。Metric engine 将表和列映射保存在元数据 Region 中，并将逻辑表的数据行保存在数据 Region 中。这两个 Region 均由 Mito 管理。 - - - - 逻辑表到物理表的映射 - 多个逻辑表共享一个物理 Region 组 - - - 逻辑表 - - 逻辑表 A - 独立的 schema 和表身份 - - 逻辑表 B - 独立的 schema 和表身份 - - 逻辑表 C - 独立的 schema 和表身份 - - - - - - - Metric 引擎 - 将逻辑请求映射到共享存储 - - 解析逻辑表 - - 映射逻辑列 - - 共享分区布局 - - - 映射 - - 数据行 - - - Mito 引擎 - 一个物理 Region 组包含两个 Mito Region - - - 元数据 Region - 逻辑表映射 - 逻辑列 → 物理列 - - - 数据 Region - 保存所映射逻辑表的数据行 - - 每行记录逻辑表身份 - - - 写入： - 附加逻辑表身份后写入共享数据 Region。 - 读取： - 扫描数据 Region 前增加逻辑表过滤条件。 + + + 逻辑表 + + 逻辑表 A + + 逻辑表 B + + 逻辑表 C + + + + + + Metric engine + 逻辑 → 物理 + + + + 映射 + + + 数据行 + + + Mito + 物理 Region 组 + + + 元数据 Region + 表映射 · 列映射 + + + 数据 Region + 共享数据行 · __table_id diff --git a/static/mito-sst-layout.svg b/static/mito-sst-layout.svg index 675c4c42a8..64415791dc 100644 --- a/static/mito-sst-layout.svg +++ b/static/mito-sst-layout.svg @@ -1,62 +1,52 @@ - - Mito Parquet SST layout - Mito records file-level metadata for a Parquet SST and divides the file into independently readable row groups. Row groups store field columns, the time index, the encoded primary key, sequence, and operation type, plus raw primary-key columns when the selected encoding includes them. An SST is not shown as belonging to exactly one compaction time window because a file can span windows. + + Default flat Mito SST layout + Mito records file-level metadata for a Parquet SST. Each row group stores optional raw primary-key columns, field columns, the time index, the encoded primary key, sequence, and operation type. - - Default flat SST layout - A Parquet SST stores table columns with Mito merge metadata - - - - Mito SST metadata - time range · primary-key range · row count · row-group count · available indexes - - - Parquet SST - row groups are the independently readable and skippable units - - - Row group 0 - - Raw primary-key columns - when present - - Field columns - cpu · memory - - Time index - ts - - Encoded key - __primary_key - - Merge metadata - __sequence · __op_type - - - Row group 1 - the same physical columns, containing the next independently readable group of rows - - column min / max / null count - - - Ordering: - for tables with primary keys, rows are ordered by primary key and then by time index. - Compaction: - an SST may span more than one compaction time window. + + + + SST metadata + time range · primary-key range · rows · row groups · indexes + + + Parquet SST + + + Row group 0 + + + Raw PK columns + optional + + + Fields + cpu · memory + + + Time + ts + + + Encoded key + __primary_key + + + Merge metadata + __sequence · __op_type + + + … Row group n diff --git a/static/mito-sst-layout.zh.svg b/static/mito-sst-layout.zh.svg index 315d01ba4a..d75ae0c09a 100644 --- a/static/mito-sst-layout.zh.svg +++ b/static/mito-sst-layout.zh.svg @@ -1,62 +1,52 @@ - - Mito Parquet SST 布局 - Mito 为 Parquet SST 记录文件级元数据，并把文件切分为可以独立读取的 row group。Row group 保存 field 列、time index、编码后的 primary key、sequence 和操作类型；所选编码需要时还会保存原始 primary-key 列。由于一个 SST 可以跨越多个 compaction time window，图中不把 SST 限定在单个 time window 内。 + + Mito 默认的 flat SST 布局 + Mito 为 Parquet SST 记录文件级元数据。每个 row group 保存可选的原始 primary-key 列、field 列、time index、编码后的 primary key、sequence 和操作类型。 - - 默认 FLAT SST 布局 - Parquet SST 同时保存表数据列和 Mito 合并元数据 - - - - Mito SST 元数据 - 时间范围 · primary-key 范围 · 行数 · row-group 数量 · 可用索引 - - - Parquet SST - row group 是可以独立读取和跳过的单位 - - - Row group 0 - - 原始 primary-key 列 - 所选编码需要时保存 - - Field 列 - cpu · memory - - Time index - ts - - 编码后的 key - __primary_key - - 合并元数据 - __sequence · __op_type - - - Row group 1 - 使用相同物理列，保存下一组可以独立读取的数据行 - - 列 min / max / null 数量 - - - 排序： - 对于带 primary key 的表，先按 primary key、再按 time index 排序。 - Compaction： - 一个 SST 可以跨越多个 compaction time window。 + + + + SST 元数据 + 时间范围 · primary-key 范围 · 行数 · row group 数量 · 索引 + + + Parquet SST + + + Row group 0 + + + 原始 PK 列 + 可选 + + + Field 列 + cpu · memory + + + 时间列 + ts + + + 编码后的 key + __primary_key + + + 合并元数据 + __sequence · __op_type + + + … Row group n diff --git a/static/parquet-row-group-statistics.svg b/static/parquet-row-group-statistics.svg index 83d400b5d1..e3850cef9d 100644 --- a/static/parquet-row-group-statistics.svg +++ b/static/parquet-row-group-statistics.svg @@ -1,61 +1,45 @@ - - Parquet row-group statistics pruning - A query for name equals Emily compares the predicate with column statistics stored in row-group column metadata. Row group zero is skipped because its maximum name is Charlie, while row group one remains a candidate because Emily falls between Doug and John. + + Parquet row-group pruning with column statistics + For the predicate name equals Emily, row group 0 can be skipped because its maximum name is Charlie. Row group 1 must be read because Emily falls between its minimum value Doug and maximum value John. - - - Parquet row-group pruning - Column statistics eliminate row groups before data is read - + - - Query predicate - name = "Emily" - - compare + + Predicate + name = "Emily" - - Parquet file metadata - ColumnMetaData for each row-group column chunk + + + min / max - - Row group 0 - - name - min Alice - max Charlie - - age - 18…23 - - Skip - Emily is greater than the maximum value Charlie. + + Parquet metadata - - Row group 1 - - name - min Doug - max John - - age - 20…30 - - Keep - Emily falls within the recorded min-max range. + + Row group 0 + + name + Alice … Charlie + + Skip - - Statistics prove when a row group cannot match; they do not prove that every remaining row matches. + + Row group 1 + + name + Doug … John + + Read diff --git a/static/parquet-row-group-statistics.zh.svg b/static/parquet-row-group-statistics.zh.svg index 2fc32ba804..780d2c8e95 100644 --- a/static/parquet-row-group-statistics.zh.svg +++ b/static/parquet-row-group-statistics.zh.svg @@ -1,61 +1,45 @@ - - Parquet row group 统计信息裁剪 - 查询 name 等于 Emily 时，系统将谓词与 row group 的列元数据统计信息比较。Row group 0 的 name 最大值为 Charlie，因此可以跳过；Emily 位于 Doug 和 John 之间，因此 row group 1 仍需读取。 + + 使用列统计信息裁剪 Parquet row group + 对于 name 等于 Emily 的谓词，row group 0 的 name 最大值为 Charlie，因此可以跳过。Emily 位于 row group 1 的最小值 Doug 和最大值 John 之间，因此仍需读取该 row group。 - - - PARQUET ROW GROUP 裁剪 - 读取数据前，先用列统计信息排除不匹配的 row group - + - - 查询谓词 - name = "Emily" - - 比较 + + 查询谓词 + name = "Emily" - - Parquet 文件元数据 - 每个 row group column chunk 的 ColumnMetaData + + + min / max - - Row group 0 - - name - min Alice - max Charlie - - age - 18…23 - - 跳过 - Emily 大于最大值 Charlie。 + + Parquet 元数据 - - Row group 1 - - name - min Doug - max John - - age - 20…30 - - 保留 - Emily 位于记录的 min-max 范围内。 + + Row group 0 + + name + Alice … Charlie + + 跳过 - - 统计信息只能证明某个 row group 不可能匹配，不能证明保留下来的每一行都匹配。 + + Row group 1 + + name + Doug … John + + 读取 From ec5696808f3b033ee46e7b5ffba28691bc39552b Mon Sep 17 00:00:00 2001 From: Dennis Zhuang Date: Fri, 28 Aug 2026 12:11:45 +0800 Subject: [PATCH 07/14] docs: clarify metasrv coordination paths --- docs/contributor-guide/metasrv/overview.md | 21 ++++++++++++------- .../contributor-guide/metasrv/overview.md | 21 ++++++++++++------- 2 files changed, 28 insertions(+), 14 deletions(-) diff --git a/docs/contributor-guide/metasrv/overview.md b/docs/contributor-guide/metasrv/overview.md index 91a85ef219..14f185a763 100644 --- a/docs/contributor-guide/metasrv/overview.md +++ b/docs/contributor-guide/metasrv/overview.md @@ -49,7 +49,7 @@ Region migration or failover changes this mapping. Frontend refreshes its cached ### Create Table 1. Frontend submits the DDL request to the Metasrv leader. -2. Metasrv derives Regions from the partition rules and selects a Datanode for each Region. +2. Metasrv derives Regions from the partition rules and [selects a Datanode for each Region](/contributor-guide/metasrv/selector.md). 3. A persisted procedure creates the Regions and records the table and route metadata. If leadership changes, the procedure can resume from its persisted state. 4. Metasrv notifies Frontends after the metadata change is committed so their caches can be refreshed. @@ -63,13 +63,20 @@ Frontend uses table and Region metadata while planning the query. Predicates on ## Metasrv Architecture -Metasrv combines several coordination mechanisms: +The main coordination paths are: -- A metadata layer stores cluster state through a key-value backend. -- Leader election ensures that one Metasrv coordinates metadata changes and cluster-management work. -- The Procedure Manager executes multi-step operations and persists enough state to resume them after failure. -- Heartbeat handlers update leases and Region statistics and deliver control messages. -- Region supervision uses lease state to detect unavailable Regions and start failover when appropriate. +```text +Leader election + | + v +Metasrv leader +├─ DDL manager -> Procedure manager +├─ Selector -> new Region placement +├─ Heartbeat handler chain -> leases and Region statistics +├─ Region supervisor -> Region migration procedures +├─ Mailbox -> cache invalidations and Region instructions +└─ Metadata managers -> KV backend +``` These mechanisms share metadata, but they have different failure boundaries. A process restart may discard caches and leader-local state; metadata and procedure state required for recovery must be durable. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md index 96ad0e69ad..fed848d4b5 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md @@ -49,7 +49,7 @@ Region 迁移或故障转移会修改这项映射。Frontend 刷新缓存路由 ### 创建表 1. Frontend 向 Metasrv leader 提交 DDL 请求。 -2. Metasrv 根据分区规则确定 Region，并为每个 Region 选择 Datanode。 +2. Metasrv 根据分区规则确定 Region，并[为每个 Region 选择 Datanode](/contributor-guide/metasrv/selector.md)。 3. 持久化的 Procedure 创建 Region，并写入表元数据和路由。发生 leader 切换后，Procedure 可以从已保存的状态继续执行。 4. 元数据提交后，Metasrv 通知 Frontend 刷新相关缓存。 @@ -63,13 +63,20 @@ Frontend 在查询规划期间使用表和 Region 元数据。分区列上的谓 ## Metasrv 架构 -Metasrv 由几类协调机制组成： +主要协调路径如下： -- 元数据层通过 key-value backend 保存集群状态。 -- Leader 选举保证同一时间只有一个 Metasrv 负责元数据变更和集群管理。 -- Procedure Manager 执行多步骤操作，并持久化恢复执行所需的状态。 -- 心跳处理链更新租约和 Region 统计信息，并传递控制消息。 -- Region 监控根据租约判断 Region 是否不可用，并在需要时启动故障转移。 +```text +Leader election + | + v +Metasrv leader +├─ DDL manager -> Procedure manager +├─ Selector -> 新 Region 的放置 +├─ Heartbeat handler chain -> 租约和 Region 统计信息 +├─ Region supervisor -> Region 迁移 Procedure +├─ Mailbox -> 缓存失效和 Region 指令 +└─ Metadata managers -> KV backend +``` 这些机制共享元数据，但故障边界不同。进程重启可以丢弃缓存和 leader 本地状态；恢复所需的元数据和 Procedure 状态必须持久化。 From 1be0a8f87b710b2fa8f4481b71909b71f16b5d7d Mon Sep 17 00:00:00 2001 From: Dennis Zhuang Date: Fri, 28 Aug 2026 15:17:43 +0800 Subject: [PATCH 08/14] docs: document Mito memtable design --- docs/contributor-guide/datanode/memtable.md | 93 +++++++++++++++++++ .../datanode/storage-engine.md | 6 +- docs/contributor-guide/datanode/wal.md | 2 +- docs/contributor-guide/metasrv/admin-api.md | 4 +- docs/contributor-guide/metasrv/overview.md | 12 ++- docs/reference/sql/create.md | 6 +- .../configuration.md | 19 +--- .../contributor-guide/datanode/memtable.md | 93 +++++++++++++++++++ .../datanode/storage-engine.md | 6 +- .../current/contributor-guide/datanode/wal.md | 2 +- .../contributor-guide/metasrv/admin-api.md | 4 +- .../contributor-guide/metasrv/overview.md | 14 +-- .../current/reference/sql/create.md | 6 +- .../configuration.md | 19 +--- sidebars.ts | 1 + 15 files changed, 224 insertions(+), 63 deletions(-) create mode 100644 docs/contributor-guide/datanode/memtable.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md diff --git a/docs/contributor-guide/datanode/memtable.md b/docs/contributor-guide/datanode/memtable.md new file mode 100644 index 0000000000..036d0f5f5f --- /dev/null +++ b/docs/contributor-guide/datanode/memtable.md @@ -0,0 +1,93 @@ +--- +keywords: [memtable, Mito engine, write buffer, flush, time partition, BulkMemtable] +description: How Mito organizes mutable Region data in memtables and moves it into SST files. +--- + +# Memtable design + +A memtable is Mito's in-memory write buffer for a Region. It makes writes available to reads before a flush creates SST files. A Region version identifies the memtables and SST files that a scan may read. Together with a committed-sequence fence, it keeps the scan consistent while writes and flushes advance the current version. + +## Write and flush lifecycle + +For a normal WAL-backed write, Mito uses this order: + +```text +write request + | + v +WAL append -> mutable memtable -> publish committed sequence + | + freeze + v + immutable memtable -> SST write -> manifest edit +``` + +The Region worker assigns sequence numbers and a WAL entry ID before appending the mutation to the [write-ahead log](wal.md). If the WAL append fails, Mito does not update the memtable. After the memtable update succeeds, Mito publishes the committed sequence and the rows become visible to new reads. A Region configured with `skip_wal` omits the WAL append, but keeps the same memtable and visibility ordering. + +A flush freezes the mutable memtables and installs a new mutable set before starting the background SST write. New writes therefore continue without changing the frozen data. The flush writes the immutable memtables to SST files, then persists a manifest edit containing the files and the flushed WAL and sequence checkpoints. Only after that edit is durable does Mito remove the flushed memtables from the current Region version. If the flush fails, the immutable memtables remain available for a later attempt. + +## Region versions and time partitions + +Each Region has one mutable `TimePartitions` container, which can hold more than one memtable: + +```text +Region version +├─ mutable TimePartitions +│ ├─ [t0, t1) -> memtable +│ └─ [t1, t2) -> memtable +├─ immutable memtables +└─ SST files +``` + +Mito routes each row to a partition by its time-index value. Partition ranges are half-open and aligned to a fixed duration. The duration follows the Region's compaction time window; Mito uses one day until a compaction window is available. An out-of-order write can create an earlier partition alongside the latest one. + +Freezing a Region freezes all mutable time partitions together. Mito moves their memtables to the immutable list and creates a new `TimePartitions` container. A failed flush can leave more than one generation of immutable memtables, so reads and later flushes must not assume that the list contains a single item. + +## Memtable implementations + +Mito selects a memtable implementation from the Region's SST format, primary-key encoding, and memtable options: + +```text +memtable.type=bulk -> BulkMemtable and flat SST +flat SST format or sparse primary-key encoding -> BulkMemtable +primary_key SST, dense encoding, primary key exists -> TimeSeriesMemtable +primary_key SST, dense encoding, no primary key -> SimpleBulkMemtable +``` + +With the default engine configuration, a Region without an explicit SST format uses `flat`, so `BulkMemtable` is the normal path. The rules prevent incompatible combinations: flat format or sparse primary-key encoding requires `BulkMemtable`, while explicitly selecting the bulk implementation forces flat format. + +### BulkMemtable + +`BulkMemtable` keeps incoming rows in the flat Arrow layout used by the flat SST path. Writes add record-batch parts instead of inserting one row at a time into a per-series structure. Small fragments first accumulate in an unordered part. Background memtable compaction merges eligible parts and can encode larger merged parts. + +Each part is exposed as a memtable range with its own statistics. A scan can prune ranges before opening their readers, while a flush can merge and deduplicate multiple ranges. An encoded range can be written as an SST without decoding and encoding its rows again. + +### TimeSeriesMemtable + +`TimeSeriesMemtable` groups rows by encoded primary key. Each series stores its timestamps, sequence numbers, operation types, and field values in column builders. When a reader requests the series, the memtable builds a batch ordered by timestamp and sequence and applies the Region's deduplication or merge mode. + +This implementation is used for the `primary_key` SST format with dense primary-key encoding unless the Region explicitly selects the bulk implementation. If the Region has no primary-key columns, the same builder creates a `SimpleBulkMemtable` instead of a series map. + +The removed `partition_tree` memtable is not a third implementation. The option parser accepts `memtable.type=partition_tree` for compatibility, but it does not recreate that implementation. The Region uses the bulk and flat path. + +## Read snapshots + +A scan obtains the Region version and committed sequence together from `VersionControl`. It selects the version before applying the sequence fence. Reading the sequence separately before the version could pair that sequence with a later version after flush or compaction removes an older input, producing an incomplete snapshot. + +The selected version supplies mutable memtables, immutable memtables, and SST files. Mito first prunes sources by time range, then asks each memtable for ranges using the scan's projection, predicate, and sequence bounds. The scan merges the resulting ranges with SST ranges and applies the same ordering, deletion, and merge semantics across all sources. References held by the scan keep an older memtable alive even after a newer Region version removes it. + +## Memory pressure + +Each memtable tracks its estimated heap allocation through the engine's write-buffer manager. Freezing a memtable removes its allocation from the mutable-memory count, but total usage includes the allocation until all references to that memtable are released. The mutable-memory count therefore tracks data that can still accept writes, while total usage continues to include memory retained by active scans. + +The global write-buffer limit causes workers to select Regions for flush. If memory remains above the configured limits, Mito stalls writes and can reject them at a higher threshold. An optional per-Region limit applies the same pressure to one hot Region without stalling unrelated Regions. Periodic, manual, and Region lifecycle operations can also request a flush. + +## Constraints for changes + +Changes to memtable code must preserve these properties: + +- For a WAL-backed Region, append to the WAL before installing rows in a memtable. Publish the committed sequence only after installation succeeds. +- Keep frozen memtables readable and retryable until the SST files and manifest edit are durable. +- Obtain the Region version and committed sequence from the same `VersionControl` snapshot; never read the sequence separately before the version. +- Preserve the ordering and metadata that scans and flushes need to apply the same deletion, deduplication, and merge rules across memtable and SST ranges. +- Charge allocations to the write-buffer manager and release them only when the underlying memory can no longer be referenced. diff --git a/docs/contributor-guide/datanode/storage-engine.md b/docs/contributor-guide/datanode/storage-engine.md index b800c218eb..36fef62833 100644 --- a/docs/contributor-guide/datanode/storage-engine.md +++ b/docs/contributor-guide/datanode/storage-engine.md @@ -23,9 +23,9 @@ The architecture is the same as a traditional LSMT engine: media. - Log records of the WAL can be stored on the local disk, or in a remote log service such as Kafka (remote WAL) that implements the `Log Store` API. -- Memtables: - - Data is written into the `active memtable`, aka `mutable memtable` first. - - When a `mutable memtable` is full, it will be changed to a `read-only memtable`, aka `immutable memtable`. +- [Memtables](memtable.md): + - Mito routes rows by time index into mutable memtables. + - A flush freezes the mutable memtables, installs a new mutable set for writes, and writes the frozen memtables to SST files. - SST - The full name of SST, aka SSTable is `Sorted String Table`. - `Immutable memtable` is flushed to persistent storage and produces an SST file. diff --git a/docs/contributor-guide/datanode/wal.md b/docs/contributor-guide/datanode/wal.md index bb6e733bae..8f898d91d6 100644 --- a/docs/contributor-guide/datanode/wal.md +++ b/docs/contributor-guide/datanode/wal.md @@ -7,7 +7,7 @@ description: Introduction to Write-Ahead Logging (WAL) in GreptimeDB, its purpos ## Introduction -Mito buffers writes in memtables before flushing them to SST files. It first appends each Region's mutations to the write-ahead log (WAL), so data that has not reached an SST can be recovered. +Mito buffers writes in [memtables](memtable.md) before flushing them to SST files. It first appends each Region's mutations to the write-ahead log (WAL), so data that has not reached an SST can be recovered. The WAL uses a common log-store abstraction with local raft-engine and remote Kafka providers. diff --git a/docs/contributor-guide/metasrv/admin-api.md b/docs/contributor-guide/metasrv/admin-api.md index 0b50d6733f..d1f2ee6dbe 100644 --- a/docs/contributor-guide/metasrv/admin-api.md +++ b/docs/contributor-guide/metasrv/admin-api.md @@ -190,7 +190,7 @@ These endpoints inspect or repair the table ID sequence: - `GET /admin/sequence/table/next-id`: return the next table ID without allocating it. - `POST /admin/sequence/table/set-next-id`: advance the next table ID. -Setting the sequence requires recovery mode. The new value must be greater than the current value; the API cannot move the sequence backwards. +Setting the sequence requires recovery mode. The new value must be greater than the current value; the API cannot move the sequence backwards. Recovery mode is an API precondition, not a DDL barrier. Follow [Manage table ID sequences](/user-guide/deployments-administration/maintenance/sequence-management.md) for the required cluster-wide procedure. ```bash curl -X POST \ @@ -199,4 +199,4 @@ curl -X POST \ http://localhost:4000/admin/sequence/table/set-next-id ``` -Changing this value affects IDs allocated to future tables. Use the endpoint only when repairing metadata after confirming the required next ID. +Changing this value affects IDs allocated to future tables. diff --git a/docs/contributor-guide/metasrv/overview.md b/docs/contributor-guide/metasrv/overview.md index 14f185a763..21f817a469 100644 --- a/docs/contributor-guide/metasrv/overview.md +++ b/docs/contributor-guide/metasrv/overview.md @@ -35,16 +35,18 @@ Datanode `-- heartbeat, lease renewal, Region stats -> Metasrv leader ``` -A table route maps each Region to its current Datanode peer. It does not contain a separate list of read replicas: +In steady state, a table route records one leader peer and zero or more follower peers for each Region. The leader is the write target. Deployments with read-replica support can route reads to followers: ```text Table route - |-- Region 0 -> Datanode A - |-- Region 1 -> Datanode B - `-- Region 2 -> Datanode C + |-- Region 0 + | |-- leader -> Datanode A + | `-- followers -> Datanode B, Datanode C + `-- Region 1 + `-- leader -> Datanode D ``` -Region migration or failover changes this mapping. Frontend refreshes its cached route before sending subsequent reads or writes to the new peer. +Region migration or failover changes peer roles and can temporarily leave a Region without a leader. Frontend refreshes its cached route before sending subsequent reads or writes to the current peers. ### Create Table diff --git a/docs/reference/sql/create.md b/docs/reference/sql/create.md index 4eccb986d0..29a9cf666f 100644 --- a/docs/reference/sql/create.md +++ b/docs/reference/sql/create.md @@ -26,7 +26,7 @@ If the `db_name` database already exists, then GreptimeDB has the following beha The database can also carry options similar to the `CREATE TABLE` statement by using the `WITH` keyword. The following options are available for databases: - `ttl` - Time-To-Live for data in all tables within the database (cannot be set to `instant`) -- `memtable.type` - Type of memtable (`time_series`, `partition_tree`) +- `memtable.type` - Type of memtable (`bulk`, `time_series`) - `append_mode` - Whether tables in the database should be append-only (`true`/`false`) - `merge_mode` - Strategy for merging duplicate rows (`last_row`, `last_non_null`) - `skip_wal` - Whether to disable Write-Ahead-Log for tables in the database (`'true'`/`'false'`) @@ -74,7 +74,7 @@ Create a database with multiple options, including append mode and custom memtab ```sql CREATE DATABASE test WITH ( ttl='30d', - 'memtable.type'='partition_tree', + 'memtable.type'='bulk', 'append_mode'='true' ); ``` @@ -154,7 +154,7 @@ Users can add table options by using `WITH`. The valid options contain the follo | `compaction.twcs.trigger_file_num` | Number of files in a specific time window to trigger a compaction | String value, such as '8'. Only available when `compaction.type` is `twcs`. You can refer to this [document](https://cassandra.apache.org/doc/latest/cassandra/managing/operating/compaction/twcs.html) to learn more about the `twcs` compaction strategy. | | `compaction.twcs.time_window` | Compaction time window | String value, such as '1d' for 1 day. The table usually partitions rows into different time windows by their timestamps. Only available when `compaction.type` is `twcs`. | | `compaction.twcs.max_output_file_size` | Maximum allowed output file size for TWCS compaction | String value, such as '1GB', '512MB'. Sets the maximum size for files produced by TWCS compaction. Only available when `compaction.type` is `twcs`. | -| `memtable.type` | Type of the memtable. | String value, supports `time_series`, `partition_tree`. | +| `memtable.type` | Type of the memtable | String value: `bulk` or `time_series`. If omitted, Mito selects the implementation from the SST format; the default flat format uses `bulk`. Setting `bulk` forces `sst_format=flat`, and flat SSTs use the bulk implementation even if `time_series` is specified. The legacy value `partition_tree` is accepted for compatibility and maps to the bulk and flat path. | | `append_mode` | Whether the table is append-only | String value. Default is 'false', which removes duplicate rows by primary keys and timestamps according to the `merge_mode`. Setting it to 'true' to enable append mode and create an append-only table which keeps duplicate rows. | | `merge_mode` | The strategy to merge duplicate rows | String value. Only available when `append_mode` is 'false'. Default is `last_row`, which keeps the last row for the same primary key and timestamp. Setting it to `last_non_null` to keep the last non-null field for the same primary key and timestamp. | | `sst_format` | The format of SST files | String value, supports `primary_key`, `flat`. Default is `flat`. `flat` is recommended for tables which have a large number of unique primary keys. | diff --git a/docs/user-guide/deployments-administration/configuration.md b/docs/user-guide/deployments-administration/configuration.md index a8b99a3014..5684faaac0 100644 --- a/docs/user-guide/deployments-administration/configuration.md +++ b/docs/user-guide/deployments-administration/configuration.md @@ -603,20 +603,9 @@ create_on_compaction = "auto" apply_on_query = "auto" mem_threshold_on_create = "64M" intermediate_path = "" - -[region_engine.mito.memtable] -type = "time_series" ``` -The `mito` engine provides an experimental memtable which optimizes for write performance and memory efficiency under large amounts of time-series. Its read performance might not as fast as the default `time_series` memtable. - -```toml -[region_engine.mito.memtable] -type = "partition_tree" -index_max_keys_per_shard = 8192 -data_freeze_threshold = 32768 -fork_dictionary_bytes = "1GiB" -``` +Mito selects the memtable implementation for each Region according to its table options and SST format. When `default_flat_format` is `true`, Regions without an explicit `sst_format` use flat SSTs and the bulk memtable. Configure `memtable.type` as a database or table option; `[region_engine.mito.memtable]` is not an engine setting. See [table options](/reference/sql/create.md#table-options). Available options: @@ -651,7 +640,7 @@ Available options: | `scan_memory_on_exhausted` | String | `fail` | Behavior when scan memory is exhausted. Options: `fail` (fail fast), `wait` or `wait()` (wait for memory). | | `min_compaction_interval` | String | `0m` | Minimum time interval between two compactions. Set to "0m" (default) to allow compactions to run immediately without restriction. | | `schedule_compaction_after_edit` | Bool | `true` | Whether to allow scheduling a compaction after a successful region edit.
Setting this to `true` is a necessary but not sufficient condition for scheduling compaction after a region edit. Other constraints, such as `min_compaction_interval`, may still prevent compaction from being scheduled.
Setting this to `false` guarantees that compaction will not be scheduled after a region edit. | -| `default_flat_format` | Bool | `true` | Whether to enable flat format as the default SST format. | +| `default_flat_format` | Bool | `true` | Whether Regions without an explicit `sst_format` use flat SSTs. Flat SSTs use the bulk memtable. | | `experimental_series_scan_v2` | Bool | `true` | Whether to enable the experimental two-phase mode for series scans of metric engine physical regions. Set to `false` to use the legacy mode. Other series scans also use the legacy mode. | | `scan_parallelism` | Integer | `0` | (Deprecated, use `max_concurrent_scan_files` instead) Legacy option for scan parallelism. | | `index` | -- | -- | The options for index in Mito engine. | @@ -668,10 +657,6 @@ Available options: | `inverted_index.apply_on_query` | String | `auto` | Whether to apply the index on query
- `auto`: automatically
- `disable`: never | | `inverted_index.mem_threshold_on_create` | String | `64M` | Memory threshold for performing an external sort during index creation.
Setting to empty will disable external sorting, forcing all sorting operations to happen in memory. | | `inverted_index.intermediate_path` | String | `""` | File system path to store intermediate files for external sorting (default `{data_home}/index_intermediate`). | -| `memtable.type` | String | `time_series` | Memtable type.
- `time_series`: time-series memtable
- `partition_tree`: partition tree memtable (experimental) | -| `memtable.index_max_keys_per_shard` | Integer | `8192` | The max number of keys in one shard.
Only available for `partition_tree` memtable. | -| `memtable.data_freeze_threshold` | Integer | `32768` | The max rows of data inside the actively writing buffer in one shard.
Only available for `partition_tree` memtable. | -| `memtable.fork_dictionary_bytes` | String | `1GiB` | Max dictionary bytes.
Only available for `partition_tree` memtable. | The `metric` engine is optimized for handling metrics data with a large number of small tables. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md new file mode 100644 index 0000000000..392fc2e9af --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md @@ -0,0 +1,93 @@ +--- +keywords: [memtable, Mito engine, 写缓冲, flush, 时间分区, BulkMemtable] +description: 介绍 Mito 如何在 memtable 中组织 Region 的可变数据，以及如何将这些数据写入 SST 文件。 +--- + +# Memtable 设计 + +Memtable 是 Mito 为每个 Region 维护的内存写缓冲。数据 flush 为 SST 文件前，读取可以先从 memtable 获取这些数据。Region version 确定 scan 可以读取的 memtable 和 SST 文件；配合 committed sequence 上限，scan 可以在写入和 flush 推进当前 version 时保持一致性。 + +## 写入和 flush 生命周期 + +对于使用 WAL 的常规写入，Mito 按以下顺序处理： + +```text +写请求 + | + v +追加 WAL -> mutable memtable -> 发布 committed sequence + | + freeze + v + immutable memtable -> 写入 SST -> manifest edit +``` + +Region worker 先分配 sequence number 和 WAL entry ID，再将 mutation 追加到[预写日志](wal.md)。如果追加失败，Mito 不会更新 memtable。Memtable 更新成功后，Mito 发布 committed sequence，新的读取随后可以看到这些数据。配置了 `skip_wal` 的 Region 会跳过 WAL，但 memtable 更新和可见性顺序不变。 + +Flush 在启动后台 SST 写入前，先冻结 mutable memtable 并安装一组新的 mutable memtable。后续写入可以继续进行，也不会修改已经冻结的数据。Flush 将 immutable memtable 写为 SST 文件，再持久化包含新文件、flushed WAL checkpoint 和 sequence checkpoint 的 manifest edit。只有 manifest edit 持久化成功后，Mito 才会从当前 Region version 中移除已 flush 的 memtable。Flush 失败时，immutable memtable 会保留，供后续任务重试。 + +## Region version 和时间分区 + +每个 Region 包含一个 mutable `TimePartitions` 容器，其中可以有多个 memtable： + +```text +Region version +├─ mutable TimePartitions +│ ├─ [t0, t1) -> memtable +│ └─ [t1, t2) -> memtable +├─ immutable memtables +└─ SST files +``` + +Mito 根据 time index 的值把每行数据路由到对应分区。分区使用左闭右开的时间范围，并按照固定时长对齐。该时长跟随 Region 的 compaction time window；在取得 compaction time window 前，Mito 使用一天作为初始值。乱序写入可能在最新分区之外创建更早的分区。 + +冻结 Region 时，Mito 会同时冻结所有 mutable 时间分区，把其中的 memtable 移入 immutable 列表，再创建新的 `TimePartitions` 容器。Flush 失败可能留下多代 immutable memtable，因此读取和后续 flush 不能假定列表中只有一个对象。 + +## Memtable 实现 + +Mito 根据 Region 的 SST format、primary key encoding 和 memtable 选项选择实现： + +```text +memtable.type=bulk -> BulkMemtable 和 flat SST +flat SST format 或 sparse primary-key encoding -> BulkMemtable +primary_key SST、dense encoding、有 primary key -> TimeSeriesMemtable +primary_key SST、dense encoding、无 primary key -> SimpleBulkMemtable +``` + +使用默认 engine 配置时，没有显式指定 SST format 的 Region 会使用 `flat`，因此通常走 `BulkMemtable` 路径。这些规则会排除不兼容的组合：flat format 或 sparse primary key encoding 必须使用 `BulkMemtable`；显式选择 bulk 实现则会强制使用 flat format。 + +### BulkMemtable + +`BulkMemtable` 按照 flat SST 读取路径使用的 Arrow 布局保存写入数据。每次写入增加一个 record batch part，而不是逐行插入按时间序列组织的数据结构。较小的 fragment 会先进入 unordered part；后台 memtable compaction 合并符合条件的 part，并可以编码合并后的较大 part。 + +每个 part 都作为独立的 memtable range 暴露，并带有自己的统计信息。Scan 可以在打开 reader 前裁剪 range，flush 则可以合并多个 range 并执行去重。已经编码的 range 可以直接写为 SST，无需再次解码并编码其中的行。 + +### TimeSeriesMemtable + +`TimeSeriesMemtable` 按编码后的 primary key 对数据行分组。每条时间序列分别保存 timestamp、sequence number、operation type 和 field value 的列构建器。Reader 读取时间序列时，memtable 按 timestamp 和 sequence 构造 batch，并应用 Region 的去重或 merge mode。 + +除非 Region 显式选择 bulk 实现，否则使用 dense primary key encoding 的 `primary_key` SST format 会选择该实现。如果 Region 没有 primary key 列，同一个 builder 会创建 `SimpleBulkMemtable`，而不创建 series map。 + +已经删除的 `partition_tree` memtable 不是第三种实现。Option parser 仍接受 `memtable.type=partition_tree` 以兼容旧配置，但不会恢复该实现。Region 最终使用 bulk 和 flat 路径。 + +## 读取快照 + +Scan 通过一次 `VersionControl` 快照同时取得 Region version 和 committed sequence，并先确定 version，再应用 sequence 上限。如果单独读取 sequence 后再获取 version，flush 或 compaction 可能在两次读取之间移除旧输入，使 scan 得到不完整的快照。 + +选定的 version 提供 mutable memtable、immutable memtable 和 SST 文件。Mito 先按照时间范围裁剪数据源，再使用 scan 的 projection、predicate 和 sequence range 从各 memtable 获取 range。Scan 将这些 range 与 SST range 合并，并在所有数据源上应用相同的排序、删除和 merge 语义。即使新的 Region version 已经移除某个 memtable，scan 持有的引用也会让该 memtable 存活到本次读取结束。 + +## 内存压力 + +每个 memtable 通过 engine 的 write-buffer manager 记录估算的堆内存分配量。冻结 memtable 后，这部分内存不再计入 mutable memory，但在所有引用释放前仍计入总用量。因此，mutable memory 只反映仍可接收写入的数据，总用量仍包含活跃 scan 保留的内存。 + +全局 write-buffer 达到限制后，worker 会选择 Region 执行 flush。如果内存用量持续超过配置限制，Mito 会阻塞写入，并在达到更高阈值后拒绝写入。可选的 Region 级限制会单独约束热点 Region，避免其阻塞无关 Region。定期任务、手动请求和 Region 生命周期操作也可以触发 flush。 + +## 修改约束 + +修改 memtable 代码时必须保持以下性质： + +- 对于使用 WAL 的 Region，先追加 WAL，再把数据安装到 memtable；只有安装成功后才能发布 committed sequence。 +- SST 文件和 manifest edit 持久化前，冻结的 memtable 必须保持可读，并能在 flush 失败后重试。 +- 从同一个 `VersionControl` 快照取得 Region version 和 committed sequence，不得先单独读取 sequence 再获取 version。 +- 保留 scan 和 flush 在 memtable range 与 SST range 之间执行统一删除、去重和 merge 所需的排序及元数据。 +- 通过 write-buffer manager 记录内存分配，并且只在底层内存不再可能被引用时释放计数。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md index 261a38bcff..d3abb58cfa 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/storage-engine.md @@ -20,9 +20,9 @@ Mito 是 GreptimeDB 的默认存储引擎，基于 [LSM tree][1]，面向时间 - 为尚未刷盘的数据提供高持久性保证。 - 基于 `LogStore` API 实现，不关心底层存储介质。 - WAL 的日志记录可以存储在本地磁盘上，也可以存储在实现了 `LogStore` API 的远程日志服务中，例如 Kafka（remote WAL）。 -- Memtable - - 数据首先写入 `active memtable`，又称 `mutable memtable`。 - - 当 `mutable memtable` 已满时，它将变为只读的 `immutable memtable`。 +- [Memtable](memtable.md) + - Mito 根据 time index 将数据行写入 mutable memtable。 + - Flush 冻结 mutable memtable，安装一组新的 mutable memtable 以接收写入，再将冻结的 memtable 写为 SST 文件。 - SST - SST 的全名为有序字符串表（`Sorted String Table`）。 - `immutable memtable` 刷到持久存储后形成一个 SST 文件。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/wal.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/wal.md index 805b294091..36e0689430 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/wal.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/wal.md @@ -9,7 +9,7 @@ description: 介绍了 GreptimeDB 的预写日志（WAL）机制，包括其命 ## 介绍 -Mito 在将数据 flush 为 SST 文件前，先在 memtable 中缓冲写入。每个 Region 的 mutation 会先追加到预写日志（WAL），从而恢复尚未进入 SST 的数据。 +Mito 在将数据 flush 为 SST 文件前，先在 [memtable](memtable.md) 中缓冲写入。每个 Region 的 mutation 会先追加到预写日志（WAL），从而恢复尚未进入 SST 的数据。 WAL 通过统一的 log-store 抽象访问，可以使用本地 raft-engine 或远端 Kafka。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/admin-api.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/admin-api.md index bd8b2551ca..bd28206c6f 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/admin-api.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/admin-api.md @@ -190,7 +190,7 @@ Recovery mode 控制手动修改 table ID sequence 等元数据修复端点。 - `GET /admin/sequence/table/next-id`：返回下一个 table ID，但不执行分配。 - `POST /admin/sequence/table/set-next-id`：推进下一个 table ID。 -设置 sequence 前必须开启 recovery mode。新值必须大于当前值，不能通过该 API 回退 sequence。 +设置 sequence 前必须开启 recovery mode。新值必须大于当前值，不能通过该 API 回退 sequence。Recovery mode 只是该 API 的前置条件，不能阻止 DDL。执行该操作时，必须遵循[管理 Table ID Sequence](/user-guide/deployments-administration/maintenance/sequence-management.md)中的完整集群操作流程。 ```bash curl -X POST \ @@ -199,4 +199,4 @@ curl -X POST \ http://localhost:4000/admin/sequence/table/set-next-id ``` -该操作会影响后续新表分配到的 ID。只有在确认所需 next ID 后，才能用它修复元数据。 +该操作会影响后续新表分配到的 ID。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md index fed848d4b5..3402eadb66 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md @@ -12,7 +12,7 @@ Metasrv 是 GreptimeDB 分布式集群中的元数据和协调服务，不参与 - 存储 Catalog、Schema、Table、Region、路由和节点元数据； - 为新 Region 选择 Datanode，并维护表路由； - 选举一个 Metasrv leader 负责协调元数据变更； -- 通过可恢复的 Procedure 执行 DDL、Region 迁移、故障转移和 repartition； +- 通过可恢复的 Procedure 执行 DDL、Region 迁移、故障转移和重分区； - 通过心跳维护节点租约和 Region 统计信息； - 在缓存元数据或 Region 状态变化时通知 Frontend 和 Datanode。 @@ -35,16 +35,18 @@ Datanode `-- 心跳、租约续期和 Region 统计信息 ----> Metasrv leader ``` -表路由把每个 Region 映射到当前 Datanode peer，其中没有单独的只读副本列表： +在稳定状态下，表路由为每个 Region 记录一个 leader peer 和零个或多个 follower peer。Leader 是写入目标；支持只读副本的部署可以把读取路由到 follower： ```text Table route - |-- Region 0 -> Datanode A - |-- Region 1 -> Datanode B - `-- Region 2 -> Datanode C + |-- Region 0 + | |-- leader -> Datanode A + | `-- followers -> Datanode B, Datanode C + `-- Region 1 + `-- leader -> Datanode D ``` -Region 迁移或故障转移会修改这项映射。Frontend 刷新缓存路由后，再把后续读写发送给新的 peer。 +Region 迁移或故障转移会改变 peer 角色，并可能使 Region 暂时没有 leader。Frontend 刷新缓存路由后，再把后续读写发送给当前 peer。 ### 创建表 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference/sql/create.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference/sql/create.md index 5e1d660bf0..660ca1751e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/reference/sql/create.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference/sql/create.md @@ -26,7 +26,7 @@ CREATE DATABASE [IF NOT EXISTS] db_name [WITH ] 数据库也可以通过使用 `WITH` 关键字配置与 `CREATE TABLE` 语句类似的选项。数据库支持以下选项： - `ttl` - 数据库中所有表的数据存活时间（不能设置为 `instant`） -- `memtable.type` - 内存表类型（`time_series`、`partition_tree`） +- `memtable.type` - memtable 类型（`bulk`、`time_series`） - `append_mode` - 数据库中的表是否为仅追加模式（`true`/`false`） - `merge_mode` - 合并重复行的策略（`last_row`、`last_non_null`） - `skip_wal` - 是否为数据库中的表禁用预写日志（`'true'`/`'false'`） @@ -74,7 +74,7 @@ CREATE DATABASE test WITH (ttl='7d'); ```sql CREATE DATABASE test WITH ( ttl='30d', - 'memtable.type'='partition_tree', + 'memtable.type'='bulk', 'append_mode'='true' ); ``` @@ -156,7 +156,7 @@ GreptimeDB 提供了丰富的索引实现来加速查询，请在[索引](/user- | `compaction.twcs.trigger_file_num` | 某个窗口内触发 compaction 的最小文件数量阈值 | 字符串值，如 '8'。只在 `compaction.type` 为 `twcs` 时可用 | | `compaction.twcs.time_window` | Compaction 时间窗口 | 字符串值，如 '1d' 表示 1 天。该表会根据时间戳将数据分区到不同的时间窗口中。只在 `compaction.type` 为 `twcs` 时可用 | | `compaction.twcs.max_output_file_size` | TWCS compaction 的最大输出文件大小 | 字符串值，如 '1GB'、'512MB'。设置 TWCS compaction 产生的文件的最大大小。只在 `compaction.type` 为 `twcs` 时可用 | -| `memtable.type` | memtable 的类型 | 字符串值，支持 `time_series`，`partition_tree` | +| `memtable.type` | memtable 类型 | 字符串值：`bulk` 或 `time_series`。未设置时，Mito 根据 SST format 选择实现；默认的 flat format 使用 `bulk`。设置 `bulk` 会强制使用 `sst_format=flat`；使用 flat SST 时，即使设置了 `time_series`，Mito 也会选择 bulk 实现。旧值 `partition_tree` 仅为兼容保留，并映射到 bulk 和 flat 路径。 | | `append_mode` | 该表是否时 append-only 的 | 字符串值。默认值为 'false'，根据 'merge_mode' 按主键和时间戳删除重复行。设置为 'true' 可以开启 append 模式和创建 append-only 表，保留所有重复的行 | | `merge_mode` | 合并重复行的策略 | 字符串值。只有当 `append_mode` 为 'false' 时可用。默认值为 `last_row`，保留相同主键和时间戳的最后一行。设置为 `last_non_null` 则保留相同主键和时间戳的最后一个非空字段。 | | `sst_format` | SST 文件的格式 | 字符串值，支持 `primary_key`，`flat`。默认为 `flat`。`flat` 格式建议用于具有高基数主键的表。 | diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/configuration.md b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/configuration.md index 9b1d7d7b3b..f2d9ba3de1 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/configuration.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/configuration.md @@ -607,20 +607,9 @@ create_on_compaction = "auto" apply_on_query = "auto" mem_threshold_on_create = "64M" intermediate_path = "" - -[region_engine.mito.memtable] -type = "time_series" ``` -此外，`mito` 也提供了一个实验性质的 memtable。该 memtable 主要优化大量时间序列下的写入性能和内存占用。其查询性能可能会不如默认的 `time_series` memtable。 - -```toml -[region_engine.mito.memtable] -type = "partition_tree" -index_max_keys_per_shard = 8192 -data_freeze_threshold = 32768 -fork_dictionary_bytes = "1GiB" -``` +Mito 根据表选项和 SST format 为每个 Region 选择 memtable 实现。`default_flat_format` 为 `true` 时，没有显式设置 `sst_format` 的 Region 使用 flat SST 和 bulk memtable。`memtable.type` 是数据库或表选项，不是 `[region_engine.mito.memtable]` 引擎配置。详见[表选项](/reference/sql/create.md#表选项)。 以下是可供使用的选项 @@ -655,7 +644,7 @@ fork_dictionary_bytes = "1GiB" | `scan_memory_on_exhausted` | 字符串 | `fail` | 扫描内存耗尽时的行为。选项：`fail`（快速失败），`wait` 或 `wait()`（等待内存）。 | | `min_compaction_interval` | 字符串 | `0m` | 两次 compaction 之间的最小时间间隔。设为 "0m"（默认）允许 compactions 立即运行，无限制。 | | `schedule_compaction_after_edit` | 布尔值 | `true` | 是否允许在成功的 region edit 之后调度 compaction。
设为 `true` 是在 region edit 后调度 compaction 的必要但不充分条件，`min_compaction_interval` 等其他约束仍可能阻止 compaction 被调度。
设为 `false` 则保证 region edit 后不会调度 compaction。 | -| `default_flat_format` | 布尔值 | `true` | 是否启用 Flat 格式作为默认 SST 格式。 | +| `default_flat_format` | 布尔值 | `true` | 没有显式设置 `sst_format` 的 Region 是否使用 flat SST。Flat SST 使用 bulk memtable。 | | `experimental_series_scan_v2` | 布尔值 | `true` | 是否为 metric 引擎物理 region 的 series scan 启用实验性的 two-phase 模式。设为 `false` 时使用 legacy 模式，其他 series scan 也使用 legacy 模式。 | | `scan_parallelism` | 整数 | `0` | （已弃用，请使用 `max_concurrent_scan_files`）旧版扫描并发度选项。 | | `index` | -- | -- | Mito 引擎中索引的选项。 | @@ -671,10 +660,6 @@ fork_dictionary_bytes = "1GiB" | `inverted_index.apply_on_query` | 字符串 | `auto` | 是否在查询时使用索引
- `auto`: 自动
- `disable`: 从不 | | `inverted_index.mem_threshold_on_create` | 字符串 | `64M` | 创建索引时如果超过该内存阈值则改为使用外部排序
设置为空会关闭外排，在内存中完成所有排序 | | `inverted_index.intermediate_path` | 字符串 | `""` | 存放外排临时文件的路径 (默认 `{data_home}/index_intermediate`). | -| `memtable.type` | 字符串 | `time_series` | Memtable type.
- `time_series`: time-series memtable
- `partition_tree`: partition tree memtable (实验性功能) | -| `memtable.index_max_keys_per_shard` | 整数 | `8192` | 一个 shard 内的主键数
只对 `partition_tree` memtable 生效 | -| `memtable.data_freeze_threshold` | 整数 | `32768` | 一个 shard 内写缓存可容纳的最大行数
只对 `partition_tree` memtable 生效 | -| `memtable.fork_dictionary_bytes` | 字符串 | `1GiB` | 主键字典的大小
只对 `partition_tree` memtable 生效 | `metric` 引擎针对包含大量小表的 metrics 数据进行了优化。 diff --git a/sidebars.ts b/sidebars.ts index 6955ab0039..0a82e3abc3 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -762,6 +762,7 @@ const sidebars: SidebarsConfig = { label: 'Overview', }, 'contributor-guide/datanode/storage-engine', + 'contributor-guide/datanode/memtable', 'contributor-guide/datanode/query-engine', 'contributor-guide/datanode/data-persistence-indexing', 'contributor-guide/datanode/wal', From 530014295f509788a0fd9f46fb2641acbbc3fef9 Mon Sep 17 00:00:00 2001 From: Dennis Zhuang Date: Fri, 28 Aug 2026 16:12:14 +0800 Subject: [PATCH 09/14] docs: clarify BulkMemtable layout --- docs/contributor-guide/datanode/memtable.md | 14 ++++++++++++-- .../current/contributor-guide/datanode/memtable.md | 14 ++++++++++++-- 2 files changed, 24 insertions(+), 4 deletions(-) diff --git a/docs/contributor-guide/datanode/memtable.md b/docs/contributor-guide/datanode/memtable.md index 036d0f5f5f..0718840b12 100644 --- a/docs/contributor-guide/datanode/memtable.md +++ b/docs/contributor-guide/datanode/memtable.md @@ -58,9 +58,19 @@ With the default engine configuration, a Region without an explicit SST format u ### BulkMemtable -`BulkMemtable` keeps incoming rows in the flat Arrow layout used by the flat SST path. Writes add record-batch parts instead of inserting one row at a time into a per-series structure. Small fragments first accumulate in an unordered part. Background memtable compaction merges eligible parts and can encode larger merged parts. +`BulkMemtable` stores writes as parts in the flat Arrow layout instead of inserting rows into per-series buffers: -Each part is exposed as a memtable range with its own statistics. A scan can prune ranges before opening their readers, while a flush can merge and deduplicate multiple ranges. An encoded range can be written as an SST without decoding and encoding its rows again. +```text +BulkMemtable +├─ unordered_part +│ └─ small BulkPart batches +└─ parts + ├─ BulkPart (Arrow RecordBatch) + ├─ MultiBulkPart (raw RecordBatches) + └─ EncodedBulkPart (in-memory Parquet) +``` + +Small parts accumulate in `unordered_part`; larger parts enter `parts` directly. Background memtable compaction merge-sorts eligible parts into a `MultiBulkPart` or encodes them as an `EncodedBulkPart`. Scans use part statistics to prune ranges, and flush can write encoded ranges to SST without decoding and encoding the rows again. For the design rationale and performance results, see [Scaling Time Series to Millions of Cardinalities: GreptimeDB's Flat Format](https://www.greptime.com/blogs/2025-12-22-flat-format). ### TimeSeriesMemtable diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md index 392fc2e9af..ec60e64524 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md @@ -58,9 +58,19 @@ primary_key SST、dense encoding、无 primary key -> SimpleBulkMemtable ### BulkMemtable -`BulkMemtable` 按照 flat SST 读取路径使用的 Arrow 布局保存写入数据。每次写入增加一个 record batch part，而不是逐行插入按时间序列组织的数据结构。较小的 fragment 会先进入 unordered part；后台 memtable compaction 合并符合条件的 part，并可以编码合并后的较大 part。 +`BulkMemtable` 使用 flat Arrow 布局把写入保存为 part，而不是将数据行插入按时间序列组织的缓冲区： -每个 part 都作为独立的 memtable range 暴露，并带有自己的统计信息。Scan 可以在打开 reader 前裁剪 range，flush 则可以合并多个 range 并执行去重。已经编码的 range 可以直接写为 SST，无需再次解码并编码其中的行。 +```text +BulkMemtable +├─ unordered_part +│ └─ 小批量 BulkPart +└─ parts + ├─ BulkPart (Arrow RecordBatch) + ├─ MultiBulkPart (未编码的 RecordBatch) + └─ EncodedBulkPart (内存中的 Parquet 数据) +``` + +小 part 先积累在 `unordered_part` 中，较大的 part 则直接进入 `parts`。后台 memtable compaction 对符合条件的 part 执行 merge sort，生成 `MultiBulkPart` 或编码为 `EncodedBulkPart`。Scan 利用 part 的统计信息裁剪 range；flush 可以将已编码的 range 写入 SST，无需再次解码和编码数据行。设计动机和性能数据见 [Scaling Time Series to Millions of Cardinalities: GreptimeDB's Flat Format](https://www.greptime.com/blogs/2025-12-22-flat-format)。 ### TimeSeriesMemtable From f677db3dedd205b8716d51164277a0e168a6d616 Mon Sep 17 00:00:00 2001 From: Dennis Zhuang Date: Wed, 2 Sep 2026 11:08:55 +0800 Subject: [PATCH 10/14] docs: address contributor guide review feedback --- docs/contributor-guide/datanode/metric-engine.md | 2 ++ docs/contributor-guide/flownode/batching_mode.md | 8 ++++---- docs/contributor-guide/flownode/dataflow.md | 2 +- docs/contributor-guide/getting-started.md | 3 ++- docs/contributor-guide/metasrv/overview.md | 16 +++++++++++----- docs/contributor-guide/overview.md | 2 +- .../contributor-guide/datanode/metric-engine.md | 2 ++ .../contributor-guide/flownode/batching_mode.md | 8 ++++---- .../contributor-guide/flownode/dataflow.md | 2 +- .../current/contributor-guide/getting-started.md | 3 ++- .../contributor-guide/metasrv/overview.md | 16 +++++++++++----- .../current/contributor-guide/overview.md | 2 +- 12 files changed, 42 insertions(+), 24 deletions(-) diff --git a/docs/contributor-guide/datanode/metric-engine.md b/docs/contributor-guide/datanode/metric-engine.md index 0073de4497..fde3c178a3 100644 --- a/docs/contributor-guide/datanode/metric-engine.md +++ b/docs/contributor-guide/datanode/metric-engine.md @@ -35,6 +35,8 @@ The `Metric` engine delegates physical storage and queries to the `Mito` engine. Logical tables associated with the same physical table share its partition layout. During writes, the engine records the logical table identity with each row. During reads, it adds a logical-table filter before scanning the physical Region. +A logical table's route stores only the ID of its physical table; the physical table route resolves that to the Datanodes holding the Regions. Because logical routes do not name peers, migrating a physical Region rewrites one physical route instead of every logical route that maps to it. + Logical tables support normal INSERT, DELETE, and SELECT operations. Direct writes to a physical Region are rejected because they would bypass the logical-table mapping; querying a physical table remains supported. Batch DDL operations reduce metadata work when many logical tables are created or updated together, such as during Prometheus Remote Write auto-creation or physical Region migration. diff --git a/docs/contributor-guide/flownode/batching_mode.md b/docs/contributor-guide/flownode/batching_mode.md index 4dc0c42195..aa8099d1ad 100644 --- a/docs/contributor-guide/flownode/batching_mode.md +++ b/docs/contributor-guide/flownode/batching_mode.md @@ -15,7 +15,7 @@ The core idea is to: 1. Define a `flow` with a SQL query that aggregates data from a source table into a sink table. 2. The query typically includes a time window function (e.g., `date_bin`) on a timestamp column. 3. When new data is inserted into the source table, the system marks the corresponding time windows as "dirty." -4. A background task periodically wakes up, identifies these dirty windows, and re-runs the aggregation query for those specific time ranges. +4. A background task runs on its own cadence, consumes the pending dirty windows at its next evaluation, and re-runs the aggregation query for those time ranges. 5. The results are then inserted into the sink table, effectively updating the aggregated view. ## Architecture @@ -39,7 +39,7 @@ A `BatchingTask` represents a single, independent data flow. Each task is associ - **State (`TaskState`)**: This contains the dynamic, mutable state of the task, most importantly the `DirtyTimeWindows`. - **Execution Loop**: The task runs an infinite loop (`start_executing_loop`) that: 1. Checks for a shutdown signal. - 2. Waits for a scheduled interval or until it's woken up. + 2. Sleeps until its next evaluation time. A task with an evaluation schedule sleeps until the next scheduled time; an adaptive task sleeps for a polling interval derived from the time window size and the minimum refresh duration. 3. Generates a new query plan (`gen_insert_plan`) based on the current set of dirty time windows. 4. Executes the query (`execute_logical_plan`) against the database. 5. Cleans up the processed dirty windows. @@ -63,8 +63,8 @@ The same calculation is used to mark dirty windows and generate the source-table Here's a simplified step-by-step walkthrough of how a query is executed in batch mode: 1. **Data Ingestion**: New data is written to a source table. -2. **Marking Dirty**: The `BatchingEngine` receives a notification about the new data. It uses the `TimeWindowExpr` associated with each relevant flow to determine which time windows are affected by the new data points. These windows are then added to the `DirtyTimeWindows` set in the corresponding `TaskState`. -3. **Task Wake-up**: The `BatchingTask`'s execution loop wakes up, either due to its periodic schedule or because it was notified of a large backlog of dirty windows. +2. **Marking Dirty**: The `BatchingEngine` receives a notification about the new data. It uses the `TimeWindowExpr` associated with each relevant flow to determine which time windows are affected by the new data points. These windows are then added to the `DirtyTimeWindows` set in the corresponding `TaskState`. Marking a window dirty does not wake the task. +3. **Next Evaluation**: The `BatchingTask`'s execution loop reaches its next evaluation, either at a scheduled time or after its adaptive polling interval, and consumes the pending dirty windows. 4. **Plan Generation**: The task calls `gen_insert_plan`. This method: - Inspects the `DirtyTimeWindows`. - Generates a series of `OR`'d `WHERE` clauses (e.g., `(ts >= 't1' AND ts < 't2') OR (ts >= 't3' AND ts < 't4') ...`) that cover the dirty windows. diff --git a/docs/contributor-guide/flownode/dataflow.md b/docs/contributor-guide/flownode/dataflow.md index e335d152a9..c876054d6d 100644 --- a/docs/contributor-guide/flownode/dataflow.md +++ b/docs/contributor-guide/flownode/dataflow.md @@ -17,7 +17,7 @@ Users do not select the mode directly. When a Flow is created, GreptimeDB choose Batching mode reuses GreptimeDB's query engine instead of maintaining an operator graph for every incoming row. For a time-windowed Flow, its main loop is: 1. A source-table write marks the affected time windows as dirty. -2. A `BatchingTask` wakes on its schedule or after a notification and collects pending dirty windows. +2. A `BatchingTask` runs on its evaluation schedule or adaptive polling cadence and collects the pending dirty windows at that evaluation. Marking a window dirty does not wake the task. 3. The task adds time predicates for those windows to the Flow query and asks Frontend to execute it against the source tables. 4. The query result is inserted into the sink table, updating the materialized result for windows that were evaluated. 5. Successfully processed windows are removed from the dirty set. Failed work remains available for a later evaluation. diff --git a/docs/contributor-guide/getting-started.md b/docs/contributor-guide/getting-started.md index bf6ec2a8fb..7e8cca3ad8 100644 --- a/docs/contributor-guide/getting-started.md +++ b/docs/contributor-guide/getting-started.md @@ -15,12 +15,13 @@ At the moment, GreptimeDB supports Linux (both amd64 and arm64), macOS (both amd ### Build Dependencies -- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) +- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) (optional; needed to clone the repository, not to build it) - C/C++ Toolchain: provides essential tools for compiling and linking. This is available either as `build-essential` on ubuntu or a similar name on other platforms. - [Rustup][1]. The repository pins the required nightly toolchain in `rust-toolchain.toml`. - Protobuf ([guide][2]) - Compile the proto file - Note that the version needs to be >= 3.15. You can check it with `protoc --version` +- Machine: 16GB of memory or more is recommended. On a smaller machine, use [mold](https://github.com/rui314/mold) to reduce memory usage during linking. [1]: [2]: diff --git a/docs/contributor-guide/metasrv/overview.md b/docs/contributor-guide/metasrv/overview.md index 21f817a469..6458b6380d 100644 --- a/docs/contributor-guide/metasrv/overview.md +++ b/docs/contributor-guide/metasrv/overview.md @@ -14,7 +14,8 @@ Metasrv is the metadata and coordination service in a distributed GreptimeDB clu - electing one Metasrv leader to coordinate metadata changes; - running recoverable procedures for DDL, Region migration, failover, and repartitioning; - tracking node leases and Region statistics through heartbeats; -- notifying Frontends and Datanodes when cached metadata or Region state changes. +- broadcasting cache invalidations to Frontends, Datanodes, and Flownodes when metadata changes; +- sending Region lifecycle instructions to Datanodes. ## How the Frontend interacts with Metasrv @@ -28,8 +29,8 @@ Frontend `-- Region reads and writes ------------> Datanode Metasrv leader - |-- Region lifecycle procedures --------> Datanode - `-- cache and Region-state notifications -> Frontend / Datanode + |-- Region lifecycle instructions ------> Datanode + `-- cache invalidations ----------------> Frontend / Datanode / Flownode Datanode `-- heartbeat, lease renewal, Region stats -> Metasrv leader @@ -86,10 +87,15 @@ These mechanisms share metadata, but they have different failure boundaries. A p Metasrv separates leader election from metadata storage. Only the elected Metasrv leader performs coordination and metadata-changing operations. Other Metasrv nodes direct clients to the current leader. -The key-value backend stores table metadata, routes, procedure state, and other information that must survive a leader change. Metasrv does not use this election to create leader and follower replicas for Datanode Regions; Region availability is managed through leases, heartbeats, and failover procedures. +The key-value backend stores table metadata, routes, procedure state, and other information that must survive a leader change. Metasrv does not use this election to create leader and follower replicas for Datanode Regions; Region availability is managed through heartbeats, Region failure detection, and failover procedures. ## Heartbeat Management Datanodes maintain heartbeat streams to the Metasrv leader. Heartbeat requests report node identity, lease information, Region statistics, and other state used for placement and supervision. Responses carry control messages such as Region lifecycle instructions and cache invalidations. -Metasrv treats a heartbeat as a lease renewal, not merely as a metrics sample. Lease expiration is therefore part of failure detection and can lead to a Region failover procedure. Changes to heartbeat timing must remain consistent with the lease and supervision intervals. +A heartbeat drives two independent mechanisms, and a change to heartbeat timing affects both: + +- **Node lease.** The keep-lease handler renews the sending Datanode's lease. Selectors and the `/node-lease` endpoint use these leases to decide whether a Datanode is still active. +- **Region failure detection.** The Region supervisor keeps a per-Region Phi Accrual detector over heartbeat arrival intervals. Its verdict is independent of lease expiry. + +A failure verdict submits a failover migration only when Region failover is enabled; it is disabled by default and requires remote WAL unless explicitly allowed on local WAL. Maintenance mode also suppresses failover. See [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md) for the prerequisites and how to enable it. diff --git a/docs/contributor-guide/overview.md b/docs/contributor-guide/overview.md index 944c541019..6e6ec669c9 100644 --- a/docs/contributor-guide/overview.md +++ b/docs/contributor-guide/overview.md @@ -5,7 +5,7 @@ description: Overview of GreptimeDB's architecture, key components, and how they # Contributor Guide -This guide explains the internal design of GreptimeDB for contributors. Build, test, and submission instructions are maintained in the source repository's [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md). +This guide explains the internal design of GreptimeDB for contributors. Start with [Getting Started](/contributor-guide/getting-started.md) to build and run it from source. Submission requirements, including the CLA, license headers, formatting, and the checks a pull request must pass, are maintained in the source repository's [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md). ## Architecture diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md index 325002f7c5..57baecf400 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/metric-engine.md @@ -33,6 +33,8 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述 关联到同一物理表的逻辑表使用相同的分区布局。写入时，Metric 引擎为每行数据记录逻辑表身份；读取时，它在扫描物理 Region 前增加逻辑表过滤条件。 +逻辑表的路由只保存所属物理表的 ID，再由物理表路由解析出持有 Region 的 Datanode。逻辑路由本身不记录 peer，因此迁移物理 Region 只需改写一条物理路由，而不必改写映射到它的每一条逻辑路由。 + 逻辑表支持普通的 INSERT、DELETE 和 SELECT 操作。直接写入物理 Region 会绕过逻辑表映射，因此会被拒绝；物理表仍然可以查询。 批量 DDL 用于减少大量逻辑表同时创建或更新时的元数据操作，例如 Prometheus Remote Write 自动建表或物理 Region 迁移。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md index f725aaba6b..8e2ecdb80b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md @@ -15,7 +15,7 @@ description: Flownode 批处理模式概述，这是持续数据聚合当前使 1. 定义一个带有 SQL 查询的 `flow`，该查询将数据从源表聚合到目标表。 2. 查询通常在时间戳列上包含一个时间窗口函数（例如 `date_bin`）。 3. 当新数据插入源表时，系统会将相应的时间窗口标记为“脏”(dirty)。 -4. 一个后台任务会周期性地唤醒，识别这些脏窗口，并为那些特定的时间范围重新运行聚合查询。 +4. 一个后台任务按自身的节奏运行，在下一次求值时取出待处理的脏窗口，并对这些时间范围重新运行聚合查询。 5. 然后将结果插入到目标表中，从而有效地更新聚合视图。 ## 架构 @@ -39,7 +39,7 @@ description: Flownode 批处理模式概述，这是持续数据聚合当前使 - **状态 (`TaskState`)**: 包含任务的动态、可变状态，最重要的是 `DirtyTimeWindows`。 - **执行循环**: 任务运行一个无限循环 (`start_executing_loop`)，该循环： 1. 检查关闭信号。 - 2. 等待一个预定的时间间隔或直到被唤醒。 + 2. 睡眠到下一次求值时间。设置了求值调度的任务睡眠到下一个调度时间点；自适应任务则按时间窗口大小和最小刷新间隔计算出的轮询间隔睡眠。 3. 基于当前的脏时间窗口集合生成一个新的查询计划 (`gen_insert_plan`)。 4. 对数据库执行查询 (`execute_logical_plan`)。 5. 清理已处理的脏窗口。 @@ -63,8 +63,8 @@ description: Flownode 批处理模式概述，这是持续数据聚合当前使 以下是批处理模式下查询执行的简化分步演练： 1. **数据摄取**: 新数据被写入源表。 -2. **标记为脏**: `BatchingEngine` 收到有关新数据的通知。它使用与每个相关 flow 关联的 `TimeWindowExpr` 来确定哪些时间窗口受到新数据点的影响。然后将这些窗口添加到相应 `TaskState` 中的 `DirtyTimeWindows` 集合中。 -3. **任务唤醒**: `BatchingTask` 的执行循环被唤醒，原因可能是其周期性调度，也可能是因为它被通知有大量积压的脏窗口。 +2. **标记为脏**: `BatchingEngine` 收到有关新数据的通知。它使用与每个相关 flow 关联的 `TimeWindowExpr` 来确定哪些时间窗口受到新数据点的影响。然后将这些窗口添加到相应 `TaskState` 中的 `DirtyTimeWindows` 集合中。标记脏窗口不会唤醒任务。 +3. **下一次求值**: `BatchingTask` 的执行循环在调度时间点或自适应轮询间隔结束后进入下一次求值，取出待处理的脏窗口。 4. **计划生成**: 任务调用 `gen_insert_plan`。此方法： - 检查 `DirtyTimeWindows`。 - 生成一系列 `OR` 连接的 `WHERE` 子句（例如 `(ts >= 't1' AND ts < 't2') OR (ts >= 't3' AND ts < 't4') ...`），覆盖所有脏窗口。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md index 7d5ad54780..95def5e844 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md @@ -17,7 +17,7 @@ Flownode 内部有两条执行路径： Batching mode 复用 GreptimeDB 的查询引擎，不需要为每一行输入维护一张算子图。对于基于时间窗口的 Flow，主循环如下： 1. Source table 收到写入后，把受影响的时间窗口标记为 dirty。 -2. `BatchingTask` 按调度周期或通知唤醒，并收集待处理的 dirty window。 +2. `BatchingTask` 按求值调度或自适应轮询节奏运行，并在该次求值时收集待处理的 dirty window。标记 dirty window 不会唤醒任务。 3. 任务把这些窗口转换成时间谓词，加入 Flow 查询，再请求 Frontend 查询 source table。 4. 查询结果写入 sink table，更新已重新计算窗口对应的物化结果。 5. 成功处理的窗口从 dirty set 中移除；执行失败的工作仍可在后续调度中处理。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/getting-started.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/getting-started.md index d6828a8a17..999f696ed0 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/getting-started.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/getting-started.md @@ -15,12 +15,13 @@ description: 介绍如何在本地环境中从源代码编译和运行 GreptimeD ### 构建依赖项 -- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) +- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line)（可选；克隆仓库需要，构建本身不需要） - C/C++ 工具链：提供编译和链接的基本工具。在 Ubuntu 上，这可用作 `build-essential`。在其他平台上，也有类似的命令。 - [Rustup][1]。仓库通过 `rust-toolchain.toml` 指定所需的 nightly 工具链。 - Protobuf（[指南][2]） - 编译 proto 文件 - 请注意，版本需要 >= 3.15。你可以使用 `protoc --version` 检查它。 +- 机器：建议 16GB 以上内存。内存较小时，可使用 [mold](https://github.com/rui314/mold) 降低链接阶段的内存占用。 [1]: [2]: diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md index 3402eadb66..e9f6144f26 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/metasrv/overview.md @@ -14,7 +14,8 @@ Metasrv 是 GreptimeDB 分布式集群中的元数据和协调服务，不参与 - 选举一个 Metasrv leader 负责协调元数据变更； - 通过可恢复的 Procedure 执行 DDL、Region 迁移、故障转移和重分区； - 通过心跳维护节点租约和 Region 统计信息； -- 在缓存元数据或 Region 状态变化时通知 Frontend 和 Datanode。 +- 元数据变更时向 Frontend、Datanode 和 Flownode 广播缓存失效； +- 向 Datanode 下发 Region 生命周期指令。 ## 前端如何与 Metasrv 交互 @@ -28,8 +29,8 @@ Frontend `-- Region 读写 ------------------------> Datanode Metasrv leader - |-- Region 生命周期 Procedure ----------> Datanode - `-- 缓存和 Region 状态通知 -------------> Frontend / Datanode + |-- Region 生命周期指令 ----------------> Datanode + `-- 缓存失效 --------------------------> Frontend / Datanode / Flownode Datanode `-- 心跳、租约续期和 Region 统计信息 ----> Metasrv leader @@ -86,10 +87,15 @@ Metasrv leader Metasrv 将 leader 选举与元数据存储分开。只有选出的 Metasrv leader 执行协调和元数据变更操作，其他 Metasrv 节点会把 client 引导到当前 leader。 -Key-value backend 保存表元数据、路由、Procedure 状态以及其他必须跨 leader 切换保留的信息。Metasrv 不使用这套选举为 Datanode Region 创建读写副本；Region 可用性由租约、心跳和故障转移 Procedure 管理。 +Key-value backend 保存表元数据、路由、Procedure 状态以及其他必须跨 leader 切换保留的信息。Metasrv 不使用这套选举为 Datanode Region 创建读写副本；Region 可用性由心跳、Region 故障检测和故障转移 Procedure 管理。 ## 心跳管理 Datanode 与 Metasrv leader 保持心跳流。心跳请求报告节点身份、租约、Region 统计信息以及放置和监控所需的其他状态；响应则携带 Region 生命周期指令、缓存失效等控制消息。 -对 Metasrv 而言，心跳不仅是指标上报，也是租约续期。租约过期会参与故障检测，并可能触发 Region 故障转移。因此，修改心跳周期时必须同时考虑租约和监控周期。 +心跳驱动两套相互独立的机制，调整心跳周期会同时影响两者： + +- **节点租约**：keep-lease handler 为发送心跳的 Datanode 续期。Selector 和 `/node-lease` 端点据此判断 Datanode 是否仍然存活。 +- **Region 故障检测**：Region supervisor 为每个 Region 维护一个基于心跳到达间隔的 Phi Accrual 检测器，其判定与租约是否过期无关。 + +只有开启 Region 故障转移时，故障判定才会提交故障转移迁移。该功能默认关闭，并且要求使用 remote WAL，除非显式允许在本地 WAL 上执行。维护模式同样会抑制故障转移。前置条件和开启方式参见 [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/overview.md index 71c62609ee..badd6ad62d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/overview.md @@ -5,7 +5,7 @@ description: 介绍 GreptimeDB 的架构、关键概念和工作原理，包括 # 贡献者指南 -本指南面向 GreptimeDB 贡献者，介绍理解内部实现所需的设计机制。构建、测试和提交要求以源码仓库的 [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md) 为准。 +本指南面向 GreptimeDB 贡献者，介绍理解内部实现所需的设计机制。从源码构建和运行参见[快速开始](/contributor-guide/getting-started.md)。提交要求（CLA、license header、代码格式，以及 PR 必须通过的检查）以源码仓库的 [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md) 为准。 ## 架构 From bc05f67e90d6af0b06376ac861d060dc0142de7b Mon Sep 17 00:00:00 2001 From: Dennis Zhuang Date: Wed, 2 Sep 2026 11:09:01 +0800 Subject: [PATCH 11/14] docs: fix SST layout diagram text overflow --- static/mito-sst-layout.svg | 8 ++++---- static/mito-sst-layout.zh.svg | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/static/mito-sst-layout.svg b/static/mito-sst-layout.svg index 64415791dc..1b88d85c21 100644 --- a/static/mito-sst-layout.svg +++ b/static/mito-sst-layout.svg @@ -39,11 +39,11 @@ Time ts - - Encoded key - __primary_key + + Encoded key + __primary_key - + Merge metadata __sequence · __op_type diff --git a/static/mito-sst-layout.zh.svg b/static/mito-sst-layout.zh.svg index d75ae0c09a..e01d5f2911 100644 --- a/static/mito-sst-layout.zh.svg +++ b/static/mito-sst-layout.zh.svg @@ -39,11 +39,11 @@ 时间列 ts - - 编码后的 key - __primary_key + + 编码后的 key + __primary_key - + 合并元数据 __sequence · __op_type From ca0cd4e8f23aad616fa6b6f393ae979244cc1e4c Mon Sep 17 00:00:00 2001 From: Dennis Zhuang Date: Wed, 2 Sep 2026 11:19:08 +0800 Subject: [PATCH 12/14] chore: ignore local MCP configuration .mcp.json must stay at the repository root for the MCP client to read it, and it holds connection credentials, so ignore it rather than relocate it. --- .gitignore | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.gitignore b/.gitignore index 96028f1992..2b5e33dd3a 100644 --- a/.gitignore +++ b/.gitignore @@ -35,3 +35,6 @@ static/skills/ # Used by Codex to store temporary artifacts tmp/ + +# Local MCP server configuration. Contains connection credentials. +/.mcp.json From 1f7eea83cd54dc8ec7cbc3792a6afd8e4682e35b Mon Sep 17 00:00:00 2001 From: Dennis Zhuang Date: Wed, 2 Sep 2026 14:12:57 +0800 Subject: [PATCH 13/14] docs: foreground BulkMemtable in the memtable guide Collapse the legacy TimeSeriesMemtable and SimpleBulkMemtable into one paragraph instead of giving them a selection-table row each and their own section. They remain reachable for regions on the legacy primary_key SST format, so the page still names them, but the bulk and flat path is now clearly the subject. --- docs/contributor-guide/datanode/memtable.md | 15 ++++++--------- .../contributor-guide/datanode/memtable.md | 15 ++++++--------- 2 files changed, 12 insertions(+), 18 deletions(-) diff --git a/docs/contributor-guide/datanode/memtable.md b/docs/contributor-guide/datanode/memtable.md index 0718840b12..baf5f66bd9 100644 --- a/docs/contributor-guide/datanode/memtable.md +++ b/docs/contributor-guide/datanode/memtable.md @@ -48,13 +48,12 @@ Freezing a Region freezes all mutable time partitions together. Mito moves their Mito selects a memtable implementation from the Region's SST format, primary-key encoding, and memtable options: ```text -memtable.type=bulk -> BulkMemtable and flat SST -flat SST format or sparse primary-key encoding -> BulkMemtable -primary_key SST, dense encoding, primary key exists -> TimeSeriesMemtable -primary_key SST, dense encoding, no primary key -> SimpleBulkMemtable +flat SST format (the default) or sparse primary-key encoding -> BulkMemtable +memtable.type=bulk -> BulkMemtable, and forces flat SST +primary_key SST with dense encoding (legacy) -> a legacy implementation ``` -With the default engine configuration, a Region without an explicit SST format uses `flat`, so `BulkMemtable` is the normal path. The rules prevent incompatible combinations: flat format or sparse primary-key encoding requires `BulkMemtable`, while explicitly selecting the bulk implementation forces flat format. +With the default engine configuration, a Region without an explicit SST format uses `flat`, so `BulkMemtable` is the normal path and the rest of this page describes it. The rules exist to prevent incompatible combinations: flat format or sparse primary-key encoding requires `BulkMemtable`, and explicitly selecting the bulk implementation forces flat format. ### BulkMemtable @@ -72,11 +71,9 @@ BulkMemtable Small parts accumulate in `unordered_part`; larger parts enter `parts` directly. Background memtable compaction merge-sorts eligible parts into a `MultiBulkPart` or encodes them as an `EncodedBulkPart`. Scans use part statistics to prune ranges, and flush can write encoded ranges to SST without decoding and encoding the rows again. For the design rationale and performance results, see [Scaling Time Series to Millions of Cardinalities: GreptimeDB's Flat Format](https://www.greptime.com/blogs/2025-12-22-flat-format). -### TimeSeriesMemtable +### Legacy implementations -`TimeSeriesMemtable` groups rows by encoded primary key. Each series stores its timestamps, sequence numbers, operation types, and field values in column builders. When a reader requests the series, the memtable builds a batch ordered by timestamp and sequence and applies the Region's deduplication or merge mode. - -This implementation is used for the `primary_key` SST format with dense primary-key encoding unless the Region explicitly selects the bulk implementation. If the Region has no primary-key columns, the same builder creates a `SimpleBulkMemtable` instead of a series map. +Regions on the legacy `primary_key` SST format with dense primary-key encoding still use `TimeSeriesMemtable`, which groups rows by encoded primary key rather than storing flat parts. A Region with no primary-key columns gets `SimpleBulkMemtable` from the same builder. Both are compatibility code for existing tables and may be removed once the `primary_key` format is retired; new work targets the bulk and flat path. The removed `partition_tree` memtable is not a third implementation. The option parser accepts `memtable.type=partition_tree` for compatibility, but it does not recreate that implementation. The Region uses the bulk and flat path. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md index ec60e64524..71f2476f9c 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md @@ -48,13 +48,12 @@ Mito 根据 time index 的值把每行数据路由到对应分区。分区使用 Mito 根据 Region 的 SST format、primary key encoding 和 memtable 选项选择实现： ```text -memtable.type=bulk -> BulkMemtable 和 flat SST -flat SST format 或 sparse primary-key encoding -> BulkMemtable -primary_key SST、dense encoding、有 primary key -> TimeSeriesMemtable -primary_key SST、dense encoding、无 primary key -> SimpleBulkMemtable +flat SST format（默认）或 sparse primary-key encoding -> BulkMemtable +memtable.type=bulk -> BulkMemtable，并强制 flat SST +primary_key SST + dense encoding（遗留） -> 遗留实现 ``` -使用默认 engine 配置时，没有显式指定 SST format 的 Region 会使用 `flat`，因此通常走 `BulkMemtable` 路径。这些规则会排除不兼容的组合：flat format 或 sparse primary key encoding 必须使用 `BulkMemtable`；显式选择 bulk 实现则会强制使用 flat format。 +使用默认 engine 配置时，没有显式指定 SST format 的 Region 会使用 `flat`，因此通常走 `BulkMemtable` 路径，本页其余内容也以它为准。这些规则用于排除不兼容的组合：flat format 或 sparse primary key encoding 必须使用 `BulkMemtable`；显式选择 bulk 实现则会强制使用 flat format。 ### BulkMemtable @@ -72,11 +71,9 @@ BulkMemtable 小 part 先积累在 `unordered_part` 中，较大的 part 则直接进入 `parts`。后台 memtable compaction 对符合条件的 part 执行 merge sort，生成 `MultiBulkPart` 或编码为 `EncodedBulkPart`。Scan 利用 part 的统计信息裁剪 range；flush 可以将已编码的 range 写入 SST，无需再次解码和编码数据行。设计动机和性能数据见 [Scaling Time Series to Millions of Cardinalities: GreptimeDB's Flat Format](https://www.greptime.com/blogs/2025-12-22-flat-format)。 -### TimeSeriesMemtable +### 遗留实现 -`TimeSeriesMemtable` 按编码后的 primary key 对数据行分组。每条时间序列分别保存 timestamp、sequence number、operation type 和 field value 的列构建器。Reader 读取时间序列时，memtable 按 timestamp 和 sequence 构造 batch，并应用 Region 的去重或 merge mode。 - -除非 Region 显式选择 bulk 实现，否则使用 dense primary key encoding 的 `primary_key` SST format 会选择该实现。如果 Region 没有 primary key 列，同一个 builder 会创建 `SimpleBulkMemtable`，而不创建 series map。 +使用遗留 `primary_key` SST format 且为 dense primary key encoding 的 Region 仍然使用 `TimeSeriesMemtable`，它按编码后的 primary key 对数据行分组，而不是保存 flat part。如果 Region 没有 primary key 列，同一个 builder 会创建 `SimpleBulkMemtable`。两者都是为已有表保留的兼容代码，`primary_key` format 退役后可能一并移除；新的工作应面向 bulk 和 flat 路径。 已经删除的 `partition_tree` memtable 不是第三种实现。Option parser 仍接受 `memtable.type=partition_tree` 以兼容旧配置，但不会恢复该实现。Region 最终使用 bulk 和 flat 路径。 From e499e618af397310fdb7b27d66244cb1f7f93d47 Mon Sep 17 00:00:00 2001 From: Dennis Zhuang Date: Wed, 2 Sep 2026 21:27:53 +0800 Subject: [PATCH 14/14] docs: backport contributor guide corrections to v1.0, v1.1 and v1.2 Apply the Nightly contributor-guide rewrite to the three supported released versions, and gate each version-sensitive claim on what that release ships. v1.2 (GreptimeDB v1.2.0-beta.2) takes the change unchanged. v1.1 (v1.1.4) drops the compatibility-test entry, because tests/compatibility and 'cargo sqlness compat' arrived after that release, and drops x-greptime-err-retry-hint, which v1.1 does not send. v1.0 (v1.0.2) additionally omits the new Memtable page: MemtableOptions there still has a PartitionTree variant, so the page's selection rules and its statement that partition_tree is not a third implementation do not hold. The links to that page degrade to plain text. It also drops the pending-batching-flow sentence, since defer_on_missing_source does not exist yet, and keeps its own existing wording where the rewrite did not change meaning. reference/sql/create.md and the Mito memtable configuration section change only in v1.1 and v1.2. v1.0.2 still has MitoConfig::memtable and accepts memtable.type=partition_tree, so the original text is correct there. Also fix the blog link domain on the new Memtable page: Chinese pages use greptime.cn and English pages use greptime.com without the www prefix, matching the existing link to the same article in features-that-you-concern.md. --- docs/contributor-guide/datanode/memtable.md | 2 +- .../contributor-guide/datanode/memtable.md | 2 +- .../datanode/data-persistence-indexing.md | 30 +-- .../datanode/metric-engine.md | 20 +- .../contributor-guide/datanode/overview.md | 34 +-- .../datanode/python-scripts.md | 30 --- .../datanode/query-engine.md | 30 +-- .../datanode/storage-engine.md | 10 +- .../contributor-guide/datanode/wal.md | 22 +- .../contributor-guide/flownode/arrangement.md | 2 + .../flownode/batching_mode.md | 16 +- .../contributor-guide/flownode/dataflow.md | 31 ++- .../frontend/distributed-querying.md | 39 +--- .../contributor-guide/frontend/overview.md | 44 ++-- .../frontend/table-sharding.md | 20 +- .../contributor-guide/getting-started.md | 7 +- .../how-to/how-to-write-sdk.md | 46 +++-- .../contributor-guide/metasrv/admin-api.md | 61 +++++- .../contributor-guide/metasrv/overview.md | 195 ++++++------------ .../contributor-guide/metasrv/selector.md | 14 +- .../version-1.0/contributor-guide/overview.md | 10 +- .../tests/integration-test.md | 14 +- .../contributor-guide/tests/overview.md | 9 +- .../contributor-guide/tests/sqlness-test.md | 18 +- .../contributor-guide/tests/unit-test.md | 16 +- .../datanode/data-persistence-indexing.md | 30 +-- .../contributor-guide/datanode/memtable.md | 100 +++++++++ .../datanode/metric-engine.md | 20 +- .../contributor-guide/datanode/overview.md | 34 +-- .../datanode/python-scripts.md | 30 --- .../datanode/query-engine.md | 30 +-- .../datanode/storage-engine.md | 12 +- .../contributor-guide/datanode/wal.md | 22 +- .../contributor-guide/flownode/arrangement.md | 2 + .../flownode/batching_mode.md | 16 +- .../contributor-guide/flownode/dataflow.md | 31 ++- .../frontend/distributed-querying.md | 39 +--- .../contributor-guide/frontend/overview.md | 44 ++-- .../frontend/table-sharding.md | 20 +- .../contributor-guide/getting-started.md | 7 +- .../how-to/how-to-write-sdk.md | 46 +++-- .../contributor-guide/metasrv/admin-api.md | 61 +++++- .../contributor-guide/metasrv/overview.md | 195 ++++++------------ .../contributor-guide/metasrv/selector.md | 14 +- .../version-1.1/contributor-guide/overview.md | 10 +- .../tests/integration-test.md | 14 +- .../contributor-guide/tests/overview.md | 9 +- .../contributor-guide/tests/sqlness-test.md | 18 +- .../contributor-guide/tests/unit-test.md | 16 +- .../version-1.1/reference/sql/create.md | 6 +- .../configuration.md | 19 +- .../datanode/data-persistence-indexing.md | 30 +-- .../contributor-guide/datanode/memtable.md | 100 +++++++++ .../datanode/metric-engine.md | 20 +- .../contributor-guide/datanode/overview.md | 34 +-- .../datanode/python-scripts.md | 30 --- .../datanode/query-engine.md | 30 +-- .../datanode/storage-engine.md | 12 +- .../contributor-guide/datanode/wal.md | 22 +- .../contributor-guide/flownode/arrangement.md | 2 + .../flownode/batching_mode.md | 16 +- .../contributor-guide/flownode/dataflow.md | 31 ++- .../frontend/distributed-querying.md | 39 +--- .../contributor-guide/frontend/overview.md | 44 ++-- .../frontend/table-sharding.md | 20 +- .../contributor-guide/getting-started.md | 7 +- .../how-to/how-to-write-sdk.md | 46 +++-- .../contributor-guide/metasrv/admin-api.md | 61 +++++- .../contributor-guide/metasrv/overview.md | 195 ++++++------------ .../contributor-guide/metasrv/selector.md | 14 +- .../version-1.2/contributor-guide/overview.md | 10 +- .../tests/integration-test.md | 14 +- .../contributor-guide/tests/overview.md | 10 +- .../contributor-guide/tests/sqlness-test.md | 18 +- .../contributor-guide/tests/unit-test.md | 16 +- .../version-1.2/reference/sql/create.md | 6 +- .../configuration.md | 19 +- .../datanode/data-persistence-indexing.md | 30 +-- .../datanode/metric-engine.md | 20 +- .../contributor-guide/datanode/overview.md | 40 ++-- .../datanode/python-scripts.md | 35 ---- .../datanode/query-engine.md | 45 ++-- .../datanode/storage-engine.md | 10 +- .../contributor-guide/datanode/wal.md | 32 ++- .../contributor-guide/flownode/arrangement.md | 2 + .../flownode/batching_mode.md | 16 +- .../contributor-guide/flownode/dataflow.md | 31 ++- .../frontend/distributed-querying.md | 21 +- .../contributor-guide/frontend/overview.md | 42 +++- .../frontend/table-sharding.md | 23 ++- .../contributor-guide/getting-started.md | 7 +- .../how-to/how-to-write-sdk.md | 50 ++--- .../contributor-guide/metasrv/admin-api.md | 59 +++++- .../contributor-guide/metasrv/overview.md | 192 ++++++----------- .../contributor-guide/metasrv/selector.md | 14 +- .../version-1.0/contributor-guide/overview.md | 9 +- .../tests/integration-test.md | 15 +- .../contributor-guide/tests/overview.md | 10 +- .../contributor-guide/tests/sqlness-test.md | 24 +-- .../contributor-guide/tests/unit-test.md | 19 +- .../datanode/data-persistence-indexing.md | 30 +-- .../contributor-guide/datanode/memtable.md | 100 +++++++++ .../datanode/metric-engine.md | 20 +- .../contributor-guide/datanode/overview.md | 40 ++-- .../datanode/python-scripts.md | 35 ---- .../datanode/query-engine.md | 45 ++-- .../datanode/storage-engine.md | 12 +- .../contributor-guide/datanode/wal.md | 32 ++- .../contributor-guide/flownode/arrangement.md | 2 + .../flownode/batching_mode.md | 16 +- .../contributor-guide/flownode/dataflow.md | 31 ++- .../frontend/distributed-querying.md | 21 +- .../contributor-guide/frontend/overview.md | 42 +++- .../frontend/table-sharding.md | 23 ++- .../contributor-guide/getting-started.md | 7 +- .../how-to/how-to-write-sdk.md | 50 ++--- .../contributor-guide/metasrv/admin-api.md | 59 +++++- .../contributor-guide/metasrv/overview.md | 192 ++++++----------- .../contributor-guide/metasrv/selector.md | 14 +- .../version-1.1/contributor-guide/overview.md | 9 +- .../tests/integration-test.md | 15 +- .../contributor-guide/tests/overview.md | 10 +- .../contributor-guide/tests/sqlness-test.md | 24 +-- .../contributor-guide/tests/unit-test.md | 19 +- .../version-1.1/reference/sql/create.md | 6 +- .../configuration.md | 19 +- .../datanode/data-persistence-indexing.md | 30 +-- .../contributor-guide/datanode/memtable.md | 100 +++++++++ .../datanode/metric-engine.md | 20 +- .../contributor-guide/datanode/overview.md | 40 ++-- .../datanode/python-scripts.md | 35 ---- .../datanode/query-engine.md | 45 ++-- .../datanode/storage-engine.md | 12 +- .../contributor-guide/datanode/wal.md | 32 ++- .../contributor-guide/flownode/arrangement.md | 2 + .../flownode/batching_mode.md | 16 +- .../contributor-guide/flownode/dataflow.md | 31 ++- .../frontend/distributed-querying.md | 21 +- .../contributor-guide/frontend/overview.md | 42 +++- .../frontend/table-sharding.md | 23 ++- .../contributor-guide/getting-started.md | 7 +- .../how-to/how-to-write-sdk.md | 50 ++--- .../contributor-guide/metasrv/admin-api.md | 59 +++++- .../contributor-guide/metasrv/overview.md | 192 ++++++----------- .../contributor-guide/metasrv/selector.md | 14 +- .../version-1.2/contributor-guide/overview.md | 9 +- .../tests/integration-test.md | 15 +- .../contributor-guide/tests/overview.md | 11 +- .../contributor-guide/tests/sqlness-test.md | 24 +-- .../contributor-guide/tests/unit-test.md | 19 +- .../version-1.2/reference/sql/create.md | 6 +- .../configuration.md | 19 +- versioned_sidebars/version-1.1-sidebars.json | 1 + versioned_sidebars/version-1.2-sidebars.json | 1 + 154 files changed, 2611 insertions(+), 2297 deletions(-) delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/python-scripts.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/memtable.md delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/python-scripts.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/memtable.md delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/python-scripts.md delete mode 100644 versioned_docs/version-1.0/contributor-guide/datanode/python-scripts.md create mode 100644 versioned_docs/version-1.1/contributor-guide/datanode/memtable.md delete mode 100644 versioned_docs/version-1.1/contributor-guide/datanode/python-scripts.md create mode 100644 versioned_docs/version-1.2/contributor-guide/datanode/memtable.md delete mode 100644 versioned_docs/version-1.2/contributor-guide/datanode/python-scripts.md diff --git a/docs/contributor-guide/datanode/memtable.md b/docs/contributor-guide/datanode/memtable.md index baf5f66bd9..92e3eb58de 100644 --- a/docs/contributor-guide/datanode/memtable.md +++ b/docs/contributor-guide/datanode/memtable.md @@ -69,7 +69,7 @@ BulkMemtable └─ EncodedBulkPart (in-memory Parquet) ``` -Small parts accumulate in `unordered_part`; larger parts enter `parts` directly. Background memtable compaction merge-sorts eligible parts into a `MultiBulkPart` or encodes them as an `EncodedBulkPart`. Scans use part statistics to prune ranges, and flush can write encoded ranges to SST without decoding and encoding the rows again. For the design rationale and performance results, see [Scaling Time Series to Millions of Cardinalities: GreptimeDB's Flat Format](https://www.greptime.com/blogs/2025-12-22-flat-format). +Small parts accumulate in `unordered_part`; larger parts enter `parts` directly. Background memtable compaction merge-sorts eligible parts into a `MultiBulkPart` or encodes them as an `EncodedBulkPart`. Scans use part statistics to prune ranges, and flush can write encoded ranges to SST without decoding and encoding the rows again. For the design rationale and performance results, see [Scaling Time Series to Millions of Cardinalities: GreptimeDB's Flat Format](https://greptime.com/blogs/2025-12-22-flat-format). ### Legacy implementations diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md index 71f2476f9c..3905a1aecd 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/datanode/memtable.md @@ -69,7 +69,7 @@ BulkMemtable └─ EncodedBulkPart (内存中的 Parquet 数据) ``` -小 part 先积累在 `unordered_part` 中，较大的 part 则直接进入 `parts`。后台 memtable compaction 对符合条件的 part 执行 merge sort，生成 `MultiBulkPart` 或编码为 `EncodedBulkPart`。Scan 利用 part 的统计信息裁剪 range；flush 可以将已编码的 range 写入 SST，无需再次解码和编码数据行。设计动机和性能数据见 [Scaling Time Series to Millions of Cardinalities: GreptimeDB's Flat Format](https://www.greptime.com/blogs/2025-12-22-flat-format)。 +小 part 先积累在 `unordered_part` 中，较大的 part 则直接进入 `parts`。后台 memtable compaction 对符合条件的 part 执行 merge sort，生成 `MultiBulkPart` 或编码为 `EncodedBulkPart`。Scan 利用 part 的统计信息裁剪 range；flush 可以将已编码的 range 写入 SST，无需再次解码和编码数据行。设计动机和性能数据见 [Scaling Time Series to Millions of Cardinalities: GreptimeDB's Flat Format](https://greptime.cn/blogs/2025-12-22-flat-format)。 ### 遗留实现 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/data-persistence-indexing.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/data-persistence-indexing.md index fbd986781f..6ace5b8d65 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/data-persistence-indexing.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/data-persistence-indexing.md @@ -5,19 +5,23 @@ description: 介绍了 GreptimeDB 的数据持久化和索引机制，包括 SST # 数据持久化与索引 -与所有类似 LSMT 的存储引擎一样，MemTables 中的数据被持久化到耐久性存储，例如本地磁盘文件系统或对象存储服务。GreptimeDB 采用 [Apache Parquet][1] 作为其持久文件格式。 +与其他 LSM-tree 存储引擎类似，GreptimeDB 将 memtable 中的数据持久化到本地文件系统或对象存储，并使用 [Apache Parquet][1] 作为持久化文件格式。 ## SST 文件格式 Parquet 是一种提供快速数据查询的开源列式存储格式，已经被许多项目采用，例如 Delta Lake。 -Parquet 具有层次结构，类似于“行组 - 列-数据页”。Parquet 文件中的数据被水平分区为行组（row group），在其中相同列的所有值一起存储以形成数据页（data pages）。数据页是最小的存储单元。这种结构极大地提高了性能。 +Parquet 按 row group、column chunk 和 page 组织数据。每个 row group 为每一列保存一个 column chunk，每个 column chunk 再包含一个或多个 page。Page 是编码和压缩单元，读取指定列时则以 column chunk 为 I/O 单元。 首先，数据按列聚集，这使得文件扫描更加高效，特别是当查询只涉及少数列时，这在分析系统中非常常见。 -其次，相同列的数据往往是同质的（比如具备近似的值），这有助于在采用字典和 Run-Length Encoding（RLE）等技术进行压缩。 +其次，同一列中的值通常比较相似，有利于字典编码和 Run-Length Encoding（RLE）等压缩技术发挥作用。 -Parquet file format +下面这张来自 Apache Parquet 规范的图进一步展示了物理文件布局：column chunk 按 row group 写入，文件元数据及其长度则保存在 footer 中。 + +Apache Parquet 文件布局 + +*来源：Apache Parquet [FileLayout.gif](https://github.com/apache/parquet-format/blob/master/doc/images/FileLayout.gif)。Copyright 2014 The Apache Software Foundation，依据 [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) 使用。* ## 数据持久化 @@ -26,18 +30,18 @@ GreptimeDB 提供了 `region_engine.mito.global_write_buffer_size` 的配置项 ## SST 文件中的索引数据 -Apache Parquet 文件格式在列块和数据页的头部提供了内置的统计信息，用于剪枝和跳过。 +Parquet 在每个 column chunk 的元数据中保存 row group 级列统计信息，例如最小值、最大值和 null 数量。Page 元数据和可选的 column index 可以提供粒度更细的统计信息。 -Column chunk header +![查询 name 列时，Parquet 列统计信息排除了一个 row group，并将另一个保留为待读取对象。](/parquet-row-group-statistics.zh.svg) -例如，在上述 Parquet 文件中，如果你想要过滤 `name` 等于 `Emily` 的行，你可以轻松跳过行组 0，因为 `name` 字段的最大值是 `Charlie`。这些统计信息减少了 IO 操作。 +例如，查询 `name` 等于 `Emily` 的行时，可以跳过 row group 0，因为其中 `name` 的最大值是 `Charlie`，无需读取该 row group。 ## 索引文件 -对于每个 SST 文件，GreptimeDB 不但维护 SST 文件内部索引，还会单独生成一个文件用于存储针对该 SST 文件的索引结构。 +当一个 SST 存在已配置且适用的索引输出时，GreptimeDB 将这些索引写入与该 SST 关联的 Puffin 文件。没有适用索引的 SST 不需要生成 Puffin 文件。 -索引文件采用 [Puffin][3] 格式，这种格式具有较大的灵活性，能够存储更多的元数据，并支持更多的索引结构。 +Puffin 是索引 Blob 及其元数据的容器，使不同索引结构可以共用一个文件。 ![Puffin](/puffin.png) @@ -58,13 +62,13 @@ GreptimeDB 会将多种索引结构作为 Blob 存储在 Puffin 文件中，包 ![Inverted index searching](/inverted-index-searching.png) -例如，上述查询使用倒排索引来定位数据段，数据段满足条件：`job` 等于 `apiserver`，`handler` 符合正则匹配 `.*users` 及 `status` 符合正则匹配 `4..`，然后扫描这些数据段以产生满足所有条件的最终结果，从而显着减少 IO 操作的次数。 +上述查询使用倒排索引定位 `job` 等于 `apiserver`、`handler` 匹配 `.*users` 且 `status` 匹配 `4..` 的数据段。Mito 只扫描这些数据段，再应用剩余过滤条件。 ### 倒排索引格式 -![Inverted index format](/inverted-index-format.png) +![倒排索引 Blob 先保存各列索引，再保存 footer 元数据；每个列索引包含 null bitmap、posting bitmap 和 FST。](/inverted-index-blob-layout.zh.svg) -GreptimeDB 按列构建倒排索引，每个倒排索引包含一个 FST 和多个 Bitmap。 +GreptimeDB 按列构建倒排索引。每个列索引包含一个 null bitmap、多个 posting bitmap 和一个 FST。Blob footer 记录定位和解码各列索引所需的 offset、size 和元数据。 FST（Finite State Transducer）允许 GreptimeDB 以紧凑的格式存储列值到 Bitmap 位置的映射，并且提供了优秀的搜索性能和支持复杂搜索（例如正则表达式匹配）；Bitmap 则维护了数据段 ID 列表，每个位表示一个数据段。 @@ -80,7 +84,7 @@ GreptimeDB 把一个 SST 文件分割成多个索引数据段，每个数据段 ## 统一数据访问层：OpenDAL -GreptimeDB 使用 [OpenDAL][2] 提供统一的数据访问层，因此，存储引擎无需与不同的存储 API 交互，数据可以无缝迁移到基于云的存储，如 AWS S3。 +GreptimeDB 使用 [OpenDAL][2] 为本地文件系统和对象存储提供统一访问层。修改配置的存储 backend 不会迁移已有数据。 [1]: https://parquet.apache.org [2]: https://github.com/datafuselabs/opendal diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/metric-engine.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/metric-engine.md index c0cbf681b9..57baecf400 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/metric-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/metric-engine.md @@ -7,9 +7,9 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述 ## 概述 -`Metric` 引擎是 GreptimeDB 的一个组件，属于存储引擎的一种实现，主要针对可观测 metrics 等存在大量小表的场景。 +`Metric` 引擎用于存储包含大量小型指标表的负载。 -它的主要特点是利用合成的物理宽表来存储大量的小表数据，实现相同列复用和元数据复用等效果，从而达到减少小表的存储开销以及提高列式压缩效率等目标。表这一概念在 `Metric` 引擎下变得更更加轻量。 +它将这些逻辑表映射到共享的物理宽表，使其复用列和元数据，从而降低每张表的存储开销并改善列式压缩。 ## 概念 @@ -18,7 +18,7 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述 ### 逻辑表 逻辑表，即用户定义的表。与普通的表都完全一样，逻辑表的定义包括表的名称、列的定义、索引的定义等。用户的查询、写入等操作都是基于逻辑表进行的。用户在使用过程中不需要关心逻辑表和普通表的区别。 -从实现层面来说，逻辑表是一个虚拟的表，它并不直接读写物理的数据，而是通过将读写请求映射成对应物理表的请求来实现数据的存储与查询。 +逻辑表是虚拟表，本身不直接存储数据。Metric 引擎将其读写请求映射为对应物理表的请求。 ### 物理表 物理表是真实存储数据的表，它拥有若干个由分区规则定义的物理 Region。 @@ -27,16 +27,14 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述 `Metric` 引擎的主要设计架构如下： -![Arch](/metric-engine-arch.png) +![多个逻辑表通过 Metric 引擎映射到由 Mito 管理的共享数据 Region 和元数据 Region。](/metric-engine-architecture.zh.svg) -在目前版本的实现中，`Metric` 引擎复用了 `Mito` 引擎来实现物理数据的存储及查询能力，并在此之上同时提供物理表与逻辑表的访问能力。 +`Metric` 引擎将物理存储和查询交给 `Mito` 引擎。每个物理 Region 组包含一个数据 Region 和一个元数据 Region：数据 Region 保存映射到该 Region 组的逻辑表数据，元数据 Region 保存逻辑表及逻辑列的映射。 -在分区方面，逻辑表拥有与物理表完全一致的分区规则及 Region 分布。这是非常自然的，因为逻辑表的数据直接存储在物理表中，所以分区规则也是一致的。 +关联到同一物理表的逻辑表使用相同的分区布局。写入时，Metric 引擎为每行数据记录逻辑表身份；读取时，它在扫描物理 Region 前增加逻辑表过滤条件。 -在路由元数据方面，逻辑表的路由地址为逻辑地址，即该逻辑表所对应的物理表是什么，而后通过该物理表进行二次路由取得真正的物理地址。这一间接路由方式能够显著减少 `Metric` 引擎的 Region 发生迁移调度时所需要修改的元数据数量。 +逻辑表的路由只保存所属物理表的 ID，再由物理表路由解析出持有 Region 的 Datanode。逻辑路由本身不记录 peer，因此迁移物理 Region 只需改写一条物理路由，而不必改写映射到它的每一条逻辑路由。 -在操作方面，`Metric` 引擎支持对逻辑表进行标准的 DML 操作（INSERT、DELETE、SELECT）。然而，对物理表的操作进行了有限的支持以防止误操作，例如禁止直接写入物理表等操作防止影响用户逻辑表的数据。总体上可以认为物理表是对用户只读的。 +逻辑表支持普通的 INSERT、DELETE 和 SELECT 操作。直接写入物理 Region 会绕过逻辑表映射，因此会被拒绝；物理表仍然可以查询。 -为了提升对大量表同时进行 DDL（Data Definition Language，数据操作语言）操作时性能，如 Prometheus Remote Write 冷启动时大量 metrics 带来的自动建表请求，以及前面提到的迁移物理 Region 时大量路由表的修改请求等，`Metric` 引擎引入了一些批量 DDL 操作。这些批量 DDL 操作能够将大量的 DDL 操作合并成一个请求，从而减少了元数据的查询及修改次数，提升了性能。 - -除了物理表的物理数据 Region 之外，`Metric` 引擎还额外为每一个物理数据 Region 创建了一个物理的元数据 Region，用于存储 `Metric` 引擎自身为了维护映射等状态所需要的一些元数据。这些元数据包括逻辑表与物理表的映射关系，逻辑列与物理列的映射关系等等。 +批量 DDL 用于减少大量逻辑表同时创建或更新时的元数据操作，例如 Prometheus Remote Write 自动建表或物理 Region 迁移。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/overview.md index 3f44a4543b..c7684eb165 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/overview.md @@ -7,22 +7,26 @@ description: 介绍了 Datanode 的主要职责和组件，包括 gRPC 服务、 ## Introduction -`Datanode` 主要的职责是为 GreptimeDB 存储数据，我们知道在 GreptimeDB 中一个 `table` 可以有一个或者多个 `Region`, -而 `Datanode` 的职责便是管理这些 `Region` 的读写。`Datanode` 不感知 `table`，可以认为它是一个 `region server`。 -所以 `Frontend` 和 `Metasrv` 按照 `Region` 粒度来操作 `Datanode`。 +Datanode 存储并处理 Region 数据。一张表可以包含多个 Region，但 Datanode 不负责表级路由。Frontend 按 Region 发送数据请求，Metasrv 则控制 Region 的放置和生命周期。 -![Datanode](/datanode.png) +这个边界使同一个 Region server 可以承载不同的存储引擎，而不向 Frontend 或 Metasrv 暴露引擎实现。 + +![Frontend 向 Datanode Region server 发送 Region 请求，Metasrv 通过 heartbeat task 与 Datanode 交换生命周期指令。Region server 使用本地 query engine，并将请求分发给 Mito、Metric 或 File Region engine。](/datanode-architecture.zh.svg) ## Components -一个 datanode 包含了 region server 所需的全部组件。这里列出了比较重要的部分： - -- 一个 gRPC 服务来提供对 `Region` 数据的读写，`Frontend` 便是使用这个服务来从 `Datanode` 读写数据。 -- 一个 HTTP 服务，可以通过它来获得当前节点的 metrics、配置信息等 -- `Heartbeat Task` 用来向 `Metasrv` 发送心跳，心跳在 GreptimeDB 的分布式架构中发挥着至关重要的作用， - 是分布式协调和调度的基础通信通道，心跳的上行消息中包含了重要信息比如 `Region` 的负载，如果 `Metasrv` 做出了调度 - 决定（比如 Region 转移），它会通过心跳的下行消息发送指令到 `Datanode` -- `Datanode` 不负责解析用户 SQL 或进行分布式规划，用户对一个或多个 `Table` 的查询请求会在 `Frontend` 中被转换为 - `Region` 查询请求，`Datanode` 负责用本地 query engine 执行这些 `Region` 查询计划 -- 一个 `Region Manager` 用来管理 `Datanode` 上的所有 `Region`s -- GreptimeDB 支持可插拔的多引擎架构，目前已有的 engine 包括 `File Engine` 和 `Mito Engine` +Datanode 包含以下主要组件： + +- Region server 记录已打开的 Region，并把读写和生命周期请求分发给该 Region 注册的 engine。 +- `Mito` 是主要的时序 Region engine。`Metric` 将多个逻辑指标 Region 映射到共享的 Mito Region，`File` 通过 Region 接口访问外部文件。 +- 本地 query engine 执行 Region 查询计划。它不解析客户端 SQL，也不进行集群级规划。 +- Heartbeat task 向 Metasrv 上报节点和 Region 状态，并接收 open、close、upgrade、downgrade 和迁移步骤等指令。 +- gRPC 承载发往 Datanode 的 Region 请求；HTTP 提供 metrics 和配置等节点诊断信息。 + +## Region 请求生命周期 + +Mito 写入到达 Region server 后，Region server 根据 Region 元数据选择 Mito。Mito 将 mutation 追加到 WAL，写入 memtable，并在之后把 memtable flush 为 SST 文件。Metric 写入会先补充逻辑表标识，再委托给对应的物理 Mito Region。 + +读取时，本地 query engine 在 Region engine 提供的 table provider 上执行 Region 计划。Mito scan 获取不可变的 Region version，读取相关 memtable 和 SST 文件，合并并去重数据，最后返回 Arrow record batch 流。 + +Region 所有权可以在不重启 Datanode 的情况下改变。Metasrv 通过心跳流下发生命周期指令；Region server 将指令应用到对应 engine，并在后续心跳中上报新的 Region role 和统计信息。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/python-scripts.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/python-scripts.md deleted file mode 100644 index 831278e80d..0000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/python-scripts.md +++ /dev/null @@ -1,30 +0,0 @@ ---- -keywords: [Python 脚本, 数据分析, CPython, RustPython] -description: 介绍了在 GreptimeDB 中使用 Python 脚本进行数据分析的两种后端实现：CPython 和嵌入式 RustPython 解释器。 ---- - -# Python 脚本 - -## 简介 - -Python 脚本是分析本地数据库中的数据的便捷方式， -通过将脚本直接在数据库内运行而不是从数据库拉取数据的方式，可以节省大量的数据传输时间。 -下图描述了 Python 脚本的工作原理。 -`RecordBatch`（基本上是表中的一列，带有类型和元数据）可以来自数据库中的任何地方， -而返回的 `RecordBatch` 可以用 Python 语法注释以指示其元数据，例如类型或空。 -脚本将尽其所能将返回的对象转换为 `RecordBatch`，无论它是 Python 列表、从参数计算出的 `RecordBatch` 还是常量（它被扩展到与输入参数相同的长度）。 - -![Python Coprocessor](/python-coprocessor.png) - -## 两种可选的后端 - -### CPython 后端 - -该后端由 [PyO3](https://pyo3.rs/v0.18.1/) 提供支持，可以使用您最喜欢的 Python 库（如 NumPy、Pandas 等），并允许 Conda 管理您的 Python 环境。 - -但是使用它也涉及一些复杂性。您必须设置正确的 Python 共享库，这可能有点棘手。一般来说，您只需要安装 `python-dev` 包。但是，如果您使用 Homebrew 在 macOS 上安装 Python，则必须创建一个适当的软链接到 `Library/Frameworks/Python.framework`。有关使用 PyO3 crate 与不同 Python 版本的详细说明，请参见 [这里](https://pyo3.rs/v0.18.1/building_and_distribution#configuring-the-python-version) - -### 嵌入式 RustPython 解释器 - -可以运行脚本的实验性 [python 解释器](https://github.com/RustPython/RustPython)，它支持 Python 3.10 语法。您可以使用所有的 Python 语法，更多信息请参见 [Python 脚本的用户指南](/user-guide/python-scripts/overview.md). - diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/query-engine.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/query-engine.md index 62d7563bb2..84445759d4 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/query-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/query-engine.md @@ -7,33 +7,33 @@ description: 介绍了 GreptimeDB 的查询引擎架构，基于 Apache DataFusi ## 介绍 -GreptimeDB 的查询引擎是基于[Apache DataFusion][1]（属于[Apache Arrow][2]的子项目）构建的，它是一个用 Rust 编写的出色的查询引擎。它提供了一整套功能齐全的组件，从逻辑计划、物理计划到执行运行时。下面将解释每个组件如何被整合在一起，以及在执行过程中它们的位置。 +GreptimeDB 的查询引擎基于 [Apache DataFusion][1]。DataFusion 提供逻辑计划、物理计划、优化器框架和执行运行时；GreptimeDB 在此基础上增加各查询语言的 planner、存储相关优化规则、自定义计划节点和分布式执行。 -![Execution Procedure](/execution-procedure.png) +DDL 和其他控制面操作由 statement executor 分发。Query engine 接收数据处理计划，包括 `INSERT ... SELECT` 等操作中读取输入数据的部分。 -入口点是逻辑计划，它被用作查询或执行逻辑等的通用中间表示。逻辑计划的两个主要来源是：1. 用户查询，例如通过 SQL 解析器和规划器的 SQL；2. Frontend 的分布式查询，这将在下一节中详细解释。 +## 查询生命周期 -接下来是物理计划，或称执行计划。与包含所有逻辑计划变体（除特殊扩展计划节点外）的大型枚举的逻辑计划不同，物理计划实际上是一个定义了在执行过程中调用的一组方法的特性。所有数据处理逻辑都包装在实现该特性的相应结构中。它们是对数据执行的实际操作，如聚合器 `MIN` 或 `AVG` ，以及表扫描 `SELECT ... FROM`。 +1. SQL、PromQL 或日志查询 planner 通过 catalog 解析表，并生成 DataFusion logical plan。DataFusion 不直接支持的操作由 GreptimeDB plan extension 表示。 +2. DataFusion 的 analyzer 和 optimizer rule 与 GreptimeDB rule 共同运行。这些规则规范化表达式和类型、改写时间范围操作、将 projection 和 filter 下推到 scan，并在需要时引入分布式计划节点。 +3. Physical planner 将优化后的 logical plan 转换为流式 operator。GreptimeDB 随后应用 scan 并行度、排序和分布式执行相关的 physical rule。 +4. 执行阶段通过 physical plan 拉取 Arrow record batch。存储 scan 接收 projection 和 predicate，下游 operator 消费数据流，无需先物化完整结果。 -优化阶段通过转换逻辑计划和物理计划来提高执行性能，现在全部基于规则。它也被称为“基于规则的优化”。一些规则是 DataFusion 原生的，其他一些是在 GreptimeDB 中自定义的。在未来，我们计划添加更多规则，并利用数据统计进行基于成本的优化 (CBO)。 - -最后一个阶段"执行"是一个动词，代表从存储读取数据、进行计算并生成预期结果的过程。虽然它比之前提到的概念更抽象，但你可以简单地将它想象为执行一个 Rust 异步函数，并且它确实是一个异步流。 - -当你想知道 SQL 是如何通过逻辑计划或物理计划中表示时，`EXPLAIN [VERBOSE] ` 是非常有用的。 +使用 [`EXPLAIN`](/reference/sql/explain.md) 查看逻辑和物理计划。`EXPLAIN ANALYZE` 还会执行计划并报告运行时指标。 ## 数据表示 -GreptimeDB 使用 [Apache Arrow][2]作为内存中的数据表示格式。它是面向列的，以跨平台格式，也包含许多高性能的基础操作。这些特性使得在许多不同的环境中共享数据和实现计算逻辑变得容易。 +GreptimeDB 使用 [Apache Arrow][2] record batch 作为内存数据表示。一个 record batch 包含等长的列数组和 schema。查询 operator 交换这些 batch 组成的数据流，使 Region scan 到结果编码的执行路径保持列式处理。 ## 索引 -在时序数据中，有两个重要的维度：时间戳和标签列（或者类似于关系数据库中的主键）。GreptimeDB 将数据分组到时间桶中，因此能在非常低的成本下定位和提取预期时间范围内的数据。GreptimeDB 中主要使用的持久文件格式 [Apache Parquet][3] 提供了多级索引和过滤器，使得在查询过程中很容易修剪数据。在未来，我们将更多地利用这个特性，并开发我们的分离索引来处理更复杂的用例。 +索引构建和持久化格式属于存储引擎。查询层向 scan 提供 predicate 和 projection，Mito 再利用时间范围、Parquet 统计信息和索引跳过不可能匹配的数据。参见[数据持久化与索引](./data-persistence-indexing.md)。 + + -## 分布式查询 +## 分布式执行 -参考 [Distributed Querying][6]. +分布式模式下，Frontend 规划集群级查询，Datanode 执行 Region 本地子计划。[`MergeScan`][6] 是两个阶段之间的边界。 -[1]: https://github.com/apache/arrow-datafusion +[1]: https://datafusion.apache.org/ [2]: https://arrow.apache.org/ -[3]: https://parquet.apache.org [6]: ../frontend/distributed-querying.md diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/storage-engine.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/storage-engine.md index 67f98216fe..1b0f0eb689 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/storage-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/storage-engine.md @@ -7,7 +7,7 @@ description: 详细介绍了 GreptimeDB 的存储引擎架构、数据模型和 ## 概述 -`存储引擎` 负责存储数据库的数据。Mito 是我们默认使用的存储引擎，基于 [LSMT][1]（Log-structured Merge-tree）。我们针对处理时间序列数据的场景做了很多优化，因此 mito 这个存储引擎并不适用于通用用途。 +Mito 是 GreptimeDB 的默认存储引擎，基于 [LSM tree][1]，面向时间序列负载设计，而不是通用的嵌入式存储引擎。 ## 架构 下图展示了存储引擎的架构和处理数据的流程。 @@ -21,8 +21,8 @@ description: 详细介绍了 GreptimeDB 的存储引擎架构、数据模型和 - 基于 `LogStore` API 实现，不关心底层存储介质。 - WAL 的日志记录可以存储在本地磁盘上，也可以存储在实现了 `LogStore` API 的远程日志服务中，例如 Kafka（remote WAL）。 - Memtable - - 数据首先写入 `active memtable`，又称 `mutable memtable`。 - - 当 `mutable memtable` 已满时，它将变为只读的 `immutable memtable`。 + - Mito 根据 time index 将数据行写入 mutable memtable。 + - Flush 冻结 mutable memtable，安装一组新的 mutable memtable 以接收写入，再将冻结的 memtable 写为 SST 文件。 - SST - SST 的全名为有序字符串表（`Sorted String Table`）。 - `immutable memtable` 刷到持久存储后形成一个 SST 文件。 @@ -100,7 +100,9 @@ Mito 会按 primary key 对行分组，并按时间排序，因此 SST 中的数 Mito 支持两种 SST 格式：`flat` 和 `primary_key`。`flat` 是新表的默认格式，适用于各种 primary key 基数，包括高基数 key。`primary_key` 是为了兼容旧表而保留的遗留格式。更多详情请参考 [SST format](/reference/sql/create.md#创建指定-sst-格式的表) 和[表设计指南](/user-guide/deployments-administration/performance-tuning/design-table.md#sst-格式)。 -SST layout +![Mito 默认的 flat SST 布局将文件级元数据与包含数据列和合并元数据的 Parquet row group 组合在一起。](/mito-sst-layout.zh.svg) + +一个 SST 可能跨越多个 compaction time window。 ## 扫描裁剪 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/wal.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/wal.md index 529adcdf50..805b294091 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/wal.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/datanode/wal.md @@ -9,20 +9,26 @@ description: 介绍了 GreptimeDB 的预写日志（WAL）机制，包括其命 ## 介绍 -我们的存储引擎受到了日志结构合并树（Log-structured Merge Tree，LSMT）的启发。对数据的变更操作直接应用于 MemTable 而不是持久化到磁盘上的数据页，这显著提高了性能，但也带来了持久化相关的问题，特别是在 Datanode 意外崩溃时。与所有类似 LSMT 的存储引擎一样，GreptimeDB 使用预写日志（Write-Ahead Log，WAL）来确保数据被可靠地持久化，并且保证崩溃时的数据完整性。 +Mito 在将数据 flush 为 SST 文件前，先在 memtable 中缓冲写入。每个 Region 的 mutation 会先追加到预写日志（WAL），从而恢复尚未进入 SST 的数据。 -预写日志是一个仅提供追加写的文件组。所有的 INSERT 和 DELETE 操作都被转换为操作日志，然后追加到 WAL。一旦操作日志被持久化到底层文件，该操作才可以进一步应用到 MemTable。 +WAL 通过统一的 log-store 抽象访问，可以使用本地 raft-engine 或远端 Kafka。 -当数据节点重新启动时，WAL 中的操作条目将被重放，以重建正确的 MemTable 状态。 +## 写入与恢复流程 -![WAL in Datanode](/wal.png) +正常写入遵循以下顺序： + +1. Region worker 分配 sequence number 和 WAL entry ID。 +2. 将 mutation 追加到 WAL。追加失败时，不会把 mutation 写入 memtable。 +3. WAL 追加成功后，Mito 将 mutation 写入 memtable，并发布新的 committed sequence。 +4. Flush 将不可变 memtable 写为 SST 文件，并持久化包含新文件和 `flushed_entry_id` 的 manifest edit。 +5. Manifest edit 持久化后，`flushed_entry_id` 及以前的 WAL entry 被标记为 obsolete；log store 可以稍后再回收物理空间。 + +Manifest 是恢复边界。正常重新打开 Region 时，Mito 根据 manifest 重建 Region，并从 `flushed_entry_id + 1` 开始重放 WAL。Region 状态切换可以指定更晚的 replay checkpoint，但不会重放早于已持久化 flush 边界的 entry。 ## 命名空间 -WAL 的命名空间用于区分来自不同 region 的条目。追加和读取操作必须提供一个命名空间。目前，region ID 被用作命名空间，因为每个 region 都有一个在数据节点重新启动时需要重构的 MemTable。 +WAL entry 按 Region 隔离，而不是按表隔离。追加和读取都需要指定 Region namespace，使单个 Region 可以独立重放或截断。本地 raft-engine 使用 Region ID 作为 namespace ID；Kafka provider 则在基于 topic 的日志中保留 Region 标识。 ## 同步/异步刷盘 -默认情况下，WAL 的追加写是异步的，这意味着写入方不会等待操作日志被刷入到磁盘并持久化。这个默认设置提供了更高的性能，但在服务器意外关闭时可能会丢失数据。另一方面，同步刷新提供了更高的可靠性，但其代价是性能更低。 - -在 v0.4 版本中，新的 region worker 架构可以使用批处理来减轻同步刷盘的开销。 +对于本地 raft-engine，`sync_write` 控制追加写是否等待日志同步到持久化存储，默认值为 `false`。异步写入延迟较低，但主机在缓冲数据同步前故障时，可能丢失最近确认的 entry。Kafka WAL 的持久性由 producer 和集群配置决定，不受这个本地选项控制。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/arrangement.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/arrangement.md index dd3b6de090..7b35b50259 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/arrangement.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/arrangement.md @@ -5,6 +5,8 @@ description: 描述了 Arrangement 在数据流进程中的状态存储功能， # Arrangement +本页介绍 Flownode 旧 streaming 模式使用的状态结构；batching 模式不使用 Arrangement。 + Arrangement 存储数据流进程中的状态，存储 flow 的更新流（stream）以供进一步查询和更新。 Arrangement 本质上存储的是带有时间戳的键值对。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/batching_mode.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/batching_mode.md index 257cca80b9..f83c4e5da1 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/batching_mode.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/batching_mode.md @@ -9,13 +9,13 @@ description: Flownode 批处理模式概述，一个为数据库提供持续数 ## 概述 -`flownode` 中的批处理模式专为持续数据聚合而设计。它在离散的、微小的时间窗口上周期性地执行用户定义的 SQL 查询。这与数据在到达时即被处理的流处理模式形成对比。 +`flownode` 中的批处理模式专为持续数据聚合而设计。它在离散的小时间窗口上周期性执行用户定义的 SQL 查询。旧 streaming 路径则在数据到达时进行处理，目前仅为兼容已有 workload 而保留，不推荐新 workload 使用。 其核心思想是： 1. 定义一个带有 SQL 查询的 `flow`，该查询将数据从源表聚合到目标表。 2. 查询通常在时间戳列上包含一个时间窗口函数（例如 `date_bin`）。 3. 当新数据插入源表时，系统会将相应的时间窗口标记为“脏”(dirty)。 -4. 一个后台任务会周期性地唤醒，识别这些脏窗口，并为那些特定的时间范围重新运行聚合查询。 +4. 一个后台任务按自身的节奏运行，在下一次求值时取出待处理的脏窗口，并对这些时间范围重新运行聚合查询。 5. 然后将结果插入到目标表中，从而有效地更新聚合视图。 ## 架构 @@ -39,15 +39,15 @@ description: Flownode 批处理模式概述，一个为数据库提供持续数 - **状态 (`TaskState`)**: 包含任务的动态、可变状态，最重要的是 `DirtyTimeWindows`。 - **执行循环**: 任务运行一个无限循环 (`start_executing_loop`)，该循环： 1. 检查关闭信号。 - 2. 等待一个预定的时间间隔或直到被唤醒。 + 2. 睡眠到下一次求值时间。设置了求值调度的任务睡眠到下一个调度时间点；自适应任务则按时间窗口大小和最小刷新间隔计算出的轮询间隔睡眠。 3. 基于当前的脏时间窗口集合生成一个新的查询计划 (`gen_insert_plan`)。 4. 对数据库执行查询 (`execute_logical_plan`)。 5. 清理已处理的脏窗口。 ### `TaskState` 和 `DirtyTimeWindows` -- **`TaskState`**: 此结构体跟踪 `BatchingTask` 的运行时状态。它包括 `dirty_time_windows`，这对于确定需要完成哪些操作至关重要。 -- **`DirtyTimeWindows`**: 这是一个关键的数据结构，用于跟踪自上次查询执行以来哪些时间窗口接收到了新数据。它存储一组不重叠的时间范围。当任务的执行循环运行时，它会参考此结构来构建一个 `WHERE` 子句，该子句仅过滤源表中的脏时间窗口。 +- **`TaskState`**: 此结构体跟踪 `BatchingTask` 的运行时状态，包括用于确定待处理工作的 `dirty_time_windows`。 +- **`DirtyTimeWindows`**: 此数据结构跟踪上次查询执行后接收到新数据的时间窗口，并保存一组不重叠的时间范围。执行循环根据它构造 `WHERE` 子句，只从源表选择脏窗口。 ### `TimeWindowExpr` @@ -56,15 +56,15 @@ description: Flownode 批处理模式概述，一个为数据库提供持续数 - **求值**: 它可以接受一个时间戳并对时间窗口表达式求值，以确定该时间戳所属窗口的开始和结束。 - **窗口大小**: 它还可以从表达式中确定时间窗口的大小（持续时间）。 -这对于标记窗口为脏以及在查询源表时生成正确的过滤条件都至关重要。 +标记脏窗口和生成源表过滤条件使用同一套计算。 ## 查询执行流程 以下是批处理模式下查询执行的简化分步演练： 1. **数据摄取**: 新数据被写入源表。 -2. **标记为脏**: `BatchingEngine` 收到有关新数据的通知。它使用与每个相关 flow 关联的 `TimeWindowExpr` 来确定哪些时间窗口受到新数据点的影响。然后将这些窗口添加到相应 `TaskState` 中的 `DirtyTimeWindows` 集合中。 -3. **任务唤醒**: `BatchingTask` 的执行循环被唤醒，原因可能是其周期性调度，也可能是因为它被通知有大量积压的脏窗口。 +2. **标记为脏**: `BatchingEngine` 收到有关新数据的通知。它使用与每个相关 flow 关联的 `TimeWindowExpr` 来确定哪些时间窗口受到新数据点的影响。然后将这些窗口添加到相应 `TaskState` 中的 `DirtyTimeWindows` 集合中。标记脏窗口不会唤醒任务。 +3. **下一次求值**: `BatchingTask` 的执行循环在调度时间点或自适应轮询间隔结束后进入下一次求值，取出待处理的脏窗口。 4. **计划生成**: 任务调用 `gen_insert_plan`。此方法： - 检查 `DirtyTimeWindows`。 - 生成一系列 `OR` 连接的 `WHERE` 子句（例如 `(ts >= 't1' AND ts < 't2') OR (ts >= 't3' AND ts < 't4') ...`），覆盖所有脏窗口。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/dataflow.md index 9d07922542..ee17b9c785 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/dataflow.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/dataflow.md @@ -1,18 +1,39 @@ --- -keywords: [Dataflow, SQL 查询, 执行计划, 数据流, map, reduce] -description: 解释了 Dataflow 模块的核心计算功能，包括 SQL 查询转换、内部执行计划、数据流的触发运行和支持的操作。 +keywords: [Flownode, batching mode, streaming mode, Dataflow, 脏时间窗口] +description: 介绍 Flownode 如何选择并运行 batching 和旧 streaming 两条执行路径。 --- # 数据流 +Flownode 内部有两条执行路径： + +- **Batching mode** 是聚合和 TQL workload 的主要执行路径。它查询已经持久化的 source 数据，并将物化结果写入 sink table。 +- **Streaming mode** 是为兼容已有 workload 而保留的旧执行路径，不推荐新 workload 使用。Frontend 会把新到达的行同步给它进行增量处理。 + +用户不能直接选择执行模式。创建 Flow 时，GreptimeDB 根据查询和 source table 的属性选择执行路径。聚合、`DISTINCT` 和 TQL 查询使用 batching mode；简单的非聚合查询，以及任何 source table 使用 `ttl = 'instant'` 的 Flow，目前仍使用 streaming mode。 + +## Batching mode + +Batching mode 复用 GreptimeDB 的查询引擎，不需要为每一行输入维护一张算子图。对于基于时间窗口的 Flow，主循环如下： + +1. Source table 收到写入后，把受影响的时间窗口标记为 dirty。 +2. `BatchingTask` 按求值调度或自适应轮询节奏运行，并在该次求值时收集待处理的 dirty window。标记 dirty window 不会唤醒任务。 +3. 任务把这些窗口转换成时间谓词，加入 Flow 查询，再请求 Frontend 查询 source table。 +4. 查询结果写入 sink table，更新已重新计算窗口对应的物化结果。 +5. 成功处理的窗口从 dirty set 中移除；执行失败的工作仍可在后续调度中处理。 + +设置了 evaluation interval、但查询中没有时间窗口表达式的 Flow，会在每次调度时执行完整查询。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见 [Flownode 批处理模式开发者指南](./batching_mode.md)。 + +## Streaming mode + Dataflow 模块（参见 `flow::compute` 模块）是 `flow` 的核心计算模块。 它接收 SQL 查询并将其转换为 `flow` 的内部执行计划。 然后，该执行计划被转化为实际的数据流，而数据流本质上是一个由带有输入和输出端口的函数组成的有向无环图（DAG）。 -数据流会在需要时被触发运行。 +新到达的行变更会增量驱动这张图执行。 -目前该数据流只支持 `map`和 `reduce` 操作，未来将添加对 `join` 等操作的支持。 +Renderer 支持 map/filter/project 和 reduce 操作。执行计划中已经有 join 和 union 节点，但 streaming renderer 尚未实现它们。 在内部，数据流使用 `tuple(row, time, diff)` 以行格式处理数据。 这里 `row` 表示实际传递的数据，可能包含多个 `value` 对象。 `time` 是系统时间，用于跟踪数据流的进度，`diff` 通常表示行的插入或删除（+1 或 -1）。 -因此，`tuple` 表示给定系统时间的 `row` 的插入/删除操作。 +因此，`tuple` 表示给定系统时间的 `row` 的插入/删除操作。有状态算子通过 [Arrangement](./arrangement.md) 保存这些变更的索引 trace。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/distributed-querying.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/distributed-querying.md index 612174662b..e6d0c35160 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/distributed-querying.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/distributed-querying.md @@ -1,41 +1,20 @@ --- -keywords: [分布式查询, 查询拆分, 查询合并, TableScan, 物理计划] -description: 介绍 GreptimeDB 中的分布式查询方法，包括查询的拆分和合并过程，以及 TableScan 节点的作用。 +keywords: [分布式查询, 逻辑计划, MergeScan, Substrait, Region 裁剪] +description: 介绍 GreptimeDB 如何把逻辑查询计划划分为 Frontend 和 Datanode 上的执行任务。 --- # 分布式查询 -我们知道在 GreptimeDB 中数据是如何分布的（参见“[表分片][1]”），那么如何查询呢？在 GreptimeDB 中，分布式查询非常简单。简单来说，我们只需将查询拆分为子查询，每个子查询负责查询表数据的一个部分，然后将所有结果合并为最终结果。这是一种典型的“拆分 - 合并”方法。具体来说，让我们从查询到达 `frontend` 开始。 +Frontend 和 Datanode 使用同一套基于 DataFusion 的查询引擎。在分布式模式下，Frontend 会增加一个规划步骤，将 Datanode 上执行的工作与 Frontend 上完成的工作分开。 -当查询到达 `frontend` 时，它首先被解析为 SQL 抽象语法树（AST）。我们遍历 AST，并从中生成逻辑计划。顾名思义，逻辑计划只是如何“逻辑地”执行查询的“提示”，它不能被直接运行，因此我们进一步从中生成可执行的物理计划。物理计划是一种类似树形的数据结构，每个节点实际上表示查询的执行方法。一旦我们从上到下运行物理计划树，结果数据将从叶子到根流动，被合并或计算。最终，我们在根节点的输出处得到了查询的结果。 +![Frontend query](/frontend-query.png) -到目前为止，这只是一个典型的“volcano”查询执行模型，你可以在几乎每个 SQL 数据库中看到这种模型。那么“分布式”是在哪里发生的呢？这全部发生在一个名为“TableScan”的物理计划节点中。TableScan 是物理计划树中的一个叶子节点，它负责扫描表的数据（就像它的名称所暗示的）。当 `frontend` 即将扫描表时，它首先需要根据每个 `region` 的数据范围将表扫描拆分为较小的扫描。 +## 分布式规划 -[1]: ./table-sharding.md +分布式规划器重写逻辑计划，把可以下推的算子移向表扫描，并用 `MergeScan` 节点包装远端子计划。分区列上的谓词还会在任务调度前用于裁剪 Region。 -表的所有 `region` 都有它们存储数据的范围。以下表为例： +算子能否下推取决于计划形态和算子本身的性质。不支持的部分会保留在 Frontend。初始设计及交换律规则参见[分布式规划器 RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md)。 -```sql -CREATE TABLE my_table ( - a INT, - others STRING, - ts TIMESTAMP TIME INDEX, -) -PARTITION ON COLUMNS (a) ( - a < 10, - a >= 10 AND a < 20, - a >= 20 -); -``` +## 分布式计划 -`my_table` 表创建时被设定了 3 个分区。在 GreptimeDB 的当前实现中，将为该表创建 3 个 `region`（分区与 `region` 的比例为 1:1）。这 3 个区域将分别包含以下范围："[-∞, 10)", "[10, 20)" 和 "[20, +∞)"。例如，如果提供了值 "42"，我们将搜索这些范围，并找到包含该值的相应的 `region`（在此示例中为第 3 个 `region`）。 - -对于查询，我们使用“过滤器”来查找 `region`。 "过滤器"是 "WHERE" 子句中的条件。例如，查询 `SELECT * FROM my_table WHERE a < 10 AND others = 'x'`，其“过滤器”为“a < 10 AND others = 'x'”。然后我们检查这些范围，找出包含满足过滤器条件的值的所有 `region`。 - -> 如果某个查询没有任何过滤器，则将其视为全表扫描。 - -找到所需的区域后，我们只需在其中组装子扫描。通过这种方式，我们将查询拆分为子查询，每个子查询都获取表数据的一部分。子查询在 `datanode` 中执行，并在 `frontend` 中等待完成。它们的结果将合并为表扫描请求的最终返回。 - -下面这张图片总结了分布式查询执行的过程： - -![Distributed Querying](/distributed-querying.png) +远端输入是完整的逻辑子计划，并不局限于表扫描。Frontend 使用 [Substrait](https://substrait.io) 序列化子计划，再向持有相应数据的 Datanode 发送 Region 级请求。Datanode 在本地规划并执行子计划，将结果流返回 Frontend。Frontend 合并远端数据流，并执行没有下推的算子。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/overview.md index ee48da3ce3..6befdc55a9 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/overview.md @@ -5,31 +5,51 @@ description: GreptimeDB Frontend 组件概述 - 为客户端请求提供服务 # Frontend -**Frontend** 是一个无状态服务，作为 GreptimeDB 中客户端请求的入口点。它为多种数据库协议提供统一接口，并充当代理，将读写请求转发到分布式系统中的相应 Datanode。 +Frontend 是 GreptimeDB 中负责请求编排的无状态服务。Server 层负责终止协议并转换线上消息；Frontend 为这些协议处理器提供数据库行为，包括权限检查、语句执行、路由和分布式查询规划。 + +Frontend 不存储表数据。它缓存从 Metasrv 获取的 catalog 和路由元数据；元数据发生变化时，Metasrv 通过心跳响应通知 Frontend 失效相应缓存。 ## 核心功能 -- **协议支持**：支持多种数据库协议，包括 SQL、PromQL、MySQL 和 PostgreSQL。详见[协议][1] -- **请求路由**：基于元数据将请求路由到相应的 Datanode -- **查询分发**：将分布式查询拆分到多个节点 -- **响应聚合**：合并来自多个 Datanode 的结果 -- **认证授权**：安全和访问控制验证 +- 为支持的[协议][1]提供查询和写入行为。 +- 解析 catalog、schema、table 和 Region 路由。 +- 在执行请求前完成权限检查。 +- 规划分布式查询并合并 Datanode 返回的结果。 +- 将表级写入和删除转换为 Region 请求。 ## 架构 ### 关键组件 -- **协议处理器**：处理不同的数据库协议 -- **目录管理器**：缓存来自 Metasrv 的元数据以实现高效的请求路由和 Schema 校验 -- **分布式规划器**：将逻辑计划转换为分布式执行计划 -- **请求路由器**：为每个请求确定目标 Datanodes + +- 协议处理器将 SQL、PromQL、gRPC 写入和可观测性协议转换为 Frontend 的内部请求接口。 +- Catalog manager 和 partition manager 提供表元数据、分区规则和 Region 路由。 +- Statement executor 将查询、DML 和 DDL 分发到各自的执行路径。 +- 分布式规划器把表扫描替换为可跨 Datanode 执行的 `MergeScan` 计划。 ### 请求流程 -![request flow](/request_flow.png) +不同操作会走不同的请求路径。 + +#### 查询 + +1. 协议处理器创建查询上下文，并完成认证和权限检查。 +2. 对应查询语言的 planner 生成逻辑计划。分布式模式下，planner 根据分区元数据选择 Region 并生成分布式计划。 +3. Frontend 将 Region 子计划发送到对应 Datanode。Datanode 在本地 Region engine 上执行，并返回 Arrow record batch 流。 +4. Frontend 执行剩余算子、合并数据流，再按客户端协议编码结果。 + +#### 写入和删除 + +1. Frontend 根据表 schema 校验请求。支持 schema-on-write 的协议可以先创建缺失的表或新增列，再重试写入。 +2. 分区规则把每一行分配给 Region。Frontend 为各目标 Region 构造请求，并路由到当前 Region leader。 +3. Datanode 的 Region server 将请求分发到对应的 Region engine。单机模式下，请求直接发送给内嵌的 Region server。 + +#### DDL + +Statement executor 将 DDL 转换为 task。分布式模式下，Metasrv 以持久化 procedure 执行 task、更新元数据，并协调 Datanode 上的 Region 操作。单机模式复用相同的语句边界，但使用本地元数据和 procedure 实现。 ### 部署 -下图是 GreptimeDB 在云上的一个典型的部署。`Frontend` 实例组成了一个集群处理来自客户端的请求： +下图展示了 GreptimeDB 的一种云上部署。多个 Frontend 实例共同处理客户端请求： ![frontend](/frontend.png) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/table-sharding.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/table-sharding.md index 63a8ac10a6..cf39afe27b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/table-sharding.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/frontend/table-sharding.md @@ -5,21 +5,17 @@ description: 介绍 GreptimeDB 中表数据的分片方法，包括分区和 Reg # 表分片 -对于任何分布式数据库来说，数据的分片都是必不可少的。本文将描述 GreptimeDB 中的表数据如何进行分片。 +GreptimeDB 将一张表分为多个 Region。分区表达式定义每行数据属于哪个 Region，Region 路由则定义当前由哪个 Datanode 持有该 Region。 ## 分区 -有关创建分区表的语法，请参阅用户指南中的[表分片](/user-guide/deployments-administration/manage-data/table-sharding.md)部分。 +分区是由一个或多个列上的表达式描述的逻辑行集合。分区布局需要覆盖表的输入域，使每一行都能找到唯一的目标 Region。SQL 语法和支持的表达式参见[表分片](/user-guide/deployments-administration/manage-data/table-sharding.md)。 ## Region -在创建分区后，表中的数据被逻辑上分割。你可能会问："在 GreptimeDB 中，被逻辑上分区的数据是如何存储的？" 答案是保存在 `Region` 当中。 - -每个 `Region` 对应一个分区，并保存分区的数据。所有的 `Region` 分布在各个 `Datanode` 之中。 -`Metasrv` 管理 `Region` 到 `Datanode` 的路由信息。如果建表后需要调整分区布局， -GreptimeDB 支持通过显式的 [repartition](/user-guide/deployments-administration/manage-data/repartition.md) 操作拆分或合并分区。 +每个分区对应一个 Region。Region ID 是 Frontend、Datanode 和 Metasrv 用于存储和路由的标识。同一张表的多个 Region 可以放在同一个 Datanode 上。 分区和 Region 的关系参见下图： @@ -54,3 +50,13 @@ GreptimeDB 支持通过显式的 [repartition](/user-guide/deployments-administr └──────────────────────────────────┘ 可以放在同一个 Datanode 之中 ``` + +## 路由与剪枝 + +写入时，Frontend 对每行数据计算分区规则，按 Region 分组，再根据路由表把 Region 请求发送到当前 leader。 + +查询时，分布式 planner 将查询谓词与分区表达式比较，只扫描可能满足谓词的 Region。如果分区元数据缺失或无法安全解释，planner 会退化为扫描所有 Region，避免漏掉数据。 + +## 调整分区布局 + +[Repartition](/user-guide/deployments-administration/manage-data/repartition.md) 通过显式的 split 或 merge 调整已有布局。Metasrv 以持久化 procedure 执行变更，更新 Region 路由和分区表达式，并使旧的表路由缓存失效。Frontend 刷新到新元数据后，后续请求使用新的布局。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/getting-started.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/getting-started.md index e5aa2ca855..999f696ed0 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/getting-started.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/getting-started.md @@ -15,14 +15,13 @@ description: 介绍如何在本地环境中从源代码编译和运行 GreptimeD ### 构建依赖项 -- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line)（可选） +- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line)（可选；克隆仓库需要，构建本身不需要） - C/C++ 工具链：提供编译和链接的基本工具。在 Ubuntu 上，这可用作 `build-essential`。在其他平台上，也有类似的命令。 -- Rust nightly 工具链（[指南][1]） - - 编译源代码 +- [Rustup][1]。仓库通过 `rust-toolchain.toml` 指定所需的 nightly 工具链。 - Protobuf（[指南][2]） - 编译 proto 文件 - 请注意，版本需要 >= 3.15。你可以使用 `protoc --version` 检查它。 -- 机器：建议内存在 16GB 以上 或者 使用[mold](https://github.com/rui314/mold)工具以降低链接时的内存使用。 +- 机器：建议 16GB 以上内存。内存较小时，可使用 [mold](https://github.com/rui314/mold) 降低链接阶段的内存占用。 [1]: [2]: diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/how-to/how-to-write-sdk.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/how-to/how-to-write-sdk.md index 28ff757763..31e2d45881 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/how-to/how-to-write-sdk.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/how-to/how-to-write-sdk.md @@ -1,21 +1,17 @@ --- keywords: [gRPC SDK, GreptimeDatabase, GreptimeRequest, GreptimeResponse, 插入请求] -description: 介绍如何为 GreptimeDB 开发一个 gRPC SDK，包括 GreptimeDatabase 服务的定义、GreptimeRequest 和 GreptimeResponse 的结构。 +description: 介绍 GreptimeDB gRPC 写入 SDK 需要遵守的协议契约和错误处理要求。 --- # 如何为 GreptimeDB 开发一个 gRPC SDK -GreptimeDB 的 gRPC SDK 只需要处理写请求即可。读请求是标准 SQL 或 PromQL，可以由任何 JDBC 客户端或 Prometheus -客户端处理。这也是为什么所有的 GreptimeDB SDK 都命名为 "`greptimedb-ingester-`"。请确保你的 GreptimeDB SDK -遵循相同的命名约定。 +GreptimeDB 的公开 gRPC SDK 是写入客户端。查询通常通过标准 SQL 或 PromQL 客户端完成。除非有单独需求，新 SDK 应聚焦写入和删除，并遵循 `greptimedb-ingester-` 命名约定。面向用户的 API 参见 [gRPC SDK 概述](/user-guide/ingest-data/for-iot/grpc-sdks/overview.md)。 ## `GreptimeDatabase` 服务 -GreptimeDB 自定义了一个 gRPC 服务：`GreptimeDatabase` -。你只需要实现这个服务即可。你可以在[这里](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto) -找到它的 Protobuf 定义。 +从 [`GreptimeDatabase` Protobuf 定义](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto)生成客户端 stub，不要在 SDK 中手写一份 message 或 service 定义。 -`GreptimeDatabase` 有 2 个 RPC 方法： +该 service 提供一个 unary method 和一个 client-streaming method： ```protobuf service GreptimeDatabase { @@ -25,13 +21,9 @@ service GreptimeDatabase { } ``` -`Handle` 方法是一个 unary 调用：当 GreptimeDB 服务接收到一个 `GreptimeRequest` 请求后，它立刻处理该请求并返回一个相应的 -`GreptimeResponse`。 +`Handle` 对一个请求返回一个响应，是 SDK insert 和 delete API 通常使用的方法。 -`HandleRequests` 方法则是一个 "[Client Streaming RPC][3]" 方式的调用。 -它可以接受一个连续的 `GreptimeRequest` 请求流，持续地发给 GreptimeDB 服务。 -GreptimeDB 服务会在收到流中的每个请求时立刻进行处理，并最终（流结束时）返回一个总结性的 `GreptimeResponse`。 -通过 `HandleRequests`，我们可以获得一个非常高的请求吞吐量。 +`HandleRequests` 是 [client-streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc)。客户端关闭请求流后，服务端才返回累计响应。SDK 如果暴露 streaming API，需要明确这个确认边界，并将一个 stream 绑定到一个 endpoint。 ### `GreptimeRequest` @@ -51,11 +43,13 @@ message GreptimeRequest { } ``` -`RequestHeader` 是必需，它包含了一些上下文，鉴权和其他信息。"oneof" 的字段包含了发往 GreptimeDB 服务的请求。 +客户端需要在 `RequestHeader` 中填写服务端要求的 database context 和认证信息，并且只能设置一个 request variant。 -注意我们有两种类型的插入请求，一种是以 "列" 的形式（`InsertRequests`），另一种是以 "行" 的形式（`RowInsertRequests` -）。通常我们建议使用 "行" 的形式，因为它对于表的插入更自然，更容易使用。但是，如果需要一次插入大量列，或者有大量的 "null" -值需要插入，那么最好使用 "列" 的形式。 +该 message 还包含供内部调用者使用的 query 和 DDL variant。公开 ingester API 不应暴露它们，因为 `GreptimeDatabase` 不返回 query result stream。 + +GreptimeDB 同时接受行式 `RowInsertRequests` 和列式 `InsertRequests`。公开写入 API 默认使用行式请求。面向列的客户端可以使用列式请求，但转换过程中必须保持列长度一致，并保留 null、时间戳精度、数据类型和列 semantic type。 + +删除同样区分行式和列式。SDK 只应暴露能够在不丢失类型信息的前提下完成映射的形式。 ### `GreptimeResponse` @@ -68,6 +62,18 @@ message GreptimeResponse { } ``` -`ResponseHeader` 包含了返回值的状态码，以及错误信息（如果有的话）。"oneof" 的字段目前只有 "affected rows"。 +成功响应包含 success header 和 `affected_rows`。该值表示服务端确认的行数；关闭请求流时返回的是累计值。 + +请求失败通过 gRPC status 返回。Trailing metadata 中的 `x-greptime-err-code` 在存在时提供 GreptimeDB error code，错误文本则由 status message 携带。SDK 应保留 gRPC status 并暴露 GreptimeDB error code，不能用一个通用 SDK error 将其覆盖。 + +## 重试与交付语义 + +重试次数必须有上限，并且对调用者可见。只有错误被标记为 retryable 且 deadline 仍允许时，才能重试 unary request。Cancellation 和 deadline expiration 不应重试。 + +响应丢失不代表服务端拒绝了写入。除非调用者的数据模型保证操作幂等，重试这类请求可能插入重复行。SDK 需要说明这一点，并在交付结果不确定时返回最终错误。 + +不要自动重试只发送了一部分的 `HandleRequests` stream。即使客户端尚未收到累计响应，服务端也可能已经接受了部分请求。此时应关闭失败的 stream，并将不确定状态返回给调用者。 + +Arrow Flight bulk ingestion 与 `GreptimeDatabase` RPC 应使用不同的 API。它的 batching 和 partial acceptance 需要独立的契约。 -GreptimeDB 现在有很多 SDK，你可以参考[这里](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=)获取一些示例。 +可以参考现有 [GreptimeDB ingester 仓库](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=)的公开 API 约定，但线上行为应以当前 Protobuf 定义和服务端契约为准。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/admin-api.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/admin-api.md index 3ac3d44f3b..bd28206c6f 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/admin-api.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/admin-api.md @@ -1,20 +1,25 @@ --- -keywords: [Admin API, 健康检查, leader 查询, 心跳检测, 维护模式] -description: 介绍 Metasrv 的 Admin API，包括健康检查、leader 查询、心跳检测、维护模式和 Procedure Manager 控制等功能。 +keywords: [Admin API, 健康检查, leader 查询, 心跳检测, 维护模式, 恢复模式, table id sequence] +description: 介绍 Metasrv 用于状态检查、集群控制和元数据恢复的 Admin API。 --- # Admin API -Admin 提供了一种简单的方法来查看和管理集群信息，包括 metasrv 健康检测、metasrv leader 查询、数据节点心跳检测、维护模式和 Procedure Manager 控制。 +:::tip +本页所有 Admin API 都监听 Metasrv 的 `HTTP_PORT`，默认值为 `4000`。 +::: -Admin API 是一个 HTTP 服务，提供一组可以通过 HTTP 请求调用的 RESTful API。Admin API 简单、用户友好且安全。 +Admin API 通过 HTTP 提供 Metasrv 状态、集群控制和元数据恢复操作。该 API 不提供认证，且部分端点会改变集群行为或元数据分配，部署时必须通过网络策略保护 HTTP 端口。 本页介绍以下 API： - /health - /leader - /heartbeat +- /node-lease - /maintenance - /procedure-manager +- /recovery +- /sequence/table 所有这些 API 都在父资源 `/admin` 下。 @@ -22,7 +27,7 @@ Admin API 是一个 HTTP 服务，提供一组可以通过 HTTP 请求调用的 ## /health HTTP 端点 -`/health` 端点接受 GET HTTP 请求，你可以使用此端点检查你的 metasrv 实例的健康状况。 +`/health` 端点接受 GET 请求。HTTP 服务运行时返回 `OK`，但不会检查当前 Metasrv 是否为 leader，也不会检查外部依赖是否可用。 ### 定义 @@ -116,9 +121,17 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100' ] ``` +## /node-lease HTTP 端点 + +`/node-lease` 返回 Metasrv 当前记录的 Datanode lease，可用于判断 Metasrv 是否仍将某个 Datanode 视为存活。 + +```bash +curl -X GET http://localhost:4000/admin/node-lease +``` + ## /maintenance HTTP 端点 -集群维护模式是 GreptimeDB 中的一项安全功能，它可以临时禁用自动集群管理操作。此模式在集群升级、计划停机以及任何可能暂时影响集群稳定性的操作期间特别有用。有关更多详细信息，请参阅[集群维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。 +维护模式在升级、计划停机等操作期间临时禁用自动集群管理。它对集群的具体影响参见[集群维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。 `/maintenance` 端点支持以下 HTTP 请求： @@ -151,3 +164,39 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100' "status": "running" } ``` + +## /recovery HTTP 端点 + +Recovery mode 控制手动修改 table ID sequence 等元数据修复端点。它只用于恢复工作，不用于常规维护。 + +- `GET /admin/recovery/status`：查询 recovery mode 是否开启。 +- `POST /admin/recovery/enable`：开启 recovery mode。 +- `POST /admin/recovery/disable`：关闭 recovery mode。 + +响应体格式如下： + +```json +{ + "enabled": true +} +``` + +修复完成后应关闭 recovery mode。如果只是计划暂停自动集群操作，应使用[维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。 + +## /sequence/table HTTP 端点 + +这些端点用于检查或修复 table ID sequence： + +- `GET /admin/sequence/table/next-id`：返回下一个 table ID，但不执行分配。 +- `POST /admin/sequence/table/set-next-id`：推进下一个 table ID。 + +设置 sequence 前必须开启 recovery mode。新值必须大于当前值，不能通过该 API 回退 sequence。Recovery mode 只是该 API 的前置条件，不能阻止 DDL。执行该操作时，必须遵循[管理 Table ID Sequence](/user-guide/deployments-administration/maintenance/sequence-management.md)中的完整集群操作流程。 + +```bash +curl -X POST \ + -H 'Content-Type: application/json' \ + -d '{"next_table_id": 2048}' \ + http://localhost:4000/admin/sequence/table/set-next-id +``` + +该操作会影响后续新表分配到的 ID。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/overview.md index 2e3c525baa..e9f6144f26 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/overview.md @@ -1,162 +1,101 @@ --- -keywords: [Metasrv, 元数据存储, 请求路由, 负载均衡, 高可用性] -description: 介绍 Metasrv 的功能、架构和与前端的交互方式。 +keywords: [Metasrv, 元数据, 路由, Leader 选举, Procedure, 心跳] +description: 介绍 Metasrv 提供的元数据及集群协调机制。 --- # Metasrv -![meta](/meta.png) - ## Metasrv 包含什么 -- 存储元数据（Catalog, Schema, Table, Region 等） -- 请求路由器。它告诉前端在哪里写入和读取数据。 -- 数据节点的负载均衡，决定谁应该处理新的表创建请求，更准确地说，它做出资源分配决策。 -- 选举与高可用性，GreptimeDB 设计为 Leader-Follower 架构，只有 leader 节点可以写入，而 follower 节点可以读取，follower 节点的数量通常 >= 1，当 leader 不可用时，follower 节点需要能够快速切换为 leader。 -- 统计数据收集（通过每个节点上的心跳报告），如 CPU、负载、节点上的表数量、平均/峰值数据读写大小等，可用作分布式调度的基础。 +Metasrv 是 GreptimeDB 分布式集群中的元数据和协调服务，不参与数据读写链路。它主要负责： + +- 存储 Catalog、Schema、Table、Region、路由和节点元数据； +- 为新 Region 选择 Datanode，并维护表路由； +- 选举一个 Metasrv leader 负责协调元数据变更； +- 通过可恢复的 Procedure 执行 DDL、Region 迁移、故障转移和重分区； +- 通过心跳维护节点租约和 Region 统计信息； +- 元数据变更时向 Frontend、Datanode 和 Flownode 广播缓存失效； +- 向 Datanode 下发 Region 生命周期指令。 ## 前端如何与 Metasrv 交互 -首先，请求路由器中的路由表结构如下（注意这只是逻辑结构，实际存储结构可能不同，例如端点可能有字典压缩）。 - -```txt - table_A - table_name - table_schema // 用于物理计划 - regions - region_1 - mutate_endpoint - select_endpoint_1, select_endpoint_2 - region_2 - mutate_endpoint - select_endpoint_1, select_endpoint_2, select_endpoint_3 - region_xxx - table_B - ... -``` +Frontend 从 Metasrv 获取表元数据和 Region 路由，并缓存在本地。修改元数据的语句会发送给 Metasrv leader；普通读写则使用缓存的路由直接访问 Datanode。 -### 创建表 +控制链路和数据链路相互分离： -1. 前端发送 `CREATE TABLE` 请求到 Metasrv。 -2. 根据请求中包含的分区规则规划 Region 数量。 -3. 检查数据节点可用资源的全局视图（通过心跳收集）并为每个 Region 分配一个节点。 -4. 前端创建表并在成功创建后将 `Schema` 存储到 Metasrv。 +```text +Frontend + |-- 元数据查询和 DDL -------------------> Metasrv leader + `-- Region 读写 ------------------------> Datanode -### `Insert` +Metasrv leader + |-- Region 生命周期指令 ----------------> Datanode + `-- 缓存失效 --------------------------> Frontend / Datanode / Flownode -1. 前端从 Metasrv 获取指定表的路由。注意，最小的路由单元是表的路由（多个 Region），即包含该表所有 Region 的地址。 -2. 最佳实践是前端首先从本地缓存中获取路由并将请求转发到数据节点。如果路由不再有效，则数据节点有义务返回 `Invalid Route` 错误，前端重新从 Metasrv 获取最新数据并更新其缓存。路由信息不经常变化，因此，前端使用惰性策略维护缓存是足够的。 -3. 前端处理可能包含多个表和多个 Region 的一批写入，因此前端需要根据“路由表”拆分用户请求。 +Datanode + `-- 心跳、租约续期和 Region 统计信息 ----> Metasrv leader +``` -### `Select` +在稳定状态下，表路由为每个 Region 记录一个 leader peer 和零个或多个 follower peer。Leader 是写入目标；支持只读副本的部署可以把读取路由到 follower： -1. 与 `Insert` 类似，前端首先从本地缓存中获取路由表。 -2. 与 `Insert` 不同，对于 `Select`，前端需要从路由表中提取只读节点（follower），然后根据优先级将请求分发到 leader 或 follower 节点。 -3. 前端的分布式查询引擎根据路由信息分发多个子查询任务并聚合查询结果。 +```text +Table route + |-- Region 0 + | |-- leader -> Datanode A + | `-- followers -> Datanode B, Datanode C + `-- Region 1 + `-- leader -> Datanode D +``` -## Metasrv 架构 +Region 迁移或故障转移会改变 peer 角色，并可能使 Region 暂时没有 leader。Frontend 刷新缓存路由后，再把后续读写发送给当前 peer。 -![metasrv-architecture](/metasrv-architecture.png) +### 创建表 -## 分布式共识 +1. Frontend 向 Metasrv leader 提交 DDL 请求。 +2. Metasrv 根据分区规则确定 Region，并[为每个 Region 选择 Datanode](/contributor-guide/metasrv/selector.md)。 +3. 持久化的 Procedure 创建 Region，并写入表元数据和路由。发生 leader 切换后，Procedure 可以从已保存的状态继续执行。 +4. 元数据提交后，Metasrv 通知 Frontend 刷新相关缓存。 -如你所见，Metasrv 依赖于分布式共识，因为： +### `Insert` -1. 首先，Metasrv 必须选举一个 leader，数据节点只向 leader 发送心跳，我们只使用单个 Metasrv 节点接收心跳，这使得基于全局信息进行一些计算或调度变得容易且快速。至于数据节点如何连接到 leader，这由 MetaClient 决定（使用重定向，心跳请求变为 gRPC 流，使用重定向比转发更不容易出错），这对数据节点是透明的。 -2. 其次，Metasrv 必须为数据节点提供选举 API，以选举“写入”和“只读”节点，并帮助数据节点实现高可用性。 -3. 最后，`Metadata`、`Schema` 和其他数据必须在 Metasrv 上可靠且一致地存储。因此，基于共识的算法是存储它们的理想方法。 +Frontend 解析表路由，按照分区规则拆分数据行，再把各 Region 的写入发送到对应 Datanode。路由发生变化时，相关缓存会失效，Frontend 随后从 Metasrv 重新获取元数据。 -对于 Metasrv 的第一个版本，我们选择 Etcd 作为共识算法组件（Metasrv 设计时考虑适应不同的实现，甚至创建一个新的轮子），原因如下： +### `Select` -1. Etcd 提供了我们需要的 API，例如 `Watch`、`Election`、`KV` 等。 -2. 我们只执行两个分布式共识任务：选举（使用 `Watch` 机制）和存储（少量元数据），这两者都不需要我们定制自己的状态机，也不需要基于 raft 定制自己的状态机；少量数据也不需要多 raft 组支持。 -3. Metasrv 的初始版本使用 Etcd，使我们能够专注于 Metasrv 的功能，而不需要在分布式共识算法上花费太多精力，这提高了系统设计（避免与共识算法耦合）并有助于初期的快速开发，同时通过良好的架构设计，未来可以轻松接入优秀的共识算法实现。 +Frontend 在查询规划期间使用表和 Region 元数据。分区列上的谓词用于裁剪 Region，分布式查询引擎再把任务发送给持有这些 Region 的 Datanode。参见[分布式查询](../frontend/distributed-querying.md)。 -## 心跳管理 +## Metasrv 架构 -数据节点与 Metasrv 之间的主要通信方式是心跳请求/响应流，我们希望这是唯一的通信方式。这个想法受到 [TiKV PD](https://github.com/tikv/pd) 设计的启发，我们在 [RheaKV](https://github.com/sofastack/sofa-jraft/tree/master/jraft-rheakv/rheakv-pd) 中有实际经验。请求发送其状态，而 Metasrv 通过心跳响应发送不同的调度指令。 +主要协调路径如下： + +```text +Leader election + | + v +Metasrv leader +├─ DDL manager -> Procedure manager +├─ Selector -> 新 Region 的放置 +├─ Heartbeat handler chain -> 租约和 Region 统计信息 +├─ Region supervisor -> Region 迁移 Procedure +├─ Mailbox -> 缓存失效和 Region 指令 +└─ Metadata managers -> KV backend +``` -心跳可能携带以下数据，但这不是最终设计，我们仍在讨论和探索究竟应该收集哪些数据。 +这些机制共享元数据，但故障边界不同。进程重启可以丢弃缓存和 leader 本地状态；恢复所需的元数据和 Procedure 状态必须持久化。 -``` -service Heartbeat { - // 心跳，心跳可能有很多内容，例如： - // 1. 要注册到 Metasrv 并可被其他节点发现的元数据。 - // 2. 一些性能指标，例如负载、CPU 使用率等。 - // 3. 正在执行的计算任务数量。 - rpc Heartbeat(stream HeartbeatRequest) returns (stream HeartbeatResponse) {} -} - -message HeartbeatRequest { - RequestHeader header = 1; - - // 自身节点 - Peer peer = 2; - // leader 节点 - bool is_leader = 3; - // 实际报告时间间隔 - TimeInterval report_interval = 4; - // 节点状态 - NodeStat node_stat = 5; - // 此节点中的 Region 状态 - repeated RegionStat region_stats = 6; - // follower 节点和状态，在 follower 节点上为空 - repeated ReplicaStat replica_stats = 7; -} - -message NodeStat { - // 此期间的读取容量单位 - uint64 rcus = 1; - // 此期间的写入容量单位 - uint64 wcus = 2; - // 此节点中的表数量 - uint64 table_num = 3; - // 此节点中的 Region 数量 - uint64 region_num = 4; - - double cpu_usage = 5; - double load = 6; - // 节点中的读取磁盘 I/O - double read_io_rate = 7; - // 节点中的写入磁盘 I/O - double write_io_rate = 8; - - // 其他 - map attrs = 100; -} - -message RegionStat { - uint64 region_id = 1; - TableName table_name = 2; - // 此期间的读取容量单位 - uint64 rcus = 3; - // 此期间的写入容量单位 - uint64 wcus = 4; - // 近似 Region 大小 - uint64 approximate_size = 5; - // 近似行数 - uint64 approximate_rows = 6; - - // 其他 - map attrs = 100; -} - -message ReplicaStat { - Peer peer = 1; - bool in_sync = 2; - bool is_learner = 3; -} -``` +## 分布式共识 -## Central Nervous System (CNS) +Metasrv 将 leader 选举与元数据存储分开。只有选出的 Metasrv leader 执行协调和元数据变更操作，其他 Metasrv 节点会把 client 引导到当前 leader。 -我们要构建一个算法系统，该系统依赖于每个节点的实时和历史心跳数据，应该做出一些更智能的调度决策并将其发送到 Metasrv 的 Autoadmin 单元，该单元分发调度决策，由数据节点本身或更可能由 PaaS 平台执行。 +Key-value backend 保存表元数据、路由、Procedure 状态以及其他必须跨 leader 切换保留的信息。Metasrv 不使用这套选举为 Datanode Region 创建读写副本；Region 可用性由心跳、Region 故障检测和故障转移 Procedure 管理。 -## 工作负载抽象 +## 心跳管理 -工作负载抽象的级别决定了 Metasrv 生成的调度策略（如资源分配）的效率和质量。 +Datanode 与 Metasrv leader 保持心跳流。心跳请求报告节点身份、租约、Region 统计信息以及放置和监控所需的其他状态；响应则携带 Region 生命周期指令、缓存失效等控制消息。 -DynamoDB 定义了 RCUs 和 WCUs（读取容量单位/写入容量单位），解释说 RCU 是一个 4KB 数据的读取请求，WCU 是一个 1KB 数据的写入请求。当使用 RCU 和 WCU 描述工作负载时，更容易实现性能可测量性并获得更有信息量的资源预分配，因为我们可以将不同的硬件能力抽象为 RCU 和 WCU 的组合。 +心跳驱动两套相互独立的机制，调整心跳周期会同时影响两者： -然而，GreptimeDB 面临比 DynamoDB 更复杂的情况，特别是 RCU 不适合描述需要大量计算的 GreptimeDB 读取工作负载。我们正在努力解决这个问题。 +- **节点租约**：keep-lease handler 为发送心跳的 Datanode 续期。Selector 和 `/node-lease` 端点据此判断 Datanode 是否仍然存活。 +- **Region 故障检测**：Region supervisor 为每个 Region 维护一个基于心跳到达间隔的 Phi Accrual 检测器，其判定与租约是否过期无关。 +只有开启 Region 故障转移时，故障判定才会提交故障转移迁移。该功能默认关闭，并且要求使用 remote WAL，除非显式允许在本地 WAL 上执行。维护模式同样会抑制故障转移。前置条件和开启方式参见 [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/selector.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/selector.md index 9048e0daa4..29b521212b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/selector.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/metasrv/selector.md @@ -7,11 +7,7 @@ description: 介绍 Metasrv 中的 Selector，包括其类型和配置方法。 ## 介绍 -什么是 `Selector`？顾名思义，它允许用户从给定的 `namespace` 和 `context` 中选择 `Item`s。有一个相关的 `trait`，也叫做 `Selector`，其定义可以在[这里][0]找到。 - -[0]: https://github.com/GreptimeTeam/greptimedb/blob/main/src/meta-srv/src/selector.rs - -在 `Metasrv` 中存在一个特定的场景。当 `Frontend` 向 `Metasrv` 发送建表请求时，`Metasrv` 会创建一个路由表（表的创建细节不在这里赘述）。在创建路由表时，`Metasrv` 需要选择适当的 `Datanode`s，这时候就需要用到 `Selector`。 +建表时，Metasrv 使用 `Selector` 为各 Region 选择 Datanode。Selector 根据当前节点租约进行选择；部分实现还会使用 Region 统计信息。 @@ -19,22 +15,22 @@ description: 介绍 Metasrv 中的 Selector，包括其类型和配置方法。 `Metasrv` 目前提供以下几种类型的 `Selectors`: -### LeasebasedSelector +### LeaseBasedSelector -`LeasebasedSelector` 从所有可用的（也就是在租约期间内）`Datanode` 中随机选择，其特点是简单和快速。 +`LeaseBasedSelector` 从租约有效的 Datanode 中随机选择。 ### LoadBasedSelector `LoadBasedSelector` 按照负载来选择，负载值则由每个 `Datanode` 上的 region 数量决定，较少的 region 表示较低的负载，`LoadBasedSelector` 优先选择低负载的 `Datanode`。 ### RoundRobinSelector [默认选项] -`RoundRobinSelector` 以轮询的方式选择 `Datanode`。在大多数情况下，这是默认的且推荐的选项。如果你不确定选择哪个，通常它就是正确的选择。 +`RoundRobinSelector` 以轮询方式选择 Datanode，是默认选项，也适用于大多数部署。 ## 配置 您可以在启动 `Metasrv` 服务时通过名称配置 `Selector`。 -- LeasebasedSelector: `lease_based` 或 `LeaseBased` +- LeaseBasedSelector: `lease_based` 或 `LeaseBased` - LoadBasedSelector: `load_based` 或 `LoadBased` - RoundRobinSelector: `round_robin` 或 `RoundRobin` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/overview.md index 9f82a72171..badd6ad62d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/overview.md @@ -5,9 +5,7 @@ description: 介绍 GreptimeDB 的架构、关键概念和工作原理，包括 # 贡献者指南 -DeepWiki 对 GreptimeDB 的架构和实现进行了详细且清晰的描述，强烈推荐阅读： - -[https://deepwiki.com/GreptimeTeam/greptimedb](https://deepwiki.com/GreptimeTeam/greptimedb) +本指南面向 GreptimeDB 贡献者，介绍理解内部实现所需的设计机制。从源码构建和运行参见[快速开始](/contributor-guide/getting-started.md)。提交要求（CLA、license header、代码格式，以及 PR 必须通过的检查）以源码仓库的 [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md) 为准。 ## 架构 @@ -18,7 +16,13 @@ DeepWiki 对 GreptimeDB 的架构和实现进行了详细且清晰的描述， - [frontend][1] - [datanode][2] - [metasrv][3] +- [flownode][4] [1]: /contributor-guide/frontend/overview.md [2]: /contributor-guide/datanode/overview.md [3]: /contributor-guide/metasrv/overview.md +[4]: /contributor-guide/flownode/overview.md + +## 补充参考 + +[DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb) 提供了自动生成的 GreptimeDB 源码导读，可用于了解不熟悉的模块。它属于辅助资料；涉及具体版本的行为时，仍应以对应源码为准。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/integration-test.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/integration-test.md index 63ffa56bc0..767109f773 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/integration-test.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/integration-test.md @@ -7,8 +7,14 @@ description: 介绍 GreptimeDB 的集成测试，包括测试范围和如何运 ## 介绍 -集成测试使用 Rust 测试工具（`#[test]`）编写，与单元测试不同，它们被单独放置在 -[这里](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration)。 -它涵盖了涉及多个组件的场景，其中一个典型案例是与 HTTP/gRPC 相关的功能。你可以查看 -其[文档](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md)以获取更多信息。 +集成测试覆盖跨 crate 或服务边界的行为，例如 HTTP 和 gRPC 处理、分布式组件或外部存储。测试使用 Rust test harness，位于 [`tests-integration`](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration) package。 +运行命令如下： + +```shell +cargo nextest run -p tests-integration +``` + +部分 case 依赖外部服务的环境变量或 fixture。运行前按照 package 的[准备说明](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md)配置环境。 + +只有 crate 级测试或 Sqlness case 无法覆盖所需边界时才使用集成测试。隔离的逻辑仍放在单元测试中，便于快速复现失败。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/overview.md index 81b6defa45..f38b625082 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/overview.md @@ -5,5 +5,12 @@ description: GreptimeDB 的测试 # 测试 -我们的团队进行了大量测试，以确保 GreptimeDB 的行为。本章将介绍几种用于测试 GreptimeDB 的重要方法，以及如何使用它们。 +选择能够覆盖本次改动的最小测试范围： +| 测试类型 | 适用场景 | 常用命令 | +| --- | --- | --- | +| [单元测试](unit-test.md) | 单个 crate 或组件内的逻辑 | `cargo nextest run -p ` | +| [Sqlness 测试](sqlness-test.md) | SQL、协议、planner、执行和端到端回归 | `cargo sqlness bare -t ` | +| [集成测试](integration-test.md) | 跨组件或依赖外部服务的行为 | `cargo nextest run -p tests-integration` | + +需要运行完整 Rust workspace 测试时使用 `make test`。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/sqlness-test.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/sqlness-test.md index d29b69e835..06503a434c 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/sqlness-test.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/sqlness-test.md @@ -7,36 +7,34 @@ description: 介绍 GreptimeDB 的 Sqlness 测试，包括测试文件类型、 ## 介绍 -SQL 是 `GreptimeDB` 的一个重要用户接口。我们为它提供了一个单独的测试套件（名为 `sqlness`）。 +Sqlness 是 GreptimeDB 针对 SQL 和协议行为的端到端回归测试。每个 case 向运行中的 GreptimeDB 发送语句，并将输出与仓库中的结果文件比较。 ## Sqlness 手册 ### 测试文件 -Sqlness 有两种类型的文件 +每个 case 使用两类文件： - `.sql`：测试输入，仅包含 SQL - `.result`：预期的测试输出，包含 SQL 和其结果 -`.result` 文件是预期的执行输出。如果 `.result` 文件发生变化，意味着测试结果不同，测试可能失败。你应该检查变更日志来解决问题。 - -你只需要在 `.sql` 文件中编写测试 SQL，然后运行测试。 +在 `.sql` 文件中编写输入，运行测试后生成或更新 `.result`。必须检查每一处结果差异，只有行为变化符合预期时才能接受。 ### 组织测试案例 -输入案例的根目录是 `tests/cases`。它包含几个子目录，代表不同的测试模式。例如，`standalone/` 包含所有在 `greptimedb standalone start` 模式下运行的测试。 +输入 case 位于 `tests/cases`。第一级目录选择运行环境，例如 `standalone/` 表示使用单机 GreptimeDB。 -在第一级子目录下（例如 `cases/standalone`），你可以随意组织你的测试案例。Sqlness 会递归地遍历每个文件并运行它们。 +在环境目录内，新 case 应与它覆盖的功能放在一起。Sqlness 会递归发现 case 文件。 ## 运行测试 -与其他测试不同，这个测试工具是以二进制目标形式存在的。你可以用以下命令运行它 +运行命令如下： ```shell -cargo run --bin sqlness-runner bare +cargo sqlness bare ``` -它会自动完成以下步骤：编译 `GreptimeDB`，启动它，抓取测试并将其发送到服务器，然后收集和比较结果。你只需要检查是否有 `.result` 文件发生变化。如果没有，恭喜你，测试通过了 🥳！ +该命令会构建并启动 GreptimeDB、执行选中的 case，再比较输出。`.result` 发生变化只是待审查的结果，不代表新输出一定正确。 ### 运行特定测试 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/unit-test.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/unit-test.md index 79c73775b4..b63b29df94 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/unit-test.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/tests/unit-test.md @@ -7,22 +7,26 @@ description: 介绍 GreptimeDB 的单元测试，包括如何编写、运行和 ## 介绍 -单元测试嵌入在代码库中，通常放置在被测试逻辑的旁边。它们使用 Rust 的 `#[test]` 属性编写，并可以使用 `cargo nextest run` 运行。 +单元测试通常放在被测逻辑旁边，使用 Rust 的 `#[test]` 属性编写。GreptimeDB 主要使用 [`cargo-nextest`](https://nexte.st/) 运行 Rust 测试。 -GreptimeDB 代码库不支持默认的 `cargo` 测试运行器。推荐使用 [`nextest`](https://nexte.st/)。你可以通过以下命令安装它： +安装命令如下： ```shell cargo install cargo-nextest --locked ``` -然后运行测试（这里 `--workspace` 不是必须的） +开发时先运行本次修改的 package： ```shell -cargo nextest run +cargo nextest run -p ``` -注意，如果你的 Rust 是通过 `rustup` 安装的，请确保使用 `cargo` 安装 `nextest`，而不是像 `homebrew` 这样的包管理器，否则会弄乱你的本地环境。 +可以继续使用测试名称或 nextest filter 缩小范围。影响范围较广的改动在提交前运行完整 workspace 测试： + +```shell +make test +``` ## 覆盖率 -我们的持续集成（CI）作业有一个“覆盖率检查”步骤。它会报告有多少代码被单元测试覆盖。请在你的补丁中添加必要的单元测试。 +CI 会报告单元测试覆盖率。测试应覆盖本次改变的行为和可能回归的失败路径，而不是只追求覆盖率数字。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/data-persistence-indexing.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/data-persistence-indexing.md index e0ef9e8e54..e0ce16721d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/data-persistence-indexing.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/data-persistence-indexing.md @@ -5,19 +5,23 @@ description: 介绍了 GreptimeDB 的数据持久化和索引机制，包括 SST # 数据持久化与索引 -与所有类似 LSMT 的存储引擎一样，MemTables 中的数据被持久化到耐久性存储，例如本地磁盘文件系统或对象存储服务。GreptimeDB 采用 [Apache Parquet][1] 作为其持久文件格式。 +与其他 LSM-tree 存储引擎类似，GreptimeDB 将 memtable 中的数据持久化到本地文件系统或对象存储，并使用 [Apache Parquet][1] 作为持久化文件格式。 ## SST 文件格式 Parquet 是一种提供快速数据查询的开源列式存储格式，已经被许多项目采用，例如 Delta Lake。 -Parquet 具有层次结构，类似于“行组 - 列-数据页”。Parquet 文件中的数据被水平分区为行组（row group），在其中相同列的所有值一起存储以形成数据页（data pages）。数据页是最小的存储单元。这种结构极大地提高了性能。 +Parquet 按 row group、column chunk 和 page 组织数据。每个 row group 为每一列保存一个 column chunk，每个 column chunk 再包含一个或多个 page。Page 是编码和压缩单元，读取指定列时则以 column chunk 为 I/O 单元。 首先，数据按列聚集，这使得文件扫描更加高效，特别是当查询只涉及少数列时，这在分析系统中非常常见。 -其次，相同列的数据往往是同质的（比如具备近似的值），这有助于在采用字典和 Run-Length Encoding（RLE）等技术进行压缩。 +其次，同一列中的值通常比较相似，有利于字典编码和 Run-Length Encoding（RLE）等压缩技术发挥作用。 -Parquet file format +下面这张来自 Apache Parquet 规范的图进一步展示了物理文件布局：column chunk 按 row group 写入，文件元数据及其长度则保存在 footer 中。 + +Apache Parquet 文件布局 + +*来源：Apache Parquet [FileLayout.gif](https://github.com/apache/parquet-format/blob/master/doc/images/FileLayout.gif)。Copyright 2014 The Apache Software Foundation，依据 [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) 使用。* ## 数据持久化 @@ -26,18 +30,18 @@ GreptimeDB 提供了 `region_engine.mito.global_write_buffer_size` 的配置项 ## SST 文件中的索引数据 -Apache Parquet 文件格式在列块和数据页的头部提供了内置的统计信息，用于剪枝和跳过。 +Parquet 在每个 column chunk 的元数据中保存 row group 级列统计信息，例如最小值、最大值和 null 数量。Page 元数据和可选的 column index 可以提供粒度更细的统计信息。 -Column chunk header +![查询 name 列时，Parquet 列统计信息排除了一个 row group，并将另一个保留为待读取对象。](/parquet-row-group-statistics.zh.svg) -例如，在上述 Parquet 文件中，如果你想要过滤 `name` 等于 `Emily` 的行，你可以轻松跳过行组 0，因为 `name` 字段的最大值是 `Charlie`。这些统计信息减少了 IO 操作。 +例如，查询 `name` 等于 `Emily` 的行时，可以跳过 row group 0，因为其中 `name` 的最大值是 `Charlie`，无需读取该 row group。 ## 索引文件 -对于每个 SST 文件，GreptimeDB 不但维护 SST 文件内部索引，还会单独生成一个文件用于存储针对该 SST 文件的索引结构。 +当一个 SST 存在已配置且适用的索引输出时，GreptimeDB 将这些索引写入与该 SST 关联的 Puffin 文件。没有适用索引的 SST 不需要生成 Puffin 文件。 -索引文件采用 [Puffin][3] 格式，这种格式具有较大的灵活性，能够存储更多的元数据，并支持更多的索引结构。 +Puffin 是索引 Blob 及其元数据的容器，使不同索引结构可以共用一个文件。 ![Puffin](/puffin.png) @@ -58,13 +62,13 @@ GreptimeDB 会将多种索引结构作为 Blob 存储在 Puffin 文件中，包 ![Inverted index searching](/inverted-index-searching.png) -例如，上述查询使用倒排索引来定位数据段，数据段满足条件：`job` 等于 `apiserver`，`handler` 符合正则匹配 `.*users` 及 `status` 符合正则匹配 `4..`，然后扫描这些数据段以产生满足所有条件的最终结果，从而显着减少 IO 操作的次数。 +上述查询使用倒排索引定位 `job` 等于 `apiserver`、`handler` 匹配 `.*users` 且 `status` 匹配 `4..` 的数据段。Mito 只扫描这些数据段，再应用剩余过滤条件。 ### 倒排索引格式 -![Inverted index format](/inverted-index-format.png) +![倒排索引 Blob 先保存各列索引，再保存 footer 元数据；每个列索引包含 null bitmap、posting bitmap 和 FST。](/inverted-index-blob-layout.zh.svg) -GreptimeDB 按列构建倒排索引，每个倒排索引包含一个 FST 和多个 Bitmap。 +GreptimeDB 按列构建倒排索引。每个列索引包含一个 null bitmap、多个 posting bitmap 和一个 FST。Blob footer 记录定位和解码各列索引所需的 offset、size 和元数据。 FST（Finite State Transducer）允许 GreptimeDB 以紧凑的格式存储列值到 Bitmap 位置的映射，并且提供了优秀的搜索性能和支持复杂搜索（例如正则表达式匹配）；Bitmap 则维护了数据段 ID 列表，每个位表示一个数据段。 @@ -80,7 +84,7 @@ GreptimeDB 把一个 SST 文件分割成多个索引数据段，每个数据段 ## 统一数据访问层：OpenDAL -GreptimeDB 使用 [OpenDAL][2] 提供统一的数据访问层，因此，存储引擎无需与不同的存储 API 交互，数据可以无缝迁移到基于云的存储，如 AWS S3。 +GreptimeDB 使用 [OpenDAL][2] 为本地文件系统和对象存储提供统一访问层。修改配置的存储 backend 不会迁移已有数据。 [1]: https://parquet.apache.org [2]: https://github.com/datafuselabs/opendal diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/memtable.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/memtable.md new file mode 100644 index 0000000000..3905a1aecd --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/memtable.md @@ -0,0 +1,100 @@ +--- +keywords: [memtable, Mito engine, 写缓冲, flush, 时间分区, BulkMemtable] +description: 介绍 Mito 如何在 memtable 中组织 Region 的可变数据，以及如何将这些数据写入 SST 文件。 +--- + +# Memtable 设计 + +Memtable 是 Mito 为每个 Region 维护的内存写缓冲。数据 flush 为 SST 文件前，读取可以先从 memtable 获取这些数据。Region version 确定 scan 可以读取的 memtable 和 SST 文件；配合 committed sequence 上限，scan 可以在写入和 flush 推进当前 version 时保持一致性。 + +## 写入和 flush 生命周期 + +对于使用 WAL 的常规写入，Mito 按以下顺序处理： + +```text +写请求 + | + v +追加 WAL -> mutable memtable -> 发布 committed sequence + | + freeze + v + immutable memtable -> 写入 SST -> manifest edit +``` + +Region worker 先分配 sequence number 和 WAL entry ID，再将 mutation 追加到[预写日志](wal.md)。如果追加失败，Mito 不会更新 memtable。Memtable 更新成功后，Mito 发布 committed sequence，新的读取随后可以看到这些数据。配置了 `skip_wal` 的 Region 会跳过 WAL，但 memtable 更新和可见性顺序不变。 + +Flush 在启动后台 SST 写入前，先冻结 mutable memtable 并安装一组新的 mutable memtable。后续写入可以继续进行，也不会修改已经冻结的数据。Flush 将 immutable memtable 写为 SST 文件，再持久化包含新文件、flushed WAL checkpoint 和 sequence checkpoint 的 manifest edit。只有 manifest edit 持久化成功后，Mito 才会从当前 Region version 中移除已 flush 的 memtable。Flush 失败时，immutable memtable 会保留，供后续任务重试。 + +## Region version 和时间分区 + +每个 Region 包含一个 mutable `TimePartitions` 容器，其中可以有多个 memtable： + +```text +Region version +├─ mutable TimePartitions +│ ├─ [t0, t1) -> memtable +│ └─ [t1, t2) -> memtable +├─ immutable memtables +└─ SST files +``` + +Mito 根据 time index 的值把每行数据路由到对应分区。分区使用左闭右开的时间范围，并按照固定时长对齐。该时长跟随 Region 的 compaction time window；在取得 compaction time window 前，Mito 使用一天作为初始值。乱序写入可能在最新分区之外创建更早的分区。 + +冻结 Region 时，Mito 会同时冻结所有 mutable 时间分区，把其中的 memtable 移入 immutable 列表，再创建新的 `TimePartitions` 容器。Flush 失败可能留下多代 immutable memtable，因此读取和后续 flush 不能假定列表中只有一个对象。 + +## Memtable 实现 + +Mito 根据 Region 的 SST format、primary key encoding 和 memtable 选项选择实现： + +```text +flat SST format（默认）或 sparse primary-key encoding -> BulkMemtable +memtable.type=bulk -> BulkMemtable，并强制 flat SST +primary_key SST + dense encoding（遗留） -> 遗留实现 +``` + +使用默认 engine 配置时，没有显式指定 SST format 的 Region 会使用 `flat`，因此通常走 `BulkMemtable` 路径，本页其余内容也以它为准。这些规则用于排除不兼容的组合：flat format 或 sparse primary key encoding 必须使用 `BulkMemtable`；显式选择 bulk 实现则会强制使用 flat format。 + +### BulkMemtable + +`BulkMemtable` 使用 flat Arrow 布局把写入保存为 part，而不是将数据行插入按时间序列组织的缓冲区： + +```text +BulkMemtable +├─ unordered_part +│ └─ 小批量 BulkPart +└─ parts + ├─ BulkPart (Arrow RecordBatch) + ├─ MultiBulkPart (未编码的 RecordBatch) + └─ EncodedBulkPart (内存中的 Parquet 数据) +``` + +小 part 先积累在 `unordered_part` 中，较大的 part 则直接进入 `parts`。后台 memtable compaction 对符合条件的 part 执行 merge sort，生成 `MultiBulkPart` 或编码为 `EncodedBulkPart`。Scan 利用 part 的统计信息裁剪 range；flush 可以将已编码的 range 写入 SST，无需再次解码和编码数据行。设计动机和性能数据见 [Scaling Time Series to Millions of Cardinalities: GreptimeDB's Flat Format](https://greptime.cn/blogs/2025-12-22-flat-format)。 + +### 遗留实现 + +使用遗留 `primary_key` SST format 且为 dense primary key encoding 的 Region 仍然使用 `TimeSeriesMemtable`，它按编码后的 primary key 对数据行分组，而不是保存 flat part。如果 Region 没有 primary key 列，同一个 builder 会创建 `SimpleBulkMemtable`。两者都是为已有表保留的兼容代码，`primary_key` format 退役后可能一并移除；新的工作应面向 bulk 和 flat 路径。 + +已经删除的 `partition_tree` memtable 不是第三种实现。Option parser 仍接受 `memtable.type=partition_tree` 以兼容旧配置，但不会恢复该实现。Region 最终使用 bulk 和 flat 路径。 + +## 读取快照 + +Scan 通过一次 `VersionControl` 快照同时取得 Region version 和 committed sequence，并先确定 version，再应用 sequence 上限。如果单独读取 sequence 后再获取 version，flush 或 compaction 可能在两次读取之间移除旧输入，使 scan 得到不完整的快照。 + +选定的 version 提供 mutable memtable、immutable memtable 和 SST 文件。Mito 先按照时间范围裁剪数据源，再使用 scan 的 projection、predicate 和 sequence range 从各 memtable 获取 range。Scan 将这些 range 与 SST range 合并，并在所有数据源上应用相同的排序、删除和 merge 语义。即使新的 Region version 已经移除某个 memtable，scan 持有的引用也会让该 memtable 存活到本次读取结束。 + +## 内存压力 + +每个 memtable 通过 engine 的 write-buffer manager 记录估算的堆内存分配量。冻结 memtable 后，这部分内存不再计入 mutable memory，但在所有引用释放前仍计入总用量。因此，mutable memory 只反映仍可接收写入的数据，总用量仍包含活跃 scan 保留的内存。 + +全局 write-buffer 达到限制后，worker 会选择 Region 执行 flush。如果内存用量持续超过配置限制，Mito 会阻塞写入，并在达到更高阈值后拒绝写入。可选的 Region 级限制会单独约束热点 Region，避免其阻塞无关 Region。定期任务、手动请求和 Region 生命周期操作也可以触发 flush。 + +## 修改约束 + +修改 memtable 代码时必须保持以下性质： + +- 对于使用 WAL 的 Region，先追加 WAL，再把数据安装到 memtable；只有安装成功后才能发布 committed sequence。 +- SST 文件和 manifest edit 持久化前，冻结的 memtable 必须保持可读，并能在 flush 失败后重试。 +- 从同一个 `VersionControl` 快照取得 Region version 和 committed sequence，不得先单独读取 sequence 再获取 version。 +- 保留 scan 和 flush 在 memtable range 与 SST range 之间执行统一删除、去重和 merge 所需的排序及元数据。 +- 通过 write-buffer manager 记录内存分配，并且只在底层内存不再可能被引用时释放计数。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/metric-engine.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/metric-engine.md index c0cbf681b9..57baecf400 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/metric-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/metric-engine.md @@ -7,9 +7,9 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述 ## 概述 -`Metric` 引擎是 GreptimeDB 的一个组件，属于存储引擎的一种实现，主要针对可观测 metrics 等存在大量小表的场景。 +`Metric` 引擎用于存储包含大量小型指标表的负载。 -它的主要特点是利用合成的物理宽表来存储大量的小表数据，实现相同列复用和元数据复用等效果，从而达到减少小表的存储开销以及提高列式压缩效率等目标。表这一概念在 `Metric` 引擎下变得更更加轻量。 +它将这些逻辑表映射到共享的物理宽表，使其复用列和元数据，从而降低每张表的存储开销并改善列式压缩。 ## 概念 @@ -18,7 +18,7 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述 ### 逻辑表 逻辑表，即用户定义的表。与普通的表都完全一样，逻辑表的定义包括表的名称、列的定义、索引的定义等。用户的查询、写入等操作都是基于逻辑表进行的。用户在使用过程中不需要关心逻辑表和普通表的区别。 -从实现层面来说，逻辑表是一个虚拟的表，它并不直接读写物理的数据，而是通过将读写请求映射成对应物理表的请求来实现数据的存储与查询。 +逻辑表是虚拟表，本身不直接存储数据。Metric 引擎将其读写请求映射为对应物理表的请求。 ### 物理表 物理表是真实存储数据的表，它拥有若干个由分区规则定义的物理 Region。 @@ -27,16 +27,14 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述 `Metric` 引擎的主要设计架构如下： -![Arch](/metric-engine-arch.png) +![多个逻辑表通过 Metric 引擎映射到由 Mito 管理的共享数据 Region 和元数据 Region。](/metric-engine-architecture.zh.svg) -在目前版本的实现中，`Metric` 引擎复用了 `Mito` 引擎来实现物理数据的存储及查询能力，并在此之上同时提供物理表与逻辑表的访问能力。 +`Metric` 引擎将物理存储和查询交给 `Mito` 引擎。每个物理 Region 组包含一个数据 Region 和一个元数据 Region：数据 Region 保存映射到该 Region 组的逻辑表数据，元数据 Region 保存逻辑表及逻辑列的映射。 -在分区方面，逻辑表拥有与物理表完全一致的分区规则及 Region 分布。这是非常自然的，因为逻辑表的数据直接存储在物理表中，所以分区规则也是一致的。 +关联到同一物理表的逻辑表使用相同的分区布局。写入时，Metric 引擎为每行数据记录逻辑表身份；读取时，它在扫描物理 Region 前增加逻辑表过滤条件。 -在路由元数据方面，逻辑表的路由地址为逻辑地址，即该逻辑表所对应的物理表是什么，而后通过该物理表进行二次路由取得真正的物理地址。这一间接路由方式能够显著减少 `Metric` 引擎的 Region 发生迁移调度时所需要修改的元数据数量。 +逻辑表的路由只保存所属物理表的 ID，再由物理表路由解析出持有 Region 的 Datanode。逻辑路由本身不记录 peer，因此迁移物理 Region 只需改写一条物理路由，而不必改写映射到它的每一条逻辑路由。 -在操作方面，`Metric` 引擎支持对逻辑表进行标准的 DML 操作（INSERT、DELETE、SELECT）。然而，对物理表的操作进行了有限的支持以防止误操作，例如禁止直接写入物理表等操作防止影响用户逻辑表的数据。总体上可以认为物理表是对用户只读的。 +逻辑表支持普通的 INSERT、DELETE 和 SELECT 操作。直接写入物理 Region 会绕过逻辑表映射，因此会被拒绝；物理表仍然可以查询。 -为了提升对大量表同时进行 DDL（Data Definition Language，数据操作语言）操作时性能，如 Prometheus Remote Write 冷启动时大量 metrics 带来的自动建表请求，以及前面提到的迁移物理 Region 时大量路由表的修改请求等，`Metric` 引擎引入了一些批量 DDL 操作。这些批量 DDL 操作能够将大量的 DDL 操作合并成一个请求，从而减少了元数据的查询及修改次数，提升了性能。 - -除了物理表的物理数据 Region 之外，`Metric` 引擎还额外为每一个物理数据 Region 创建了一个物理的元数据 Region，用于存储 `Metric` 引擎自身为了维护映射等状态所需要的一些元数据。这些元数据包括逻辑表与物理表的映射关系，逻辑列与物理列的映射关系等等。 +批量 DDL 用于减少大量逻辑表同时创建或更新时的元数据操作，例如 Prometheus Remote Write 自动建表或物理 Region 迁移。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/overview.md index 3f44a4543b..c7684eb165 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/overview.md @@ -7,22 +7,26 @@ description: 介绍了 Datanode 的主要职责和组件，包括 gRPC 服务、 ## Introduction -`Datanode` 主要的职责是为 GreptimeDB 存储数据，我们知道在 GreptimeDB 中一个 `table` 可以有一个或者多个 `Region`, -而 `Datanode` 的职责便是管理这些 `Region` 的读写。`Datanode` 不感知 `table`，可以认为它是一个 `region server`。 -所以 `Frontend` 和 `Metasrv` 按照 `Region` 粒度来操作 `Datanode`。 +Datanode 存储并处理 Region 数据。一张表可以包含多个 Region，但 Datanode 不负责表级路由。Frontend 按 Region 发送数据请求，Metasrv 则控制 Region 的放置和生命周期。 -![Datanode](/datanode.png) +这个边界使同一个 Region server 可以承载不同的存储引擎，而不向 Frontend 或 Metasrv 暴露引擎实现。 + +![Frontend 向 Datanode Region server 发送 Region 请求，Metasrv 通过 heartbeat task 与 Datanode 交换生命周期指令。Region server 使用本地 query engine，并将请求分发给 Mito、Metric 或 File Region engine。](/datanode-architecture.zh.svg) ## Components -一个 datanode 包含了 region server 所需的全部组件。这里列出了比较重要的部分： - -- 一个 gRPC 服务来提供对 `Region` 数据的读写，`Frontend` 便是使用这个服务来从 `Datanode` 读写数据。 -- 一个 HTTP 服务，可以通过它来获得当前节点的 metrics、配置信息等 -- `Heartbeat Task` 用来向 `Metasrv` 发送心跳，心跳在 GreptimeDB 的分布式架构中发挥着至关重要的作用， - 是分布式协调和调度的基础通信通道，心跳的上行消息中包含了重要信息比如 `Region` 的负载，如果 `Metasrv` 做出了调度 - 决定（比如 Region 转移），它会通过心跳的下行消息发送指令到 `Datanode` -- `Datanode` 不负责解析用户 SQL 或进行分布式规划，用户对一个或多个 `Table` 的查询请求会在 `Frontend` 中被转换为 - `Region` 查询请求，`Datanode` 负责用本地 query engine 执行这些 `Region` 查询计划 -- 一个 `Region Manager` 用来管理 `Datanode` 上的所有 `Region`s -- GreptimeDB 支持可插拔的多引擎架构，目前已有的 engine 包括 `File Engine` 和 `Mito Engine` +Datanode 包含以下主要组件： + +- Region server 记录已打开的 Region，并把读写和生命周期请求分发给该 Region 注册的 engine。 +- `Mito` 是主要的时序 Region engine。`Metric` 将多个逻辑指标 Region 映射到共享的 Mito Region，`File` 通过 Region 接口访问外部文件。 +- 本地 query engine 执行 Region 查询计划。它不解析客户端 SQL，也不进行集群级规划。 +- Heartbeat task 向 Metasrv 上报节点和 Region 状态，并接收 open、close、upgrade、downgrade 和迁移步骤等指令。 +- gRPC 承载发往 Datanode 的 Region 请求；HTTP 提供 metrics 和配置等节点诊断信息。 + +## Region 请求生命周期 + +Mito 写入到达 Region server 后，Region server 根据 Region 元数据选择 Mito。Mito 将 mutation 追加到 WAL，写入 memtable，并在之后把 memtable flush 为 SST 文件。Metric 写入会先补充逻辑表标识，再委托给对应的物理 Mito Region。 + +读取时，本地 query engine 在 Region engine 提供的 table provider 上执行 Region 计划。Mito scan 获取不可变的 Region version，读取相关 memtable 和 SST 文件，合并并去重数据，最后返回 Arrow record batch 流。 + +Region 所有权可以在不重启 Datanode 的情况下改变。Metasrv 通过心跳流下发生命周期指令；Region server 将指令应用到对应 engine，并在后续心跳中上报新的 Region role 和统计信息。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/python-scripts.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/python-scripts.md deleted file mode 100644 index 831278e80d..0000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/python-scripts.md +++ /dev/null @@ -1,30 +0,0 @@ ---- -keywords: [Python 脚本, 数据分析, CPython, RustPython] -description: 介绍了在 GreptimeDB 中使用 Python 脚本进行数据分析的两种后端实现：CPython 和嵌入式 RustPython 解释器。 ---- - -# Python 脚本 - -## 简介 - -Python 脚本是分析本地数据库中的数据的便捷方式， -通过将脚本直接在数据库内运行而不是从数据库拉取数据的方式，可以节省大量的数据传输时间。 -下图描述了 Python 脚本的工作原理。 -`RecordBatch`（基本上是表中的一列，带有类型和元数据）可以来自数据库中的任何地方， -而返回的 `RecordBatch` 可以用 Python 语法注释以指示其元数据，例如类型或空。 -脚本将尽其所能将返回的对象转换为 `RecordBatch`，无论它是 Python 列表、从参数计算出的 `RecordBatch` 还是常量（它被扩展到与输入参数相同的长度）。 - -![Python Coprocessor](/python-coprocessor.png) - -## 两种可选的后端 - -### CPython 后端 - -该后端由 [PyO3](https://pyo3.rs/v0.18.1/) 提供支持，可以使用您最喜欢的 Python 库（如 NumPy、Pandas 等），并允许 Conda 管理您的 Python 环境。 - -但是使用它也涉及一些复杂性。您必须设置正确的 Python 共享库，这可能有点棘手。一般来说，您只需要安装 `python-dev` 包。但是，如果您使用 Homebrew 在 macOS 上安装 Python，则必须创建一个适当的软链接到 `Library/Frameworks/Python.framework`。有关使用 PyO3 crate 与不同 Python 版本的详细说明，请参见 [这里](https://pyo3.rs/v0.18.1/building_and_distribution#configuring-the-python-version) - -### 嵌入式 RustPython 解释器 - -可以运行脚本的实验性 [python 解释器](https://github.com/RustPython/RustPython)，它支持 Python 3.10 语法。您可以使用所有的 Python 语法，更多信息请参见 [Python 脚本的用户指南](/user-guide/python-scripts/overview.md). - diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/query-engine.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/query-engine.md index 62d7563bb2..84445759d4 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/query-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/query-engine.md @@ -7,33 +7,33 @@ description: 介绍了 GreptimeDB 的查询引擎架构，基于 Apache DataFusi ## 介绍 -GreptimeDB 的查询引擎是基于[Apache DataFusion][1]（属于[Apache Arrow][2]的子项目）构建的，它是一个用 Rust 编写的出色的查询引擎。它提供了一整套功能齐全的组件，从逻辑计划、物理计划到执行运行时。下面将解释每个组件如何被整合在一起，以及在执行过程中它们的位置。 +GreptimeDB 的查询引擎基于 [Apache DataFusion][1]。DataFusion 提供逻辑计划、物理计划、优化器框架和执行运行时；GreptimeDB 在此基础上增加各查询语言的 planner、存储相关优化规则、自定义计划节点和分布式执行。 -![Execution Procedure](/execution-procedure.png) +DDL 和其他控制面操作由 statement executor 分发。Query engine 接收数据处理计划，包括 `INSERT ... SELECT` 等操作中读取输入数据的部分。 -入口点是逻辑计划，它被用作查询或执行逻辑等的通用中间表示。逻辑计划的两个主要来源是：1. 用户查询，例如通过 SQL 解析器和规划器的 SQL；2. Frontend 的分布式查询，这将在下一节中详细解释。 +## 查询生命周期 -接下来是物理计划，或称执行计划。与包含所有逻辑计划变体（除特殊扩展计划节点外）的大型枚举的逻辑计划不同，物理计划实际上是一个定义了在执行过程中调用的一组方法的特性。所有数据处理逻辑都包装在实现该特性的相应结构中。它们是对数据执行的实际操作，如聚合器 `MIN` 或 `AVG` ，以及表扫描 `SELECT ... FROM`。 +1. SQL、PromQL 或日志查询 planner 通过 catalog 解析表，并生成 DataFusion logical plan。DataFusion 不直接支持的操作由 GreptimeDB plan extension 表示。 +2. DataFusion 的 analyzer 和 optimizer rule 与 GreptimeDB rule 共同运行。这些规则规范化表达式和类型、改写时间范围操作、将 projection 和 filter 下推到 scan，并在需要时引入分布式计划节点。 +3. Physical planner 将优化后的 logical plan 转换为流式 operator。GreptimeDB 随后应用 scan 并行度、排序和分布式执行相关的 physical rule。 +4. 执行阶段通过 physical plan 拉取 Arrow record batch。存储 scan 接收 projection 和 predicate，下游 operator 消费数据流，无需先物化完整结果。 -优化阶段通过转换逻辑计划和物理计划来提高执行性能，现在全部基于规则。它也被称为“基于规则的优化”。一些规则是 DataFusion 原生的，其他一些是在 GreptimeDB 中自定义的。在未来，我们计划添加更多规则，并利用数据统计进行基于成本的优化 (CBO)。 - -最后一个阶段"执行"是一个动词，代表从存储读取数据、进行计算并生成预期结果的过程。虽然它比之前提到的概念更抽象，但你可以简单地将它想象为执行一个 Rust 异步函数，并且它确实是一个异步流。 - -当你想知道 SQL 是如何通过逻辑计划或物理计划中表示时，`EXPLAIN [VERBOSE] ` 是非常有用的。 +使用 [`EXPLAIN`](/reference/sql/explain.md) 查看逻辑和物理计划。`EXPLAIN ANALYZE` 还会执行计划并报告运行时指标。 ## 数据表示 -GreptimeDB 使用 [Apache Arrow][2]作为内存中的数据表示格式。它是面向列的，以跨平台格式，也包含许多高性能的基础操作。这些特性使得在许多不同的环境中共享数据和实现计算逻辑变得容易。 +GreptimeDB 使用 [Apache Arrow][2] record batch 作为内存数据表示。一个 record batch 包含等长的列数组和 schema。查询 operator 交换这些 batch 组成的数据流，使 Region scan 到结果编码的执行路径保持列式处理。 ## 索引 -在时序数据中，有两个重要的维度：时间戳和标签列（或者类似于关系数据库中的主键）。GreptimeDB 将数据分组到时间桶中，因此能在非常低的成本下定位和提取预期时间范围内的数据。GreptimeDB 中主要使用的持久文件格式 [Apache Parquet][3] 提供了多级索引和过滤器，使得在查询过程中很容易修剪数据。在未来，我们将更多地利用这个特性，并开发我们的分离索引来处理更复杂的用例。 +索引构建和持久化格式属于存储引擎。查询层向 scan 提供 predicate 和 projection，Mito 再利用时间范围、Parquet 统计信息和索引跳过不可能匹配的数据。参见[数据持久化与索引](./data-persistence-indexing.md)。 + + -## 分布式查询 +## 分布式执行 -参考 [Distributed Querying][6]. +分布式模式下，Frontend 规划集群级查询，Datanode 执行 Region 本地子计划。[`MergeScan`][6] 是两个阶段之间的边界。 -[1]: https://github.com/apache/arrow-datafusion +[1]: https://datafusion.apache.org/ [2]: https://arrow.apache.org/ -[3]: https://parquet.apache.org [6]: ../frontend/distributed-querying.md diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/storage-engine.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/storage-engine.md index 67f98216fe..d3abb58cfa 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/storage-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/storage-engine.md @@ -7,7 +7,7 @@ description: 详细介绍了 GreptimeDB 的存储引擎架构、数据模型和 ## 概述 -`存储引擎` 负责存储数据库的数据。Mito 是我们默认使用的存储引擎，基于 [LSMT][1]（Log-structured Merge-tree）。我们针对处理时间序列数据的场景做了很多优化，因此 mito 这个存储引擎并不适用于通用用途。 +Mito 是 GreptimeDB 的默认存储引擎，基于 [LSM tree][1]，面向时间序列负载设计，而不是通用的嵌入式存储引擎。 ## 架构 下图展示了存储引擎的架构和处理数据的流程。 @@ -20,9 +20,9 @@ description: 详细介绍了 GreptimeDB 的存储引擎架构、数据模型和 - 为尚未刷盘的数据提供高持久性保证。 - 基于 `LogStore` API 实现，不关心底层存储介质。 - WAL 的日志记录可以存储在本地磁盘上，也可以存储在实现了 `LogStore` API 的远程日志服务中，例如 Kafka（remote WAL）。 -- Memtable - - 数据首先写入 `active memtable`，又称 `mutable memtable`。 - - 当 `mutable memtable` 已满时，它将变为只读的 `immutable memtable`。 +- [Memtable](memtable.md) + - Mito 根据 time index 将数据行写入 mutable memtable。 + - Flush 冻结 mutable memtable，安装一组新的 mutable memtable 以接收写入，再将冻结的 memtable 写为 SST 文件。 - SST - SST 的全名为有序字符串表（`Sorted String Table`）。 - `immutable memtable` 刷到持久存储后形成一个 SST 文件。 @@ -100,7 +100,9 @@ Mito 会按 primary key 对行分组，并按时间排序，因此 SST 中的数 Mito 支持两种 SST 格式：`flat` 和 `primary_key`。`flat` 是新表的默认格式，适用于各种 primary key 基数，包括高基数 key。`primary_key` 是为了兼容旧表而保留的遗留格式。更多详情请参考 [SST format](/reference/sql/create.md#创建指定-sst-格式的表) 和[表设计指南](/user-guide/deployments-administration/performance-tuning/design-table.md#sst-格式)。 -SST layout +![Mito 默认的 flat SST 布局将文件级元数据与包含数据列和合并元数据的 Parquet row group 组合在一起。](/mito-sst-layout.zh.svg) + +一个 SST 可能跨越多个 compaction time window。 ## 扫描裁剪 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/wal.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/wal.md index 529adcdf50..36e0689430 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/wal.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/datanode/wal.md @@ -9,20 +9,26 @@ description: 介绍了 GreptimeDB 的预写日志（WAL）机制，包括其命 ## 介绍 -我们的存储引擎受到了日志结构合并树（Log-structured Merge Tree，LSMT）的启发。对数据的变更操作直接应用于 MemTable 而不是持久化到磁盘上的数据页，这显著提高了性能，但也带来了持久化相关的问题，特别是在 Datanode 意外崩溃时。与所有类似 LSMT 的存储引擎一样，GreptimeDB 使用预写日志（Write-Ahead Log，WAL）来确保数据被可靠地持久化，并且保证崩溃时的数据完整性。 +Mito 在将数据 flush 为 SST 文件前，先在 [memtable](memtable.md) 中缓冲写入。每个 Region 的 mutation 会先追加到预写日志（WAL），从而恢复尚未进入 SST 的数据。 -预写日志是一个仅提供追加写的文件组。所有的 INSERT 和 DELETE 操作都被转换为操作日志，然后追加到 WAL。一旦操作日志被持久化到底层文件，该操作才可以进一步应用到 MemTable。 +WAL 通过统一的 log-store 抽象访问，可以使用本地 raft-engine 或远端 Kafka。 -当数据节点重新启动时，WAL 中的操作条目将被重放，以重建正确的 MemTable 状态。 +## 写入与恢复流程 -![WAL in Datanode](/wal.png) +正常写入遵循以下顺序： + +1. Region worker 分配 sequence number 和 WAL entry ID。 +2. 将 mutation 追加到 WAL。追加失败时，不会把 mutation 写入 memtable。 +3. WAL 追加成功后，Mito 将 mutation 写入 memtable，并发布新的 committed sequence。 +4. Flush 将不可变 memtable 写为 SST 文件，并持久化包含新文件和 `flushed_entry_id` 的 manifest edit。 +5. Manifest edit 持久化后，`flushed_entry_id` 及以前的 WAL entry 被标记为 obsolete；log store 可以稍后再回收物理空间。 + +Manifest 是恢复边界。正常重新打开 Region 时，Mito 根据 manifest 重建 Region，并从 `flushed_entry_id + 1` 开始重放 WAL。Region 状态切换可以指定更晚的 replay checkpoint，但不会重放早于已持久化 flush 边界的 entry。 ## 命名空间 -WAL 的命名空间用于区分来自不同 region 的条目。追加和读取操作必须提供一个命名空间。目前，region ID 被用作命名空间，因为每个 region 都有一个在数据节点重新启动时需要重构的 MemTable。 +WAL entry 按 Region 隔离，而不是按表隔离。追加和读取都需要指定 Region namespace，使单个 Region 可以独立重放或截断。本地 raft-engine 使用 Region ID 作为 namespace ID；Kafka provider 则在基于 topic 的日志中保留 Region 标识。 ## 同步/异步刷盘 -默认情况下，WAL 的追加写是异步的，这意味着写入方不会等待操作日志被刷入到磁盘并持久化。这个默认设置提供了更高的性能，但在服务器意外关闭时可能会丢失数据。另一方面，同步刷新提供了更高的可靠性，但其代价是性能更低。 - -在 v0.4 版本中，新的 region worker 架构可以使用批处理来减轻同步刷盘的开销。 +对于本地 raft-engine，`sync_write` 控制追加写是否等待日志同步到持久化存储，默认值为 `false`。异步写入延迟较低，但主机在缓冲数据同步前故障时，可能丢失最近确认的 entry。Kafka WAL 的持久性由 producer 和集群配置决定，不受这个本地选项控制。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/arrangement.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/arrangement.md index dd3b6de090..7b35b50259 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/arrangement.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/arrangement.md @@ -5,6 +5,8 @@ description: 描述了 Arrangement 在数据流进程中的状态存储功能， # Arrangement +本页介绍 Flownode 旧 streaming 模式使用的状态结构；batching 模式不使用 Arrangement。 + Arrangement 存储数据流进程中的状态，存储 flow 的更新流（stream）以供进一步查询和更新。 Arrangement 本质上存储的是带有时间戳的键值对。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/batching_mode.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/batching_mode.md index bec092b0af..8e2ecdb80b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/batching_mode.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/batching_mode.md @@ -9,13 +9,13 @@ description: Flownode 批处理模式概述，这是持续数据聚合当前使 ## 概述 -`flownode` 中的批处理模式专为持续数据聚合而设计。它在离散的、微小的时间窗口上周期性地执行用户定义的 SQL 查询。这与原始的流处理模式形成对比；流处理模式现在已经废弃，在该模式下数据会在到达时即被处理。 +`flownode` 中的批处理模式专为持续数据聚合而设计。它在离散的小时间窗口上周期性执行用户定义的 SQL 查询。旧 streaming 路径则在数据到达时进行处理，目前仅为兼容已有 workload 而保留，不推荐新 workload 使用。 其核心思想是： 1. 定义一个带有 SQL 查询的 `flow`，该查询将数据从源表聚合到目标表。 2. 查询通常在时间戳列上包含一个时间窗口函数（例如 `date_bin`）。 3. 当新数据插入源表时，系统会将相应的时间窗口标记为“脏”(dirty)。 -4. 一个后台任务会周期性地唤醒，识别这些脏窗口，并为那些特定的时间范围重新运行聚合查询。 +4. 一个后台任务按自身的节奏运行，在下一次求值时取出待处理的脏窗口，并对这些时间范围重新运行聚合查询。 5. 然后将结果插入到目标表中，从而有效地更新聚合视图。 ## 架构 @@ -39,15 +39,15 @@ description: Flownode 批处理模式概述，这是持续数据聚合当前使 - **状态 (`TaskState`)**: 包含任务的动态、可变状态，最重要的是 `DirtyTimeWindows`。 - **执行循环**: 任务运行一个无限循环 (`start_executing_loop`)，该循环： 1. 检查关闭信号。 - 2. 等待一个预定的时间间隔或直到被唤醒。 + 2. 睡眠到下一次求值时间。设置了求值调度的任务睡眠到下一个调度时间点；自适应任务则按时间窗口大小和最小刷新间隔计算出的轮询间隔睡眠。 3. 基于当前的脏时间窗口集合生成一个新的查询计划 (`gen_insert_plan`)。 4. 对数据库执行查询 (`execute_logical_plan`)。 5. 清理已处理的脏窗口。 ### `TaskState` 和 `DirtyTimeWindows` -- **`TaskState`**: 此结构体跟踪 `BatchingTask` 的运行时状态。它包括 `dirty_time_windows`，这对于确定需要完成哪些操作至关重要。 -- **`DirtyTimeWindows`**: 这是一个关键的数据结构，用于跟踪自上次查询执行以来哪些时间窗口接收到了新数据。它存储一组不重叠的时间范围。当任务的执行循环运行时，它会参考此结构来构建一个 `WHERE` 子句，该子句仅过滤源表中的脏时间窗口。 +- **`TaskState`**: 此结构体跟踪 `BatchingTask` 的运行时状态，包括用于确定待处理工作的 `dirty_time_windows`。 +- **`DirtyTimeWindows`**: 此数据结构跟踪上次查询执行后接收到新数据的时间窗口，并保存一组不重叠的时间范围。执行循环根据它构造 `WHERE` 子句，只从源表选择脏窗口。 ### `TimeWindowExpr` @@ -56,15 +56,15 @@ description: Flownode 批处理模式概述，这是持续数据聚合当前使 - **求值**: 它可以接受一个时间戳并对时间窗口表达式求值，以确定该时间戳所属窗口的开始和结束。 - **窗口大小**: 它还可以从表达式中确定时间窗口的大小（持续时间）。 -这对于标记窗口为脏以及在查询源表时生成正确的过滤条件都至关重要。 +标记脏窗口和生成源表过滤条件使用同一套计算。 ## 查询执行流程 以下是批处理模式下查询执行的简化分步演练： 1. **数据摄取**: 新数据被写入源表。 -2. **标记为脏**: `BatchingEngine` 收到有关新数据的通知。它使用与每个相关 flow 关联的 `TimeWindowExpr` 来确定哪些时间窗口受到新数据点的影响。然后将这些窗口添加到相应 `TaskState` 中的 `DirtyTimeWindows` 集合中。 -3. **任务唤醒**: `BatchingTask` 的执行循环被唤醒，原因可能是其周期性调度，也可能是因为它被通知有大量积压的脏窗口。 +2. **标记为脏**: `BatchingEngine` 收到有关新数据的通知。它使用与每个相关 flow 关联的 `TimeWindowExpr` 来确定哪些时间窗口受到新数据点的影响。然后将这些窗口添加到相应 `TaskState` 中的 `DirtyTimeWindows` 集合中。标记脏窗口不会唤醒任务。 +3. **下一次求值**: `BatchingTask` 的执行循环在调度时间点或自适应轮询间隔结束后进入下一次求值，取出待处理的脏窗口。 4. **计划生成**: 任务调用 `gen_insert_plan`。此方法： - 检查 `DirtyTimeWindows`。 - 生成一系列 `OR` 连接的 `WHERE` 子句（例如 `(ts >= 't1' AND ts < 't2') OR (ts >= 't3' AND ts < 't4') ...`），覆盖所有脏窗口。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/dataflow.md index 9d07922542..95def5e844 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/dataflow.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/dataflow.md @@ -1,18 +1,39 @@ --- -keywords: [Dataflow, SQL 查询, 执行计划, 数据流, map, reduce] -description: 解释了 Dataflow 模块的核心计算功能，包括 SQL 查询转换、内部执行计划、数据流的触发运行和支持的操作。 +keywords: [Flownode, batching mode, streaming mode, Dataflow, 脏时间窗口] +description: 介绍 Flownode 如何选择并运行 batching 和旧 streaming 两条执行路径。 --- # 数据流 +Flownode 内部有两条执行路径： + +- **Batching mode** 是聚合和 TQL workload 的主要执行路径。它查询已经持久化的 source 数据，并将物化结果写入 sink table。 +- **Streaming mode** 是为兼容已有 workload 而保留的旧执行路径，不推荐新 workload 使用。Frontend 会把新到达的行同步给它进行增量处理。 + +用户不能直接选择执行模式。创建 Flow 时，GreptimeDB 根据查询和 source table 的属性选择执行路径。聚合、`DISTINCT` 和 TQL 查询使用 batching mode；简单的非聚合查询，以及任何 source table 使用 `ttl = 'instant'` 的 Flow，目前仍使用 streaming mode。如果 source table 尚不存在并选择延迟创建，Flow 会先成为 pending batching Flow。 + +## Batching mode + +Batching mode 复用 GreptimeDB 的查询引擎，不需要为每一行输入维护一张算子图。对于基于时间窗口的 Flow，主循环如下： + +1. Source table 收到写入后，把受影响的时间窗口标记为 dirty。 +2. `BatchingTask` 按求值调度或自适应轮询节奏运行，并在该次求值时收集待处理的 dirty window。标记 dirty window 不会唤醒任务。 +3. 任务把这些窗口转换成时间谓词，加入 Flow 查询，再请求 Frontend 查询 source table。 +4. 查询结果写入 sink table，更新已重新计算窗口对应的物化结果。 +5. 成功处理的窗口从 dirty set 中移除；执行失败的工作仍可在后续调度中处理。 + +设置了 evaluation interval、但查询中没有时间窗口表达式的 Flow，会在每次调度时执行完整查询。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见 [Flownode 批处理模式开发者指南](./batching_mode.md)。 + +## Streaming mode + Dataflow 模块（参见 `flow::compute` 模块）是 `flow` 的核心计算模块。 它接收 SQL 查询并将其转换为 `flow` 的内部执行计划。 然后，该执行计划被转化为实际的数据流，而数据流本质上是一个由带有输入和输出端口的函数组成的有向无环图（DAG）。 -数据流会在需要时被触发运行。 +新到达的行变更会增量驱动这张图执行。 -目前该数据流只支持 `map`和 `reduce` 操作，未来将添加对 `join` 等操作的支持。 +Renderer 支持 map/filter/project 和 reduce 操作。执行计划中已经有 join 和 union 节点，但 streaming renderer 尚未实现它们。 在内部，数据流使用 `tuple(row, time, diff)` 以行格式处理数据。 这里 `row` 表示实际传递的数据，可能包含多个 `value` 对象。 `time` 是系统时间，用于跟踪数据流的进度，`diff` 通常表示行的插入或删除（+1 或 -1）。 -因此，`tuple` 表示给定系统时间的 `row` 的插入/删除操作。 +因此，`tuple` 表示给定系统时间的 `row` 的插入/删除操作。有状态算子通过 [Arrangement](./arrangement.md) 保存这些变更的索引 trace。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/distributed-querying.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/distributed-querying.md index 612174662b..e6d0c35160 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/distributed-querying.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/distributed-querying.md @@ -1,41 +1,20 @@ --- -keywords: [分布式查询, 查询拆分, 查询合并, TableScan, 物理计划] -description: 介绍 GreptimeDB 中的分布式查询方法，包括查询的拆分和合并过程，以及 TableScan 节点的作用。 +keywords: [分布式查询, 逻辑计划, MergeScan, Substrait, Region 裁剪] +description: 介绍 GreptimeDB 如何把逻辑查询计划划分为 Frontend 和 Datanode 上的执行任务。 --- # 分布式查询 -我们知道在 GreptimeDB 中数据是如何分布的（参见“[表分片][1]”），那么如何查询呢？在 GreptimeDB 中，分布式查询非常简单。简单来说，我们只需将查询拆分为子查询，每个子查询负责查询表数据的一个部分，然后将所有结果合并为最终结果。这是一种典型的“拆分 - 合并”方法。具体来说，让我们从查询到达 `frontend` 开始。 +Frontend 和 Datanode 使用同一套基于 DataFusion 的查询引擎。在分布式模式下，Frontend 会增加一个规划步骤，将 Datanode 上执行的工作与 Frontend 上完成的工作分开。 -当查询到达 `frontend` 时，它首先被解析为 SQL 抽象语法树（AST）。我们遍历 AST，并从中生成逻辑计划。顾名思义，逻辑计划只是如何“逻辑地”执行查询的“提示”，它不能被直接运行，因此我们进一步从中生成可执行的物理计划。物理计划是一种类似树形的数据结构，每个节点实际上表示查询的执行方法。一旦我们从上到下运行物理计划树，结果数据将从叶子到根流动，被合并或计算。最终，我们在根节点的输出处得到了查询的结果。 +![Frontend query](/frontend-query.png) -到目前为止，这只是一个典型的“volcano”查询执行模型，你可以在几乎每个 SQL 数据库中看到这种模型。那么“分布式”是在哪里发生的呢？这全部发生在一个名为“TableScan”的物理计划节点中。TableScan 是物理计划树中的一个叶子节点，它负责扫描表的数据（就像它的名称所暗示的）。当 `frontend` 即将扫描表时，它首先需要根据每个 `region` 的数据范围将表扫描拆分为较小的扫描。 +## 分布式规划 -[1]: ./table-sharding.md +分布式规划器重写逻辑计划，把可以下推的算子移向表扫描，并用 `MergeScan` 节点包装远端子计划。分区列上的谓词还会在任务调度前用于裁剪 Region。 -表的所有 `region` 都有它们存储数据的范围。以下表为例： +算子能否下推取决于计划形态和算子本身的性质。不支持的部分会保留在 Frontend。初始设计及交换律规则参见[分布式规划器 RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md)。 -```sql -CREATE TABLE my_table ( - a INT, - others STRING, - ts TIMESTAMP TIME INDEX, -) -PARTITION ON COLUMNS (a) ( - a < 10, - a >= 10 AND a < 20, - a >= 20 -); -``` +## 分布式计划 -`my_table` 表创建时被设定了 3 个分区。在 GreptimeDB 的当前实现中，将为该表创建 3 个 `region`（分区与 `region` 的比例为 1:1）。这 3 个区域将分别包含以下范围："[-∞, 10)", "[10, 20)" 和 "[20, +∞)"。例如，如果提供了值 "42"，我们将搜索这些范围，并找到包含该值的相应的 `region`（在此示例中为第 3 个 `region`）。 - -对于查询，我们使用“过滤器”来查找 `region`。 "过滤器"是 "WHERE" 子句中的条件。例如，查询 `SELECT * FROM my_table WHERE a < 10 AND others = 'x'`，其“过滤器”为“a < 10 AND others = 'x'”。然后我们检查这些范围，找出包含满足过滤器条件的值的所有 `region`。 - -> 如果某个查询没有任何过滤器，则将其视为全表扫描。 - -找到所需的区域后，我们只需在其中组装子扫描。通过这种方式，我们将查询拆分为子查询，每个子查询都获取表数据的一部分。子查询在 `datanode` 中执行，并在 `frontend` 中等待完成。它们的结果将合并为表扫描请求的最终返回。 - -下面这张图片总结了分布式查询执行的过程： - -![Distributed Querying](/distributed-querying.png) +远端输入是完整的逻辑子计划，并不局限于表扫描。Frontend 使用 [Substrait](https://substrait.io) 序列化子计划，再向持有相应数据的 Datanode 发送 Region 级请求。Datanode 在本地规划并执行子计划，将结果流返回 Frontend。Frontend 合并远端数据流，并执行没有下推的算子。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/overview.md index ee48da3ce3..6befdc55a9 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/overview.md @@ -5,31 +5,51 @@ description: GreptimeDB Frontend 组件概述 - 为客户端请求提供服务 # Frontend -**Frontend** 是一个无状态服务，作为 GreptimeDB 中客户端请求的入口点。它为多种数据库协议提供统一接口，并充当代理，将读写请求转发到分布式系统中的相应 Datanode。 +Frontend 是 GreptimeDB 中负责请求编排的无状态服务。Server 层负责终止协议并转换线上消息；Frontend 为这些协议处理器提供数据库行为，包括权限检查、语句执行、路由和分布式查询规划。 + +Frontend 不存储表数据。它缓存从 Metasrv 获取的 catalog 和路由元数据；元数据发生变化时，Metasrv 通过心跳响应通知 Frontend 失效相应缓存。 ## 核心功能 -- **协议支持**：支持多种数据库协议，包括 SQL、PromQL、MySQL 和 PostgreSQL。详见[协议][1] -- **请求路由**：基于元数据将请求路由到相应的 Datanode -- **查询分发**：将分布式查询拆分到多个节点 -- **响应聚合**：合并来自多个 Datanode 的结果 -- **认证授权**：安全和访问控制验证 +- 为支持的[协议][1]提供查询和写入行为。 +- 解析 catalog、schema、table 和 Region 路由。 +- 在执行请求前完成权限检查。 +- 规划分布式查询并合并 Datanode 返回的结果。 +- 将表级写入和删除转换为 Region 请求。 ## 架构 ### 关键组件 -- **协议处理器**：处理不同的数据库协议 -- **目录管理器**：缓存来自 Metasrv 的元数据以实现高效的请求路由和 Schema 校验 -- **分布式规划器**：将逻辑计划转换为分布式执行计划 -- **请求路由器**：为每个请求确定目标 Datanodes + +- 协议处理器将 SQL、PromQL、gRPC 写入和可观测性协议转换为 Frontend 的内部请求接口。 +- Catalog manager 和 partition manager 提供表元数据、分区规则和 Region 路由。 +- Statement executor 将查询、DML 和 DDL 分发到各自的执行路径。 +- 分布式规划器把表扫描替换为可跨 Datanode 执行的 `MergeScan` 计划。 ### 请求流程 -![request flow](/request_flow.png) +不同操作会走不同的请求路径。 + +#### 查询 + +1. 协议处理器创建查询上下文，并完成认证和权限检查。 +2. 对应查询语言的 planner 生成逻辑计划。分布式模式下，planner 根据分区元数据选择 Region 并生成分布式计划。 +3. Frontend 将 Region 子计划发送到对应 Datanode。Datanode 在本地 Region engine 上执行，并返回 Arrow record batch 流。 +4. Frontend 执行剩余算子、合并数据流，再按客户端协议编码结果。 + +#### 写入和删除 + +1. Frontend 根据表 schema 校验请求。支持 schema-on-write 的协议可以先创建缺失的表或新增列，再重试写入。 +2. 分区规则把每一行分配给 Region。Frontend 为各目标 Region 构造请求，并路由到当前 Region leader。 +3. Datanode 的 Region server 将请求分发到对应的 Region engine。单机模式下，请求直接发送给内嵌的 Region server。 + +#### DDL + +Statement executor 将 DDL 转换为 task。分布式模式下，Metasrv 以持久化 procedure 执行 task、更新元数据，并协调 Datanode 上的 Region 操作。单机模式复用相同的语句边界，但使用本地元数据和 procedure 实现。 ### 部署 -下图是 GreptimeDB 在云上的一个典型的部署。`Frontend` 实例组成了一个集群处理来自客户端的请求： +下图展示了 GreptimeDB 的一种云上部署。多个 Frontend 实例共同处理客户端请求： ![frontend](/frontend.png) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/table-sharding.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/table-sharding.md index 63a8ac10a6..cf39afe27b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/table-sharding.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/frontend/table-sharding.md @@ -5,21 +5,17 @@ description: 介绍 GreptimeDB 中表数据的分片方法，包括分区和 Reg # 表分片 -对于任何分布式数据库来说，数据的分片都是必不可少的。本文将描述 GreptimeDB 中的表数据如何进行分片。 +GreptimeDB 将一张表分为多个 Region。分区表达式定义每行数据属于哪个 Region，Region 路由则定义当前由哪个 Datanode 持有该 Region。 ## 分区 -有关创建分区表的语法，请参阅用户指南中的[表分片](/user-guide/deployments-administration/manage-data/table-sharding.md)部分。 +分区是由一个或多个列上的表达式描述的逻辑行集合。分区布局需要覆盖表的输入域，使每一行都能找到唯一的目标 Region。SQL 语法和支持的表达式参见[表分片](/user-guide/deployments-administration/manage-data/table-sharding.md)。 ## Region -在创建分区后，表中的数据被逻辑上分割。你可能会问："在 GreptimeDB 中，被逻辑上分区的数据是如何存储的？" 答案是保存在 `Region` 当中。 - -每个 `Region` 对应一个分区，并保存分区的数据。所有的 `Region` 分布在各个 `Datanode` 之中。 -`Metasrv` 管理 `Region` 到 `Datanode` 的路由信息。如果建表后需要调整分区布局， -GreptimeDB 支持通过显式的 [repartition](/user-guide/deployments-administration/manage-data/repartition.md) 操作拆分或合并分区。 +每个分区对应一个 Region。Region ID 是 Frontend、Datanode 和 Metasrv 用于存储和路由的标识。同一张表的多个 Region 可以放在同一个 Datanode 上。 分区和 Region 的关系参见下图： @@ -54,3 +50,13 @@ GreptimeDB 支持通过显式的 [repartition](/user-guide/deployments-administr └──────────────────────────────────┘ 可以放在同一个 Datanode 之中 ``` + +## 路由与剪枝 + +写入时，Frontend 对每行数据计算分区规则，按 Region 分组，再根据路由表把 Region 请求发送到当前 leader。 + +查询时，分布式 planner 将查询谓词与分区表达式比较，只扫描可能满足谓词的 Region。如果分区元数据缺失或无法安全解释，planner 会退化为扫描所有 Region，避免漏掉数据。 + +## 调整分区布局 + +[Repartition](/user-guide/deployments-administration/manage-data/repartition.md) 通过显式的 split 或 merge 调整已有布局。Metasrv 以持久化 procedure 执行变更，更新 Region 路由和分区表达式，并使旧的表路由缓存失效。Frontend 刷新到新元数据后，后续请求使用新的布局。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/getting-started.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/getting-started.md index e5aa2ca855..999f696ed0 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/getting-started.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/getting-started.md @@ -15,14 +15,13 @@ description: 介绍如何在本地环境中从源代码编译和运行 GreptimeD ### 构建依赖项 -- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line)（可选） +- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line)（可选；克隆仓库需要，构建本身不需要） - C/C++ 工具链：提供编译和链接的基本工具。在 Ubuntu 上，这可用作 `build-essential`。在其他平台上，也有类似的命令。 -- Rust nightly 工具链（[指南][1]） - - 编译源代码 +- [Rustup][1]。仓库通过 `rust-toolchain.toml` 指定所需的 nightly 工具链。 - Protobuf（[指南][2]） - 编译 proto 文件 - 请注意，版本需要 >= 3.15。你可以使用 `protoc --version` 检查它。 -- 机器：建议内存在 16GB 以上 或者 使用[mold](https://github.com/rui314/mold)工具以降低链接时的内存使用。 +- 机器：建议 16GB 以上内存。内存较小时，可使用 [mold](https://github.com/rui314/mold) 降低链接阶段的内存占用。 [1]: [2]: diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/how-to/how-to-write-sdk.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/how-to/how-to-write-sdk.md index 28ff757763..31e2d45881 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/how-to/how-to-write-sdk.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/how-to/how-to-write-sdk.md @@ -1,21 +1,17 @@ --- keywords: [gRPC SDK, GreptimeDatabase, GreptimeRequest, GreptimeResponse, 插入请求] -description: 介绍如何为 GreptimeDB 开发一个 gRPC SDK，包括 GreptimeDatabase 服务的定义、GreptimeRequest 和 GreptimeResponse 的结构。 +description: 介绍 GreptimeDB gRPC 写入 SDK 需要遵守的协议契约和错误处理要求。 --- # 如何为 GreptimeDB 开发一个 gRPC SDK -GreptimeDB 的 gRPC SDK 只需要处理写请求即可。读请求是标准 SQL 或 PromQL，可以由任何 JDBC 客户端或 Prometheus -客户端处理。这也是为什么所有的 GreptimeDB SDK 都命名为 "`greptimedb-ingester-`"。请确保你的 GreptimeDB SDK -遵循相同的命名约定。 +GreptimeDB 的公开 gRPC SDK 是写入客户端。查询通常通过标准 SQL 或 PromQL 客户端完成。除非有单独需求，新 SDK 应聚焦写入和删除，并遵循 `greptimedb-ingester-` 命名约定。面向用户的 API 参见 [gRPC SDK 概述](/user-guide/ingest-data/for-iot/grpc-sdks/overview.md)。 ## `GreptimeDatabase` 服务 -GreptimeDB 自定义了一个 gRPC 服务：`GreptimeDatabase` -。你只需要实现这个服务即可。你可以在[这里](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto) -找到它的 Protobuf 定义。 +从 [`GreptimeDatabase` Protobuf 定义](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto)生成客户端 stub，不要在 SDK 中手写一份 message 或 service 定义。 -`GreptimeDatabase` 有 2 个 RPC 方法： +该 service 提供一个 unary method 和一个 client-streaming method： ```protobuf service GreptimeDatabase { @@ -25,13 +21,9 @@ service GreptimeDatabase { } ``` -`Handle` 方法是一个 unary 调用：当 GreptimeDB 服务接收到一个 `GreptimeRequest` 请求后，它立刻处理该请求并返回一个相应的 -`GreptimeResponse`。 +`Handle` 对一个请求返回一个响应，是 SDK insert 和 delete API 通常使用的方法。 -`HandleRequests` 方法则是一个 "[Client Streaming RPC][3]" 方式的调用。 -它可以接受一个连续的 `GreptimeRequest` 请求流，持续地发给 GreptimeDB 服务。 -GreptimeDB 服务会在收到流中的每个请求时立刻进行处理，并最终（流结束时）返回一个总结性的 `GreptimeResponse`。 -通过 `HandleRequests`，我们可以获得一个非常高的请求吞吐量。 +`HandleRequests` 是 [client-streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc)。客户端关闭请求流后，服务端才返回累计响应。SDK 如果暴露 streaming API，需要明确这个确认边界，并将一个 stream 绑定到一个 endpoint。 ### `GreptimeRequest` @@ -51,11 +43,13 @@ message GreptimeRequest { } ``` -`RequestHeader` 是必需，它包含了一些上下文，鉴权和其他信息。"oneof" 的字段包含了发往 GreptimeDB 服务的请求。 +客户端需要在 `RequestHeader` 中填写服务端要求的 database context 和认证信息，并且只能设置一个 request variant。 -注意我们有两种类型的插入请求，一种是以 "列" 的形式（`InsertRequests`），另一种是以 "行" 的形式（`RowInsertRequests` -）。通常我们建议使用 "行" 的形式，因为它对于表的插入更自然，更容易使用。但是，如果需要一次插入大量列，或者有大量的 "null" -值需要插入，那么最好使用 "列" 的形式。 +该 message 还包含供内部调用者使用的 query 和 DDL variant。公开 ingester API 不应暴露它们，因为 `GreptimeDatabase` 不返回 query result stream。 + +GreptimeDB 同时接受行式 `RowInsertRequests` 和列式 `InsertRequests`。公开写入 API 默认使用行式请求。面向列的客户端可以使用列式请求，但转换过程中必须保持列长度一致，并保留 null、时间戳精度、数据类型和列 semantic type。 + +删除同样区分行式和列式。SDK 只应暴露能够在不丢失类型信息的前提下完成映射的形式。 ### `GreptimeResponse` @@ -68,6 +62,18 @@ message GreptimeResponse { } ``` -`ResponseHeader` 包含了返回值的状态码，以及错误信息（如果有的话）。"oneof" 的字段目前只有 "affected rows"。 +成功响应包含 success header 和 `affected_rows`。该值表示服务端确认的行数；关闭请求流时返回的是累计值。 + +请求失败通过 gRPC status 返回。Trailing metadata 中的 `x-greptime-err-code` 在存在时提供 GreptimeDB error code，错误文本则由 status message 携带。SDK 应保留 gRPC status 并暴露 GreptimeDB error code，不能用一个通用 SDK error 将其覆盖。 + +## 重试与交付语义 + +重试次数必须有上限，并且对调用者可见。只有错误被标记为 retryable 且 deadline 仍允许时，才能重试 unary request。Cancellation 和 deadline expiration 不应重试。 + +响应丢失不代表服务端拒绝了写入。除非调用者的数据模型保证操作幂等，重试这类请求可能插入重复行。SDK 需要说明这一点，并在交付结果不确定时返回最终错误。 + +不要自动重试只发送了一部分的 `HandleRequests` stream。即使客户端尚未收到累计响应，服务端也可能已经接受了部分请求。此时应关闭失败的 stream，并将不确定状态返回给调用者。 + +Arrow Flight bulk ingestion 与 `GreptimeDatabase` RPC 应使用不同的 API。它的 batching 和 partial acceptance 需要独立的契约。 -GreptimeDB 现在有很多 SDK，你可以参考[这里](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=)获取一些示例。 +可以参考现有 [GreptimeDB ingester 仓库](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=)的公开 API 约定，但线上行为应以当前 Protobuf 定义和服务端契约为准。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/admin-api.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/admin-api.md index 3ac3d44f3b..bd28206c6f 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/admin-api.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/admin-api.md @@ -1,20 +1,25 @@ --- -keywords: [Admin API, 健康检查, leader 查询, 心跳检测, 维护模式] -description: 介绍 Metasrv 的 Admin API，包括健康检查、leader 查询、心跳检测、维护模式和 Procedure Manager 控制等功能。 +keywords: [Admin API, 健康检查, leader 查询, 心跳检测, 维护模式, 恢复模式, table id sequence] +description: 介绍 Metasrv 用于状态检查、集群控制和元数据恢复的 Admin API。 --- # Admin API -Admin 提供了一种简单的方法来查看和管理集群信息，包括 metasrv 健康检测、metasrv leader 查询、数据节点心跳检测、维护模式和 Procedure Manager 控制。 +:::tip +本页所有 Admin API 都监听 Metasrv 的 `HTTP_PORT`，默认值为 `4000`。 +::: -Admin API 是一个 HTTP 服务，提供一组可以通过 HTTP 请求调用的 RESTful API。Admin API 简单、用户友好且安全。 +Admin API 通过 HTTP 提供 Metasrv 状态、集群控制和元数据恢复操作。该 API 不提供认证，且部分端点会改变集群行为或元数据分配，部署时必须通过网络策略保护 HTTP 端口。 本页介绍以下 API： - /health - /leader - /heartbeat +- /node-lease - /maintenance - /procedure-manager +- /recovery +- /sequence/table 所有这些 API 都在父资源 `/admin` 下。 @@ -22,7 +27,7 @@ Admin API 是一个 HTTP 服务，提供一组可以通过 HTTP 请求调用的 ## /health HTTP 端点 -`/health` 端点接受 GET HTTP 请求，你可以使用此端点检查你的 metasrv 实例的健康状况。 +`/health` 端点接受 GET 请求。HTTP 服务运行时返回 `OK`，但不会检查当前 Metasrv 是否为 leader，也不会检查外部依赖是否可用。 ### 定义 @@ -116,9 +121,17 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100' ] ``` +## /node-lease HTTP 端点 + +`/node-lease` 返回 Metasrv 当前记录的 Datanode lease，可用于判断 Metasrv 是否仍将某个 Datanode 视为存活。 + +```bash +curl -X GET http://localhost:4000/admin/node-lease +``` + ## /maintenance HTTP 端点 -集群维护模式是 GreptimeDB 中的一项安全功能，它可以临时禁用自动集群管理操作。此模式在集群升级、计划停机以及任何可能暂时影响集群稳定性的操作期间特别有用。有关更多详细信息，请参阅[集群维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。 +维护模式在升级、计划停机等操作期间临时禁用自动集群管理。它对集群的具体影响参见[集群维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。 `/maintenance` 端点支持以下 HTTP 请求： @@ -151,3 +164,39 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100' "status": "running" } ``` + +## /recovery HTTP 端点 + +Recovery mode 控制手动修改 table ID sequence 等元数据修复端点。它只用于恢复工作，不用于常规维护。 + +- `GET /admin/recovery/status`：查询 recovery mode 是否开启。 +- `POST /admin/recovery/enable`：开启 recovery mode。 +- `POST /admin/recovery/disable`：关闭 recovery mode。 + +响应体格式如下： + +```json +{ + "enabled": true +} +``` + +修复完成后应关闭 recovery mode。如果只是计划暂停自动集群操作，应使用[维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。 + +## /sequence/table HTTP 端点 + +这些端点用于检查或修复 table ID sequence： + +- `GET /admin/sequence/table/next-id`：返回下一个 table ID，但不执行分配。 +- `POST /admin/sequence/table/set-next-id`：推进下一个 table ID。 + +设置 sequence 前必须开启 recovery mode。新值必须大于当前值，不能通过该 API 回退 sequence。Recovery mode 只是该 API 的前置条件，不能阻止 DDL。执行该操作时，必须遵循[管理 Table ID Sequence](/user-guide/deployments-administration/maintenance/sequence-management.md)中的完整集群操作流程。 + +```bash +curl -X POST \ + -H 'Content-Type: application/json' \ + -d '{"next_table_id": 2048}' \ + http://localhost:4000/admin/sequence/table/set-next-id +``` + +该操作会影响后续新表分配到的 ID。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/overview.md index 2e3c525baa..e9f6144f26 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/overview.md @@ -1,162 +1,101 @@ --- -keywords: [Metasrv, 元数据存储, 请求路由, 负载均衡, 高可用性] -description: 介绍 Metasrv 的功能、架构和与前端的交互方式。 +keywords: [Metasrv, 元数据, 路由, Leader 选举, Procedure, 心跳] +description: 介绍 Metasrv 提供的元数据及集群协调机制。 --- # Metasrv -![meta](/meta.png) - ## Metasrv 包含什么 -- 存储元数据（Catalog, Schema, Table, Region 等） -- 请求路由器。它告诉前端在哪里写入和读取数据。 -- 数据节点的负载均衡，决定谁应该处理新的表创建请求，更准确地说，它做出资源分配决策。 -- 选举与高可用性，GreptimeDB 设计为 Leader-Follower 架构，只有 leader 节点可以写入，而 follower 节点可以读取，follower 节点的数量通常 >= 1，当 leader 不可用时，follower 节点需要能够快速切换为 leader。 -- 统计数据收集（通过每个节点上的心跳报告），如 CPU、负载、节点上的表数量、平均/峰值数据读写大小等，可用作分布式调度的基础。 +Metasrv 是 GreptimeDB 分布式集群中的元数据和协调服务，不参与数据读写链路。它主要负责： + +- 存储 Catalog、Schema、Table、Region、路由和节点元数据； +- 为新 Region 选择 Datanode，并维护表路由； +- 选举一个 Metasrv leader 负责协调元数据变更； +- 通过可恢复的 Procedure 执行 DDL、Region 迁移、故障转移和重分区； +- 通过心跳维护节点租约和 Region 统计信息； +- 元数据变更时向 Frontend、Datanode 和 Flownode 广播缓存失效； +- 向 Datanode 下发 Region 生命周期指令。 ## 前端如何与 Metasrv 交互 -首先，请求路由器中的路由表结构如下（注意这只是逻辑结构，实际存储结构可能不同，例如端点可能有字典压缩）。 - -```txt - table_A - table_name - table_schema // 用于物理计划 - regions - region_1 - mutate_endpoint - select_endpoint_1, select_endpoint_2 - region_2 - mutate_endpoint - select_endpoint_1, select_endpoint_2, select_endpoint_3 - region_xxx - table_B - ... -``` +Frontend 从 Metasrv 获取表元数据和 Region 路由，并缓存在本地。修改元数据的语句会发送给 Metasrv leader；普通读写则使用缓存的路由直接访问 Datanode。 -### 创建表 +控制链路和数据链路相互分离： -1. 前端发送 `CREATE TABLE` 请求到 Metasrv。 -2. 根据请求中包含的分区规则规划 Region 数量。 -3. 检查数据节点可用资源的全局视图（通过心跳收集）并为每个 Region 分配一个节点。 -4. 前端创建表并在成功创建后将 `Schema` 存储到 Metasrv。 +```text +Frontend + |-- 元数据查询和 DDL -------------------> Metasrv leader + `-- Region 读写 ------------------------> Datanode -### `Insert` +Metasrv leader + |-- Region 生命周期指令 ----------------> Datanode + `-- 缓存失效 --------------------------> Frontend / Datanode / Flownode -1. 前端从 Metasrv 获取指定表的路由。注意，最小的路由单元是表的路由（多个 Region），即包含该表所有 Region 的地址。 -2. 最佳实践是前端首先从本地缓存中获取路由并将请求转发到数据节点。如果路由不再有效，则数据节点有义务返回 `Invalid Route` 错误，前端重新从 Metasrv 获取最新数据并更新其缓存。路由信息不经常变化，因此，前端使用惰性策略维护缓存是足够的。 -3. 前端处理可能包含多个表和多个 Region 的一批写入，因此前端需要根据“路由表”拆分用户请求。 +Datanode + `-- 心跳、租约续期和 Region 统计信息 ----> Metasrv leader +``` -### `Select` +在稳定状态下，表路由为每个 Region 记录一个 leader peer 和零个或多个 follower peer。Leader 是写入目标；支持只读副本的部署可以把读取路由到 follower： -1. 与 `Insert` 类似，前端首先从本地缓存中获取路由表。 -2. 与 `Insert` 不同，对于 `Select`，前端需要从路由表中提取只读节点（follower），然后根据优先级将请求分发到 leader 或 follower 节点。 -3. 前端的分布式查询引擎根据路由信息分发多个子查询任务并聚合查询结果。 +```text +Table route + |-- Region 0 + | |-- leader -> Datanode A + | `-- followers -> Datanode B, Datanode C + `-- Region 1 + `-- leader -> Datanode D +``` -## Metasrv 架构 +Region 迁移或故障转移会改变 peer 角色，并可能使 Region 暂时没有 leader。Frontend 刷新缓存路由后，再把后续读写发送给当前 peer。 -![metasrv-architecture](/metasrv-architecture.png) +### 创建表 -## 分布式共识 +1. Frontend 向 Metasrv leader 提交 DDL 请求。 +2. Metasrv 根据分区规则确定 Region，并[为每个 Region 选择 Datanode](/contributor-guide/metasrv/selector.md)。 +3. 持久化的 Procedure 创建 Region，并写入表元数据和路由。发生 leader 切换后，Procedure 可以从已保存的状态继续执行。 +4. 元数据提交后，Metasrv 通知 Frontend 刷新相关缓存。 -如你所见，Metasrv 依赖于分布式共识，因为： +### `Insert` -1. 首先，Metasrv 必须选举一个 leader，数据节点只向 leader 发送心跳，我们只使用单个 Metasrv 节点接收心跳，这使得基于全局信息进行一些计算或调度变得容易且快速。至于数据节点如何连接到 leader，这由 MetaClient 决定（使用重定向，心跳请求变为 gRPC 流，使用重定向比转发更不容易出错），这对数据节点是透明的。 -2. 其次，Metasrv 必须为数据节点提供选举 API，以选举“写入”和“只读”节点，并帮助数据节点实现高可用性。 -3. 最后，`Metadata`、`Schema` 和其他数据必须在 Metasrv 上可靠且一致地存储。因此，基于共识的算法是存储它们的理想方法。 +Frontend 解析表路由，按照分区规则拆分数据行，再把各 Region 的写入发送到对应 Datanode。路由发生变化时，相关缓存会失效，Frontend 随后从 Metasrv 重新获取元数据。 -对于 Metasrv 的第一个版本，我们选择 Etcd 作为共识算法组件（Metasrv 设计时考虑适应不同的实现，甚至创建一个新的轮子），原因如下： +### `Select` -1. Etcd 提供了我们需要的 API，例如 `Watch`、`Election`、`KV` 等。 -2. 我们只执行两个分布式共识任务：选举（使用 `Watch` 机制）和存储（少量元数据），这两者都不需要我们定制自己的状态机，也不需要基于 raft 定制自己的状态机；少量数据也不需要多 raft 组支持。 -3. Metasrv 的初始版本使用 Etcd，使我们能够专注于 Metasrv 的功能，而不需要在分布式共识算法上花费太多精力，这提高了系统设计（避免与共识算法耦合）并有助于初期的快速开发，同时通过良好的架构设计，未来可以轻松接入优秀的共识算法实现。 +Frontend 在查询规划期间使用表和 Region 元数据。分区列上的谓词用于裁剪 Region，分布式查询引擎再把任务发送给持有这些 Region 的 Datanode。参见[分布式查询](../frontend/distributed-querying.md)。 -## 心跳管理 +## Metasrv 架构 -数据节点与 Metasrv 之间的主要通信方式是心跳请求/响应流，我们希望这是唯一的通信方式。这个想法受到 [TiKV PD](https://github.com/tikv/pd) 设计的启发，我们在 [RheaKV](https://github.com/sofastack/sofa-jraft/tree/master/jraft-rheakv/rheakv-pd) 中有实际经验。请求发送其状态，而 Metasrv 通过心跳响应发送不同的调度指令。 +主要协调路径如下： + +```text +Leader election + | + v +Metasrv leader +├─ DDL manager -> Procedure manager +├─ Selector -> 新 Region 的放置 +├─ Heartbeat handler chain -> 租约和 Region 统计信息 +├─ Region supervisor -> Region 迁移 Procedure +├─ Mailbox -> 缓存失效和 Region 指令 +└─ Metadata managers -> KV backend +``` -心跳可能携带以下数据，但这不是最终设计，我们仍在讨论和探索究竟应该收集哪些数据。 +这些机制共享元数据，但故障边界不同。进程重启可以丢弃缓存和 leader 本地状态；恢复所需的元数据和 Procedure 状态必须持久化。 -``` -service Heartbeat { - // 心跳，心跳可能有很多内容，例如： - // 1. 要注册到 Metasrv 并可被其他节点发现的元数据。 - // 2. 一些性能指标，例如负载、CPU 使用率等。 - // 3. 正在执行的计算任务数量。 - rpc Heartbeat(stream HeartbeatRequest) returns (stream HeartbeatResponse) {} -} - -message HeartbeatRequest { - RequestHeader header = 1; - - // 自身节点 - Peer peer = 2; - // leader 节点 - bool is_leader = 3; - // 实际报告时间间隔 - TimeInterval report_interval = 4; - // 节点状态 - NodeStat node_stat = 5; - // 此节点中的 Region 状态 - repeated RegionStat region_stats = 6; - // follower 节点和状态，在 follower 节点上为空 - repeated ReplicaStat replica_stats = 7; -} - -message NodeStat { - // 此期间的读取容量单位 - uint64 rcus = 1; - // 此期间的写入容量单位 - uint64 wcus = 2; - // 此节点中的表数量 - uint64 table_num = 3; - // 此节点中的 Region 数量 - uint64 region_num = 4; - - double cpu_usage = 5; - double load = 6; - // 节点中的读取磁盘 I/O - double read_io_rate = 7; - // 节点中的写入磁盘 I/O - double write_io_rate = 8; - - // 其他 - map attrs = 100; -} - -message RegionStat { - uint64 region_id = 1; - TableName table_name = 2; - // 此期间的读取容量单位 - uint64 rcus = 3; - // 此期间的写入容量单位 - uint64 wcus = 4; - // 近似 Region 大小 - uint64 approximate_size = 5; - // 近似行数 - uint64 approximate_rows = 6; - - // 其他 - map attrs = 100; -} - -message ReplicaStat { - Peer peer = 1; - bool in_sync = 2; - bool is_learner = 3; -} -``` +## 分布式共识 -## Central Nervous System (CNS) +Metasrv 将 leader 选举与元数据存储分开。只有选出的 Metasrv leader 执行协调和元数据变更操作，其他 Metasrv 节点会把 client 引导到当前 leader。 -我们要构建一个算法系统，该系统依赖于每个节点的实时和历史心跳数据，应该做出一些更智能的调度决策并将其发送到 Metasrv 的 Autoadmin 单元，该单元分发调度决策，由数据节点本身或更可能由 PaaS 平台执行。 +Key-value backend 保存表元数据、路由、Procedure 状态以及其他必须跨 leader 切换保留的信息。Metasrv 不使用这套选举为 Datanode Region 创建读写副本；Region 可用性由心跳、Region 故障检测和故障转移 Procedure 管理。 -## 工作负载抽象 +## 心跳管理 -工作负载抽象的级别决定了 Metasrv 生成的调度策略（如资源分配）的效率和质量。 +Datanode 与 Metasrv leader 保持心跳流。心跳请求报告节点身份、租约、Region 统计信息以及放置和监控所需的其他状态；响应则携带 Region 生命周期指令、缓存失效等控制消息。 -DynamoDB 定义了 RCUs 和 WCUs（读取容量单位/写入容量单位），解释说 RCU 是一个 4KB 数据的读取请求，WCU 是一个 1KB 数据的写入请求。当使用 RCU 和 WCU 描述工作负载时，更容易实现性能可测量性并获得更有信息量的资源预分配，因为我们可以将不同的硬件能力抽象为 RCU 和 WCU 的组合。 +心跳驱动两套相互独立的机制，调整心跳周期会同时影响两者： -然而，GreptimeDB 面临比 DynamoDB 更复杂的情况，特别是 RCU 不适合描述需要大量计算的 GreptimeDB 读取工作负载。我们正在努力解决这个问题。 +- **节点租约**：keep-lease handler 为发送心跳的 Datanode 续期。Selector 和 `/node-lease` 端点据此判断 Datanode 是否仍然存活。 +- **Region 故障检测**：Region supervisor 为每个 Region 维护一个基于心跳到达间隔的 Phi Accrual 检测器，其判定与租约是否过期无关。 +只有开启 Region 故障转移时，故障判定才会提交故障转移迁移。该功能默认关闭，并且要求使用 remote WAL，除非显式允许在本地 WAL 上执行。维护模式同样会抑制故障转移。前置条件和开启方式参见 [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/selector.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/selector.md index 9048e0daa4..29b521212b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/selector.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/metasrv/selector.md @@ -7,11 +7,7 @@ description: 介绍 Metasrv 中的 Selector，包括其类型和配置方法。 ## 介绍 -什么是 `Selector`？顾名思义，它允许用户从给定的 `namespace` 和 `context` 中选择 `Item`s。有一个相关的 `trait`，也叫做 `Selector`，其定义可以在[这里][0]找到。 - -[0]: https://github.com/GreptimeTeam/greptimedb/blob/main/src/meta-srv/src/selector.rs - -在 `Metasrv` 中存在一个特定的场景。当 `Frontend` 向 `Metasrv` 发送建表请求时，`Metasrv` 会创建一个路由表（表的创建细节不在这里赘述）。在创建路由表时，`Metasrv` 需要选择适当的 `Datanode`s，这时候就需要用到 `Selector`。 +建表时，Metasrv 使用 `Selector` 为各 Region 选择 Datanode。Selector 根据当前节点租约进行选择；部分实现还会使用 Region 统计信息。 @@ -19,22 +15,22 @@ description: 介绍 Metasrv 中的 Selector，包括其类型和配置方法。 `Metasrv` 目前提供以下几种类型的 `Selectors`: -### LeasebasedSelector +### LeaseBasedSelector -`LeasebasedSelector` 从所有可用的（也就是在租约期间内）`Datanode` 中随机选择，其特点是简单和快速。 +`LeaseBasedSelector` 从租约有效的 Datanode 中随机选择。 ### LoadBasedSelector `LoadBasedSelector` 按照负载来选择，负载值则由每个 `Datanode` 上的 region 数量决定，较少的 region 表示较低的负载，`LoadBasedSelector` 优先选择低负载的 `Datanode`。 ### RoundRobinSelector [默认选项] -`RoundRobinSelector` 以轮询的方式选择 `Datanode`。在大多数情况下，这是默认的且推荐的选项。如果你不确定选择哪个，通常它就是正确的选择。 +`RoundRobinSelector` 以轮询方式选择 Datanode，是默认选项，也适用于大多数部署。 ## 配置 您可以在启动 `Metasrv` 服务时通过名称配置 `Selector`。 -- LeasebasedSelector: `lease_based` 或 `LeaseBased` +- LeaseBasedSelector: `lease_based` 或 `LeaseBased` - LoadBasedSelector: `load_based` 或 `LoadBased` - RoundRobinSelector: `round_robin` 或 `RoundRobin` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/overview.md index 9f82a72171..badd6ad62d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/overview.md @@ -5,9 +5,7 @@ description: 介绍 GreptimeDB 的架构、关键概念和工作原理，包括 # 贡献者指南 -DeepWiki 对 GreptimeDB 的架构和实现进行了详细且清晰的描述，强烈推荐阅读： - -[https://deepwiki.com/GreptimeTeam/greptimedb](https://deepwiki.com/GreptimeTeam/greptimedb) +本指南面向 GreptimeDB 贡献者，介绍理解内部实现所需的设计机制。从源码构建和运行参见[快速开始](/contributor-guide/getting-started.md)。提交要求（CLA、license header、代码格式，以及 PR 必须通过的检查）以源码仓库的 [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md) 为准。 ## 架构 @@ -18,7 +16,13 @@ DeepWiki 对 GreptimeDB 的架构和实现进行了详细且清晰的描述， - [frontend][1] - [datanode][2] - [metasrv][3] +- [flownode][4] [1]: /contributor-guide/frontend/overview.md [2]: /contributor-guide/datanode/overview.md [3]: /contributor-guide/metasrv/overview.md +[4]: /contributor-guide/flownode/overview.md + +## 补充参考 + +[DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb) 提供了自动生成的 GreptimeDB 源码导读，可用于了解不熟悉的模块。它属于辅助资料；涉及具体版本的行为时，仍应以对应源码为准。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/integration-test.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/integration-test.md index 63ffa56bc0..767109f773 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/integration-test.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/integration-test.md @@ -7,8 +7,14 @@ description: 介绍 GreptimeDB 的集成测试，包括测试范围和如何运 ## 介绍 -集成测试使用 Rust 测试工具（`#[test]`）编写，与单元测试不同，它们被单独放置在 -[这里](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration)。 -它涵盖了涉及多个组件的场景，其中一个典型案例是与 HTTP/gRPC 相关的功能。你可以查看 -其[文档](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md)以获取更多信息。 +集成测试覆盖跨 crate 或服务边界的行为，例如 HTTP 和 gRPC 处理、分布式组件或外部存储。测试使用 Rust test harness，位于 [`tests-integration`](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration) package。 +运行命令如下： + +```shell +cargo nextest run -p tests-integration +``` + +部分 case 依赖外部服务的环境变量或 fixture。运行前按照 package 的[准备说明](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md)配置环境。 + +只有 crate 级测试或 Sqlness case 无法覆盖所需边界时才使用集成测试。隔离的逻辑仍放在单元测试中，便于快速复现失败。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/overview.md index 81b6defa45..f38b625082 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/overview.md @@ -5,5 +5,12 @@ description: GreptimeDB 的测试 # 测试 -我们的团队进行了大量测试，以确保 GreptimeDB 的行为。本章将介绍几种用于测试 GreptimeDB 的重要方法，以及如何使用它们。 +选择能够覆盖本次改动的最小测试范围： +| 测试类型 | 适用场景 | 常用命令 | +| --- | --- | --- | +| [单元测试](unit-test.md) | 单个 crate 或组件内的逻辑 | `cargo nextest run -p ` | +| [Sqlness 测试](sqlness-test.md) | SQL、协议、planner、执行和端到端回归 | `cargo sqlness bare -t ` | +| [集成测试](integration-test.md) | 跨组件或依赖外部服务的行为 | `cargo nextest run -p tests-integration` | + +需要运行完整 Rust workspace 测试时使用 `make test`。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/sqlness-test.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/sqlness-test.md index d29b69e835..06503a434c 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/sqlness-test.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/sqlness-test.md @@ -7,36 +7,34 @@ description: 介绍 GreptimeDB 的 Sqlness 测试，包括测试文件类型、 ## 介绍 -SQL 是 `GreptimeDB` 的一个重要用户接口。我们为它提供了一个单独的测试套件（名为 `sqlness`）。 +Sqlness 是 GreptimeDB 针对 SQL 和协议行为的端到端回归测试。每个 case 向运行中的 GreptimeDB 发送语句，并将输出与仓库中的结果文件比较。 ## Sqlness 手册 ### 测试文件 -Sqlness 有两种类型的文件 +每个 case 使用两类文件： - `.sql`：测试输入，仅包含 SQL - `.result`：预期的测试输出，包含 SQL 和其结果 -`.result` 文件是预期的执行输出。如果 `.result` 文件发生变化，意味着测试结果不同，测试可能失败。你应该检查变更日志来解决问题。 - -你只需要在 `.sql` 文件中编写测试 SQL，然后运行测试。 +在 `.sql` 文件中编写输入，运行测试后生成或更新 `.result`。必须检查每一处结果差异，只有行为变化符合预期时才能接受。 ### 组织测试案例 -输入案例的根目录是 `tests/cases`。它包含几个子目录，代表不同的测试模式。例如，`standalone/` 包含所有在 `greptimedb standalone start` 模式下运行的测试。 +输入 case 位于 `tests/cases`。第一级目录选择运行环境，例如 `standalone/` 表示使用单机 GreptimeDB。 -在第一级子目录下（例如 `cases/standalone`），你可以随意组织你的测试案例。Sqlness 会递归地遍历每个文件并运行它们。 +在环境目录内，新 case 应与它覆盖的功能放在一起。Sqlness 会递归发现 case 文件。 ## 运行测试 -与其他测试不同，这个测试工具是以二进制目标形式存在的。你可以用以下命令运行它 +运行命令如下： ```shell -cargo run --bin sqlness-runner bare +cargo sqlness bare ``` -它会自动完成以下步骤：编译 `GreptimeDB`，启动它，抓取测试并将其发送到服务器，然后收集和比较结果。你只需要检查是否有 `.result` 文件发生变化。如果没有，恭喜你，测试通过了 🥳！ +该命令会构建并启动 GreptimeDB、执行选中的 case，再比较输出。`.result` 发生变化只是待审查的结果，不代表新输出一定正确。 ### 运行特定测试 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/unit-test.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/unit-test.md index 79c73775b4..b63b29df94 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/unit-test.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/tests/unit-test.md @@ -7,22 +7,26 @@ description: 介绍 GreptimeDB 的单元测试，包括如何编写、运行和 ## 介绍 -单元测试嵌入在代码库中，通常放置在被测试逻辑的旁边。它们使用 Rust 的 `#[test]` 属性编写，并可以使用 `cargo nextest run` 运行。 +单元测试通常放在被测逻辑旁边，使用 Rust 的 `#[test]` 属性编写。GreptimeDB 主要使用 [`cargo-nextest`](https://nexte.st/) 运行 Rust 测试。 -GreptimeDB 代码库不支持默认的 `cargo` 测试运行器。推荐使用 [`nextest`](https://nexte.st/)。你可以通过以下命令安装它： +安装命令如下： ```shell cargo install cargo-nextest --locked ``` -然后运行测试（这里 `--workspace` 不是必须的） +开发时先运行本次修改的 package： ```shell -cargo nextest run +cargo nextest run -p ``` -注意，如果你的 Rust 是通过 `rustup` 安装的，请确保使用 `cargo` 安装 `nextest`，而不是像 `homebrew` 这样的包管理器，否则会弄乱你的本地环境。 +可以继续使用测试名称或 nextest filter 缩小范围。影响范围较广的改动在提交前运行完整 workspace 测试： + +```shell +make test +``` ## 覆盖率 -我们的持续集成（CI）作业有一个“覆盖率检查”步骤。它会报告有多少代码被单元测试覆盖。请在你的补丁中添加必要的单元测试。 +CI 会报告单元测试覆盖率。测试应覆盖本次改变的行为和可能回归的失败路径，而不是只追求覆盖率数字。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/reference/sql/create.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/reference/sql/create.md index f7fb21170d..cb2d9bdb30 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/reference/sql/create.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/reference/sql/create.md @@ -26,7 +26,7 @@ CREATE DATABASE [IF NOT EXISTS] db_name [WITH ] 数据库也可以通过使用 `WITH` 关键字配置与 `CREATE TABLE` 语句类似的选项。数据库支持以下选项： - `ttl` - 数据库中所有表的数据存活时间（不能设置为 `instant`） -- `memtable.type` - 内存表类型（`time_series`、`partition_tree`） +- `memtable.type` - memtable 类型（`bulk`、`time_series`） - `append_mode` - 数据库中的表是否为仅追加模式（`true`/`false`） - `merge_mode` - 合并重复行的策略（`last_row`、`last_non_null`） - `skip_wal` - 是否为数据库中的表禁用预写日志（`'true'`/`'false'`） @@ -74,7 +74,7 @@ CREATE DATABASE test WITH (ttl='7d'); ```sql CREATE DATABASE test WITH ( ttl='30d', - 'memtable.type'='partition_tree', + 'memtable.type'='bulk', 'append_mode'='true' ); ``` @@ -156,7 +156,7 @@ GreptimeDB 提供了丰富的索引实现来加速查询，请在[索引](/user- | `compaction.twcs.trigger_file_num` | 某个窗口内触发 compaction 的最小文件数量阈值 | 字符串值，如 '8'。只在 `compaction.type` 为 `twcs` 时可用 | | `compaction.twcs.time_window` | Compaction 时间窗口 | 字符串值，如 '1d' 表示 1 天。该表会根据时间戳将数据分区到不同的时间窗口中。只在 `compaction.type` 为 `twcs` 时可用 | | `compaction.twcs.max_output_file_size` | TWCS compaction 的最大输出文件大小 | 字符串值，如 '1GB'、'512MB'。设置 TWCS compaction 产生的文件的最大大小。只在 `compaction.type` 为 `twcs` 时可用 | -| `memtable.type` | memtable 的类型 | 字符串值，支持 `time_series`，`partition_tree` | +| `memtable.type` | memtable 类型 | 字符串值：`bulk` 或 `time_series`。未设置时，Mito 根据 SST format 选择实现；默认的 flat format 使用 `bulk`。设置 `bulk` 会强制使用 `sst_format=flat`；使用 flat SST 时，即使设置了 `time_series`，Mito 也会选择 bulk 实现。旧值 `partition_tree` 仅为兼容保留，并映射到 bulk 和 flat 路径。 | | `append_mode` | 该表是否时 append-only 的 | 字符串值。默认值为 'false'，根据 'merge_mode' 按主键和时间戳删除重复行。设置为 'true' 可以开启 append 模式和创建 append-only 表，保留所有重复的行 | | `merge_mode` | 合并重复行的策略 | 字符串值。只有当 `append_mode` 为 'false' 时可用。默认值为 `last_row`，保留相同主键和时间戳的最后一行。设置为 `last_non_null` 则保留相同主键和时间戳的最后一个非空字段。 | | `sst_format` | SST 文件的格式 | 字符串值，支持 `primary_key`，`flat`。默认为 `flat`。`flat` 格式建议用于具有高基数主键的表。 | diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/user-guide/deployments-administration/configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/user-guide/deployments-administration/configuration.md index faaf932d65..c6b52a42e9 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/user-guide/deployments-administration/configuration.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/user-guide/deployments-administration/configuration.md @@ -481,20 +481,9 @@ create_on_compaction = "auto" apply_on_query = "auto" mem_threshold_on_create = "64M" intermediate_path = "" - -[region_engine.mito.memtable] -type = "time_series" ``` -此外，`mito` 也提供了一个实验性质的 memtable。该 memtable 主要优化大量时间序列下的写入性能和内存占用。其查询性能可能会不如默认的 `time_series` memtable。 - -```toml -[region_engine.mito.memtable] -type = "partition_tree" -index_max_keys_per_shard = 8192 -data_freeze_threshold = 32768 -fork_dictionary_bytes = "1GiB" -``` +Mito 根据表选项和 SST format 为每个 Region 选择 memtable 实现。`default_flat_format` 为 `true` 时，没有显式设置 `sst_format` 的 Region 使用 flat SST 和 bulk memtable。`memtable.type` 是数据库或表选项，不是 `[region_engine.mito.memtable]` 引擎配置。详见[表选项](/reference/sql/create.md#表选项)。 以下是可供使用的选项 @@ -528,7 +517,7 @@ fork_dictionary_bytes = "1GiB" | `scan_memory_on_exhausted` | 字符串 | `fail` | 扫描内存耗尽时的行为。选项：`fail`（快速失败），`wait` 或 `wait()`（等待内存）。 | | `min_compaction_interval` | 字符串 | `0m` | 两次 compaction 之间的最小时间间隔。设为 "0m"（默认）允许 compactions 立即运行，无限制。 | | `schedule_compaction_after_edit` | 布尔值 | `true` | 是否允许在成功的 region edit 之后调度 compaction。
设为 `true` 是在 region edit 后调度 compaction 的必要但不充分条件，`min_compaction_interval` 等其他约束仍可能阻止 compaction 被调度。
设为 `false` 则保证 region edit 后不会调度 compaction。 | -| `default_flat_format` | 布尔值 | `true` | 是否启用 Flat 格式作为默认 SST 格式。 | +| `default_flat_format` | 布尔值 | `true` | 没有显式设置 `sst_format` 的 Region 是否使用 flat SST。Flat SST 使用 bulk memtable。 | | `scan_parallelism` | 整数 | `0` | （已弃用，请使用 `max_concurrent_scan_files`）旧版扫描并发度选项。 | | `index` | -- | -- | Mito 引擎中索引的选项。 | | `index.aux_path` | 字符串 | `""` | 文件系统中索引的辅助目录路径，用于存储创建索引的中间文件和搜索索引的暂存文件，默认为 `{data_home}/index_intermediate`。为了向后兼容，该目录的默认名称为 `index_intermediate`。此路径包含两个子目录：- `__intm`: 用于存储创建索引时使用的中间文件。- `staging`: 用于存储搜索索引时使用的暂存文件。 | @@ -543,10 +532,6 @@ fork_dictionary_bytes = "1GiB" | `inverted_index.apply_on_query` | 字符串 | `auto` | 是否在查询时使用索引
- `auto`: 自动
- `disable`: 从不 | | `inverted_index.mem_threshold_on_create` | 字符串 | `64M` | 创建索引时如果超过该内存阈值则改为使用外部排序
设置为空会关闭外排，在内存中完成所有排序 | | `inverted_index.intermediate_path` | 字符串 | `""` | 存放外排临时文件的路径 (默认 `{data_home}/index_intermediate`). | -| `memtable.type` | 字符串 | `time_series` | Memtable type.
- `time_series`: time-series memtable
- `partition_tree`: partition tree memtable (实验性功能) | -| `memtable.index_max_keys_per_shard` | 整数 | `8192` | 一个 shard 内的主键数
只对 `partition_tree` memtable 生效 | -| `memtable.data_freeze_threshold` | 整数 | `32768` | 一个 shard 内写缓存可容纳的最大行数
只对 `partition_tree` memtable 生效 | -| `memtable.fork_dictionary_bytes` | 字符串 | `1GiB` | 主键字典的大小
只对 `partition_tree` memtable 生效 | `metric` 引擎针对包含大量小表的 metrics 数据进行了优化： diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/data-persistence-indexing.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/data-persistence-indexing.md index e0ef9e8e54..e0ce16721d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/data-persistence-indexing.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/data-persistence-indexing.md @@ -5,19 +5,23 @@ description: 介绍了 GreptimeDB 的数据持久化和索引机制，包括 SST # 数据持久化与索引 -与所有类似 LSMT 的存储引擎一样，MemTables 中的数据被持久化到耐久性存储，例如本地磁盘文件系统或对象存储服务。GreptimeDB 采用 [Apache Parquet][1] 作为其持久文件格式。 +与其他 LSM-tree 存储引擎类似，GreptimeDB 将 memtable 中的数据持久化到本地文件系统或对象存储，并使用 [Apache Parquet][1] 作为持久化文件格式。 ## SST 文件格式 Parquet 是一种提供快速数据查询的开源列式存储格式，已经被许多项目采用，例如 Delta Lake。 -Parquet 具有层次结构，类似于“行组 - 列-数据页”。Parquet 文件中的数据被水平分区为行组（row group），在其中相同列的所有值一起存储以形成数据页（data pages）。数据页是最小的存储单元。这种结构极大地提高了性能。 +Parquet 按 row group、column chunk 和 page 组织数据。每个 row group 为每一列保存一个 column chunk，每个 column chunk 再包含一个或多个 page。Page 是编码和压缩单元，读取指定列时则以 column chunk 为 I/O 单元。 首先，数据按列聚集，这使得文件扫描更加高效，特别是当查询只涉及少数列时，这在分析系统中非常常见。 -其次，相同列的数据往往是同质的（比如具备近似的值），这有助于在采用字典和 Run-Length Encoding（RLE）等技术进行压缩。 +其次，同一列中的值通常比较相似，有利于字典编码和 Run-Length Encoding（RLE）等压缩技术发挥作用。 -Parquet file format +下面这张来自 Apache Parquet 规范的图进一步展示了物理文件布局：column chunk 按 row group 写入，文件元数据及其长度则保存在 footer 中。 + +Apache Parquet 文件布局 + +*来源：Apache Parquet [FileLayout.gif](https://github.com/apache/parquet-format/blob/master/doc/images/FileLayout.gif)。Copyright 2014 The Apache Software Foundation，依据 [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) 使用。* ## 数据持久化 @@ -26,18 +30,18 @@ GreptimeDB 提供了 `region_engine.mito.global_write_buffer_size` 的配置项 ## SST 文件中的索引数据 -Apache Parquet 文件格式在列块和数据页的头部提供了内置的统计信息，用于剪枝和跳过。 +Parquet 在每个 column chunk 的元数据中保存 row group 级列统计信息，例如最小值、最大值和 null 数量。Page 元数据和可选的 column index 可以提供粒度更细的统计信息。 -Column chunk header +![查询 name 列时，Parquet 列统计信息排除了一个 row group，并将另一个保留为待读取对象。](/parquet-row-group-statistics.zh.svg) -例如，在上述 Parquet 文件中，如果你想要过滤 `name` 等于 `Emily` 的行，你可以轻松跳过行组 0，因为 `name` 字段的最大值是 `Charlie`。这些统计信息减少了 IO 操作。 +例如，查询 `name` 等于 `Emily` 的行时，可以跳过 row group 0，因为其中 `name` 的最大值是 `Charlie`，无需读取该 row group。 ## 索引文件 -对于每个 SST 文件，GreptimeDB 不但维护 SST 文件内部索引，还会单独生成一个文件用于存储针对该 SST 文件的索引结构。 +当一个 SST 存在已配置且适用的索引输出时，GreptimeDB 将这些索引写入与该 SST 关联的 Puffin 文件。没有适用索引的 SST 不需要生成 Puffin 文件。 -索引文件采用 [Puffin][3] 格式，这种格式具有较大的灵活性，能够存储更多的元数据，并支持更多的索引结构。 +Puffin 是索引 Blob 及其元数据的容器，使不同索引结构可以共用一个文件。 ![Puffin](/puffin.png) @@ -58,13 +62,13 @@ GreptimeDB 会将多种索引结构作为 Blob 存储在 Puffin 文件中，包 ![Inverted index searching](/inverted-index-searching.png) -例如，上述查询使用倒排索引来定位数据段，数据段满足条件：`job` 等于 `apiserver`，`handler` 符合正则匹配 `.*users` 及 `status` 符合正则匹配 `4..`，然后扫描这些数据段以产生满足所有条件的最终结果，从而显着减少 IO 操作的次数。 +上述查询使用倒排索引定位 `job` 等于 `apiserver`、`handler` 匹配 `.*users` 且 `status` 匹配 `4..` 的数据段。Mito 只扫描这些数据段，再应用剩余过滤条件。 ### 倒排索引格式 -![Inverted index format](/inverted-index-format.png) +![倒排索引 Blob 先保存各列索引，再保存 footer 元数据；每个列索引包含 null bitmap、posting bitmap 和 FST。](/inverted-index-blob-layout.zh.svg) -GreptimeDB 按列构建倒排索引，每个倒排索引包含一个 FST 和多个 Bitmap。 +GreptimeDB 按列构建倒排索引。每个列索引包含一个 null bitmap、多个 posting bitmap 和一个 FST。Blob footer 记录定位和解码各列索引所需的 offset、size 和元数据。 FST（Finite State Transducer）允许 GreptimeDB 以紧凑的格式存储列值到 Bitmap 位置的映射，并且提供了优秀的搜索性能和支持复杂搜索（例如正则表达式匹配）；Bitmap 则维护了数据段 ID 列表，每个位表示一个数据段。 @@ -80,7 +84,7 @@ GreptimeDB 把一个 SST 文件分割成多个索引数据段，每个数据段 ## 统一数据访问层：OpenDAL -GreptimeDB 使用 [OpenDAL][2] 提供统一的数据访问层，因此，存储引擎无需与不同的存储 API 交互，数据可以无缝迁移到基于云的存储，如 AWS S3。 +GreptimeDB 使用 [OpenDAL][2] 为本地文件系统和对象存储提供统一访问层。修改配置的存储 backend 不会迁移已有数据。 [1]: https://parquet.apache.org [2]: https://github.com/datafuselabs/opendal diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/memtable.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/memtable.md new file mode 100644 index 0000000000..3905a1aecd --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/memtable.md @@ -0,0 +1,100 @@ +--- +keywords: [memtable, Mito engine, 写缓冲, flush, 时间分区, BulkMemtable] +description: 介绍 Mito 如何在 memtable 中组织 Region 的可变数据，以及如何将这些数据写入 SST 文件。 +--- + +# Memtable 设计 + +Memtable 是 Mito 为每个 Region 维护的内存写缓冲。数据 flush 为 SST 文件前，读取可以先从 memtable 获取这些数据。Region version 确定 scan 可以读取的 memtable 和 SST 文件；配合 committed sequence 上限，scan 可以在写入和 flush 推进当前 version 时保持一致性。 + +## 写入和 flush 生命周期 + +对于使用 WAL 的常规写入，Mito 按以下顺序处理： + +```text +写请求 + | + v +追加 WAL -> mutable memtable -> 发布 committed sequence + | + freeze + v + immutable memtable -> 写入 SST -> manifest edit +``` + +Region worker 先分配 sequence number 和 WAL entry ID，再将 mutation 追加到[预写日志](wal.md)。如果追加失败，Mito 不会更新 memtable。Memtable 更新成功后，Mito 发布 committed sequence，新的读取随后可以看到这些数据。配置了 `skip_wal` 的 Region 会跳过 WAL，但 memtable 更新和可见性顺序不变。 + +Flush 在启动后台 SST 写入前，先冻结 mutable memtable 并安装一组新的 mutable memtable。后续写入可以继续进行，也不会修改已经冻结的数据。Flush 将 immutable memtable 写为 SST 文件，再持久化包含新文件、flushed WAL checkpoint 和 sequence checkpoint 的 manifest edit。只有 manifest edit 持久化成功后，Mito 才会从当前 Region version 中移除已 flush 的 memtable。Flush 失败时，immutable memtable 会保留，供后续任务重试。 + +## Region version 和时间分区 + +每个 Region 包含一个 mutable `TimePartitions` 容器，其中可以有多个 memtable： + +```text +Region version +├─ mutable TimePartitions +│ ├─ [t0, t1) -> memtable +│ └─ [t1, t2) -> memtable +├─ immutable memtables +└─ SST files +``` + +Mito 根据 time index 的值把每行数据路由到对应分区。分区使用左闭右开的时间范围，并按照固定时长对齐。该时长跟随 Region 的 compaction time window；在取得 compaction time window 前，Mito 使用一天作为初始值。乱序写入可能在最新分区之外创建更早的分区。 + +冻结 Region 时，Mito 会同时冻结所有 mutable 时间分区，把其中的 memtable 移入 immutable 列表，再创建新的 `TimePartitions` 容器。Flush 失败可能留下多代 immutable memtable，因此读取和后续 flush 不能假定列表中只有一个对象。 + +## Memtable 实现 + +Mito 根据 Region 的 SST format、primary key encoding 和 memtable 选项选择实现： + +```text +flat SST format（默认）或 sparse primary-key encoding -> BulkMemtable +memtable.type=bulk -> BulkMemtable，并强制 flat SST +primary_key SST + dense encoding（遗留） -> 遗留实现 +``` + +使用默认 engine 配置时，没有显式指定 SST format 的 Region 会使用 `flat`，因此通常走 `BulkMemtable` 路径，本页其余内容也以它为准。这些规则用于排除不兼容的组合：flat format 或 sparse primary key encoding 必须使用 `BulkMemtable`；显式选择 bulk 实现则会强制使用 flat format。 + +### BulkMemtable + +`BulkMemtable` 使用 flat Arrow 布局把写入保存为 part，而不是将数据行插入按时间序列组织的缓冲区： + +```text +BulkMemtable +├─ unordered_part +│ └─ 小批量 BulkPart +└─ parts + ├─ BulkPart (Arrow RecordBatch) + ├─ MultiBulkPart (未编码的 RecordBatch) + └─ EncodedBulkPart (内存中的 Parquet 数据) +``` + +小 part 先积累在 `unordered_part` 中，较大的 part 则直接进入 `parts`。后台 memtable compaction 对符合条件的 part 执行 merge sort，生成 `MultiBulkPart` 或编码为 `EncodedBulkPart`。Scan 利用 part 的统计信息裁剪 range；flush 可以将已编码的 range 写入 SST，无需再次解码和编码数据行。设计动机和性能数据见 [Scaling Time Series to Millions of Cardinalities: GreptimeDB's Flat Format](https://greptime.cn/blogs/2025-12-22-flat-format)。 + +### 遗留实现 + +使用遗留 `primary_key` SST format 且为 dense primary key encoding 的 Region 仍然使用 `TimeSeriesMemtable`，它按编码后的 primary key 对数据行分组，而不是保存 flat part。如果 Region 没有 primary key 列，同一个 builder 会创建 `SimpleBulkMemtable`。两者都是为已有表保留的兼容代码，`primary_key` format 退役后可能一并移除；新的工作应面向 bulk 和 flat 路径。 + +已经删除的 `partition_tree` memtable 不是第三种实现。Option parser 仍接受 `memtable.type=partition_tree` 以兼容旧配置，但不会恢复该实现。Region 最终使用 bulk 和 flat 路径。 + +## 读取快照 + +Scan 通过一次 `VersionControl` 快照同时取得 Region version 和 committed sequence，并先确定 version，再应用 sequence 上限。如果单独读取 sequence 后再获取 version，flush 或 compaction 可能在两次读取之间移除旧输入，使 scan 得到不完整的快照。 + +选定的 version 提供 mutable memtable、immutable memtable 和 SST 文件。Mito 先按照时间范围裁剪数据源，再使用 scan 的 projection、predicate 和 sequence range 从各 memtable 获取 range。Scan 将这些 range 与 SST range 合并，并在所有数据源上应用相同的排序、删除和 merge 语义。即使新的 Region version 已经移除某个 memtable，scan 持有的引用也会让该 memtable 存活到本次读取结束。 + +## 内存压力 + +每个 memtable 通过 engine 的 write-buffer manager 记录估算的堆内存分配量。冻结 memtable 后，这部分内存不再计入 mutable memory，但在所有引用释放前仍计入总用量。因此，mutable memory 只反映仍可接收写入的数据，总用量仍包含活跃 scan 保留的内存。 + +全局 write-buffer 达到限制后，worker 会选择 Region 执行 flush。如果内存用量持续超过配置限制，Mito 会阻塞写入，并在达到更高阈值后拒绝写入。可选的 Region 级限制会单独约束热点 Region，避免其阻塞无关 Region。定期任务、手动请求和 Region 生命周期操作也可以触发 flush。 + +## 修改约束 + +修改 memtable 代码时必须保持以下性质： + +- 对于使用 WAL 的 Region，先追加 WAL，再把数据安装到 memtable；只有安装成功后才能发布 committed sequence。 +- SST 文件和 manifest edit 持久化前，冻结的 memtable 必须保持可读，并能在 flush 失败后重试。 +- 从同一个 `VersionControl` 快照取得 Region version 和 committed sequence，不得先单独读取 sequence 再获取 version。 +- 保留 scan 和 flush 在 memtable range 与 SST range 之间执行统一删除、去重和 merge 所需的排序及元数据。 +- 通过 write-buffer manager 记录内存分配，并且只在底层内存不再可能被引用时释放计数。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/metric-engine.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/metric-engine.md index c0cbf681b9..57baecf400 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/metric-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/metric-engine.md @@ -7,9 +7,9 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述 ## 概述 -`Metric` 引擎是 GreptimeDB 的一个组件，属于存储引擎的一种实现，主要针对可观测 metrics 等存在大量小表的场景。 +`Metric` 引擎用于存储包含大量小型指标表的负载。 -它的主要特点是利用合成的物理宽表来存储大量的小表数据，实现相同列复用和元数据复用等效果，从而达到减少小表的存储开销以及提高列式压缩效率等目标。表这一概念在 `Metric` 引擎下变得更更加轻量。 +它将这些逻辑表映射到共享的物理宽表，使其复用列和元数据，从而降低每张表的存储开销并改善列式压缩。 ## 概念 @@ -18,7 +18,7 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述 ### 逻辑表 逻辑表，即用户定义的表。与普通的表都完全一样，逻辑表的定义包括表的名称、列的定义、索引的定义等。用户的查询、写入等操作都是基于逻辑表进行的。用户在使用过程中不需要关心逻辑表和普通表的区别。 -从实现层面来说，逻辑表是一个虚拟的表，它并不直接读写物理的数据，而是通过将读写请求映射成对应物理表的请求来实现数据的存储与查询。 +逻辑表是虚拟表，本身不直接存储数据。Metric 引擎将其读写请求映射为对应物理表的请求。 ### 物理表 物理表是真实存储数据的表，它拥有若干个由分区规则定义的物理 Region。 @@ -27,16 +27,14 @@ description: 介绍了 Metric 引擎的概念、架构及设计，重点描述 `Metric` 引擎的主要设计架构如下： -![Arch](/metric-engine-arch.png) +![多个逻辑表通过 Metric 引擎映射到由 Mito 管理的共享数据 Region 和元数据 Region。](/metric-engine-architecture.zh.svg) -在目前版本的实现中，`Metric` 引擎复用了 `Mito` 引擎来实现物理数据的存储及查询能力，并在此之上同时提供物理表与逻辑表的访问能力。 +`Metric` 引擎将物理存储和查询交给 `Mito` 引擎。每个物理 Region 组包含一个数据 Region 和一个元数据 Region：数据 Region 保存映射到该 Region 组的逻辑表数据，元数据 Region 保存逻辑表及逻辑列的映射。 -在分区方面，逻辑表拥有与物理表完全一致的分区规则及 Region 分布。这是非常自然的，因为逻辑表的数据直接存储在物理表中，所以分区规则也是一致的。 +关联到同一物理表的逻辑表使用相同的分区布局。写入时，Metric 引擎为每行数据记录逻辑表身份；读取时，它在扫描物理 Region 前增加逻辑表过滤条件。 -在路由元数据方面，逻辑表的路由地址为逻辑地址，即该逻辑表所对应的物理表是什么，而后通过该物理表进行二次路由取得真正的物理地址。这一间接路由方式能够显著减少 `Metric` 引擎的 Region 发生迁移调度时所需要修改的元数据数量。 +逻辑表的路由只保存所属物理表的 ID，再由物理表路由解析出持有 Region 的 Datanode。逻辑路由本身不记录 peer，因此迁移物理 Region 只需改写一条物理路由，而不必改写映射到它的每一条逻辑路由。 -在操作方面，`Metric` 引擎支持对逻辑表进行标准的 DML 操作（INSERT、DELETE、SELECT）。然而，对物理表的操作进行了有限的支持以防止误操作，例如禁止直接写入物理表等操作防止影响用户逻辑表的数据。总体上可以认为物理表是对用户只读的。 +逻辑表支持普通的 INSERT、DELETE 和 SELECT 操作。直接写入物理 Region 会绕过逻辑表映射，因此会被拒绝；物理表仍然可以查询。 -为了提升对大量表同时进行 DDL（Data Definition Language，数据操作语言）操作时性能，如 Prometheus Remote Write 冷启动时大量 metrics 带来的自动建表请求，以及前面提到的迁移物理 Region 时大量路由表的修改请求等，`Metric` 引擎引入了一些批量 DDL 操作。这些批量 DDL 操作能够将大量的 DDL 操作合并成一个请求，从而减少了元数据的查询及修改次数，提升了性能。 - -除了物理表的物理数据 Region 之外，`Metric` 引擎还额外为每一个物理数据 Region 创建了一个物理的元数据 Region，用于存储 `Metric` 引擎自身为了维护映射等状态所需要的一些元数据。这些元数据包括逻辑表与物理表的映射关系，逻辑列与物理列的映射关系等等。 +批量 DDL 用于减少大量逻辑表同时创建或更新时的元数据操作，例如 Prometheus Remote Write 自动建表或物理 Region 迁移。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/overview.md index 3f44a4543b..c7684eb165 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/overview.md @@ -7,22 +7,26 @@ description: 介绍了 Datanode 的主要职责和组件，包括 gRPC 服务、 ## Introduction -`Datanode` 主要的职责是为 GreptimeDB 存储数据，我们知道在 GreptimeDB 中一个 `table` 可以有一个或者多个 `Region`, -而 `Datanode` 的职责便是管理这些 `Region` 的读写。`Datanode` 不感知 `table`，可以认为它是一个 `region server`。 -所以 `Frontend` 和 `Metasrv` 按照 `Region` 粒度来操作 `Datanode`。 +Datanode 存储并处理 Region 数据。一张表可以包含多个 Region，但 Datanode 不负责表级路由。Frontend 按 Region 发送数据请求，Metasrv 则控制 Region 的放置和生命周期。 -![Datanode](/datanode.png) +这个边界使同一个 Region server 可以承载不同的存储引擎，而不向 Frontend 或 Metasrv 暴露引擎实现。 + +![Frontend 向 Datanode Region server 发送 Region 请求，Metasrv 通过 heartbeat task 与 Datanode 交换生命周期指令。Region server 使用本地 query engine，并将请求分发给 Mito、Metric 或 File Region engine。](/datanode-architecture.zh.svg) ## Components -一个 datanode 包含了 region server 所需的全部组件。这里列出了比较重要的部分： - -- 一个 gRPC 服务来提供对 `Region` 数据的读写，`Frontend` 便是使用这个服务来从 `Datanode` 读写数据。 -- 一个 HTTP 服务，可以通过它来获得当前节点的 metrics、配置信息等 -- `Heartbeat Task` 用来向 `Metasrv` 发送心跳，心跳在 GreptimeDB 的分布式架构中发挥着至关重要的作用， - 是分布式协调和调度的基础通信通道，心跳的上行消息中包含了重要信息比如 `Region` 的负载，如果 `Metasrv` 做出了调度 - 决定（比如 Region 转移），它会通过心跳的下行消息发送指令到 `Datanode` -- `Datanode` 不负责解析用户 SQL 或进行分布式规划，用户对一个或多个 `Table` 的查询请求会在 `Frontend` 中被转换为 - `Region` 查询请求，`Datanode` 负责用本地 query engine 执行这些 `Region` 查询计划 -- 一个 `Region Manager` 用来管理 `Datanode` 上的所有 `Region`s -- GreptimeDB 支持可插拔的多引擎架构，目前已有的 engine 包括 `File Engine` 和 `Mito Engine` +Datanode 包含以下主要组件： + +- Region server 记录已打开的 Region，并把读写和生命周期请求分发给该 Region 注册的 engine。 +- `Mito` 是主要的时序 Region engine。`Metric` 将多个逻辑指标 Region 映射到共享的 Mito Region，`File` 通过 Region 接口访问外部文件。 +- 本地 query engine 执行 Region 查询计划。它不解析客户端 SQL，也不进行集群级规划。 +- Heartbeat task 向 Metasrv 上报节点和 Region 状态，并接收 open、close、upgrade、downgrade 和迁移步骤等指令。 +- gRPC 承载发往 Datanode 的 Region 请求；HTTP 提供 metrics 和配置等节点诊断信息。 + +## Region 请求生命周期 + +Mito 写入到达 Region server 后，Region server 根据 Region 元数据选择 Mito。Mito 将 mutation 追加到 WAL，写入 memtable，并在之后把 memtable flush 为 SST 文件。Metric 写入会先补充逻辑表标识，再委托给对应的物理 Mito Region。 + +读取时，本地 query engine 在 Region engine 提供的 table provider 上执行 Region 计划。Mito scan 获取不可变的 Region version，读取相关 memtable 和 SST 文件，合并并去重数据，最后返回 Arrow record batch 流。 + +Region 所有权可以在不重启 Datanode 的情况下改变。Metasrv 通过心跳流下发生命周期指令；Region server 将指令应用到对应 engine，并在后续心跳中上报新的 Region role 和统计信息。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/python-scripts.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/python-scripts.md deleted file mode 100644 index 831278e80d..0000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/python-scripts.md +++ /dev/null @@ -1,30 +0,0 @@ ---- -keywords: [Python 脚本, 数据分析, CPython, RustPython] -description: 介绍了在 GreptimeDB 中使用 Python 脚本进行数据分析的两种后端实现：CPython 和嵌入式 RustPython 解释器。 ---- - -# Python 脚本 - -## 简介 - -Python 脚本是分析本地数据库中的数据的便捷方式， -通过将脚本直接在数据库内运行而不是从数据库拉取数据的方式，可以节省大量的数据传输时间。 -下图描述了 Python 脚本的工作原理。 -`RecordBatch`（基本上是表中的一列，带有类型和元数据）可以来自数据库中的任何地方， -而返回的 `RecordBatch` 可以用 Python 语法注释以指示其元数据，例如类型或空。 -脚本将尽其所能将返回的对象转换为 `RecordBatch`，无论它是 Python 列表、从参数计算出的 `RecordBatch` 还是常量（它被扩展到与输入参数相同的长度）。 - -![Python Coprocessor](/python-coprocessor.png) - -## 两种可选的后端 - -### CPython 后端 - -该后端由 [PyO3](https://pyo3.rs/v0.18.1/) 提供支持，可以使用您最喜欢的 Python 库（如 NumPy、Pandas 等），并允许 Conda 管理您的 Python 环境。 - -但是使用它也涉及一些复杂性。您必须设置正确的 Python 共享库，这可能有点棘手。一般来说，您只需要安装 `python-dev` 包。但是，如果您使用 Homebrew 在 macOS 上安装 Python，则必须创建一个适当的软链接到 `Library/Frameworks/Python.framework`。有关使用 PyO3 crate 与不同 Python 版本的详细说明，请参见 [这里](https://pyo3.rs/v0.18.1/building_and_distribution#configuring-the-python-version) - -### 嵌入式 RustPython 解释器 - -可以运行脚本的实验性 [python 解释器](https://github.com/RustPython/RustPython)，它支持 Python 3.10 语法。您可以使用所有的 Python 语法，更多信息请参见 [Python 脚本的用户指南](/user-guide/python-scripts/overview.md). - diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/query-engine.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/query-engine.md index 62d7563bb2..84445759d4 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/query-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/query-engine.md @@ -7,33 +7,33 @@ description: 介绍了 GreptimeDB 的查询引擎架构，基于 Apache DataFusi ## 介绍 -GreptimeDB 的查询引擎是基于[Apache DataFusion][1]（属于[Apache Arrow][2]的子项目）构建的，它是一个用 Rust 编写的出色的查询引擎。它提供了一整套功能齐全的组件，从逻辑计划、物理计划到执行运行时。下面将解释每个组件如何被整合在一起，以及在执行过程中它们的位置。 +GreptimeDB 的查询引擎基于 [Apache DataFusion][1]。DataFusion 提供逻辑计划、物理计划、优化器框架和执行运行时；GreptimeDB 在此基础上增加各查询语言的 planner、存储相关优化规则、自定义计划节点和分布式执行。 -![Execution Procedure](/execution-procedure.png) +DDL 和其他控制面操作由 statement executor 分发。Query engine 接收数据处理计划，包括 `INSERT ... SELECT` 等操作中读取输入数据的部分。 -入口点是逻辑计划，它被用作查询或执行逻辑等的通用中间表示。逻辑计划的两个主要来源是：1. 用户查询，例如通过 SQL 解析器和规划器的 SQL；2. Frontend 的分布式查询，这将在下一节中详细解释。 +## 查询生命周期 -接下来是物理计划，或称执行计划。与包含所有逻辑计划变体（除特殊扩展计划节点外）的大型枚举的逻辑计划不同，物理计划实际上是一个定义了在执行过程中调用的一组方法的特性。所有数据处理逻辑都包装在实现该特性的相应结构中。它们是对数据执行的实际操作，如聚合器 `MIN` 或 `AVG` ，以及表扫描 `SELECT ... FROM`。 +1. SQL、PromQL 或日志查询 planner 通过 catalog 解析表，并生成 DataFusion logical plan。DataFusion 不直接支持的操作由 GreptimeDB plan extension 表示。 +2. DataFusion 的 analyzer 和 optimizer rule 与 GreptimeDB rule 共同运行。这些规则规范化表达式和类型、改写时间范围操作、将 projection 和 filter 下推到 scan，并在需要时引入分布式计划节点。 +3. Physical planner 将优化后的 logical plan 转换为流式 operator。GreptimeDB 随后应用 scan 并行度、排序和分布式执行相关的 physical rule。 +4. 执行阶段通过 physical plan 拉取 Arrow record batch。存储 scan 接收 projection 和 predicate，下游 operator 消费数据流，无需先物化完整结果。 -优化阶段通过转换逻辑计划和物理计划来提高执行性能，现在全部基于规则。它也被称为“基于规则的优化”。一些规则是 DataFusion 原生的，其他一些是在 GreptimeDB 中自定义的。在未来，我们计划添加更多规则，并利用数据统计进行基于成本的优化 (CBO)。 - -最后一个阶段"执行"是一个动词，代表从存储读取数据、进行计算并生成预期结果的过程。虽然它比之前提到的概念更抽象，但你可以简单地将它想象为执行一个 Rust 异步函数，并且它确实是一个异步流。 - -当你想知道 SQL 是如何通过逻辑计划或物理计划中表示时，`EXPLAIN [VERBOSE] ` 是非常有用的。 +使用 [`EXPLAIN`](/reference/sql/explain.md) 查看逻辑和物理计划。`EXPLAIN ANALYZE` 还会执行计划并报告运行时指标。 ## 数据表示 -GreptimeDB 使用 [Apache Arrow][2]作为内存中的数据表示格式。它是面向列的，以跨平台格式，也包含许多高性能的基础操作。这些特性使得在许多不同的环境中共享数据和实现计算逻辑变得容易。 +GreptimeDB 使用 [Apache Arrow][2] record batch 作为内存数据表示。一个 record batch 包含等长的列数组和 schema。查询 operator 交换这些 batch 组成的数据流，使 Region scan 到结果编码的执行路径保持列式处理。 ## 索引 -在时序数据中，有两个重要的维度：时间戳和标签列（或者类似于关系数据库中的主键）。GreptimeDB 将数据分组到时间桶中，因此能在非常低的成本下定位和提取预期时间范围内的数据。GreptimeDB 中主要使用的持久文件格式 [Apache Parquet][3] 提供了多级索引和过滤器，使得在查询过程中很容易修剪数据。在未来，我们将更多地利用这个特性，并开发我们的分离索引来处理更复杂的用例。 +索引构建和持久化格式属于存储引擎。查询层向 scan 提供 predicate 和 projection，Mito 再利用时间范围、Parquet 统计信息和索引跳过不可能匹配的数据。参见[数据持久化与索引](./data-persistence-indexing.md)。 + + -## 分布式查询 +## 分布式执行 -参考 [Distributed Querying][6]. +分布式模式下，Frontend 规划集群级查询，Datanode 执行 Region 本地子计划。[`MergeScan`][6] 是两个阶段之间的边界。 -[1]: https://github.com/apache/arrow-datafusion +[1]: https://datafusion.apache.org/ [2]: https://arrow.apache.org/ -[3]: https://parquet.apache.org [6]: ../frontend/distributed-querying.md diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/storage-engine.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/storage-engine.md index 67f98216fe..d3abb58cfa 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/storage-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/storage-engine.md @@ -7,7 +7,7 @@ description: 详细介绍了 GreptimeDB 的存储引擎架构、数据模型和 ## 概述 -`存储引擎` 负责存储数据库的数据。Mito 是我们默认使用的存储引擎，基于 [LSMT][1]（Log-structured Merge-tree）。我们针对处理时间序列数据的场景做了很多优化，因此 mito 这个存储引擎并不适用于通用用途。 +Mito 是 GreptimeDB 的默认存储引擎，基于 [LSM tree][1]，面向时间序列负载设计，而不是通用的嵌入式存储引擎。 ## 架构 下图展示了存储引擎的架构和处理数据的流程。 @@ -20,9 +20,9 @@ description: 详细介绍了 GreptimeDB 的存储引擎架构、数据模型和 - 为尚未刷盘的数据提供高持久性保证。 - 基于 `LogStore` API 实现，不关心底层存储介质。 - WAL 的日志记录可以存储在本地磁盘上，也可以存储在实现了 `LogStore` API 的远程日志服务中，例如 Kafka（remote WAL）。 -- Memtable - - 数据首先写入 `active memtable`，又称 `mutable memtable`。 - - 当 `mutable memtable` 已满时，它将变为只读的 `immutable memtable`。 +- [Memtable](memtable.md) + - Mito 根据 time index 将数据行写入 mutable memtable。 + - Flush 冻结 mutable memtable，安装一组新的 mutable memtable 以接收写入，再将冻结的 memtable 写为 SST 文件。 - SST - SST 的全名为有序字符串表（`Sorted String Table`）。 - `immutable memtable` 刷到持久存储后形成一个 SST 文件。 @@ -100,7 +100,9 @@ Mito 会按 primary key 对行分组，并按时间排序，因此 SST 中的数 Mito 支持两种 SST 格式：`flat` 和 `primary_key`。`flat` 是新表的默认格式，适用于各种 primary key 基数，包括高基数 key。`primary_key` 是为了兼容旧表而保留的遗留格式。更多详情请参考 [SST format](/reference/sql/create.md#创建指定-sst-格式的表) 和[表设计指南](/user-guide/deployments-administration/performance-tuning/design-table.md#sst-格式)。 -SST layout +![Mito 默认的 flat SST 布局将文件级元数据与包含数据列和合并元数据的 Parquet row group 组合在一起。](/mito-sst-layout.zh.svg) + +一个 SST 可能跨越多个 compaction time window。 ## 扫描裁剪 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/wal.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/wal.md index 529adcdf50..36e0689430 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/wal.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/datanode/wal.md @@ -9,20 +9,26 @@ description: 介绍了 GreptimeDB 的预写日志（WAL）机制，包括其命 ## 介绍 -我们的存储引擎受到了日志结构合并树（Log-structured Merge Tree，LSMT）的启发。对数据的变更操作直接应用于 MemTable 而不是持久化到磁盘上的数据页，这显著提高了性能，但也带来了持久化相关的问题，特别是在 Datanode 意外崩溃时。与所有类似 LSMT 的存储引擎一样，GreptimeDB 使用预写日志（Write-Ahead Log，WAL）来确保数据被可靠地持久化，并且保证崩溃时的数据完整性。 +Mito 在将数据 flush 为 SST 文件前，先在 [memtable](memtable.md) 中缓冲写入。每个 Region 的 mutation 会先追加到预写日志（WAL），从而恢复尚未进入 SST 的数据。 -预写日志是一个仅提供追加写的文件组。所有的 INSERT 和 DELETE 操作都被转换为操作日志，然后追加到 WAL。一旦操作日志被持久化到底层文件，该操作才可以进一步应用到 MemTable。 +WAL 通过统一的 log-store 抽象访问，可以使用本地 raft-engine 或远端 Kafka。 -当数据节点重新启动时，WAL 中的操作条目将被重放，以重建正确的 MemTable 状态。 +## 写入与恢复流程 -![WAL in Datanode](/wal.png) +正常写入遵循以下顺序： + +1. Region worker 分配 sequence number 和 WAL entry ID。 +2. 将 mutation 追加到 WAL。追加失败时，不会把 mutation 写入 memtable。 +3. WAL 追加成功后，Mito 将 mutation 写入 memtable，并发布新的 committed sequence。 +4. Flush 将不可变 memtable 写为 SST 文件，并持久化包含新文件和 `flushed_entry_id` 的 manifest edit。 +5. Manifest edit 持久化后，`flushed_entry_id` 及以前的 WAL entry 被标记为 obsolete；log store 可以稍后再回收物理空间。 + +Manifest 是恢复边界。正常重新打开 Region 时，Mito 根据 manifest 重建 Region，并从 `flushed_entry_id + 1` 开始重放 WAL。Region 状态切换可以指定更晚的 replay checkpoint，但不会重放早于已持久化 flush 边界的 entry。 ## 命名空间 -WAL 的命名空间用于区分来自不同 region 的条目。追加和读取操作必须提供一个命名空间。目前，region ID 被用作命名空间，因为每个 region 都有一个在数据节点重新启动时需要重构的 MemTable。 +WAL entry 按 Region 隔离，而不是按表隔离。追加和读取都需要指定 Region namespace，使单个 Region 可以独立重放或截断。本地 raft-engine 使用 Region ID 作为 namespace ID；Kafka provider 则在基于 topic 的日志中保留 Region 标识。 ## 同步/异步刷盘 -默认情况下，WAL 的追加写是异步的，这意味着写入方不会等待操作日志被刷入到磁盘并持久化。这个默认设置提供了更高的性能，但在服务器意外关闭时可能会丢失数据。另一方面，同步刷新提供了更高的可靠性，但其代价是性能更低。 - -在 v0.4 版本中，新的 region worker 架构可以使用批处理来减轻同步刷盘的开销。 +对于本地 raft-engine，`sync_write` 控制追加写是否等待日志同步到持久化存储，默认值为 `false`。异步写入延迟较低，但主机在缓冲数据同步前故障时，可能丢失最近确认的 entry。Kafka WAL 的持久性由 producer 和集群配置决定，不受这个本地选项控制。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/arrangement.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/arrangement.md index dd3b6de090..7b35b50259 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/arrangement.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/arrangement.md @@ -5,6 +5,8 @@ description: 描述了 Arrangement 在数据流进程中的状态存储功能， # Arrangement +本页介绍 Flownode 旧 streaming 模式使用的状态结构；batching 模式不使用 Arrangement。 + Arrangement 存储数据流进程中的状态，存储 flow 的更新流（stream）以供进一步查询和更新。 Arrangement 本质上存储的是带有时间戳的键值对。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/batching_mode.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/batching_mode.md index bec092b0af..8e2ecdb80b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/batching_mode.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/batching_mode.md @@ -9,13 +9,13 @@ description: Flownode 批处理模式概述，这是持续数据聚合当前使 ## 概述 -`flownode` 中的批处理模式专为持续数据聚合而设计。它在离散的、微小的时间窗口上周期性地执行用户定义的 SQL 查询。这与原始的流处理模式形成对比；流处理模式现在已经废弃，在该模式下数据会在到达时即被处理。 +`flownode` 中的批处理模式专为持续数据聚合而设计。它在离散的小时间窗口上周期性执行用户定义的 SQL 查询。旧 streaming 路径则在数据到达时进行处理，目前仅为兼容已有 workload 而保留，不推荐新 workload 使用。 其核心思想是： 1. 定义一个带有 SQL 查询的 `flow`，该查询将数据从源表聚合到目标表。 2. 查询通常在时间戳列上包含一个时间窗口函数（例如 `date_bin`）。 3. 当新数据插入源表时，系统会将相应的时间窗口标记为“脏”(dirty)。 -4. 一个后台任务会周期性地唤醒，识别这些脏窗口，并为那些特定的时间范围重新运行聚合查询。 +4. 一个后台任务按自身的节奏运行，在下一次求值时取出待处理的脏窗口，并对这些时间范围重新运行聚合查询。 5. 然后将结果插入到目标表中，从而有效地更新聚合视图。 ## 架构 @@ -39,15 +39,15 @@ description: Flownode 批处理模式概述，这是持续数据聚合当前使 - **状态 (`TaskState`)**: 包含任务的动态、可变状态，最重要的是 `DirtyTimeWindows`。 - **执行循环**: 任务运行一个无限循环 (`start_executing_loop`)，该循环： 1. 检查关闭信号。 - 2. 等待一个预定的时间间隔或直到被唤醒。 + 2. 睡眠到下一次求值时间。设置了求值调度的任务睡眠到下一个调度时间点；自适应任务则按时间窗口大小和最小刷新间隔计算出的轮询间隔睡眠。 3. 基于当前的脏时间窗口集合生成一个新的查询计划 (`gen_insert_plan`)。 4. 对数据库执行查询 (`execute_logical_plan`)。 5. 清理已处理的脏窗口。 ### `TaskState` 和 `DirtyTimeWindows` -- **`TaskState`**: 此结构体跟踪 `BatchingTask` 的运行时状态。它包括 `dirty_time_windows`，这对于确定需要完成哪些操作至关重要。 -- **`DirtyTimeWindows`**: 这是一个关键的数据结构，用于跟踪自上次查询执行以来哪些时间窗口接收到了新数据。它存储一组不重叠的时间范围。当任务的执行循环运行时，它会参考此结构来构建一个 `WHERE` 子句，该子句仅过滤源表中的脏时间窗口。 +- **`TaskState`**: 此结构体跟踪 `BatchingTask` 的运行时状态，包括用于确定待处理工作的 `dirty_time_windows`。 +- **`DirtyTimeWindows`**: 此数据结构跟踪上次查询执行后接收到新数据的时间窗口，并保存一组不重叠的时间范围。执行循环根据它构造 `WHERE` 子句，只从源表选择脏窗口。 ### `TimeWindowExpr` @@ -56,15 +56,15 @@ description: Flownode 批处理模式概述，这是持续数据聚合当前使 - **求值**: 它可以接受一个时间戳并对时间窗口表达式求值，以确定该时间戳所属窗口的开始和结束。 - **窗口大小**: 它还可以从表达式中确定时间窗口的大小（持续时间）。 -这对于标记窗口为脏以及在查询源表时生成正确的过滤条件都至关重要。 +标记脏窗口和生成源表过滤条件使用同一套计算。 ## 查询执行流程 以下是批处理模式下查询执行的简化分步演练： 1. **数据摄取**: 新数据被写入源表。 -2. **标记为脏**: `BatchingEngine` 收到有关新数据的通知。它使用与每个相关 flow 关联的 `TimeWindowExpr` 来确定哪些时间窗口受到新数据点的影响。然后将这些窗口添加到相应 `TaskState` 中的 `DirtyTimeWindows` 集合中。 -3. **任务唤醒**: `BatchingTask` 的执行循环被唤醒，原因可能是其周期性调度，也可能是因为它被通知有大量积压的脏窗口。 +2. **标记为脏**: `BatchingEngine` 收到有关新数据的通知。它使用与每个相关 flow 关联的 `TimeWindowExpr` 来确定哪些时间窗口受到新数据点的影响。然后将这些窗口添加到相应 `TaskState` 中的 `DirtyTimeWindows` 集合中。标记脏窗口不会唤醒任务。 +3. **下一次求值**: `BatchingTask` 的执行循环在调度时间点或自适应轮询间隔结束后进入下一次求值，取出待处理的脏窗口。 4. **计划生成**: 任务调用 `gen_insert_plan`。此方法： - 检查 `DirtyTimeWindows`。 - 生成一系列 `OR` 连接的 `WHERE` 子句（例如 `(ts >= 't1' AND ts < 't2') OR (ts >= 't3' AND ts < 't4') ...`），覆盖所有脏窗口。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/dataflow.md index 9d07922542..95def5e844 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/dataflow.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/dataflow.md @@ -1,18 +1,39 @@ --- -keywords: [Dataflow, SQL 查询, 执行计划, 数据流, map, reduce] -description: 解释了 Dataflow 模块的核心计算功能，包括 SQL 查询转换、内部执行计划、数据流的触发运行和支持的操作。 +keywords: [Flownode, batching mode, streaming mode, Dataflow, 脏时间窗口] +description: 介绍 Flownode 如何选择并运行 batching 和旧 streaming 两条执行路径。 --- # 数据流 +Flownode 内部有两条执行路径： + +- **Batching mode** 是聚合和 TQL workload 的主要执行路径。它查询已经持久化的 source 数据，并将物化结果写入 sink table。 +- **Streaming mode** 是为兼容已有 workload 而保留的旧执行路径，不推荐新 workload 使用。Frontend 会把新到达的行同步给它进行增量处理。 + +用户不能直接选择执行模式。创建 Flow 时，GreptimeDB 根据查询和 source table 的属性选择执行路径。聚合、`DISTINCT` 和 TQL 查询使用 batching mode；简单的非聚合查询，以及任何 source table 使用 `ttl = 'instant'` 的 Flow，目前仍使用 streaming mode。如果 source table 尚不存在并选择延迟创建，Flow 会先成为 pending batching Flow。 + +## Batching mode + +Batching mode 复用 GreptimeDB 的查询引擎，不需要为每一行输入维护一张算子图。对于基于时间窗口的 Flow，主循环如下： + +1. Source table 收到写入后，把受影响的时间窗口标记为 dirty。 +2. `BatchingTask` 按求值调度或自适应轮询节奏运行，并在该次求值时收集待处理的 dirty window。标记 dirty window 不会唤醒任务。 +3. 任务把这些窗口转换成时间谓词，加入 Flow 查询，再请求 Frontend 查询 source table。 +4. 查询结果写入 sink table，更新已重新计算窗口对应的物化结果。 +5. 成功处理的窗口从 dirty set 中移除；执行失败的工作仍可在后续调度中处理。 + +设置了 evaluation interval、但查询中没有时间窗口表达式的 Flow，会在每次调度时执行完整查询。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见 [Flownode 批处理模式开发者指南](./batching_mode.md)。 + +## Streaming mode + Dataflow 模块（参见 `flow::compute` 模块）是 `flow` 的核心计算模块。 它接收 SQL 查询并将其转换为 `flow` 的内部执行计划。 然后，该执行计划被转化为实际的数据流，而数据流本质上是一个由带有输入和输出端口的函数组成的有向无环图（DAG）。 -数据流会在需要时被触发运行。 +新到达的行变更会增量驱动这张图执行。 -目前该数据流只支持 `map`和 `reduce` 操作，未来将添加对 `join` 等操作的支持。 +Renderer 支持 map/filter/project 和 reduce 操作。执行计划中已经有 join 和 union 节点，但 streaming renderer 尚未实现它们。 在内部，数据流使用 `tuple(row, time, diff)` 以行格式处理数据。 这里 `row` 表示实际传递的数据，可能包含多个 `value` 对象。 `time` 是系统时间，用于跟踪数据流的进度，`diff` 通常表示行的插入或删除（+1 或 -1）。 -因此，`tuple` 表示给定系统时间的 `row` 的插入/删除操作。 +因此，`tuple` 表示给定系统时间的 `row` 的插入/删除操作。有状态算子通过 [Arrangement](./arrangement.md) 保存这些变更的索引 trace。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/distributed-querying.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/distributed-querying.md index 612174662b..e6d0c35160 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/distributed-querying.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/distributed-querying.md @@ -1,41 +1,20 @@ --- -keywords: [分布式查询, 查询拆分, 查询合并, TableScan, 物理计划] -description: 介绍 GreptimeDB 中的分布式查询方法，包括查询的拆分和合并过程，以及 TableScan 节点的作用。 +keywords: [分布式查询, 逻辑计划, MergeScan, Substrait, Region 裁剪] +description: 介绍 GreptimeDB 如何把逻辑查询计划划分为 Frontend 和 Datanode 上的执行任务。 --- # 分布式查询 -我们知道在 GreptimeDB 中数据是如何分布的（参见“[表分片][1]”），那么如何查询呢？在 GreptimeDB 中，分布式查询非常简单。简单来说，我们只需将查询拆分为子查询，每个子查询负责查询表数据的一个部分，然后将所有结果合并为最终结果。这是一种典型的“拆分 - 合并”方法。具体来说，让我们从查询到达 `frontend` 开始。 +Frontend 和 Datanode 使用同一套基于 DataFusion 的查询引擎。在分布式模式下，Frontend 会增加一个规划步骤，将 Datanode 上执行的工作与 Frontend 上完成的工作分开。 -当查询到达 `frontend` 时，它首先被解析为 SQL 抽象语法树（AST）。我们遍历 AST，并从中生成逻辑计划。顾名思义，逻辑计划只是如何“逻辑地”执行查询的“提示”，它不能被直接运行，因此我们进一步从中生成可执行的物理计划。物理计划是一种类似树形的数据结构，每个节点实际上表示查询的执行方法。一旦我们从上到下运行物理计划树，结果数据将从叶子到根流动，被合并或计算。最终，我们在根节点的输出处得到了查询的结果。 +![Frontend query](/frontend-query.png) -到目前为止，这只是一个典型的“volcano”查询执行模型，你可以在几乎每个 SQL 数据库中看到这种模型。那么“分布式”是在哪里发生的呢？这全部发生在一个名为“TableScan”的物理计划节点中。TableScan 是物理计划树中的一个叶子节点，它负责扫描表的数据（就像它的名称所暗示的）。当 `frontend` 即将扫描表时，它首先需要根据每个 `region` 的数据范围将表扫描拆分为较小的扫描。 +## 分布式规划 -[1]: ./table-sharding.md +分布式规划器重写逻辑计划，把可以下推的算子移向表扫描，并用 `MergeScan` 节点包装远端子计划。分区列上的谓词还会在任务调度前用于裁剪 Region。 -表的所有 `region` 都有它们存储数据的范围。以下表为例： +算子能否下推取决于计划形态和算子本身的性质。不支持的部分会保留在 Frontend。初始设计及交换律规则参见[分布式规划器 RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md)。 -```sql -CREATE TABLE my_table ( - a INT, - others STRING, - ts TIMESTAMP TIME INDEX, -) -PARTITION ON COLUMNS (a) ( - a < 10, - a >= 10 AND a < 20, - a >= 20 -); -``` +## 分布式计划 -`my_table` 表创建时被设定了 3 个分区。在 GreptimeDB 的当前实现中，将为该表创建 3 个 `region`（分区与 `region` 的比例为 1:1）。这 3 个区域将分别包含以下范围："[-∞, 10)", "[10, 20)" 和 "[20, +∞)"。例如，如果提供了值 "42"，我们将搜索这些范围，并找到包含该值的相应的 `region`（在此示例中为第 3 个 `region`）。 - -对于查询，我们使用“过滤器”来查找 `region`。 "过滤器"是 "WHERE" 子句中的条件。例如，查询 `SELECT * FROM my_table WHERE a < 10 AND others = 'x'`，其“过滤器”为“a < 10 AND others = 'x'”。然后我们检查这些范围，找出包含满足过滤器条件的值的所有 `region`。 - -> 如果某个查询没有任何过滤器，则将其视为全表扫描。 - -找到所需的区域后，我们只需在其中组装子扫描。通过这种方式，我们将查询拆分为子查询，每个子查询都获取表数据的一部分。子查询在 `datanode` 中执行，并在 `frontend` 中等待完成。它们的结果将合并为表扫描请求的最终返回。 - -下面这张图片总结了分布式查询执行的过程： - -![Distributed Querying](/distributed-querying.png) +远端输入是完整的逻辑子计划，并不局限于表扫描。Frontend 使用 [Substrait](https://substrait.io) 序列化子计划，再向持有相应数据的 Datanode 发送 Region 级请求。Datanode 在本地规划并执行子计划，将结果流返回 Frontend。Frontend 合并远端数据流，并执行没有下推的算子。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/overview.md index ee48da3ce3..6befdc55a9 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/overview.md @@ -5,31 +5,51 @@ description: GreptimeDB Frontend 组件概述 - 为客户端请求提供服务 # Frontend -**Frontend** 是一个无状态服务，作为 GreptimeDB 中客户端请求的入口点。它为多种数据库协议提供统一接口，并充当代理，将读写请求转发到分布式系统中的相应 Datanode。 +Frontend 是 GreptimeDB 中负责请求编排的无状态服务。Server 层负责终止协议并转换线上消息；Frontend 为这些协议处理器提供数据库行为，包括权限检查、语句执行、路由和分布式查询规划。 + +Frontend 不存储表数据。它缓存从 Metasrv 获取的 catalog 和路由元数据；元数据发生变化时，Metasrv 通过心跳响应通知 Frontend 失效相应缓存。 ## 核心功能 -- **协议支持**：支持多种数据库协议，包括 SQL、PromQL、MySQL 和 PostgreSQL。详见[协议][1] -- **请求路由**：基于元数据将请求路由到相应的 Datanode -- **查询分发**：将分布式查询拆分到多个节点 -- **响应聚合**：合并来自多个 Datanode 的结果 -- **认证授权**：安全和访问控制验证 +- 为支持的[协议][1]提供查询和写入行为。 +- 解析 catalog、schema、table 和 Region 路由。 +- 在执行请求前完成权限检查。 +- 规划分布式查询并合并 Datanode 返回的结果。 +- 将表级写入和删除转换为 Region 请求。 ## 架构 ### 关键组件 -- **协议处理器**：处理不同的数据库协议 -- **目录管理器**：缓存来自 Metasrv 的元数据以实现高效的请求路由和 Schema 校验 -- **分布式规划器**：将逻辑计划转换为分布式执行计划 -- **请求路由器**：为每个请求确定目标 Datanodes + +- 协议处理器将 SQL、PromQL、gRPC 写入和可观测性协议转换为 Frontend 的内部请求接口。 +- Catalog manager 和 partition manager 提供表元数据、分区规则和 Region 路由。 +- Statement executor 将查询、DML 和 DDL 分发到各自的执行路径。 +- 分布式规划器把表扫描替换为可跨 Datanode 执行的 `MergeScan` 计划。 ### 请求流程 -![request flow](/request_flow.png) +不同操作会走不同的请求路径。 + +#### 查询 + +1. 协议处理器创建查询上下文，并完成认证和权限检查。 +2. 对应查询语言的 planner 生成逻辑计划。分布式模式下，planner 根据分区元数据选择 Region 并生成分布式计划。 +3. Frontend 将 Region 子计划发送到对应 Datanode。Datanode 在本地 Region engine 上执行，并返回 Arrow record batch 流。 +4. Frontend 执行剩余算子、合并数据流，再按客户端协议编码结果。 + +#### 写入和删除 + +1. Frontend 根据表 schema 校验请求。支持 schema-on-write 的协议可以先创建缺失的表或新增列，再重试写入。 +2. 分区规则把每一行分配给 Region。Frontend 为各目标 Region 构造请求，并路由到当前 Region leader。 +3. Datanode 的 Region server 将请求分发到对应的 Region engine。单机模式下，请求直接发送给内嵌的 Region server。 + +#### DDL + +Statement executor 将 DDL 转换为 task。分布式模式下，Metasrv 以持久化 procedure 执行 task、更新元数据，并协调 Datanode 上的 Region 操作。单机模式复用相同的语句边界，但使用本地元数据和 procedure 实现。 ### 部署 -下图是 GreptimeDB 在云上的一个典型的部署。`Frontend` 实例组成了一个集群处理来自客户端的请求： +下图展示了 GreptimeDB 的一种云上部署。多个 Frontend 实例共同处理客户端请求： ![frontend](/frontend.png) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/table-sharding.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/table-sharding.md index 63a8ac10a6..cf39afe27b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/table-sharding.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/frontend/table-sharding.md @@ -5,21 +5,17 @@ description: 介绍 GreptimeDB 中表数据的分片方法，包括分区和 Reg # 表分片 -对于任何分布式数据库来说，数据的分片都是必不可少的。本文将描述 GreptimeDB 中的表数据如何进行分片。 +GreptimeDB 将一张表分为多个 Region。分区表达式定义每行数据属于哪个 Region，Region 路由则定义当前由哪个 Datanode 持有该 Region。 ## 分区 -有关创建分区表的语法，请参阅用户指南中的[表分片](/user-guide/deployments-administration/manage-data/table-sharding.md)部分。 +分区是由一个或多个列上的表达式描述的逻辑行集合。分区布局需要覆盖表的输入域，使每一行都能找到唯一的目标 Region。SQL 语法和支持的表达式参见[表分片](/user-guide/deployments-administration/manage-data/table-sharding.md)。 ## Region -在创建分区后，表中的数据被逻辑上分割。你可能会问："在 GreptimeDB 中，被逻辑上分区的数据是如何存储的？" 答案是保存在 `Region` 当中。 - -每个 `Region` 对应一个分区，并保存分区的数据。所有的 `Region` 分布在各个 `Datanode` 之中。 -`Metasrv` 管理 `Region` 到 `Datanode` 的路由信息。如果建表后需要调整分区布局， -GreptimeDB 支持通过显式的 [repartition](/user-guide/deployments-administration/manage-data/repartition.md) 操作拆分或合并分区。 +每个分区对应一个 Region。Region ID 是 Frontend、Datanode 和 Metasrv 用于存储和路由的标识。同一张表的多个 Region 可以放在同一个 Datanode 上。 分区和 Region 的关系参见下图： @@ -54,3 +50,13 @@ GreptimeDB 支持通过显式的 [repartition](/user-guide/deployments-administr └──────────────────────────────────┘ 可以放在同一个 Datanode 之中 ``` + +## 路由与剪枝 + +写入时，Frontend 对每行数据计算分区规则，按 Region 分组，再根据路由表把 Region 请求发送到当前 leader。 + +查询时，分布式 planner 将查询谓词与分区表达式比较，只扫描可能满足谓词的 Region。如果分区元数据缺失或无法安全解释，planner 会退化为扫描所有 Region，避免漏掉数据。 + +## 调整分区布局 + +[Repartition](/user-guide/deployments-administration/manage-data/repartition.md) 通过显式的 split 或 merge 调整已有布局。Metasrv 以持久化 procedure 执行变更，更新 Region 路由和分区表达式，并使旧的表路由缓存失效。Frontend 刷新到新元数据后，后续请求使用新的布局。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/getting-started.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/getting-started.md index e5aa2ca855..999f696ed0 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/getting-started.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/getting-started.md @@ -15,14 +15,13 @@ description: 介绍如何在本地环境中从源代码编译和运行 GreptimeD ### 构建依赖项 -- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line)（可选） +- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line)（可选；克隆仓库需要，构建本身不需要） - C/C++ 工具链：提供编译和链接的基本工具。在 Ubuntu 上，这可用作 `build-essential`。在其他平台上，也有类似的命令。 -- Rust nightly 工具链（[指南][1]） - - 编译源代码 +- [Rustup][1]。仓库通过 `rust-toolchain.toml` 指定所需的 nightly 工具链。 - Protobuf（[指南][2]） - 编译 proto 文件 - 请注意，版本需要 >= 3.15。你可以使用 `protoc --version` 检查它。 -- 机器：建议内存在 16GB 以上 或者 使用[mold](https://github.com/rui314/mold)工具以降低链接时的内存使用。 +- 机器：建议 16GB 以上内存。内存较小时，可使用 [mold](https://github.com/rui314/mold) 降低链接阶段的内存占用。 [1]: [2]: diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/how-to/how-to-write-sdk.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/how-to/how-to-write-sdk.md index 28ff757763..916aec36f3 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/how-to/how-to-write-sdk.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/how-to/how-to-write-sdk.md @@ -1,21 +1,17 @@ --- keywords: [gRPC SDK, GreptimeDatabase, GreptimeRequest, GreptimeResponse, 插入请求] -description: 介绍如何为 GreptimeDB 开发一个 gRPC SDK，包括 GreptimeDatabase 服务的定义、GreptimeRequest 和 GreptimeResponse 的结构。 +description: 介绍 GreptimeDB gRPC 写入 SDK 需要遵守的协议契约和错误处理要求。 --- # 如何为 GreptimeDB 开发一个 gRPC SDK -GreptimeDB 的 gRPC SDK 只需要处理写请求即可。读请求是标准 SQL 或 PromQL，可以由任何 JDBC 客户端或 Prometheus -客户端处理。这也是为什么所有的 GreptimeDB SDK 都命名为 "`greptimedb-ingester-`"。请确保你的 GreptimeDB SDK -遵循相同的命名约定。 +GreptimeDB 的公开 gRPC SDK 是写入客户端。查询通常通过标准 SQL 或 PromQL 客户端完成。除非有单独需求，新 SDK 应聚焦写入和删除，并遵循 `greptimedb-ingester-` 命名约定。面向用户的 API 参见 [gRPC SDK 概述](/user-guide/ingest-data/for-iot/grpc-sdks/overview.md)。 ## `GreptimeDatabase` 服务 -GreptimeDB 自定义了一个 gRPC 服务：`GreptimeDatabase` -。你只需要实现这个服务即可。你可以在[这里](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto) -找到它的 Protobuf 定义。 +从 [`GreptimeDatabase` Protobuf 定义](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto)生成客户端 stub，不要在 SDK 中手写一份 message 或 service 定义。 -`GreptimeDatabase` 有 2 个 RPC 方法： +该 service 提供一个 unary method 和一个 client-streaming method： ```protobuf service GreptimeDatabase { @@ -25,13 +21,9 @@ service GreptimeDatabase { } ``` -`Handle` 方法是一个 unary 调用：当 GreptimeDB 服务接收到一个 `GreptimeRequest` 请求后，它立刻处理该请求并返回一个相应的 -`GreptimeResponse`。 +`Handle` 对一个请求返回一个响应，是 SDK insert 和 delete API 通常使用的方法。 -`HandleRequests` 方法则是一个 "[Client Streaming RPC][3]" 方式的调用。 -它可以接受一个连续的 `GreptimeRequest` 请求流，持续地发给 GreptimeDB 服务。 -GreptimeDB 服务会在收到流中的每个请求时立刻进行处理，并最终（流结束时）返回一个总结性的 `GreptimeResponse`。 -通过 `HandleRequests`，我们可以获得一个非常高的请求吞吐量。 +`HandleRequests` 是 [client-streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc)。客户端关闭请求流后，服务端才返回累计响应。SDK 如果暴露 streaming API，需要明确这个确认边界，并将一个 stream 绑定到一个 endpoint。 ### `GreptimeRequest` @@ -51,11 +43,13 @@ message GreptimeRequest { } ``` -`RequestHeader` 是必需，它包含了一些上下文，鉴权和其他信息。"oneof" 的字段包含了发往 GreptimeDB 服务的请求。 +客户端需要在 `RequestHeader` 中填写服务端要求的 database context 和认证信息，并且只能设置一个 request variant。 -注意我们有两种类型的插入请求，一种是以 "列" 的形式（`InsertRequests`），另一种是以 "行" 的形式（`RowInsertRequests` -）。通常我们建议使用 "行" 的形式，因为它对于表的插入更自然，更容易使用。但是，如果需要一次插入大量列，或者有大量的 "null" -值需要插入，那么最好使用 "列" 的形式。 +该 message 还包含供内部调用者使用的 query 和 DDL variant。公开 ingester API 不应暴露它们，因为 `GreptimeDatabase` 不返回 query result stream。 + +GreptimeDB 同时接受行式 `RowInsertRequests` 和列式 `InsertRequests`。公开写入 API 默认使用行式请求。面向列的客户端可以使用列式请求，但转换过程中必须保持列长度一致，并保留 null、时间戳精度、数据类型和列 semantic type。 + +删除同样区分行式和列式。SDK 只应暴露能够在不丢失类型信息的前提下完成映射的形式。 ### `GreptimeResponse` @@ -68,6 +62,18 @@ message GreptimeResponse { } ``` -`ResponseHeader` 包含了返回值的状态码，以及错误信息（如果有的话）。"oneof" 的字段目前只有 "affected rows"。 +成功响应包含 success header 和 `affected_rows`。该值表示服务端确认的行数；关闭请求流时返回的是累计值。 + +请求失败通过 gRPC status 返回。Trailing metadata 中的 `x-greptime-err-code` 和 `x-greptime-err-retry-hint` 在存在时分别提供 GreptimeDB error code 和 retry classification。SDK 应保留 gRPC status 并暴露这些 GreptimeDB metadata，不能用一个通用 SDK error 将其覆盖。 + +## 重试与交付语义 + +重试次数必须有上限，并且对调用者可见。只有错误被标记为 retryable 且 deadline 仍允许时，才能重试 unary request。Cancellation 和 deadline expiration 不应重试。 + +响应丢失不代表服务端拒绝了写入。除非调用者的数据模型保证操作幂等，重试这类请求可能插入重复行。SDK 需要说明这一点，并在交付结果不确定时返回最终错误。 + +不要自动重试只发送了一部分的 `HandleRequests` stream。即使客户端尚未收到累计响应，服务端也可能已经接受了部分请求。此时应关闭失败的 stream，并将不确定状态返回给调用者。 + +Arrow Flight bulk ingestion 与 `GreptimeDatabase` RPC 应使用不同的 API。它的 batching 和 partial acceptance 需要独立的契约。 -GreptimeDB 现在有很多 SDK，你可以参考[这里](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=)获取一些示例。 +可以参考现有 [GreptimeDB ingester 仓库](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=)的公开 API 约定，但线上行为应以当前 Protobuf 定义和服务端契约为准。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/admin-api.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/admin-api.md index 3ac3d44f3b..bd28206c6f 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/admin-api.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/admin-api.md @@ -1,20 +1,25 @@ --- -keywords: [Admin API, 健康检查, leader 查询, 心跳检测, 维护模式] -description: 介绍 Metasrv 的 Admin API，包括健康检查、leader 查询、心跳检测、维护模式和 Procedure Manager 控制等功能。 +keywords: [Admin API, 健康检查, leader 查询, 心跳检测, 维护模式, 恢复模式, table id sequence] +description: 介绍 Metasrv 用于状态检查、集群控制和元数据恢复的 Admin API。 --- # Admin API -Admin 提供了一种简单的方法来查看和管理集群信息，包括 metasrv 健康检测、metasrv leader 查询、数据节点心跳检测、维护模式和 Procedure Manager 控制。 +:::tip +本页所有 Admin API 都监听 Metasrv 的 `HTTP_PORT`，默认值为 `4000`。 +::: -Admin API 是一个 HTTP 服务，提供一组可以通过 HTTP 请求调用的 RESTful API。Admin API 简单、用户友好且安全。 +Admin API 通过 HTTP 提供 Metasrv 状态、集群控制和元数据恢复操作。该 API 不提供认证，且部分端点会改变集群行为或元数据分配，部署时必须通过网络策略保护 HTTP 端口。 本页介绍以下 API： - /health - /leader - /heartbeat +- /node-lease - /maintenance - /procedure-manager +- /recovery +- /sequence/table 所有这些 API 都在父资源 `/admin` 下。 @@ -22,7 +27,7 @@ Admin API 是一个 HTTP 服务，提供一组可以通过 HTTP 请求调用的 ## /health HTTP 端点 -`/health` 端点接受 GET HTTP 请求，你可以使用此端点检查你的 metasrv 实例的健康状况。 +`/health` 端点接受 GET 请求。HTTP 服务运行时返回 `OK`，但不会检查当前 Metasrv 是否为 leader，也不会检查外部依赖是否可用。 ### 定义 @@ -116,9 +121,17 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100' ] ``` +## /node-lease HTTP 端点 + +`/node-lease` 返回 Metasrv 当前记录的 Datanode lease，可用于判断 Metasrv 是否仍将某个 Datanode 视为存活。 + +```bash +curl -X GET http://localhost:4000/admin/node-lease +``` + ## /maintenance HTTP 端点 -集群维护模式是 GreptimeDB 中的一项安全功能，它可以临时禁用自动集群管理操作。此模式在集群升级、计划停机以及任何可能暂时影响集群稳定性的操作期间特别有用。有关更多详细信息，请参阅[集群维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。 +维护模式在升级、计划停机等操作期间临时禁用自动集群管理。它对集群的具体影响参见[集群维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。 `/maintenance` 端点支持以下 HTTP 请求： @@ -151,3 +164,39 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100' "status": "running" } ``` + +## /recovery HTTP 端点 + +Recovery mode 控制手动修改 table ID sequence 等元数据修复端点。它只用于恢复工作，不用于常规维护。 + +- `GET /admin/recovery/status`：查询 recovery mode 是否开启。 +- `POST /admin/recovery/enable`：开启 recovery mode。 +- `POST /admin/recovery/disable`：关闭 recovery mode。 + +响应体格式如下： + +```json +{ + "enabled": true +} +``` + +修复完成后应关闭 recovery mode。如果只是计划暂停自动集群操作，应使用[维护模式](/user-guide/deployments-administration/maintenance/maintenance-mode.md)。 + +## /sequence/table HTTP 端点 + +这些端点用于检查或修复 table ID sequence： + +- `GET /admin/sequence/table/next-id`：返回下一个 table ID，但不执行分配。 +- `POST /admin/sequence/table/set-next-id`：推进下一个 table ID。 + +设置 sequence 前必须开启 recovery mode。新值必须大于当前值，不能通过该 API 回退 sequence。Recovery mode 只是该 API 的前置条件，不能阻止 DDL。执行该操作时，必须遵循[管理 Table ID Sequence](/user-guide/deployments-administration/maintenance/sequence-management.md)中的完整集群操作流程。 + +```bash +curl -X POST \ + -H 'Content-Type: application/json' \ + -d '{"next_table_id": 2048}' \ + http://localhost:4000/admin/sequence/table/set-next-id +``` + +该操作会影响后续新表分配到的 ID。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/overview.md index 2e3c525baa..e9f6144f26 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/overview.md @@ -1,162 +1,101 @@ --- -keywords: [Metasrv, 元数据存储, 请求路由, 负载均衡, 高可用性] -description: 介绍 Metasrv 的功能、架构和与前端的交互方式。 +keywords: [Metasrv, 元数据, 路由, Leader 选举, Procedure, 心跳] +description: 介绍 Metasrv 提供的元数据及集群协调机制。 --- # Metasrv -![meta](/meta.png) - ## Metasrv 包含什么 -- 存储元数据（Catalog, Schema, Table, Region 等） -- 请求路由器。它告诉前端在哪里写入和读取数据。 -- 数据节点的负载均衡，决定谁应该处理新的表创建请求，更准确地说，它做出资源分配决策。 -- 选举与高可用性，GreptimeDB 设计为 Leader-Follower 架构，只有 leader 节点可以写入，而 follower 节点可以读取，follower 节点的数量通常 >= 1，当 leader 不可用时，follower 节点需要能够快速切换为 leader。 -- 统计数据收集（通过每个节点上的心跳报告），如 CPU、负载、节点上的表数量、平均/峰值数据读写大小等，可用作分布式调度的基础。 +Metasrv 是 GreptimeDB 分布式集群中的元数据和协调服务，不参与数据读写链路。它主要负责： + +- 存储 Catalog、Schema、Table、Region、路由和节点元数据； +- 为新 Region 选择 Datanode，并维护表路由； +- 选举一个 Metasrv leader 负责协调元数据变更； +- 通过可恢复的 Procedure 执行 DDL、Region 迁移、故障转移和重分区； +- 通过心跳维护节点租约和 Region 统计信息； +- 元数据变更时向 Frontend、Datanode 和 Flownode 广播缓存失效； +- 向 Datanode 下发 Region 生命周期指令。 ## 前端如何与 Metasrv 交互 -首先，请求路由器中的路由表结构如下（注意这只是逻辑结构，实际存储结构可能不同，例如端点可能有字典压缩）。 - -```txt - table_A - table_name - table_schema // 用于物理计划 - regions - region_1 - mutate_endpoint - select_endpoint_1, select_endpoint_2 - region_2 - mutate_endpoint - select_endpoint_1, select_endpoint_2, select_endpoint_3 - region_xxx - table_B - ... -``` +Frontend 从 Metasrv 获取表元数据和 Region 路由，并缓存在本地。修改元数据的语句会发送给 Metasrv leader；普通读写则使用缓存的路由直接访问 Datanode。 -### 创建表 +控制链路和数据链路相互分离： -1. 前端发送 `CREATE TABLE` 请求到 Metasrv。 -2. 根据请求中包含的分区规则规划 Region 数量。 -3. 检查数据节点可用资源的全局视图（通过心跳收集）并为每个 Region 分配一个节点。 -4. 前端创建表并在成功创建后将 `Schema` 存储到 Metasrv。 +```text +Frontend + |-- 元数据查询和 DDL -------------------> Metasrv leader + `-- Region 读写 ------------------------> Datanode -### `Insert` +Metasrv leader + |-- Region 生命周期指令 ----------------> Datanode + `-- 缓存失效 --------------------------> Frontend / Datanode / Flownode -1. 前端从 Metasrv 获取指定表的路由。注意，最小的路由单元是表的路由（多个 Region），即包含该表所有 Region 的地址。 -2. 最佳实践是前端首先从本地缓存中获取路由并将请求转发到数据节点。如果路由不再有效，则数据节点有义务返回 `Invalid Route` 错误，前端重新从 Metasrv 获取最新数据并更新其缓存。路由信息不经常变化，因此，前端使用惰性策略维护缓存是足够的。 -3. 前端处理可能包含多个表和多个 Region 的一批写入，因此前端需要根据“路由表”拆分用户请求。 +Datanode + `-- 心跳、租约续期和 Region 统计信息 ----> Metasrv leader +``` -### `Select` +在稳定状态下，表路由为每个 Region 记录一个 leader peer 和零个或多个 follower peer。Leader 是写入目标；支持只读副本的部署可以把读取路由到 follower： -1. 与 `Insert` 类似，前端首先从本地缓存中获取路由表。 -2. 与 `Insert` 不同，对于 `Select`，前端需要从路由表中提取只读节点（follower），然后根据优先级将请求分发到 leader 或 follower 节点。 -3. 前端的分布式查询引擎根据路由信息分发多个子查询任务并聚合查询结果。 +```text +Table route + |-- Region 0 + | |-- leader -> Datanode A + | `-- followers -> Datanode B, Datanode C + `-- Region 1 + `-- leader -> Datanode D +``` -## Metasrv 架构 +Region 迁移或故障转移会改变 peer 角色，并可能使 Region 暂时没有 leader。Frontend 刷新缓存路由后，再把后续读写发送给当前 peer。 -![metasrv-architecture](/metasrv-architecture.png) +### 创建表 -## 分布式共识 +1. Frontend 向 Metasrv leader 提交 DDL 请求。 +2. Metasrv 根据分区规则确定 Region，并[为每个 Region 选择 Datanode](/contributor-guide/metasrv/selector.md)。 +3. 持久化的 Procedure 创建 Region，并写入表元数据和路由。发生 leader 切换后，Procedure 可以从已保存的状态继续执行。 +4. 元数据提交后，Metasrv 通知 Frontend 刷新相关缓存。 -如你所见，Metasrv 依赖于分布式共识，因为： +### `Insert` -1. 首先，Metasrv 必须选举一个 leader，数据节点只向 leader 发送心跳，我们只使用单个 Metasrv 节点接收心跳，这使得基于全局信息进行一些计算或调度变得容易且快速。至于数据节点如何连接到 leader，这由 MetaClient 决定（使用重定向，心跳请求变为 gRPC 流，使用重定向比转发更不容易出错），这对数据节点是透明的。 -2. 其次，Metasrv 必须为数据节点提供选举 API，以选举“写入”和“只读”节点，并帮助数据节点实现高可用性。 -3. 最后，`Metadata`、`Schema` 和其他数据必须在 Metasrv 上可靠且一致地存储。因此，基于共识的算法是存储它们的理想方法。 +Frontend 解析表路由，按照分区规则拆分数据行，再把各 Region 的写入发送到对应 Datanode。路由发生变化时，相关缓存会失效，Frontend 随后从 Metasrv 重新获取元数据。 -对于 Metasrv 的第一个版本，我们选择 Etcd 作为共识算法组件（Metasrv 设计时考虑适应不同的实现，甚至创建一个新的轮子），原因如下： +### `Select` -1. Etcd 提供了我们需要的 API，例如 `Watch`、`Election`、`KV` 等。 -2. 我们只执行两个分布式共识任务：选举（使用 `Watch` 机制）和存储（少量元数据），这两者都不需要我们定制自己的状态机，也不需要基于 raft 定制自己的状态机；少量数据也不需要多 raft 组支持。 -3. Metasrv 的初始版本使用 Etcd，使我们能够专注于 Metasrv 的功能，而不需要在分布式共识算法上花费太多精力，这提高了系统设计（避免与共识算法耦合）并有助于初期的快速开发，同时通过良好的架构设计，未来可以轻松接入优秀的共识算法实现。 +Frontend 在查询规划期间使用表和 Region 元数据。分区列上的谓词用于裁剪 Region，分布式查询引擎再把任务发送给持有这些 Region 的 Datanode。参见[分布式查询](../frontend/distributed-querying.md)。 -## 心跳管理 +## Metasrv 架构 -数据节点与 Metasrv 之间的主要通信方式是心跳请求/响应流，我们希望这是唯一的通信方式。这个想法受到 [TiKV PD](https://github.com/tikv/pd) 设计的启发，我们在 [RheaKV](https://github.com/sofastack/sofa-jraft/tree/master/jraft-rheakv/rheakv-pd) 中有实际经验。请求发送其状态，而 Metasrv 通过心跳响应发送不同的调度指令。 +主要协调路径如下： + +```text +Leader election + | + v +Metasrv leader +├─ DDL manager -> Procedure manager +├─ Selector -> 新 Region 的放置 +├─ Heartbeat handler chain -> 租约和 Region 统计信息 +├─ Region supervisor -> Region 迁移 Procedure +├─ Mailbox -> 缓存失效和 Region 指令 +└─ Metadata managers -> KV backend +``` -心跳可能携带以下数据，但这不是最终设计，我们仍在讨论和探索究竟应该收集哪些数据。 +这些机制共享元数据，但故障边界不同。进程重启可以丢弃缓存和 leader 本地状态；恢复所需的元数据和 Procedure 状态必须持久化。 -``` -service Heartbeat { - // 心跳，心跳可能有很多内容，例如： - // 1. 要注册到 Metasrv 并可被其他节点发现的元数据。 - // 2. 一些性能指标，例如负载、CPU 使用率等。 - // 3. 正在执行的计算任务数量。 - rpc Heartbeat(stream HeartbeatRequest) returns (stream HeartbeatResponse) {} -} - -message HeartbeatRequest { - RequestHeader header = 1; - - // 自身节点 - Peer peer = 2; - // leader 节点 - bool is_leader = 3; - // 实际报告时间间隔 - TimeInterval report_interval = 4; - // 节点状态 - NodeStat node_stat = 5; - // 此节点中的 Region 状态 - repeated RegionStat region_stats = 6; - // follower 节点和状态，在 follower 节点上为空 - repeated ReplicaStat replica_stats = 7; -} - -message NodeStat { - // 此期间的读取容量单位 - uint64 rcus = 1; - // 此期间的写入容量单位 - uint64 wcus = 2; - // 此节点中的表数量 - uint64 table_num = 3; - // 此节点中的 Region 数量 - uint64 region_num = 4; - - double cpu_usage = 5; - double load = 6; - // 节点中的读取磁盘 I/O - double read_io_rate = 7; - // 节点中的写入磁盘 I/O - double write_io_rate = 8; - - // 其他 - map attrs = 100; -} - -message RegionStat { - uint64 region_id = 1; - TableName table_name = 2; - // 此期间的读取容量单位 - uint64 rcus = 3; - // 此期间的写入容量单位 - uint64 wcus = 4; - // 近似 Region 大小 - uint64 approximate_size = 5; - // 近似行数 - uint64 approximate_rows = 6; - - // 其他 - map attrs = 100; -} - -message ReplicaStat { - Peer peer = 1; - bool in_sync = 2; - bool is_learner = 3; -} -``` +## 分布式共识 -## Central Nervous System (CNS) +Metasrv 将 leader 选举与元数据存储分开。只有选出的 Metasrv leader 执行协调和元数据变更操作，其他 Metasrv 节点会把 client 引导到当前 leader。 -我们要构建一个算法系统，该系统依赖于每个节点的实时和历史心跳数据，应该做出一些更智能的调度决策并将其发送到 Metasrv 的 Autoadmin 单元，该单元分发调度决策，由数据节点本身或更可能由 PaaS 平台执行。 +Key-value backend 保存表元数据、路由、Procedure 状态以及其他必须跨 leader 切换保留的信息。Metasrv 不使用这套选举为 Datanode Region 创建读写副本；Region 可用性由心跳、Region 故障检测和故障转移 Procedure 管理。 -## 工作负载抽象 +## 心跳管理 -工作负载抽象的级别决定了 Metasrv 生成的调度策略（如资源分配）的效率和质量。 +Datanode 与 Metasrv leader 保持心跳流。心跳请求报告节点身份、租约、Region 统计信息以及放置和监控所需的其他状态；响应则携带 Region 生命周期指令、缓存失效等控制消息。 -DynamoDB 定义了 RCUs 和 WCUs（读取容量单位/写入容量单位），解释说 RCU 是一个 4KB 数据的读取请求，WCU 是一个 1KB 数据的写入请求。当使用 RCU 和 WCU 描述工作负载时，更容易实现性能可测量性并获得更有信息量的资源预分配，因为我们可以将不同的硬件能力抽象为 RCU 和 WCU 的组合。 +心跳驱动两套相互独立的机制，调整心跳周期会同时影响两者： -然而，GreptimeDB 面临比 DynamoDB 更复杂的情况，特别是 RCU 不适合描述需要大量计算的 GreptimeDB 读取工作负载。我们正在努力解决这个问题。 +- **节点租约**：keep-lease handler 为发送心跳的 Datanode 续期。Selector 和 `/node-lease` 端点据此判断 Datanode 是否仍然存活。 +- **Region 故障检测**：Region supervisor 为每个 Region 维护一个基于心跳到达间隔的 Phi Accrual 检测器，其判定与租约是否过期无关。 +只有开启 Region 故障转移时，故障判定才会提交故障转移迁移。该功能默认关闭，并且要求使用 remote WAL，除非显式允许在本地 WAL 上执行。维护模式同样会抑制故障转移。前置条件和开启方式参见 [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/selector.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/selector.md index 9048e0daa4..29b521212b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/selector.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/metasrv/selector.md @@ -7,11 +7,7 @@ description: 介绍 Metasrv 中的 Selector，包括其类型和配置方法。 ## 介绍 -什么是 `Selector`？顾名思义，它允许用户从给定的 `namespace` 和 `context` 中选择 `Item`s。有一个相关的 `trait`，也叫做 `Selector`，其定义可以在[这里][0]找到。 - -[0]: https://github.com/GreptimeTeam/greptimedb/blob/main/src/meta-srv/src/selector.rs - -在 `Metasrv` 中存在一个特定的场景。当 `Frontend` 向 `Metasrv` 发送建表请求时，`Metasrv` 会创建一个路由表（表的创建细节不在这里赘述）。在创建路由表时，`Metasrv` 需要选择适当的 `Datanode`s，这时候就需要用到 `Selector`。 +建表时，Metasrv 使用 `Selector` 为各 Region 选择 Datanode。Selector 根据当前节点租约进行选择；部分实现还会使用 Region 统计信息。 @@ -19,22 +15,22 @@ description: 介绍 Metasrv 中的 Selector，包括其类型和配置方法。 `Metasrv` 目前提供以下几种类型的 `Selectors`: -### LeasebasedSelector +### LeaseBasedSelector -`LeasebasedSelector` 从所有可用的（也就是在租约期间内）`Datanode` 中随机选择，其特点是简单和快速。 +`LeaseBasedSelector` 从租约有效的 Datanode 中随机选择。 ### LoadBasedSelector `LoadBasedSelector` 按照负载来选择，负载值则由每个 `Datanode` 上的 region 数量决定，较少的 region 表示较低的负载，`LoadBasedSelector` 优先选择低负载的 `Datanode`。 ### RoundRobinSelector [默认选项] -`RoundRobinSelector` 以轮询的方式选择 `Datanode`。在大多数情况下，这是默认的且推荐的选项。如果你不确定选择哪个，通常它就是正确的选择。 +`RoundRobinSelector` 以轮询方式选择 Datanode，是默认选项，也适用于大多数部署。 ## 配置 您可以在启动 `Metasrv` 服务时通过名称配置 `Selector`。 -- LeasebasedSelector: `lease_based` 或 `LeaseBased` +- LeaseBasedSelector: `lease_based` 或 `LeaseBased` - LoadBasedSelector: `load_based` 或 `LoadBased` - RoundRobinSelector: `round_robin` 或 `RoundRobin` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/overview.md index 9f82a72171..badd6ad62d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/overview.md @@ -5,9 +5,7 @@ description: 介绍 GreptimeDB 的架构、关键概念和工作原理，包括 # 贡献者指南 -DeepWiki 对 GreptimeDB 的架构和实现进行了详细且清晰的描述，强烈推荐阅读： - -[https://deepwiki.com/GreptimeTeam/greptimedb](https://deepwiki.com/GreptimeTeam/greptimedb) +本指南面向 GreptimeDB 贡献者，介绍理解内部实现所需的设计机制。从源码构建和运行参见[快速开始](/contributor-guide/getting-started.md)。提交要求（CLA、license header、代码格式，以及 PR 必须通过的检查）以源码仓库的 [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md) 为准。 ## 架构 @@ -18,7 +16,13 @@ DeepWiki 对 GreptimeDB 的架构和实现进行了详细且清晰的描述， - [frontend][1] - [datanode][2] - [metasrv][3] +- [flownode][4] [1]: /contributor-guide/frontend/overview.md [2]: /contributor-guide/datanode/overview.md [3]: /contributor-guide/metasrv/overview.md +[4]: /contributor-guide/flownode/overview.md + +## 补充参考 + +[DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb) 提供了自动生成的 GreptimeDB 源码导读，可用于了解不熟悉的模块。它属于辅助资料；涉及具体版本的行为时，仍应以对应源码为准。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/integration-test.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/integration-test.md index 63ffa56bc0..767109f773 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/integration-test.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/integration-test.md @@ -7,8 +7,14 @@ description: 介绍 GreptimeDB 的集成测试，包括测试范围和如何运 ## 介绍 -集成测试使用 Rust 测试工具（`#[test]`）编写，与单元测试不同，它们被单独放置在 -[这里](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration)。 -它涵盖了涉及多个组件的场景，其中一个典型案例是与 HTTP/gRPC 相关的功能。你可以查看 -其[文档](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md)以获取更多信息。 +集成测试覆盖跨 crate 或服务边界的行为，例如 HTTP 和 gRPC 处理、分布式组件或外部存储。测试使用 Rust test harness，位于 [`tests-integration`](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration) package。 +运行命令如下： + +```shell +cargo nextest run -p tests-integration +``` + +部分 case 依赖外部服务的环境变量或 fixture。运行前按照 package 的[准备说明](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md)配置环境。 + +只有 crate 级测试或 Sqlness case 无法覆盖所需边界时才使用集成测试。隔离的逻辑仍放在单元测试中，便于快速复现失败。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/overview.md index 81b6defa45..9ac8eb2ec1 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/overview.md @@ -5,5 +5,13 @@ description: GreptimeDB 的测试 # 测试 -我们的团队进行了大量测试，以确保 GreptimeDB 的行为。本章将介绍几种用于测试 GreptimeDB 的重要方法，以及如何使用它们。 +选择能够覆盖本次改动的最小测试范围： +| 测试类型 | 适用场景 | 常用命令 | +| --- | --- | --- | +| [单元测试](unit-test.md) | 单个 crate 或组件内的逻辑 | `cargo nextest run -p ` | +| [Sqlness 测试](sqlness-test.md) | SQL、协议、planner、执行和端到端回归 | `cargo sqlness bare -t ` | +| [集成测试](integration-test.md) | 跨组件或依赖外部服务的行为 | `cargo nextest run -p tests-integration` | +| [兼容性测试](https://github.com/GreptimeTeam/greptimedb/blob/main/tests/compatibility/README.md) | 读取旧版本写入的数据或元数据 | `cargo sqlness compat --from-version ` | + +需要运行完整 Rust workspace 测试时使用 `make test`。如果改动涉及持久化元数据、WAL record、SST 文件或线上协议，并且输入可能由旧版本产生，还需要补充兼容性测试。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/sqlness-test.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/sqlness-test.md index d29b69e835..06503a434c 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/sqlness-test.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/sqlness-test.md @@ -7,36 +7,34 @@ description: 介绍 GreptimeDB 的 Sqlness 测试，包括测试文件类型、 ## 介绍 -SQL 是 `GreptimeDB` 的一个重要用户接口。我们为它提供了一个单独的测试套件（名为 `sqlness`）。 +Sqlness 是 GreptimeDB 针对 SQL 和协议行为的端到端回归测试。每个 case 向运行中的 GreptimeDB 发送语句，并将输出与仓库中的结果文件比较。 ## Sqlness 手册 ### 测试文件 -Sqlness 有两种类型的文件 +每个 case 使用两类文件： - `.sql`：测试输入，仅包含 SQL - `.result`：预期的测试输出，包含 SQL 和其结果 -`.result` 文件是预期的执行输出。如果 `.result` 文件发生变化，意味着测试结果不同，测试可能失败。你应该检查变更日志来解决问题。 - -你只需要在 `.sql` 文件中编写测试 SQL，然后运行测试。 +在 `.sql` 文件中编写输入，运行测试后生成或更新 `.result`。必须检查每一处结果差异，只有行为变化符合预期时才能接受。 ### 组织测试案例 -输入案例的根目录是 `tests/cases`。它包含几个子目录，代表不同的测试模式。例如，`standalone/` 包含所有在 `greptimedb standalone start` 模式下运行的测试。 +输入 case 位于 `tests/cases`。第一级目录选择运行环境，例如 `standalone/` 表示使用单机 GreptimeDB。 -在第一级子目录下（例如 `cases/standalone`），你可以随意组织你的测试案例。Sqlness 会递归地遍历每个文件并运行它们。 +在环境目录内，新 case 应与它覆盖的功能放在一起。Sqlness 会递归发现 case 文件。 ## 运行测试 -与其他测试不同，这个测试工具是以二进制目标形式存在的。你可以用以下命令运行它 +运行命令如下： ```shell -cargo run --bin sqlness-runner bare +cargo sqlness bare ``` -它会自动完成以下步骤：编译 `GreptimeDB`，启动它，抓取测试并将其发送到服务器，然后收集和比较结果。你只需要检查是否有 `.result` 文件发生变化。如果没有，恭喜你，测试通过了 🥳！ +该命令会构建并启动 GreptimeDB、执行选中的 case，再比较输出。`.result` 发生变化只是待审查的结果，不代表新输出一定正确。 ### 运行特定测试 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/unit-test.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/unit-test.md index 79c73775b4..b63b29df94 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/unit-test.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/tests/unit-test.md @@ -7,22 +7,26 @@ description: 介绍 GreptimeDB 的单元测试，包括如何编写、运行和 ## 介绍 -单元测试嵌入在代码库中，通常放置在被测试逻辑的旁边。它们使用 Rust 的 `#[test]` 属性编写，并可以使用 `cargo nextest run` 运行。 +单元测试通常放在被测逻辑旁边，使用 Rust 的 `#[test]` 属性编写。GreptimeDB 主要使用 [`cargo-nextest`](https://nexte.st/) 运行 Rust 测试。 -GreptimeDB 代码库不支持默认的 `cargo` 测试运行器。推荐使用 [`nextest`](https://nexte.st/)。你可以通过以下命令安装它： +安装命令如下： ```shell cargo install cargo-nextest --locked ``` -然后运行测试（这里 `--workspace` 不是必须的） +开发时先运行本次修改的 package： ```shell -cargo nextest run +cargo nextest run -p ``` -注意，如果你的 Rust 是通过 `rustup` 安装的，请确保使用 `cargo` 安装 `nextest`，而不是像 `homebrew` 这样的包管理器，否则会弄乱你的本地环境。 +可以继续使用测试名称或 nextest filter 缩小范围。影响范围较广的改动在提交前运行完整 workspace 测试： + +```shell +make test +``` ## 覆盖率 -我们的持续集成（CI）作业有一个“覆盖率检查”步骤。它会报告有多少代码被单元测试覆盖。请在你的补丁中添加必要的单元测试。 +CI 会报告单元测试覆盖率。测试应覆盖本次改变的行为和可能回归的失败路径，而不是只追求覆盖率数字。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/reference/sql/create.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/reference/sql/create.md index 5e1d660bf0..660ca1751e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/reference/sql/create.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/reference/sql/create.md @@ -26,7 +26,7 @@ CREATE DATABASE [IF NOT EXISTS] db_name [WITH ] 数据库也可以通过使用 `WITH` 关键字配置与 `CREATE TABLE` 语句类似的选项。数据库支持以下选项： - `ttl` - 数据库中所有表的数据存活时间（不能设置为 `instant`） -- `memtable.type` - 内存表类型（`time_series`、`partition_tree`） +- `memtable.type` - memtable 类型（`bulk`、`time_series`） - `append_mode` - 数据库中的表是否为仅追加模式（`true`/`false`） - `merge_mode` - 合并重复行的策略（`last_row`、`last_non_null`） - `skip_wal` - 是否为数据库中的表禁用预写日志（`'true'`/`'false'`） @@ -74,7 +74,7 @@ CREATE DATABASE test WITH (ttl='7d'); ```sql CREATE DATABASE test WITH ( ttl='30d', - 'memtable.type'='partition_tree', + 'memtable.type'='bulk', 'append_mode'='true' ); ``` @@ -156,7 +156,7 @@ GreptimeDB 提供了丰富的索引实现来加速查询，请在[索引](/user- | `compaction.twcs.trigger_file_num` | 某个窗口内触发 compaction 的最小文件数量阈值 | 字符串值，如 '8'。只在 `compaction.type` 为 `twcs` 时可用 | | `compaction.twcs.time_window` | Compaction 时间窗口 | 字符串值，如 '1d' 表示 1 天。该表会根据时间戳将数据分区到不同的时间窗口中。只在 `compaction.type` 为 `twcs` 时可用 | | `compaction.twcs.max_output_file_size` | TWCS compaction 的最大输出文件大小 | 字符串值，如 '1GB'、'512MB'。设置 TWCS compaction 产生的文件的最大大小。只在 `compaction.type` 为 `twcs` 时可用 | -| `memtable.type` | memtable 的类型 | 字符串值，支持 `time_series`，`partition_tree` | +| `memtable.type` | memtable 类型 | 字符串值：`bulk` 或 `time_series`。未设置时，Mito 根据 SST format 选择实现；默认的 flat format 使用 `bulk`。设置 `bulk` 会强制使用 `sst_format=flat`；使用 flat SST 时，即使设置了 `time_series`，Mito 也会选择 bulk 实现。旧值 `partition_tree` 仅为兼容保留，并映射到 bulk 和 flat 路径。 | | `append_mode` | 该表是否时 append-only 的 | 字符串值。默认值为 'false'，根据 'merge_mode' 按主键和时间戳删除重复行。设置为 'true' 可以开启 append 模式和创建 append-only 表，保留所有重复的行 | | `merge_mode` | 合并重复行的策略 | 字符串值。只有当 `append_mode` 为 'false' 时可用。默认值为 `last_row`，保留相同主键和时间戳的最后一行。设置为 `last_non_null` 则保留相同主键和时间戳的最后一个非空字段。 | | `sst_format` | SST 文件的格式 | 字符串值，支持 `primary_key`，`flat`。默认为 `flat`。`flat` 格式建议用于具有高基数主键的表。 | diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/user-guide/deployments-administration/configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/user-guide/deployments-administration/configuration.md index 3324915c65..b7c37a192d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/user-guide/deployments-administration/configuration.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/user-guide/deployments-administration/configuration.md @@ -600,20 +600,9 @@ create_on_compaction = "auto" apply_on_query = "auto" mem_threshold_on_create = "64M" intermediate_path = "" - -[region_engine.mito.memtable] -type = "time_series" ``` -此外，`mito` 也提供了一个实验性质的 memtable。该 memtable 主要优化大量时间序列下的写入性能和内存占用。其查询性能可能会不如默认的 `time_series` memtable。 - -```toml -[region_engine.mito.memtable] -type = "partition_tree" -index_max_keys_per_shard = 8192 -data_freeze_threshold = 32768 -fork_dictionary_bytes = "1GiB" -``` +Mito 根据表选项和 SST format 为每个 Region 选择 memtable 实现。`default_flat_format` 为 `true` 时，没有显式设置 `sst_format` 的 Region 使用 flat SST 和 bulk memtable。`memtable.type` 是数据库或表选项，不是 `[region_engine.mito.memtable]` 引擎配置。详见[表选项](/reference/sql/create.md#表选项)。 以下是可供使用的选项 @@ -648,7 +637,7 @@ fork_dictionary_bytes = "1GiB" | `scan_memory_on_exhausted` | 字符串 | `fail` | 扫描内存耗尽时的行为。选项：`fail`（快速失败），`wait` 或 `wait()`（等待内存）。 | | `min_compaction_interval` | 字符串 | `0m` | 两次 compaction 之间的最小时间间隔。设为 "0m"（默认）允许 compactions 立即运行，无限制。 | | `schedule_compaction_after_edit` | 布尔值 | `true` | 是否允许在成功的 region edit 之后调度 compaction。
设为 `true` 是在 region edit 后调度 compaction 的必要但不充分条件，`min_compaction_interval` 等其他约束仍可能阻止 compaction 被调度。
设为 `false` 则保证 region edit 后不会调度 compaction。 | -| `default_flat_format` | 布尔值 | `true` | 是否启用 Flat 格式作为默认 SST 格式。 | +| `default_flat_format` | 布尔值 | `true` | 没有显式设置 `sst_format` 的 Region 是否使用 flat SST。Flat SST 使用 bulk memtable。 | | `scan_parallelism` | 整数 | `0` | （已弃用，请使用 `max_concurrent_scan_files`）旧版扫描并发度选项。 | | `index` | -- | -- | Mito 引擎中索引的选项。 | | `index.aux_path` | 字符串 | `""` | 文件系统中索引的辅助目录路径，用于存储创建索引的中间文件和搜索索引的暂存文件，默认为 `{data_home}/index_intermediate`。为了向后兼容，该目录的默认名称为 `index_intermediate`。此路径包含两个子目录：- `__intm`: 用于存储创建索引时使用的中间文件。- `staging`: 用于存储搜索索引时使用的暂存文件。 | @@ -663,10 +652,6 @@ fork_dictionary_bytes = "1GiB" | `inverted_index.apply_on_query` | 字符串 | `auto` | 是否在查询时使用索引
- `auto`: 自动
- `disable`: 从不 | | `inverted_index.mem_threshold_on_create` | 字符串 | `64M` | 创建索引时如果超过该内存阈值则改为使用外部排序
设置为空会关闭外排，在内存中完成所有排序 | | `inverted_index.intermediate_path` | 字符串 | `""` | 存放外排临时文件的路径 (默认 `{data_home}/index_intermediate`). | -| `memtable.type` | 字符串 | `time_series` | Memtable type.
- `time_series`: time-series memtable
- `partition_tree`: partition tree memtable (实验性功能) | -| `memtable.index_max_keys_per_shard` | 整数 | `8192` | 一个 shard 内的主键数
只对 `partition_tree` memtable 生效 | -| `memtable.data_freeze_threshold` | 整数 | `32768` | 一个 shard 内写缓存可容纳的最大行数
只对 `partition_tree` memtable 生效 | -| `memtable.fork_dictionary_bytes` | 字符串 | `1GiB` | 主键字典的大小
只对 `partition_tree` memtable 生效 | `metric` 引擎针对包含大量小表的 metrics 数据进行了优化。 diff --git a/versioned_docs/version-1.0/contributor-guide/datanode/data-persistence-indexing.md b/versioned_docs/version-1.0/contributor-guide/datanode/data-persistence-indexing.md index 9f657c5edf..13f4d85e66 100644 --- a/versioned_docs/version-1.0/contributor-guide/datanode/data-persistence-indexing.md +++ b/versioned_docs/version-1.0/contributor-guide/datanode/data-persistence-indexing.md @@ -5,19 +5,23 @@ description: Explanation of data persistence and indexing in GreptimeDB, includi # Data Persistence and Indexing -Similar to all LSMT-like storage engines, data in MemTables is persisted to durable storage, for example, the local disk file system or object storage service. GreptimeDB adopts [Apache Parquet][1] as its persistent file format. +Like other LSM-tree storage engines, GreptimeDB persists data from memtables to durable storage such as a local filesystem or object storage. It uses [Apache Parquet][1] as the persistent file format. ## SST File Format Parquet is an open source columnar format that provides fast data querying and has already been adopted by many projects, such as Delta Lake. -Parquet has a hierarchical structure like "row groups-columns-data pages". Data in a Parquet file is horizontally partitioned into row groups, in which all values of the same column are stored together to form a data page. Data page is the minimal storage unit. This structure greatly improves performance. +Parquet organizes data as row groups, column chunks, and pages. A row group contains one column chunk for each column, and each column chunk contains one or more pages. Pages are the units of encoding and compression; column chunks are the I/O units for reading selected columns. First, clustering data by column makes file scanning more efficient, especially when only a few columns are queried, which is very common in analytical systems. -Second, data of the same column tends to be homogeneous which helps with compression when apply techniques like dictionary and Run-Length Encoding (RLE). +Second, values within a column tend to be similar, which improves compression with techniques such as dictionary encoding and run-length encoding (RLE). -Parquet file format +The following diagram from the Apache Parquet specification also shows the physical file layout: column chunks are stored by row group, while file metadata and its length are written in the footer. + +Apache Parquet file layout + +*Source: Apache Parquet [FileLayout.gif](https://github.com/apache/parquet-format/blob/master/doc/images/FileLayout.gif). Copyright 2014 The Apache Software Foundation, licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).* ## Data Persistence @@ -27,17 +31,17 @@ When the size of data buffered in MemTables reaches that threshold, GreptimeDB w ## Indexing Data in SST Files -Apache Parquet file format provides inherent statistics in headers of column chunks and data pages, which are used for pruning and skipping. +Parquet stores row-group column statistics such as minimum, maximum, and null count in each column chunk's metadata. Page metadata and optional column indexes can provide finer-grained statistics. -Column chunk header +![A name predicate uses Parquet column statistics to skip one row group while retaining another as a read candidate.](/parquet-row-group-statistics.svg) -For example, in the above Parquet file, if you want to filter rows where `name` = `Emily`, you can easily skip row group 0 because the max value for `name` field is `Charlie`. This statistical information reduces IO operations. +For example, a query filtering for `name` = `Emily` can skip row group 0 because the maximum `name` value is `Charlie`. This avoids reading that row group. ## Index Files -For each SST file, GreptimeDB not only maintains an internal index but also generates a separate file to store the index structures specific to that SST file. +When an SST has one or more configured index outputs, GreptimeDB writes them to a Puffin file associated with that SST. An SST with no applicable index does not need a Puffin file. -The index files utilize the [Puffin][3] format, which offers significant flexibility, allowing for the storage of additional metadata and supporting a broader range of index structures. +Puffin provides a container for index blobs and their metadata, allowing different index structures to share one file. ![Puffin](/puffin.png) @@ -57,13 +61,13 @@ The inverted index enables GreptimeDB to skip data segments that do not meet que ![Inverted index searching](/inverted-index-searching.png) -For instance, the query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches the regex `.*users`, and `status` matches the regex `4...`. It then scans these data segments to produce the final results that meet all conditions, significantly reducing the number of IO operations. +The query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches `.*users`, and `status` matches `4..`. It scans only those segments before applying the remaining filters. ### Inverted Index Format -![Inverted index format](/inverted-index-format.png) +![An inverted-index blob contains one index per column followed by footer metadata; each column index contains a null bitmap, posting bitmaps, and an FST.](/inverted-index-blob-layout.svg) -GreptimeDB builds inverted indexes by column, with each inverted index consisting of an FST and multiple Bitmaps. +GreptimeDB builds inverted indexes by column. Each column index contains a null bitmap, multiple posting bitmaps, and an FST. The blob footer records the offsets, sizes, and metadata needed to locate and decode the column indexes. The FST (Finite State Transducer) enables GreptimeDB to store mappings from column values to Bitmap positions in a compact format and provides excellent search performance and supports complex search capabilities (such as regular expression matching). The Bitmaps maintain a list of data segment IDs, with each bit representing a data segment. @@ -77,7 +81,7 @@ The number of rows in a data segment is controlled by the engine option `index.i ## Unified Data Access Layer: OpenDAL -GreptimeDB uses [OpenDAL][2] to provide a unified data access layer, thus, the storage engine does not need to interact with different storage APIs, and data can be migrated to cloud-based storage like AWS S3 seamlessly. +GreptimeDB uses [OpenDAL][2] to provide a common access layer for local filesystems and object stores. Changing the configured storage backend does not migrate existing data. [1]: https://parquet.apache.org [2]: https://github.com/datafuselabs/opendal diff --git a/versioned_docs/version-1.0/contributor-guide/datanode/metric-engine.md b/versioned_docs/version-1.0/contributor-guide/datanode/metric-engine.md index 064872ce14..fde3c178a3 100644 --- a/versioned_docs/version-1.0/contributor-guide/datanode/metric-engine.md +++ b/versioned_docs/version-1.0/contributor-guide/datanode/metric-engine.md @@ -7,9 +7,9 @@ description: Overview of the Metric engine in GreptimeDB, its concepts, architec ## Overview -The `Metric` engine is a component of GreptimeDB, and it's an implementation of the storage engine. It mainly targets scenarios with a large number of small tables for observable metrics. +The `Metric` engine stores workloads with many small metric tables. -Its main feature is to use synthetic physical wide tables to store a large amount of small table data, achieving effects such as reuse of the same column and metadata. This reduces storage overhead for small tables and improves columnar compression efficiency. The concept of a table becomes even more lightweight under the `Metric` engine. +It maps those logical tables onto shared physical wide tables so they can reuse columns and metadata. This reduces per-table storage overhead and improves columnar compression. ## Concepts @@ -19,7 +19,7 @@ The `Metric` engine introduces two new concepts: "logical table" and "physical t A logical table refers to user-defined tables. Just like any other ordinary table, its definition includes the name of the table, column definitions, index definitions etc. All operations such as queries or write-ins by users are based on these logical tables. Users don't need to worry about differences between logical and ordinary tables during usage. -From an implementation standpoint, a logical table is virtual; it doesn't directly read or write physical data but maps read/write requests into corresponding requests for physical tables in order to implement data storage and querying. +A logical table is virtual. The engine maps its read and write requests to the corresponding physical table instead of storing data for it directly. ### Physical Table @@ -29,16 +29,14 @@ A physical table is a table that actually stores data, possessing several physic The main design architecture of the `Metric` engine is as follows: -![Arch](/metric-engine-arch.png) +![Multiple logical tables map through the Metric engine to shared data and metadata Regions managed by Mito.](/metric-engine-architecture.svg) -In the current version implementation, the `Metric` engine reuses the `Mito` engine to achieve storage and query capabilities for physical data. It also provides access to both physical tables and logical tables simultaneously. +The `Metric` engine delegates physical storage and queries to the `Mito` engine. Each physical Region group contains a data Region, which stores rows from its mapped logical tables, and a metadata Region, which stores the logical-table and logical-column mappings. -Regarding partitioning, logical tables have identical partition rules and Region distribution as physical tables. This makes sense because the data of logical tables are directly stored in physical tables, so their partition rules are consistent. +Logical tables associated with the same physical table share its partition layout. During writes, the engine records the logical table identity with each row. During reads, it adds a logical-table filter before scanning the physical Region. -Concerning routing metadata, the routing address of a logical table is a logical address - what its corresponding physical table is - then through this physical table for secondary routing to obtain the real physical address. This indirect routing method can significantly reduce the number of metadata modifications required when Region migration scheduling occurs in Metric engines. +A logical table's route stores only the ID of its physical table; the physical table route resolves that to the Datanodes holding the Regions. Because logical routes do not name peers, migrating a physical Region rewrites one physical route instead of every logical route that maps to it. -Operationally speaking, The `Metric` engine supports standard DML operations (INSERT, DELETE, SELECT) on logical tables. However, it only supports limited operations on physical tables to prevent misoperations - for example, writing directly to a physical table is prohibited as it could affect user's logical table data. Generally speaking, users can consider that they have read-only access to these physical tables. +Logical tables support normal INSERT, DELETE, and SELECT operations. Direct writes to a physical Region are rejected because they would bypass the logical-table mapping; querying a physical table remains supported. -To improve performance during simultaneous DDL (Data Definition Language) operations on many tables, the 'Metric' engine has introduced some batch DDL operations. These batch DDL operations can merge lots of DDL actions into one request thereby reducing queries and modifications times for metadata thus enhancing performance. This feature is particularly beneficial in scenarios such as the automatic creation requests brought about by large amounts of metrics during Prometheus Remote Write cold start-up, as well as the modification requests for numerous route-tables mentioned earlier during migration of many physical regions. - -Apart from physical data regions belonging to physical tables, the 'Metric' engine creates an additional metadata region physically for each individual physical data region used in storing some metadata needed by itself while maintaining mapping and other states. This metadata includes the mapping relationship between logical tables and physical tables, the mapping relationship between logical columns and physical columns etc. +Batch DDL operations reduce metadata work when many logical tables are created or updated together, such as during Prometheus Remote Write auto-creation or physical Region migration. diff --git a/versioned_docs/version-1.0/contributor-guide/datanode/overview.md b/versioned_docs/version-1.0/contributor-guide/datanode/overview.md index d0afe21b34..a21c112faa 100644 --- a/versioned_docs/version-1.0/contributor-guide/datanode/overview.md +++ b/versioned_docs/version-1.0/contributor-guide/datanode/overview.md @@ -7,28 +7,26 @@ description: Overview of Datanode in GreptimeDB, its responsibilities, component ## Introduction -`Datanode` is mainly responsible for storing the actual data for GreptimeDB. As we know, in GreptimeDB, -a `table` can have one or more `Region`s, and `Datanode` is responsible for managing the reading and writing -of these `Region`s. `Datanode` is not aware of `table` and can be considered as a `region server`. Therefore, -`Frontend` and `Metasrv` operate `Datanode` at the granularity of `Region`. +A Datanode stores and processes Region data. A table can contain multiple Regions, but the Datanode does not own table-level routing. Frontend sends data requests by Region, while Metasrv controls Region placement and lifecycle. -![Datanode](/datanode.png) +This boundary lets the same Region server host different storage engines without exposing their implementation to Frontend or Metasrv. + +![Frontend sends Region requests to the Datanode Region server, while Metasrv exchanges lifecycle instructions through the heartbeat task. The Region server uses the local query engine and dispatches requests to the Mito, Metric, or File Region engine.](/datanode-architecture.svg) ## Components -A `Datanode` contains all the components needed for a `region server`. Here we list some of the vital parts: - -- A gRPC service is provided for reading and writing region data, and `Frontend` uses this service - to read and write data from `Datanode`s. -- An HTTP service, through which you can obtain metrics, configuration information, etc., of the current node. -- `Heartbeat Task` is used to send heartbeat to the `Metasrv`. The heartbeat plays a crucial role in the - distributed architecture of GreptimeDB and serves as a basic communication channel for distributed coordination. - The upstream heartbeat messages contain important information such as the workload of a `Region`. If the - `Metasrv `has made scheduling(such as `Region` migration) decisions, it will send instructions to the - `Datanode` via downstream heartbeat messages. -- The `Datanode` does not parse user SQL or perform distributed planning. The user's query requests for one or - more `Table`s will be transformed into `Region` query requests in the `Frontend`. The `Datanode` is responsible - for executing these `Region` query plans with its local query engine. -- A `Region Manager` is used to manage all `Region`s on a `Datanode`. -- GreptimeDB supports a pluggable multi-engine architecture, with existing engines including `File Engine` and - `Mito Engine`. +The main components are: + +- The Region server tracks open Regions and dispatches reads, writes, and lifecycle requests to the engine registered for each Region. +- `Mito` is the primary time-series Region engine. `Metric` maps many logical metric Regions onto shared Mito Regions, and `File` exposes external files through the Region interface. +- The local query engine executes Region query plans. It does not parse client SQL or perform cluster-wide planning. +- The heartbeat task reports node and Region state to Metasrv and receives instructions such as open, close, upgrade, downgrade, and migration steps. +- gRPC carries Region requests to the Datanode. HTTP exposes node diagnostics such as metrics and configuration. + +## Region Request Lifecycle + +For a Mito write, the Region server selects Mito from the Region metadata. Mito appends the mutation to the WAL, applies it to a memtable, and later flushes the memtable to SST files. A Metric write is first rewritten with the logical-table identity and then delegated to its physical Mito Region. + +For a read, the local query engine executes the Region plan against a table provider backed by the Region engine. A Mito scan takes an immutable Region version, reads the relevant memtables and SST files, merges and deduplicates rows, and returns a stream of Arrow record batches. + +Region ownership can change without restarting the Datanode. Metasrv sends lifecycle instructions over the heartbeat stream; the Region server applies them to the engine and reports the new Region role and statistics in subsequent heartbeats. diff --git a/versioned_docs/version-1.0/contributor-guide/datanode/python-scripts.md b/versioned_docs/version-1.0/contributor-guide/datanode/python-scripts.md deleted file mode 100644 index 98909142a6..0000000000 --- a/versioned_docs/version-1.0/contributor-guide/datanode/python-scripts.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -keywords: [Python scripts, data analysis, CPython backend, RustPython interpreter, RecordBatch] -description: Guide on using Python scripts for data analysis in GreptimeDB, including backend options and setup instructions. ---- - -# Python Scripts - -## Introduction - -Python scripts are methods for analyzing data in GreptimeDB, -by running it in the database directly instead of fetching all the data from the database and running it locally. -This approach saves a lot of data transfer costs. -The image below depicts how the script works. -The `RecordBatch` (which is basically a column in a table with type and nullability metadata) -can come from anywhere in the database, -and the returned `RecordBatch` can be annotated in Python grammar to indicate its metadata, -such as type or nullability. -The script will do its best to convert the returned object to a `RecordBatch`, -whether it is a Python list, a `RecordBatch` computed from parameters, -or a constant (which is extended to the same length as the input arguments). - -![Python Coprocessor](/python-coprocessor.png) - -## Two optional backends - -### CPython Backend powered by PyO3 - -This backend is powered by [PyO3](https://pyo3.rs/v0.18.1/), enabling the use of your favourite Python libraries (such as NumPy, Pandas, etc.) and allowing Conda to manage your Python environment. - -But using it also involves some complications. You must set up the correct Python shared library, which can be a bit challenging. In general, you just need to install the `python-dev` package. However, if you are using Homebrew to install Python on macOS, you must create a proper soft link to `Library/Frameworks/Python.framework`. Detailed instructions on using PyO3 crate with different Python Version can be found [here](https://pyo3.rs/v0.18.1/building_and_distribution#configuring-the-python-version) - -### Embedded RustPython Interpreter - -An experiment [python interpreter](https://github.com/RustPython/RustPython) to run -the coprocessor script, it supports Python 3.10 grammar. You can use all the very Python syntax, see [User Guide/Python Coprocessor](/user-guide/python-scripts/overview.md) for more! diff --git a/versioned_docs/version-1.0/contributor-guide/datanode/query-engine.md b/versioned_docs/version-1.0/contributor-guide/datanode/query-engine.md index 5b83a30613..91f307aa59 100644 --- a/versioned_docs/version-1.0/contributor-guide/datanode/query-engine.md +++ b/versioned_docs/version-1.0/contributor-guide/datanode/query-engine.md @@ -7,51 +7,30 @@ description: Overview of GreptimeDB's query engine, its architecture, data repre ## Introduction -GreptimeDB's query engine is built on [Apache DataFusion][1] (subproject under [Apache -Arrow][2]), a brilliant query engine written in Rust. It provides a set of well functional components from -logical plan, physical plan and the execution runtime. Below explains how each component is orchestrated and their positions during execution. +GreptimeDB's query engine is built on [Apache DataFusion][1]. DataFusion supplies the logical and physical plan interfaces, optimizer framework, and execution runtime. GreptimeDB adds planners for its query languages, storage-aware optimizer rules, custom plan nodes, and distributed execution. -![Execution Procedure](/execution-procedure.png) +DDL and other control-plane operations are dispatched by the statement executor. The query engine receives plans for data processing, including the input side of operations such as `INSERT ... SELECT`. -The entry point is the logical plan, which is used as the general intermediate representation of a -query or execution logic etc. Two noticeable sources of logical plan are from: 1. the user query, like -SQL through SQL parser and planner; 2. the Frontend's distributed query, which is explained in details in the following section. +## Query Lifecycle -Next is the physical plan, or the execution plan. Unlike the logical plan which is a big -enumeration containing all the logical plan variants (except the special extension plan node), the -physical plan is in fact a trait that defines a group of methods invoked during -execution. All data processing logics are packed in corresponding structures that -implement the trait. They are the actual operations performed on the data, like -aggregator `MIN` or `AVG`, and table scan `SELECT ... FROM`. +1. The SQL, PromQL, or log-query planner resolves tables through the catalog and produces a DataFusion logical plan. GreptimeDB plan extensions represent operations that DataFusion does not provide directly. +2. DataFusion analyzer and optimizer rules run together with GreptimeDB rules. These rules normalize expressions and types, rewrite time-range operations, push projections and filters toward scans, and introduce distributed plan nodes when required. +3. The physical planner converts the optimized logical plan into streaming operators. GreptimeDB then applies physical rules for scan parallelism, ordering, and distributed execution. +4. Execution pulls Arrow record batches through the physical plan. Storage scans receive the projection and predicates, and downstream operators consume the resulting stream without materializing the complete result first. -The optimization phase which improves execution performance by transforming both logical and physical plans, is now all based on rules. It is also called, "Rule Based Optimization". Some of the rules are DataFusion native and others are customized in Greptime DB. In the future, we plan to add more -rules and leverage the data statistics for Cost Based Optimization/CBO. - -The last phase "execute" is a verb, stands for the procedure that reads data from storage, performs -calculations and generates the expected results. Although it's more abstract than previously mentioned concepts, you can just -simply imagine it as executing a Rust async function. And it's indeed a future (stream). - -`EXPLAIN [VERBOSE] ` is very useful if you want to see how your SQL is represented in the logical or physical plan. +Use [`EXPLAIN`](/reference/sql/explain.md) to inspect the logical and physical plans. `EXPLAIN ANALYZE` also executes the plan and reports runtime metrics. ## Data Representation -GreptimeDB uses [Apache Arrow][2] as the in-memory data representation. It's column-oriented, in -cross-platform format, and also contains many high-performance data operators. These features -make it easy to share data in many different environments and implement calculation logic. +GreptimeDB uses [Apache Arrow][2] record batches as its in-memory data representation. A record batch contains equal-length column arrays and a schema. Query operators exchange streams of these batches, which keeps the execution path columnar from Region scans through result encoding. ## Indexing -In time series data, there are two important dimensions: timestamp and tag columns (or like -primary key in a general relational database). GreptimeDB groups data in time buckets, so it's efficient -to locate and extract data within the expected time range at a very low cost. The mainly used persistent file format [Apache Parquet][3] in GreptimeDB helps a lot -- it -provides multi-level indices and filters that make it easy to prune data during querying. In the future, we -will make more use of this feature, and develop our separated index to handle more complex use cases. +Index construction and persistent index formats belong to the storage engine. The query layer supplies predicates and projections to a scan; Mito then uses time ranges, Parquet statistics, and indexes to avoid reading data that cannot match. See [Data Persistence and Indexing](./data-persistence-indexing.md). ## Distributed Execution -Covered in [Distributed Querying][6]. +In distributed mode, the Frontend plans the cluster-wide query and Datanodes execute Region-local subplans. [`MergeScan`](../frontend/distributed-querying.md) is the boundary between those stages. -[1]: https://github.com/apache/arrow-datafusion +[1]: https://datafusion.apache.org/ [2]: https://arrow.apache.org/ -[3]: https://parquet.apache.org -[6]: ../frontend/distributed-querying.md diff --git a/versioned_docs/version-1.0/contributor-guide/datanode/storage-engine.md b/versioned_docs/version-1.0/contributor-guide/datanode/storage-engine.md index c220c72208..b6963815b3 100644 --- a/versioned_docs/version-1.0/contributor-guide/datanode/storage-engine.md +++ b/versioned_docs/version-1.0/contributor-guide/datanode/storage-engine.md @@ -7,7 +7,7 @@ description: Overview of the storage engine in GreptimeDB, its architecture, com ## Introduction -The `storage engine` is responsible for storing the data of the database. Mito, based on [LSMT][1] (Log-structured Merge-tree), is the storage engine we use by default. We have made significant optimizations for handling time-series data scenarios, so mito engine is not suitable for general purposes. +Mito is GreptimeDB's default storage engine. It uses an [LSM tree][1] and is designed for time-series workloads rather than as a general-purpose embedded storage engine. ## Architecture @@ -24,8 +24,8 @@ The architecture is the same as a traditional LSMT engine: - Log records of the WAL can be stored on the local disk, or in a remote log service such as Kafka (remote WAL) that implements the `Log Store` API. - Memtables: - - Data is written into the `active memtable`, aka `mutable memtable` first. - - When a `mutable memtable` is full, it will be changed to a `read-only memtable`, aka `immutable memtable`. + - Mito routes rows by time index into mutable memtables. + - A flush freezes the mutable memtables, installs a new mutable set for writes, and writes the frozen memtables to SST files. - SST - The full name of SST, aka SSTable is `Sorted String Table`. - `Immutable memtable` is flushed to persistent storage and produces an SST file. @@ -103,7 +103,9 @@ Each Parquet SST is split into row groups, the unit that Parquet can read or ski Mito supports two SST formats: `flat` and `primary_key`. `flat` is the default for new tables and works well across primary-key cardinalities, including high-cardinality keys. `primary_key` is the legacy format kept for compatibility with older tables. See [SST format](/reference/sql/create.md#create-a-table-with-sst-format) and the [table design guide](/user-guide/deployments-administration/performance-tuning/design-table.md#sst-format) for more details. -SST layout +![The default flat Mito SST layout combines file-level metadata with Parquet row groups containing data columns and merge metadata.](/mito-sst-layout.svg) + +An SST may span more than one compaction time window. ## Scan Pruning diff --git a/versioned_docs/version-1.0/contributor-guide/datanode/wal.md b/versioned_docs/version-1.0/contributor-guide/datanode/wal.md index 4ecb19ef02..bb6e733bae 100644 --- a/versioned_docs/version-1.0/contributor-guide/datanode/wal.md +++ b/versioned_docs/version-1.0/contributor-guide/datanode/wal.md @@ -7,30 +7,26 @@ description: Introduction to Write-Ahead Logging (WAL) in GreptimeDB, its purpos ## Introduction -Our storage engine is inspired by the Log-structured Merge Tree (LSMT). Mutating operations are -applied to a MemTable instead of persisting to disk, which significantly improves performance but -also brings durability-related issues, especially when the Datanode crashes unexpectedly. Similar -to all LSMT-like storage engines, GreptimeDB uses a write-ahead log (WAL) to ensure data durability -and is safe from crashing. +Mito buffers writes in memtables before flushing them to SST files. It first appends each Region's mutations to the write-ahead log (WAL), so data that has not reached an SST can be recovered. -WAL is an append-only file group. All `INSERT` and `DELETE` operations are transformed into -operation entries and then appended to WAL. Once operation entries are persisted to the underlying -file, the operation can be further applied to MemTable. +The WAL uses a common log-store abstraction with local raft-engine and remote Kafka providers. -When the Datanode restarts, operation entries in WAL are replayed to reconstruct the correct -in-memory state. +## Write and Recovery Cycle -![WAL in Datanode](/wal.png) +The order of a normal write is: + +1. The Region worker assigns sequence numbers and a WAL entry ID. +2. It appends the mutations to the WAL. If the append fails, the mutations are not applied to the memtable. +3. After the append succeeds, Mito writes the mutations to the memtable and publishes the new committed sequence. +4. A flush writes immutable SST files and persists a manifest edit containing the new files and `flushed_entry_id`. +5. After the manifest edit is durable, WAL entries through `flushed_entry_id` are marked obsolete. The log store may reclaim them later. + +The manifest is the recovery boundary. On a normal reopen, Mito rebuilds the Region from the manifest and replays WAL entries starting at `flushed_entry_id + 1`. Region transitions may supply a later replay checkpoint, but they never replay entries before the persisted flush boundary. ## Namespace -Namespace of WAL is used to separate entries from different tables (different regions). Append and -read operations must provide a Namespace. Currently, region ID is used as the Namespace, because -each region has a MemTable that needs to be reconstructed when Datanode restarts. +WAL entries are isolated by Region, not by table. Each append and read identifies a Region namespace so one Region can be replayed or truncated independently. The local raft-engine provider uses the Region ID as its namespace ID. Kafka keeps Region identity within the provider's topic-backed log. ## Synchronous/Asynchronous flush -By default, appending to WAL is asynchronous, which means the writer will not wait until entries are -flushed to disk. This setting provides higher performance, but may lose data when running host shutdown unexpectedly. In the other hand, synchronous flush provides higher durability at the cost of performance. - -In v0.4 version, the new region worker architecture can use batching to alleviate the overhead of sync flush. +For the local raft-engine provider, `sync_write` controls whether an append waits for the log to be synced to durable storage. It defaults to `false`. Asynchronous writes reduce latency but can lose recently acknowledged entries if the host fails before buffered data is synced. Kafka WAL durability is controlled by its producer and cluster settings instead of this local option. diff --git a/versioned_docs/version-1.0/contributor-guide/flownode/arrangement.md b/versioned_docs/version-1.0/contributor-guide/flownode/arrangement.md index aed75af777..8b472ea316 100644 --- a/versioned_docs/version-1.0/contributor-guide/flownode/arrangement.md +++ b/versioned_docs/version-1.0/contributor-guide/flownode/arrangement.md @@ -5,6 +5,8 @@ description: Details on the arrangement component in Flownode, which stores stat # Arrangement +This page describes state used by Flownode's legacy streaming mode. Batching mode does not use an Arrangement. + Arrangement stores the state in the dataflow's process. It stores the streams of update flows for further querying and updating. The arrangement essentially stores key-value pairs with timestamps to mark their change time. diff --git a/versioned_docs/version-1.0/contributor-guide/flownode/batching_mode.md b/versioned_docs/version-1.0/contributor-guide/flownode/batching_mode.md index df6cd15cf8..d90dc87bfd 100644 --- a/versioned_docs/version-1.0/contributor-guide/flownode/batching_mode.md +++ b/versioned_docs/version-1.0/contributor-guide/flownode/batching_mode.md @@ -9,13 +9,13 @@ This guide provides a brief overview of the batching mode in `flownode`. It's in ## Overview -The batching mode in `flownode` is designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. This is in contrast to a streaming mode where data is processed as it arrives. +The batching mode in `flownode` is designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. This is in contrast to the legacy streaming path, which processes data as it arrives and is retained for compatibility but deprecated for new workloads. The core idea is to: 1. Define a `flow` with a SQL query that aggregates data from a source table into a sink table. 2. The query typically includes a time window function (e.g., `date_bin`) on a timestamp column. 3. When new data is inserted into the source table, the system marks the corresponding time windows as "dirty." -4. A background task periodically wakes up, identifies these dirty windows, and re-runs the aggregation query for those specific time ranges. +4. A background task runs on its own cadence, consumes the pending dirty windows at its next evaluation, and re-runs the aggregation query for those time ranges. 5. The results are then inserted into the sink table, effectively updating the aggregated view. ## Architecture @@ -39,15 +39,15 @@ A `BatchingTask` represents a single, independent data flow. Each task is associ - **State (`TaskState`)**: This contains the dynamic, mutable state of the task, most importantly the `DirtyTimeWindows`. - **Execution Loop**: The task runs an infinite loop (`start_executing_loop`) that: 1. Checks for a shutdown signal. - 2. Waits for a scheduled interval or until it's woken up. + 2. Sleeps until its next evaluation time. A task with an evaluation schedule sleeps until the next scheduled time; an adaptive task sleeps for a polling interval derived from the time window size and the minimum refresh duration. 3. Generates a new query plan (`gen_insert_plan`) based on the current set of dirty time windows. 4. Executes the query (`execute_logical_plan`) against the database. 5. Cleans up the processed dirty windows. ### `TaskState` and `DirtyTimeWindows` -- **`TaskState`**: This struct tracks the runtime state of a `BatchingTask`. It includes `dirty_time_windows`, which is crucial for determining what work needs to be done. -- **`DirtyTimeWindows`**: This is a key data structure that keeps track of which time windows have received new data since the last query execution. It stores a set of non-overlapping time ranges. When a task's execution loop runs, it consults this structure to build a `WHERE` clause that filters the source table for only the dirty time windows. +- **`TaskState`**: This struct tracks the runtime state of a `BatchingTask`, including the `dirty_time_windows` that determine its pending work. +- **`DirtyTimeWindows`**: This data structure tracks which time windows have received new data since the last query execution. It stores a set of non-overlapping time ranges. The execution loop uses it to build a `WHERE` clause that selects only the dirty windows from the source table. ### `TimeWindowExpr` @@ -56,15 +56,15 @@ The `TimeWindowExpr` is a helper utility for dealing with time window expression - **Evaluation**: It can take a timestamp and evaluate the time window expression to determine the start and end of the window that the timestamp falls into. - **Window Size**: It can also determine the size (duration) of the time window from the expression. -This is essential for both marking windows as dirty and for generating the correct filter conditions when querying the source table. +The same calculation is used to mark dirty windows and generate the source-table filters. ## Query Execution Flow Here's a simplified step-by-step walkthrough of how a query is executed in batch mode: 1. **Data Ingestion**: New data is written to a source table. -2. **Marking Dirty**: The `BatchingEngine` receives a notification about the new data. It uses the `TimeWindowExpr` associated with each relevant flow to determine which time windows are affected by the new data points. These windows are then added to the `DirtyTimeWindows` set in the corresponding `TaskState`. -3. **Task Wake-up**: The `BatchingTask`'s execution loop wakes up, either due to its periodic schedule or because it was notified of a large backlog of dirty windows. +2. **Marking Dirty**: The `BatchingEngine` receives a notification about the new data. It uses the `TimeWindowExpr` associated with each relevant flow to determine which time windows are affected by the new data points. These windows are then added to the `DirtyTimeWindows` set in the corresponding `TaskState`. Marking a window dirty does not wake the task. +3. **Next Evaluation**: The `BatchingTask`'s execution loop reaches its next evaluation, either at a scheduled time or after its adaptive polling interval, and consumes the pending dirty windows. 4. **Plan Generation**: The task calls `gen_insert_plan`. This method: - Inspects the `DirtyTimeWindows`. - Generates a series of `OR`'d `WHERE` clauses (e.g., `(ts >= 't1' AND ts < 't2') OR (ts >= 't3' AND ts < 't4') ...`) that cover the dirty windows. diff --git a/versioned_docs/version-1.0/contributor-guide/flownode/dataflow.md b/versioned_docs/version-1.0/contributor-guide/flownode/dataflow.md index 000a65edb3..8000f14166 100644 --- a/versioned_docs/version-1.0/contributor-guide/flownode/dataflow.md +++ b/versioned_docs/version-1.0/contributor-guide/flownode/dataflow.md @@ -1,17 +1,38 @@ --- -keywords: [dataflow module, SQL query transformation, execution plan, DAG, map and reduce operations] -description: Explanation of the dataflow module in Flownode, its operations, internal data handling, and future enhancements. +keywords: [Flownode, batching mode, streaming mode, dataflow, dirty time windows] +description: How Flownode selects and runs its batching and legacy streaming execution paths. --- # Dataflow +Flownode has two internal execution paths: + +- **Batching mode** is the primary path for aggregation and TQL workloads. It evaluates queries over persisted source data and writes materialized results to a sink table. +- **Streaming mode** is the legacy path retained for compatibility and deprecated for new workloads. It incrementally processes rows mirrored from Frontend as they arrive. + +Users do not select the mode directly. When a Flow is created, GreptimeDB chooses the path from the query and source-table properties. Aggregation, `DISTINCT`, and TQL queries use batching mode. Simple non-aggregation queries, and any Flow whose source table has `ttl = 'instant'`, currently use streaming mode. + +## Batching mode + +Batching mode reuses GreptimeDB's query engine instead of maintaining an operator graph for every incoming row. For a time-windowed Flow, its main loop is: + +1. A source-table write marks the affected time windows as dirty. +2. A `BatchingTask` runs on its evaluation schedule or adaptive polling cadence and collects the pending dirty windows at that evaluation. Marking a window dirty does not wake the task. +3. The task adds time predicates for those windows to the Flow query and asks Frontend to execute it against the source tables. +4. The query result is inserted into the sink table, updating the materialized result for windows that were evaluated. +5. Successfully processed windows are removed from the dirty set. Failed work remains available for a later evaluation. + +Flows with an evaluation interval but without a time-window expression run the complete query on each scheduled evaluation. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Flownode Batching Mode Developer Guide](./batching_mode.md) for the task and dirty-window components. + +## Streaming mode + The `dataflow` module (see `flow::compute` module) is the core computing module of `flow`. It takes a SQL query and transforms it into flow's internal execution plan. This execution plan is then rendered into an actual dataflow, which is essentially a directed acyclic graph (DAG) of functions with input and output ports. -The dataflow is triggered to run when needed. +New row changes drive the graph incrementally. -Currently, this dataflow only supports `map` and `reduce` operations. Support for `join` operations will be added in the future. +The renderer supports map/filter/project and reduce operations. Join and union plan nodes exist, but their streaming renderers are not implemented. Internally, the dataflow handles data in row format, using a tuple `(row, time, diff)`. Here, `row` represents the actual data being passed, which may contain multiple `Value` objects. `time` is the system time which tracks the progress of the dataflow, and `diff` typically represents the insertion or deletion of the row (+1 or -1). -Therefore, the tuple represents the insert/delete operation of the `row` at a given system `time`. \ No newline at end of file +Therefore, the tuple represents the insert/delete operation of the `row` at a given system `time`. Stateful operators keep indexed traces of these changes in an [Arrangement](./arrangement.md). diff --git a/versioned_docs/version-1.0/contributor-guide/frontend/distributed-querying.md b/versioned_docs/version-1.0/contributor-guide/frontend/distributed-querying.md index 21ee07d7e8..ca3822e113 100644 --- a/versioned_docs/version-1.0/contributor-guide/frontend/distributed-querying.md +++ b/versioned_docs/version-1.0/contributor-guide/frontend/distributed-querying.md @@ -5,29 +5,16 @@ description: Describes the process of distributed querying in GreptimeDB, focusi # Distributed Querying -Most steps of querying in frontend and datanode are identical. The only difference is that -Frontend have a "special" step in planning phase to make the logical query plan distributed. -Let's reference it as "dist planner" in the following text. - -The modified, distributed logical plan has multiple stages, each of them is executed in different -server node. +Frontend and Datanode use the same DataFusion-based query engine. In distributed mode, Frontend adds a planning step that separates work performed by Datanodes from work completed by Frontend. ![Frontend query](/frontend-query.png) ## Dist Planner -Planner will traverse the input logical plan, and split it into multiple stages by the "[commutativity -rule](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md)". +The distributed planner rewrites the logical plan. It pushes compatible operators toward table scans and wraps remote subplans in `MergeScan` nodes. Partition predicates are also used to prune Regions before the remote work is scheduled. -This rule is under heavy development. At present it will consider things like: -- whether the operator itself is commutative -- how the partition rule is configured -- etc... +Whether an operator can be pushed down depends on the plan shape and the operator's properties. Unsupported parts remain on Frontend. The original design and its commutativity rules are described in the [distributed planner RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md). ## Dist Plan -Except the first stage, which have to read data from files in storage. All other stages' leaf node -are actually a gRPC call to its previous stage. - -Sub-plan in a stage is itself a complete logical plan, and can be executed independently without -the follow up stages. The plan is encoded in [substrait format](https://substrait.io). +A remote input is a complete logical subplan, not just a table scan. Frontend serializes the subplan in [Substrait](https://substrait.io) format and sends a Region-specific request to the Datanode that owns the data. The Datanode plans and executes it locally, then streams the result back. Frontend merges the remote streams and executes any operators that were not pushed down. diff --git a/versioned_docs/version-1.0/contributor-guide/frontend/overview.md b/versioned_docs/version-1.0/contributor-guide/frontend/overview.md index 3a2f63e7ae..18cabfc2b8 100644 --- a/versioned_docs/version-1.0/contributor-guide/frontend/overview.md +++ b/versioned_docs/version-1.0/contributor-guide/frontend/overview.md @@ -5,27 +5,47 @@ description: Overview of GreptimeDB's Frontend component - a stateless proxy ser # Frontend -The **Frontend** is a stateless service that serves as the entry point for client requests in GreptimeDB. It provides a unified interface for multiple database protocols and acts as a proxy that forwards read/write requests to appropriate Datanodes in the distributed system. +Frontend is GreptimeDB's stateless request-orchestration service. The server layer terminates protocols and converts wire messages; Frontend supplies the database behavior behind those handlers, including permission checks, statement execution, routing, and distributed query planning. + +Frontend does not store table data. It caches catalog and route metadata obtained from Metasrv, and Metasrv invalidates those caches through heartbeat responses when metadata changes. ## Core Functions -- **Protocol Support**: Multiple database protocols including SQL, PromQL, MySQL, and PostgreSQL. See [Protocols][1] for details -- **Request Routing**: Routes requests to appropriate Datanodes based on metadata -- **Query Distribution**: Splits distributed queries across multiple nodes -- **Response Aggregation**: Combines results from multiple Datanodes -- **Authorization**: Security and access control validation +- Provide query and ingestion behavior for the supported [protocols][1]. +- Resolve catalogs, schemas, tables, and Region routes. +- Validate permissions before executing a request. +- Plan distributed queries and merge results from Datanodes. +- Convert table-level writes and deletes into Region requests. ## Architecture ### Key Components -- **Protocol Handlers**: Handle different database protocols -- **Catalog Manager**: Caches metadata from Metasrv to enable efficient request routing and schema validation -- **Dist Planner**: Converts logical plans to distributed execution plans -- **Request Router**: Determines target Datanodes for each request + +- Protocol handlers adapt SQL, PromQL, gRPC ingestion, and observability protocols to Frontend's internal request interfaces. +- The catalog and partition managers provide table metadata, partition rules, and Region routes. +- The statement executor dispatches queries, DML, and DDL to their respective execution paths. +- The distributed planner replaces table scans with `MergeScan` plans that can run across Datanodes. ### Request Flow -![request flow](/request_flow.png) +The request path depends on the operation. + +#### Queries + +1. A protocol handler creates the query context and performs authentication and permission checks. +2. The language-specific planner produces a logical plan. In distributed mode, the planner uses partition metadata to select Regions and constructs a distributed plan. +3. Frontend sends Region subplans to the owning Datanodes. Datanodes execute them against local Region engines and return streams of Arrow record batches. +4. Frontend runs the remaining operators, merges the streams, and formats the result for the client protocol. + +#### Writes and deletes + +1. Frontend validates the request against the table schema. Protocols that support schema-on-write may create a missing table or add columns before retrying the write. +2. The partition rule assigns rows to Regions. Frontend builds one Region request per target and routes it to the current Region leader. +3. The Datanode's Region server dispatches each request to the Region engine. In standalone mode, the same request is sent to an embedded Region server instead of over RPC. + +#### DDL + +The statement executor converts DDL into a task. In distributed mode, Metasrv runs that task as a persisted procedure, updates metadata, and coordinates Region operations on Datanodes. Standalone mode uses the same statement boundary with local implementations of the metadata and procedure services. ### Deployment diff --git a/versioned_docs/version-1.0/contributor-guide/frontend/table-sharding.md b/versioned_docs/version-1.0/contributor-guide/frontend/table-sharding.md index a60276d14e..beaece40c0 100644 --- a/versioned_docs/version-1.0/contributor-guide/frontend/table-sharding.md +++ b/versioned_docs/version-1.0/contributor-guide/frontend/table-sharding.md @@ -5,21 +5,15 @@ description: Explains how table data in GreptimeDB is sharded and distributed, i # Table Sharding -The sharding of stored data is essential to any distributed database. This document will describe how table's data in GreptimeDB is being sharded, and distributed. +GreptimeDB shards a table into Regions. Partition expressions define which rows belong to each Region, while Region routes define which Datanode currently owns each Region. ## Partition -For the syntax of creating a partitioned table, please refer to the [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) section in the User Guide. +A partition is a logical row set described by an expression over one or more columns. The partition layout must cover the table's input domain so each row has one target Region. See [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) for the SQL syntax and supported expressions. ## Region -The data within a table is logically split after creating partitions. You may ask the question " -how are the data, after being logically partitioned, stored in the GreptimeDB? The answer is in "`Region`"s. - -Each region is corresponding to a partition, and stores the data in the partition. The regions are distributed among -`Datanode`s. `Metasrv` manages the route information that maps regions to Datanodes. -If the partition layout needs to change after table creation, GreptimeDB supports explicit -[repartitioning](/user-guide/deployments-administration/manage-data/repartition.md) through split and merge operations. +Each partition maps to one Region. Region IDs remain the storage and routing identity used by Frontend, Datanode, and Metasrv. Multiple Regions from the same table may be placed on one Datanode. The relationship between partition and region can be viewed as the following diagram: @@ -53,3 +47,14 @@ The relationship between partition and region can be viewed as the following dia │ │ └──────────────────────────────────┘ Could be placed in one Datanode +``` + +## Routing and Pruning + +For writes, Frontend evaluates the partition rule for each row, groups rows by Region, and sends Region requests to the current leaders from the route table. + +For queries, the distributed planner compares query predicates with the partition expressions. It scans only Regions that can satisfy the predicates. If partition metadata is missing or cannot be interpreted safely, the planner falls back to all Regions rather than risk omitting data. + +## Changing the Partition Layout + +[Repartitioning](/user-guide/deployments-administration/manage-data/repartition.md) changes an existing layout through explicit split and merge operations. Metasrv runs the change as a persisted procedure, updates the Region routes and partition expressions, and invalidates stale table-route caches. New requests use the published layout after their Frontend refreshes that metadata. diff --git a/versioned_docs/version-1.0/contributor-guide/getting-started.md b/versioned_docs/version-1.0/contributor-guide/getting-started.md index b17184dfa8..7e8cca3ad8 100644 --- a/versioned_docs/version-1.0/contributor-guide/getting-started.md +++ b/versioned_docs/version-1.0/contributor-guide/getting-started.md @@ -15,14 +15,13 @@ At the moment, GreptimeDB supports Linux (both amd64 and arm64), macOS (both amd ### Build Dependencies -- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) (optional) +- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) (optional; needed to clone the repository, not to build it) - C/C++ Toolchain: provides essential tools for compiling and linking. This is available either as `build-essential` on ubuntu or a similar name on other platforms. -- Rust nightly toolchain ([guide][1]) - - Compile the source code +- [Rustup][1]. The repository pins the required nightly toolchain in `rust-toolchain.toml`. - Protobuf ([guide][2]) - Compile the proto file - Note that the version needs to be >= 3.15. You can check it with `protoc --version` -- Machine: Recommended memory is 16GB or more, or use the [mold](https://github.com/rui314/mold) tool to reduce memory usage during linking. +- Machine: 16GB of memory or more is recommended. On a smaller machine, use [mold](https://github.com/rui314/mold) to reduce memory usage during linking. [1]: [2]: diff --git a/versioned_docs/version-1.0/contributor-guide/how-to/how-to-write-sdk.md b/versioned_docs/version-1.0/contributor-guide/how-to/how-to-write-sdk.md index 40d0bc3713..e306448642 100644 --- a/versioned_docs/version-1.0/contributor-guide/how-to/how-to-write-sdk.md +++ b/versioned_docs/version-1.0/contributor-guide/how-to/how-to-write-sdk.md @@ -1,21 +1,17 @@ --- keywords: [gRPC SDK, GreptimeDatabase, Handle, HandleRequests, GreptimeRequest, GreptimeResponse] -description: Explains how to write a gRPC SDK for GreptimeDB, focusing on the GreptimeDatabase service, its methods, and the structure of requests and responses. +description: Protocol contracts and error-handling requirements for a GreptimeDB gRPC ingestion SDK. --- # How to write a gRPC SDK for GreptimeDB -A GreptimeDB gRPC SDK only needs to handle the writes. The reads are standard SQL and PromQL, can be handled by any JDBC -client or Prometheus client. This is also why GreptimeDB gRPC SDKs are all named -like "`greptimedb-ingester-`". Please make sure your GreptimeDB SDK follow the same naming convention. +GreptimeDB's public gRPC SDKs are ingestion clients. Queries normally use SQL or PromQL through their standard clients. A new SDK should therefore focus on writes and deletes unless it has a separate requirement, and follow the `greptimedb-ingester-` naming convention. See the [gRPC SDK overview](/user-guide/ingest-data/for-iot/grpc-sdks/overview.md) for the user-facing API. ## `GreptimeDatabase` Service -GreptimeDB defines a custom gRPC service called `GreptimeDatabase`. All you need to do in your SDK are implement it. You -can find its Protobuf -definitions [here](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto). +Generate client stubs from the [`GreptimeDatabase` Protobuf definition](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto). Do not maintain a handwritten copy of the messages or service definition. -The service contains two RPC methods: +The service provides a unary method and a client-streaming method: ```protobuf service GreptimeDatabase { @@ -25,13 +21,9 @@ service GreptimeDatabase { } ``` -The `Handle` method is for unary call: when a `GreptimeRequest` is received and processed by a GreptimeDB -server, it responds with a `GreptimeResponse` immediately. +`Handle` returns one response for one request. It is the usual choice for an SDK's insert and delete APIs. -The `HandleRequests` acts in -a "[Client streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc)" style. It ingests a -stream of `GreptimeRequest`, and handles them on the fly. After all the requests have been handled, it returns a -summarized `GreptimeResponse`. Through `HandleRequests`, we can achieve a very high throughput of requests handling. +`HandleRequests` is a [client-streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc). The server returns a cumulative response only after the client closes the request stream. An SDK that exposes streaming must document this acknowledgement boundary and bind the stream to one endpoint. ### `GreptimeRequest` @@ -51,13 +43,13 @@ message GreptimeRequest { } ``` -A `RequestHeader` is needed, it includes some context, authentication and others. The "oneof" field contains the request -to the GreptimeDB server. +A client must populate `RequestHeader` with the database context and authentication expected by the server. Set exactly one request variant. -Note that we have two types of insertions, one is in the form of "column" (the `InsertRequests`), and the other is " -row" (`RowInsertRequests`). It's generally recommended to use the "row" form, since it's more natural for insertions on -a table, and easier to use. However, if there's a need to insert a large number of columns at once, or there're plenty -of "null" values to insert, the "column" form is better to be used. +The message also contains query and DDL variants used by internal callers. The public ingester API should not expose them: `GreptimeDatabase` does not return query result streams. + +GreptimeDB accepts row-oriented `RowInsertRequests` and column-oriented `InsertRequests`. Row-oriented requests are the default for public ingestion APIs. A column-native client may use the column form, but it must keep column lengths consistent and preserve null values, timestamp precision, data types, and column semantic types during conversion. + +Deletes have the same row-oriented and column-oriented distinction. Expose only the forms that the SDK can map without losing type information. ### `GreptimeResponse` @@ -70,8 +62,18 @@ message GreptimeResponse { } ``` -The `ResponseHeader` contains the response's status code, and error message (if there's any). The "oneof" response only -contains the affected rows for now. +On success, the response contains a successful header and `affected_rows`. Treat that value as the number acknowledged by the server, including the cumulative value returned when a request stream closes. + +Request failures are returned as a gRPC status. When present, the trailing metadata key `x-greptime-err-code` carries GreptimeDB's error code, and the status message carries the error text. Preserve the gRPC status and expose the GreptimeDB error code rather than replacing them with a generic SDK error. + +## Retry and Delivery Semantics + +Retries must be bounded and observable. A unary request may be retried only when the failure is classified as retryable and the deadline still permits it. Do not retry cancellation or deadline-expiration errors. + +A lost response does not prove that the server rejected a write. Retrying such a request can insert duplicate rows unless the caller's data model makes the operation idempotent. Document this possibility and return the final error when delivery is ambiguous. + +Do not transparently retry a partially sent `HandleRequests` stream. The server may already have accepted some requests even though the client has not received the cumulative response. Close the failed stream and report the ambiguity to the caller. + +Keep Arrow Flight bulk ingestion separate from the `GreptimeDatabase` RPCs. Its batching and partial-acceptance behavior needs its own API contract. -GreptimeDB has a lot of SDKs now, you can refer to -them [here](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=) for some examples. +Use the existing [GreptimeDB ingester repositories](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=) to compare public API conventions, but derive wire behavior from the current Protobuf definition and server contract. diff --git a/versioned_docs/version-1.0/contributor-guide/metasrv/admin-api.md b/versioned_docs/version-1.0/contributor-guide/metasrv/admin-api.md index b091b48b70..d1f2ee6dbe 100644 --- a/versioned_docs/version-1.0/contributor-guide/metasrv/admin-api.md +++ b/versioned_docs/version-1.0/contributor-guide/metasrv/admin-api.md @@ -1,6 +1,6 @@ --- -keywords: [admin api, health check, leader query, heartbeat, maintenance mode, RESTful API] -description: Details the Admin API for Metasrv, including endpoints for health checks, leader queries, heartbeat data, maintenance mode, and Procedure Manager controls. +keywords: [admin api, health check, leader query, heartbeat, maintenance mode, recovery mode, table id sequence] +description: Details the Metasrv Admin API for status inspection, cluster controls, and metadata recovery. --- # Admin API @@ -9,16 +9,17 @@ description: Details the Admin API for Metasrv, including endpoints for health c Note that all Admin API endpoints in this document listen on Metasrv's `HTTP_PORT`, which defaults to `4000`. ::: -The Admin API provides a simple way to view and manage cluster information, including metasrv health detection, metasrv leader query, datanode heartbeat detection, maintenance mode, and Procedure Manager controls. - -The Admin API is an HTTP service that provides a set of RESTful APIs that can be called through HTTP requests. The Admin API is simple, user-friendly and safe. +The Admin API exposes Metasrv status, cluster controls, and metadata recovery operations over HTTP. It does not provide authentication, and some endpoints change cluster behavior or metadata allocation. Deployments must protect the HTTP port with network-level controls. This page covers the following APIs: - /health - /leader - /heartbeat +- /node-lease - /maintenance - /procedure-manager +- /recovery +- /sequence/table All these APIs are under the parent resource `/admin`. @@ -26,7 +27,7 @@ In the following sections, we assume that your metasrv instance is running on lo ## /health HTTP endpoint -The `/health` endpoint accepts GET HTTP requests and you can use this endpoint to check the health of your metasrv instance. +The `/health` endpoint accepts GET requests and returns `OK` when the HTTP service is running. It does not check whether this Metasrv is the leader or whether external dependencies are available. ### Definition @@ -120,9 +121,17 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100' ] ``` +## /node-lease HTTP endpoint + +The `/node-lease` endpoint returns the current leases recorded for Datanodes. Use it when diagnosing whether Metasrv still considers a Datanode active. + +```bash +curl -X GET http://localhost:4000/admin/node-lease +``` + ## /maintenance HTTP endpoint -Cluster Maintenance Mode is a safety feature in GreptimeDB that temporarily disables automatic cluster management operations. This mode is particularly useful during cluster upgrades, planned downtime, and any operation that might temporarily affect cluster stability. For more details, please refer to [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md). +Maintenance mode temporarily disables automatic cluster management operations during upgrades, planned downtime, or similar work. See [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md) for its effect on the cluster. The `/maintenance` endpoint supports the following HTTP requests: @@ -155,3 +164,39 @@ The response body uses the following format: "status": "running" } ``` + +## /recovery HTTP endpoints + +Recovery mode gates metadata repair endpoints such as manual table ID sequence changes. It is intended for recovery work, not routine maintenance. + +- `GET /admin/recovery/status`: query whether recovery mode is enabled. +- `POST /admin/recovery/enable`: enable recovery mode. +- `POST /admin/recovery/disable`: disable recovery mode. + +The response body uses the following format: + +```json +{ + "enabled": true +} +``` + +Disable recovery mode after the repair is complete. Use [maintenance mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md) instead when the goal is to suspend automatic cluster operations during planned maintenance. + +## /sequence/table HTTP endpoints + +These endpoints inspect or repair the table ID sequence: + +- `GET /admin/sequence/table/next-id`: return the next table ID without allocating it. +- `POST /admin/sequence/table/set-next-id`: advance the next table ID. + +Setting the sequence requires recovery mode. The new value must be greater than the current value; the API cannot move the sequence backwards. Recovery mode is an API precondition, not a DDL barrier. Follow [Manage table ID sequences](/user-guide/deployments-administration/maintenance/sequence-management.md) for the required cluster-wide procedure. + +```bash +curl -X POST \ + -H 'Content-Type: application/json' \ + -d '{"next_table_id": 2048}' \ + http://localhost:4000/admin/sequence/table/set-next-id +``` + +Changing this value affects IDs allocated to future tables. diff --git a/versioned_docs/version-1.0/contributor-guide/metasrv/overview.md b/versioned_docs/version-1.0/contributor-guide/metasrv/overview.md index 7aa052dd7e..6458b6380d 100644 --- a/versioned_docs/version-1.0/contributor-guide/metasrv/overview.md +++ b/versioned_docs/version-1.0/contributor-guide/metasrv/overview.md @@ -1,161 +1,101 @@ --- -keywords: [metasrv, metadata, request-router, load balancing, election, high availability, heartbeat] -description: Provides an overview of the Metasrv service, its components, interactions with the Frontend, architecture, and key functionalities like distributed consensus and heartbeat management. +keywords: [metasrv, metadata, routing, leader election, procedure, heartbeat] +description: Overview of the metadata and coordination mechanisms provided by Metasrv. --- # Metasrv -![meta](/meta.png) - ## What's in Metasrv -- Store metadata (Catalog, Schema, Table, Region, etc.) -- Request-Router. It tells the Frontend where to write and read data. -- Load balancing for Datanode, determines who should handle new table creation requests, more precisely, it makes resource allocation decisions. -- Election & High Availability, GreptimeDB is designed in a Leader-Follower architecture, only Leader nodes can write while Follower nodes can read, the number of Follower nodes is usually >= 1, and Follower nodes need to be able to switch to Leader quickly when Leader is not available. -- Statistical data collection (reported via Heartbeats on each node), such as CPU, Load, number of Tables on the node, average/peak data read/write size, etc., can be used as the basis for distributed scheduling. +Metasrv is the metadata and coordination service in a distributed GreptimeDB cluster. It does not sit on the data path. Its main responsibilities are: + +- storing Catalog, Schema, Table, Region, route, and node metadata; +- choosing Datanodes for new Regions and maintaining table routes; +- electing one Metasrv leader to coordinate metadata changes; +- running recoverable procedures for DDL, Region migration, failover, and repartitioning; +- tracking node leases and Region statistics through heartbeats; +- broadcasting cache invalidations to Frontends, Datanodes, and Flownodes when metadata changes; +- sending Region lifecycle instructions to Datanodes. ## How the Frontend interacts with Metasrv -First, the routing table in Request-Router is in the following structure (note that this is only the logical structure, the actual storage structure varies, for example, endpoints may have dictionary compression). +Frontend obtains table metadata and Region routes from Metasrv and caches them locally. Metadata-changing statements are sent to the Metasrv leader, while reads and writes use the cached routes to reach Datanodes directly. + +The control and data paths are separate: + +```text +Frontend + |-- metadata lookup and DDL ------------> Metasrv leader + `-- Region reads and writes ------------> Datanode + +Metasrv leader + |-- Region lifecycle instructions ------> Datanode + `-- cache invalidations ----------------> Frontend / Datanode / Flownode +Datanode + `-- heartbeat, lease renewal, Region stats -> Metasrv leader ``` - table_A - table_name - table_schema // for physical plan - regions - region_1 - mutate_endpoint - select_endpoint_1, select_endpoint_2 - region_2 - mutate_endpoint - select_endpoint_1, select_endpoint_2, select_endpoint_3 - region_xxx - table_B - ... + +In steady state, a table route records one leader peer and zero or more follower peers for each Region. The leader is the write target. Deployments with read-replica support can route reads to followers: + +```text +Table route + |-- Region 0 + | |-- leader -> Datanode A + | `-- followers -> Datanode B, Datanode C + `-- Region 1 + `-- leader -> Datanode D ``` +Region migration or failover changes peer roles and can temporarily leave a Region without a leader. Frontend refreshes its cached route before sending subsequent reads or writes to the current peers. + ### Create Table -1. The Frontend sends `CREATE TABLE` requests to Metasrv. -2. Plan the number of Regions according to the partition rules contained in the request. -3. Check the global view of resources available to Datanodes (collected by Heartbeats) and assign one node to each region. -4. The Frontend creates the table and stores the `Schema` to Metasrv after successful creation. +1. Frontend submits the DDL request to the Metasrv leader. +2. Metasrv derives Regions from the partition rules and [selects a Datanode for each Region](/contributor-guide/metasrv/selector.md). +3. A persisted procedure creates the Regions and records the table and route metadata. If leadership changes, the procedure can resume from its persisted state. +4. Metasrv notifies Frontends after the metadata change is committed so their caches can be refreshed. ### Insert -1. The Frontend fetches the routes of the specified table from Metasrv. Note that the smallest routing unit is the route of the table (several regions), i.e., it contains the addresses of all regions of this table. -2. The best practice is that the Frontend first fetches the routes from its local cache and forwards the request to the Datanode. If the route is no longer valid, then Datanode is obliged to return an `Invalid Route` error, and the Frontend re-fetches the latest data from Metasrv and updates its cache. Route information does not change frequently, thus, it's sufficient for Frontend uses the Lazy policy to maintain the cache. -3. The Frontend processes a batch of writes that may contain multiple tables and multiple regions, so the Frontend needs to split user requests based on the 'route table'. +Frontend resolves the table route, splits rows according to the partition rules, and sends each Region write to the corresponding Datanode. Route changes cause the cached metadata to be invalidated and fetched again from Metasrv. ### Select -1. As with `Insert`, the Frontend first fetches the route table from the local cache. -2. Unlike `Insert`, for `Select`, the Frontend needs to extract the read-only node (follower) from the route table, then dispatch the request to the leader or follower node depending on the priority. -3. The distributed query engine in the Frontend distributes multiple sub-query tasks based on the routing information and aggregates the query results. +Frontend uses table and Region metadata while planning the query. Predicates on partition columns prune Regions, and the distributed query engine sends work to the Datanodes that own the selected Regions. See [Distributed Querying](../frontend/distributed-querying.md). ## Metasrv Architecture -![metasrv-architecture](/metasrv-architecture.png) - -## Distributed Consensus +The main coordination paths are: + +```text +Leader election + | + v +Metasrv leader +├─ DDL manager -> Procedure manager +├─ Selector -> new Region placement +├─ Heartbeat handler chain -> leases and Region statistics +├─ Region supervisor -> Region migration procedures +├─ Mailbox -> cache invalidations and Region instructions +└─ Metadata managers -> KV backend +``` -As you can see, Metasrv has a dependency on distributed consensus because: +These mechanisms share metadata, but they have different failure boundaries. A process restart may discard caches and leader-local state; metadata and procedure state required for recovery must be durable. -1. First, Metasrv has to elect a leader, Datanode only sends heartbeats to the leader, and we only use a single metasrv node to receive heartbeats, which makes it easy to do some calculations or scheduling accurately and quickly based on global information. As for how the Datanode connects to the leader, this is for MetaClient to decide (using a redirect, Heartbeat requests becomes a gRPC stream, and using redirect will be less error-prone than forwarding), and it is transparent to the Datanode. -2. Second, Metasrv must provide an election API for Datanode to elect "write" and "read-only" nodes and help Datanode achieve high availability. -3. Finally, `Metadata`, `Schema` and other data must be reliably and consistently stored on Metasrv. Therefore, consensus-based algorithms are the ideal approach for storing them. +## Distributed Consensus -For the first version of Metasrv, we choose Etcd as the consensus algorithm component (Metasrv is designed to consider adapting different implementations and even creating a new wheel) for the following reasons: +Metasrv separates leader election from metadata storage. Only the elected Metasrv leader performs coordination and metadata-changing operations. Other Metasrv nodes direct clients to the current leader. -1. Etcd provides exactly the API we need, such as `Watch`, `Election`, `KV`, etc. -2. We only perform two tasks with distributed consensus: elections (using the `Watch` mechanism) and storing (a small amount of metadata), and neither of them requires us to customize our own state machine, nor do we need to customize our own state machine based on raft; the small amount of data also does not require multi-raft-group support. -3. The initial version of Metasrv uses Etcd, which allows us to focus on the capabilities of Metasrv and not spend too much effort on distributed consensus algorithms, which improves the design of the system (avoiding coupling with consensus algorithms) and helps with rapid development at the beginning, as well as allows easy access to good consensus algorithm implementations in the future through good architectural designs. +The key-value backend stores table metadata, routes, procedure state, and other information that must survive a leader change. Metasrv does not use this election to create leader and follower replicas for Datanode Regions; Region availability is managed through heartbeats, Region failure detection, and failover procedures. ## Heartbeat Management -The primary means of communication between Datanode and Metasrv is the Heartbeat Request/Response Stream, and we want this to be the only way to communicate. This idea is inspired by the design of [TiKV PD](https://github.com/tikv/pd), and we have practical experience in [RheaKV](https://github.com/sofastack/sofa-jraft/tree/master/jraft-rheakv/rheakv-pd). The request sends its state, while Metasrv sends different scheduling instructions via Heartbeat Response. - -A heartbeat will probably carry the data listed below, but this is not the final design, and we are still discussing and exploring exactly which data should be mostly collected. - -``` -service Heartbeat { - // Heartbeat, there may be many contents of the heartbeat, such as: - // 1. Metadata to be registered to metasrv and discoverable by other nodes. - // 2. Some performance metrics, such as Load, CPU usage, etc. - // 3. The number of computing tasks being executed. - rpc Heartbeat(stream HeartbeatRequest) returns (stream HeartbeatResponse) {} -} - -message HeartbeatRequest { - RequestHeader header = 1; - - // Self peer - Peer peer = 2; - // Leader node - bool is_leader = 3; - // Actually reported time interval - TimeInterval report_interval = 4; - // Node stat - NodeStat node_stat = 5; - // Region stats in this node - repeated RegionStat region_stats = 6; - // Follower nodes and stats, empty on follower nodes - repeated ReplicaStat replica_stats = 7; -} - -message NodeStat { - // The read capacity units during this period - uint64 rcus = 1; - // The write capacity units during this period - uint64 wcus = 2; - // Table number in this node - uint64 table_num = 3; - // Region number in this node - uint64 region_num = 4; - - double cpu_usage = 5; - double load = 6; - // Read disk I/O in the node - double read_io_rate = 7; - // Write disk I/O in the node - double write_io_rate = 8; - - // Others - map attrs = 100; -} - -message RegionStat { - uint64 region_id = 1; - TableName table_name = 2; - // The read capacity units during this period - uint64 rcus = 3; - // The write capacity units during this period - uint64 wcus = 4; - // Approximate region size - uint64 approximate_size = 5; - // Approximate number of rows - uint64 approximate_rows = 6; - - // Others - map attrs = 100; -} - -message ReplicaStat { - Peer peer = 1; - bool in_sync = 2; - bool is_learner = 3; -} -``` - -## Central Nervous System (CNS) - -We are to build an algorithmic system, which relies on real-time and historical heartbeat data from each node, should make some smarter scheduling decisions and send them to Metasrv's Autoadmin unit, which distributes the scheduling decisions, either by the Datanode itself or more likely by the PaaS platform. - -## Abstraction of Workloads +Datanodes maintain heartbeat streams to the Metasrv leader. Heartbeat requests report node identity, lease information, Region statistics, and other state used for placement and supervision. Responses carry control messages such as Region lifecycle instructions and cache invalidations. -The level of workload abstraction determines the efficiency and quality of the scheduling strategy generated by Metasrv such as resource allocation. +A heartbeat drives two independent mechanisms, and a change to heartbeat timing affects both: -DynamoDB defines RCUs & WCUs (Read Capacity Units / Write Capacity Units), explaining that a RCU is a read request of 4KB data, and a WCU is a write request of 1KB data. When using RCU and WCU to describe workloads, it's easier to achieve performance measurability and get more informative resource preallocation because we can abstract different hardware capabilities as a combination of RCU and WCU. +- **Node lease.** The keep-lease handler renews the sending Datanode's lease. Selectors and the `/node-lease` endpoint use these leases to decide whether a Datanode is still active. +- **Region failure detection.** The Region supervisor keeps a per-Region Phi Accrual detector over heartbeat arrival intervals. Its verdict is independent of lease expiry. -However, GreptimeDB still faces a more complex situation than DynamoDB, in particular, RCU doesn't fit to describe GreptimeDB's read workloads which require a lot of computation. We are working on that. +A failure verdict submits a failover migration only when Region failover is enabled; it is disabled by default and requires remote WAL unless explicitly allowed on local WAL. Maintenance mode also suppresses failover. See [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md) for the prerequisites and how to enable it. diff --git a/versioned_docs/version-1.0/contributor-guide/metasrv/selector.md b/versioned_docs/version-1.0/contributor-guide/metasrv/selector.md index 790ffb849b..23190cd6a9 100644 --- a/versioned_docs/version-1.0/contributor-guide/metasrv/selector.md +++ b/versioned_docs/version-1.0/contributor-guide/metasrv/selector.md @@ -7,32 +7,28 @@ description: Describes the different types of selectors in the Metasrv service, ## Introduction -What is the `Selector`? As its name suggests, it allows users to select specific items from a given `namespace` and `context`. There is a related trait, also named `Selector`, whose definition can be found [below][0]. - -[0]: https://github.com/GreptimeTeam/greptimedb/blob/main/src/meta-srv/src/selector.rs - -There is a specific scenario in `Metasrv` service. When a request to create a table is sent to the `Metasrv` service, it creates a routing table (the details of table creation will not be described here). The `Metasrv` service needs to select the appropriate `Datanode` list when creating a routing table. +When a table is created, Metasrv uses a `Selector` to choose Datanodes for its Regions. Selection uses the current node leases and, depending on the selector, Region statistics. ## Selector Type The `Metasrv` service currently offers the following types of `Selectors`: -### LeasebasedSelector +### LeaseBasedSelector -`LeasebasedSelector` randomly selects from all available (in lease) `Datanode`s, its characteristic is simplicity and fast. +`LeaseBasedSelector` randomly selects from Datanodes with valid leases. ### LoadBasedSelector The `LoadBasedSelector` load value is determined by the number of regions on each `Datanode`, fewer regions indicate lower load, and `LoadBasedSelector` prioritizes selecting low-load `Datanodes`. ### RoundRobinSelector [default] -`RoundRobinSelector` selects `Datanode`s in a round-robin fashion. It is recommended and the default option in most cases. If you're unsure which to choose, it's usually the right choice. +`RoundRobinSelector` selects `Datanode`s in a round-robin fashion. It is the default and recommended choice for most deployments. ## Configuration You can configure the `Selector` by its name when starting the `Metasrv` service. -- LeasebasedSelector: `lease_based` or `LeaseBased` +- LeaseBasedSelector: `lease_based` or `LeaseBased` - LoadBasedSelector: `load_based` or `LoadBased` - RoundRobinSelector: `round_robin` or `RoundRobin` diff --git a/versioned_docs/version-1.0/contributor-guide/overview.md b/versioned_docs/version-1.0/contributor-guide/overview.md index 875b4b1e16..6e6ec669c9 100644 --- a/versioned_docs/version-1.0/contributor-guide/overview.md +++ b/versioned_docs/version-1.0/contributor-guide/overview.md @@ -5,9 +5,7 @@ description: Overview of GreptimeDB's architecture, key components, and how they # Contributor Guide -DeepWiki provides a detailed and clear explanation of GreptimeDB's architecture and implementation. Highly recommended: - -[https://deepwiki.com/GreptimeTeam/greptimedb](https://deepwiki.com/GreptimeTeam/greptimedb) +This guide explains the internal design of GreptimeDB for contributors. Start with [Getting Started](/contributor-guide/getting-started.md) to build and run it from source. Submission requirements, including the CLA, license headers, formatting, and the checks a pull request must pass, are maintained in the source repository's [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md). ## Architecture @@ -18,8 +16,13 @@ For more details on each component, see the following guides: - [frontend][1] - [datanode][2] - [metasrv][3] +- [flownode][4] [1]: /contributor-guide/frontend/overview.md [2]: /contributor-guide/datanode/overview.md [3]: /contributor-guide/metasrv/overview.md +[4]: /contributor-guide/flownode/overview.md + +## Additional reference +[DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb) provides an automatically generated walkthrough of the GreptimeDB repository. It can help when exploring an unfamiliar area, but it is a secondary reference: verify version-sensitive behavior against the source code. diff --git a/versioned_docs/version-1.0/contributor-guide/tests/integration-test.md b/versioned_docs/version-1.0/contributor-guide/tests/integration-test.md index 5d5f6cb1a5..bc52ee4854 100644 --- a/versioned_docs/version-1.0/contributor-guide/tests/integration-test.md +++ b/versioned_docs/version-1.0/contributor-guide/tests/integration-test.md @@ -7,7 +7,14 @@ description: Guide on writing and running integration tests in GreptimeDB, cover ## Introduction -Integration testing is written with Rust test harness (`#[test]`), unlike unit testing, they are placed separately -[here](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration). -It covers scenarios involving multiple components, in which one typical case is HTTP/gRPC-related features. You can check -its [documentation](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md) for more information. +Integration tests cover behavior that crosses crate or service boundaries, such as HTTP and gRPC handling, distributed components, or external storage. They use Rust's test harness and live in the [`tests-integration`](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration) package. + +Run the package with: + +```shell +cargo nextest run -p tests-integration +``` + +Some cases require environment variables or fixtures for external services. Follow the package's [setup instructions](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md) before running those cases. + +Use an integration test when a crate-level test or a Sqlness case cannot exercise the required boundary. Keep isolated logic in unit tests so that failures remain fast to reproduce. diff --git a/versioned_docs/version-1.0/contributor-guide/tests/overview.md b/versioned_docs/version-1.0/contributor-guide/tests/overview.md index feeefe2b6d..07290b79bd 100644 --- a/versioned_docs/version-1.0/contributor-guide/tests/overview.md +++ b/versioned_docs/version-1.0/contributor-guide/tests/overview.md @@ -5,4 +5,12 @@ description: Overview of the testing methods used in GreptimeDB to ensure its be # Tests -Our team has conducted lots of tests to ensure the behaviours of `GreptimeDB` . This chapter will introduce several significant methods used to test `GreptimeDB`, and how to work with them. +Choose the narrowest test that exercises the behavior you changed: + +| Test type | Use it for | Typical command | +| --- | --- | --- | +| [Unit test](unit-test.md) | Logic contained within one crate or component | `cargo nextest run -p ` | +| [Sqlness test](sqlness-test.md) | SQL, protocol, planner, execution, and end-to-end regressions | `cargo sqlness bare -t ` | +| [Integration test](integration-test.md) | Behavior that crosses components or requires external services | `cargo nextest run -p tests-integration` | + +Run `make test` when a change needs the full Rust workspace test suite. diff --git a/versioned_docs/version-1.0/contributor-guide/tests/sqlness-test.md b/versioned_docs/version-1.0/contributor-guide/tests/sqlness-test.md index b5fdcf2b0b..0262ff6851 100644 --- a/versioned_docs/version-1.0/contributor-guide/tests/sqlness-test.md +++ b/versioned_docs/version-1.0/contributor-guide/tests/sqlness-test.md @@ -7,42 +7,34 @@ description: Instructions for running SQL tests in GreptimeDB using the `sqlness ## Introduction -SQL is an important user interface for `GreptimeDB`. We have a separate test suite for it (named `sqlness`). +Sqlness is GreptimeDB's end-to-end regression suite for SQL and protocol behavior. A case sends statements to a running GreptimeDB instance and compares the output with a checked-in result file. ## Sqlness manual ### Case file -Sqlness has two types of file +Each case uses two files: - `.sql`: test input, SQL only - `.result`: expected test output, SQL and its results -The `.result` file is the expected execution output. If you see `.result` files changed, -it means the test gets a different result and indicates it may fail. You should -check the change logs to solve the problem. - -You only need to write test SQL in the `.sql` file, and run the test. +Write the input in the `.sql` file and run the test to generate or update `.result`. Review every result diff: accept it only when the behavior change is intended. ### Case organization -The root dir of input cases is `tests/cases`. It contains several sub-directories stand for different test -modes. E.g., `standalone/` contains all the tests to run under `greptimedb standalone start` mode. +Input cases live under `tests/cases`. The first directory level selects an environment. For example, `standalone/` runs against a standalone GreptimeDB instance. -Under the first level of sub-directory (e.g. the `cases/standalone`), you can organize your cases as you like. -Sqlness walks through every file recursively and runs them. +Within an environment, group a new case with the feature it exercises. Sqlness discovers case files recursively. ## Run the test -Unlike other tests, this harness is in a binary target form. You can run it with +Run the suite with: ```shell -cargo run --bin sqlness-runner bare +cargo sqlness bare ``` -It automatically finishes the following procedures: compile `GreptimeDB`, start it, grab tests and feed it to -the server, then collect and compare the results. You only need to check whether any `.result` files changed. -If not, congratulations, the test is passed 🥳! +The command builds and starts GreptimeDB, runs the selected cases, and compares their output. A changed `.result` file is part of the review, not proof that the new output is correct. ### Run a specific test diff --git a/versioned_docs/version-1.0/contributor-guide/tests/unit-test.md b/versioned_docs/version-1.0/contributor-guide/tests/unit-test.md index 82e30bf5fa..183a72aab4 100644 --- a/versioned_docs/version-1.0/contributor-guide/tests/unit-test.md +++ b/versioned_docs/version-1.0/contributor-guide/tests/unit-test.md @@ -8,25 +8,26 @@ description: Guide on writing and running unit tests in GreptimeDB using Rust's ## Introduction Unit tests are embedded into the codebase, usually placed next to the logic being tested. -They are written using Rust's `#[test]` attribute and can run with `cargo nextest run`. +They are written using Rust's `#[test]` attribute. GreptimeDB uses [`cargo-nextest`](https://nexte.st/) as its primary Rust test runner. -The default test runner ships with `cargo` is not supported in GreptimeDB codebase. It's recommended -to use [`nextest`](https://nexte.st/) instead. You can install it with +Install it with: ```shell cargo install cargo-nextest --locked ``` -And run the tests (here the `--workspace` is not necessary) +Run the package you changed first: ```shell -cargo nextest run +cargo nextest run -p ``` -Notes if your Rust is installed via `rustup`, be sure to install `nextest` with `cargo` rather -than the package manager like `homebrew`. Otherwise it will mess up your local environment. +Use a test name or nextest filter to narrow the run further while developing. Before submitting a change with broad effects, run the full workspace suite: + +```shell +make test +``` ## Coverage -Our continuous integration (CI) jobs have a "coverage checking" step. It will report how many -codes are covered by unit tests. Please add the necessary unit test to your patch. +CI reports unit-test coverage. Add tests for changed behavior and failure cases that could otherwise regress; coverage percentage alone is not the goal. diff --git a/versioned_docs/version-1.1/contributor-guide/datanode/data-persistence-indexing.md b/versioned_docs/version-1.1/contributor-guide/datanode/data-persistence-indexing.md index 6403ffc2df..1fa5d6d258 100644 --- a/versioned_docs/version-1.1/contributor-guide/datanode/data-persistence-indexing.md +++ b/versioned_docs/version-1.1/contributor-guide/datanode/data-persistence-indexing.md @@ -5,19 +5,23 @@ description: Explanation of data persistence and indexing in GreptimeDB, includi # Data Persistence and Indexing -Similar to all LSMT-like storage engines, data in MemTables is persisted to durable storage, for example, the local disk file system or object storage service. GreptimeDB adopts [Apache Parquet][1] as its persistent file format. +Like other LSM-tree storage engines, GreptimeDB persists data from memtables to durable storage such as a local filesystem or object storage. It uses [Apache Parquet][1] as the persistent file format. ## SST File Format Parquet is an open source columnar format that provides fast data querying and has already been adopted by many projects, such as Delta Lake. -Parquet has a hierarchical structure like "row groups-columns-data pages". Data in a Parquet file is horizontally partitioned into row groups, in which all values of the same column are stored together to form a data page. Data page is the minimal storage unit. This structure greatly improves performance. +Parquet organizes data as row groups, column chunks, and pages. A row group contains one column chunk for each column, and each column chunk contains one or more pages. Pages are the units of encoding and compression; column chunks are the I/O units for reading selected columns. First, clustering data by column makes file scanning more efficient, especially when only a few columns are queried, which is very common in analytical systems. -Second, data of the same column tends to be homogeneous which helps with compression when apply techniques like dictionary and Run-Length Encoding (RLE). +Second, values within a column tend to be similar, which improves compression with techniques such as dictionary encoding and run-length encoding (RLE). -Parquet file format +The following diagram from the Apache Parquet specification also shows the physical file layout: column chunks are stored by row group, while file metadata and its length are written in the footer. + +Apache Parquet file layout + +*Source: Apache Parquet [FileLayout.gif](https://github.com/apache/parquet-format/blob/master/doc/images/FileLayout.gif). Copyright 2014 The Apache Software Foundation, licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).* ## Data Persistence @@ -27,17 +31,17 @@ When the size of data buffered in MemTables reaches that threshold, GreptimeDB w ## Indexing Data in SST Files -Apache Parquet file format provides inherent statistics in headers of column chunks and data pages, which are used for pruning and skipping. +Parquet stores row-group column statistics such as minimum, maximum, and null count in each column chunk's metadata. Page metadata and optional column indexes can provide finer-grained statistics. -Column chunk header +![A name predicate uses Parquet column statistics to skip one row group while retaining another as a read candidate.](/parquet-row-group-statistics.svg) -For example, in the above Parquet file, if you want to filter rows where `name` = `Emily`, you can easily skip row group 0 because the max value for `name` field is `Charlie`. This statistical information reduces IO operations. +For example, a query filtering for `name` = `Emily` can skip row group 0 because the maximum `name` value is `Charlie`. This avoids reading that row group. ## Index Files -For each SST file, GreptimeDB not only maintains an internal index but also generates a separate file to store the index structures specific to that SST file. +When an SST has one or more configured index outputs, GreptimeDB writes them to a Puffin file associated with that SST. An SST with no applicable index does not need a Puffin file. -The index files utilize the [Puffin][3] format, which offers significant flexibility, allowing for the storage of additional metadata and supporting a broader range of index structures. +Puffin provides a container for index blobs and their metadata, allowing different index structures to share one file. ![Puffin](/puffin.png) @@ -57,13 +61,13 @@ The inverted index enables GreptimeDB to skip data segments that do not meet que ![Inverted index searching](/inverted-index-searching.png) -For instance, the query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches the regex `.*users`, and `status` matches the regex `4...`. It then scans these data segments to produce the final results that meet all conditions, significantly reducing the number of IO operations. +The query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches `.*users`, and `status` matches `4..`. It scans only those segments before applying the remaining filters. ### Inverted Index Format -![Inverted index format](/inverted-index-format.png) +![An inverted-index blob contains one index per column followed by footer metadata; each column index contains a null bitmap, posting bitmaps, and an FST.](/inverted-index-blob-layout.svg) -GreptimeDB builds inverted indexes by column, with each inverted index consisting of an FST and multiple Bitmaps. +GreptimeDB builds inverted indexes by column. Each column index contains a null bitmap, multiple posting bitmaps, and an FST. The blob footer records the offsets, sizes, and metadata needed to locate and decode the column indexes. The FST (Finite State Transducer) enables GreptimeDB to store mappings from column values to Bitmap positions in a compact format and provides excellent search performance and supports complex search capabilities (such as regular expression matching). The Bitmaps maintain a list of data segment IDs, with each bit representing a data segment. @@ -77,7 +81,7 @@ The number of rows in a data segment is controlled by the engine option `index.i ## Unified Data Access Layer: OpenDAL -GreptimeDB uses [OpenDAL][2] to provide a unified data access layer, thus, the storage engine does not need to interact with different storage APIs, and data can be migrated to cloud-based storage like AWS S3 seamlessly. +GreptimeDB uses [OpenDAL][2] to provide a common access layer for local filesystems and object stores. Changing the configured storage backend does not migrate existing data. [1]: https://parquet.apache.org [2]: https://github.com/datafuselabs/opendal diff --git a/versioned_docs/version-1.1/contributor-guide/datanode/memtable.md b/versioned_docs/version-1.1/contributor-guide/datanode/memtable.md new file mode 100644 index 0000000000..92e3eb58de --- /dev/null +++ b/versioned_docs/version-1.1/contributor-guide/datanode/memtable.md @@ -0,0 +1,100 @@ +--- +keywords: [memtable, Mito engine, write buffer, flush, time partition, BulkMemtable] +description: How Mito organizes mutable Region data in memtables and moves it into SST files. +--- + +# Memtable design + +A memtable is Mito's in-memory write buffer for a Region. It makes writes available to reads before a flush creates SST files. A Region version identifies the memtables and SST files that a scan may read. Together with a committed-sequence fence, it keeps the scan consistent while writes and flushes advance the current version. + +## Write and flush lifecycle + +For a normal WAL-backed write, Mito uses this order: + +```text +write request + | + v +WAL append -> mutable memtable -> publish committed sequence + | + freeze + v + immutable memtable -> SST write -> manifest edit +``` + +The Region worker assigns sequence numbers and a WAL entry ID before appending the mutation to the [write-ahead log](wal.md). If the WAL append fails, Mito does not update the memtable. After the memtable update succeeds, Mito publishes the committed sequence and the rows become visible to new reads. A Region configured with `skip_wal` omits the WAL append, but keeps the same memtable and visibility ordering. + +A flush freezes the mutable memtables and installs a new mutable set before starting the background SST write. New writes therefore continue without changing the frozen data. The flush writes the immutable memtables to SST files, then persists a manifest edit containing the files and the flushed WAL and sequence checkpoints. Only after that edit is durable does Mito remove the flushed memtables from the current Region version. If the flush fails, the immutable memtables remain available for a later attempt. + +## Region versions and time partitions + +Each Region has one mutable `TimePartitions` container, which can hold more than one memtable: + +```text +Region version +├─ mutable TimePartitions +│ ├─ [t0, t1) -> memtable +│ └─ [t1, t2) -> memtable +├─ immutable memtables +└─ SST files +``` + +Mito routes each row to a partition by its time-index value. Partition ranges are half-open and aligned to a fixed duration. The duration follows the Region's compaction time window; Mito uses one day until a compaction window is available. An out-of-order write can create an earlier partition alongside the latest one. + +Freezing a Region freezes all mutable time partitions together. Mito moves their memtables to the immutable list and creates a new `TimePartitions` container. A failed flush can leave more than one generation of immutable memtables, so reads and later flushes must not assume that the list contains a single item. + +## Memtable implementations + +Mito selects a memtable implementation from the Region's SST format, primary-key encoding, and memtable options: + +```text +flat SST format (the default) or sparse primary-key encoding -> BulkMemtable +memtable.type=bulk -> BulkMemtable, and forces flat SST +primary_key SST with dense encoding (legacy) -> a legacy implementation +``` + +With the default engine configuration, a Region without an explicit SST format uses `flat`, so `BulkMemtable` is the normal path and the rest of this page describes it. The rules exist to prevent incompatible combinations: flat format or sparse primary-key encoding requires `BulkMemtable`, and explicitly selecting the bulk implementation forces flat format. + +### BulkMemtable + +`BulkMemtable` stores writes as parts in the flat Arrow layout instead of inserting rows into per-series buffers: + +```text +BulkMemtable +├─ unordered_part +│ └─ small BulkPart batches +└─ parts + ├─ BulkPart (Arrow RecordBatch) + ├─ MultiBulkPart (raw RecordBatches) + └─ EncodedBulkPart (in-memory Parquet) +``` + +Small parts accumulate in `unordered_part`; larger parts enter `parts` directly. Background memtable compaction merge-sorts eligible parts into a `MultiBulkPart` or encodes them as an `EncodedBulkPart`. Scans use part statistics to prune ranges, and flush can write encoded ranges to SST without decoding and encoding the rows again. For the design rationale and performance results, see [Scaling Time Series to Millions of Cardinalities: GreptimeDB's Flat Format](https://greptime.com/blogs/2025-12-22-flat-format). + +### Legacy implementations + +Regions on the legacy `primary_key` SST format with dense primary-key encoding still use `TimeSeriesMemtable`, which groups rows by encoded primary key rather than storing flat parts. A Region with no primary-key columns gets `SimpleBulkMemtable` from the same builder. Both are compatibility code for existing tables and may be removed once the `primary_key` format is retired; new work targets the bulk and flat path. + +The removed `partition_tree` memtable is not a third implementation. The option parser accepts `memtable.type=partition_tree` for compatibility, but it does not recreate that implementation. The Region uses the bulk and flat path. + +## Read snapshots + +A scan obtains the Region version and committed sequence together from `VersionControl`. It selects the version before applying the sequence fence. Reading the sequence separately before the version could pair that sequence with a later version after flush or compaction removes an older input, producing an incomplete snapshot. + +The selected version supplies mutable memtables, immutable memtables, and SST files. Mito first prunes sources by time range, then asks each memtable for ranges using the scan's projection, predicate, and sequence bounds. The scan merges the resulting ranges with SST ranges and applies the same ordering, deletion, and merge semantics across all sources. References held by the scan keep an older memtable alive even after a newer Region version removes it. + +## Memory pressure + +Each memtable tracks its estimated heap allocation through the engine's write-buffer manager. Freezing a memtable removes its allocation from the mutable-memory count, but total usage includes the allocation until all references to that memtable are released. The mutable-memory count therefore tracks data that can still accept writes, while total usage continues to include memory retained by active scans. + +The global write-buffer limit causes workers to select Regions for flush. If memory remains above the configured limits, Mito stalls writes and can reject them at a higher threshold. An optional per-Region limit applies the same pressure to one hot Region without stalling unrelated Regions. Periodic, manual, and Region lifecycle operations can also request a flush. + +## Constraints for changes + +Changes to memtable code must preserve these properties: + +- For a WAL-backed Region, append to the WAL before installing rows in a memtable. Publish the committed sequence only after installation succeeds. +- Keep frozen memtables readable and retryable until the SST files and manifest edit are durable. +- Obtain the Region version and committed sequence from the same `VersionControl` snapshot; never read the sequence separately before the version. +- Preserve the ordering and metadata that scans and flushes need to apply the same deletion, deduplication, and merge rules across memtable and SST ranges. +- Charge allocations to the write-buffer manager and release them only when the underlying memory can no longer be referenced. diff --git a/versioned_docs/version-1.1/contributor-guide/datanode/metric-engine.md b/versioned_docs/version-1.1/contributor-guide/datanode/metric-engine.md index 064872ce14..fde3c178a3 100644 --- a/versioned_docs/version-1.1/contributor-guide/datanode/metric-engine.md +++ b/versioned_docs/version-1.1/contributor-guide/datanode/metric-engine.md @@ -7,9 +7,9 @@ description: Overview of the Metric engine in GreptimeDB, its concepts, architec ## Overview -The `Metric` engine is a component of GreptimeDB, and it's an implementation of the storage engine. It mainly targets scenarios with a large number of small tables for observable metrics. +The `Metric` engine stores workloads with many small metric tables. -Its main feature is to use synthetic physical wide tables to store a large amount of small table data, achieving effects such as reuse of the same column and metadata. This reduces storage overhead for small tables and improves columnar compression efficiency. The concept of a table becomes even more lightweight under the `Metric` engine. +It maps those logical tables onto shared physical wide tables so they can reuse columns and metadata. This reduces per-table storage overhead and improves columnar compression. ## Concepts @@ -19,7 +19,7 @@ The `Metric` engine introduces two new concepts: "logical table" and "physical t A logical table refers to user-defined tables. Just like any other ordinary table, its definition includes the name of the table, column definitions, index definitions etc. All operations such as queries or write-ins by users are based on these logical tables. Users don't need to worry about differences between logical and ordinary tables during usage. -From an implementation standpoint, a logical table is virtual; it doesn't directly read or write physical data but maps read/write requests into corresponding requests for physical tables in order to implement data storage and querying. +A logical table is virtual. The engine maps its read and write requests to the corresponding physical table instead of storing data for it directly. ### Physical Table @@ -29,16 +29,14 @@ A physical table is a table that actually stores data, possessing several physic The main design architecture of the `Metric` engine is as follows: -![Arch](/metric-engine-arch.png) +![Multiple logical tables map through the Metric engine to shared data and metadata Regions managed by Mito.](/metric-engine-architecture.svg) -In the current version implementation, the `Metric` engine reuses the `Mito` engine to achieve storage and query capabilities for physical data. It also provides access to both physical tables and logical tables simultaneously. +The `Metric` engine delegates physical storage and queries to the `Mito` engine. Each physical Region group contains a data Region, which stores rows from its mapped logical tables, and a metadata Region, which stores the logical-table and logical-column mappings. -Regarding partitioning, logical tables have identical partition rules and Region distribution as physical tables. This makes sense because the data of logical tables are directly stored in physical tables, so their partition rules are consistent. +Logical tables associated with the same physical table share its partition layout. During writes, the engine records the logical table identity with each row. During reads, it adds a logical-table filter before scanning the physical Region. -Concerning routing metadata, the routing address of a logical table is a logical address - what its corresponding physical table is - then through this physical table for secondary routing to obtain the real physical address. This indirect routing method can significantly reduce the number of metadata modifications required when Region migration scheduling occurs in Metric engines. +A logical table's route stores only the ID of its physical table; the physical table route resolves that to the Datanodes holding the Regions. Because logical routes do not name peers, migrating a physical Region rewrites one physical route instead of every logical route that maps to it. -Operationally speaking, The `Metric` engine supports standard DML operations (INSERT, DELETE, SELECT) on logical tables. However, it only supports limited operations on physical tables to prevent misoperations - for example, writing directly to a physical table is prohibited as it could affect user's logical table data. Generally speaking, users can consider that they have read-only access to these physical tables. +Logical tables support normal INSERT, DELETE, and SELECT operations. Direct writes to a physical Region are rejected because they would bypass the logical-table mapping; querying a physical table remains supported. -To improve performance during simultaneous DDL (Data Definition Language) operations on many tables, the 'Metric' engine has introduced some batch DDL operations. These batch DDL operations can merge lots of DDL actions into one request thereby reducing queries and modifications times for metadata thus enhancing performance. This feature is particularly beneficial in scenarios such as the automatic creation requests brought about by large amounts of metrics during Prometheus Remote Write cold start-up, as well as the modification requests for numerous route-tables mentioned earlier during migration of many physical regions. - -Apart from physical data regions belonging to physical tables, the 'Metric' engine creates an additional metadata region physically for each individual physical data region used in storing some metadata needed by itself while maintaining mapping and other states. This metadata includes the mapping relationship between logical tables and physical tables, the mapping relationship between logical columns and physical columns etc. +Batch DDL operations reduce metadata work when many logical tables are created or updated together, such as during Prometheus Remote Write auto-creation or physical Region migration. diff --git a/versioned_docs/version-1.1/contributor-guide/datanode/overview.md b/versioned_docs/version-1.1/contributor-guide/datanode/overview.md index d0afe21b34..a21c112faa 100644 --- a/versioned_docs/version-1.1/contributor-guide/datanode/overview.md +++ b/versioned_docs/version-1.1/contributor-guide/datanode/overview.md @@ -7,28 +7,26 @@ description: Overview of Datanode in GreptimeDB, its responsibilities, component ## Introduction -`Datanode` is mainly responsible for storing the actual data for GreptimeDB. As we know, in GreptimeDB, -a `table` can have one or more `Region`s, and `Datanode` is responsible for managing the reading and writing -of these `Region`s. `Datanode` is not aware of `table` and can be considered as a `region server`. Therefore, -`Frontend` and `Metasrv` operate `Datanode` at the granularity of `Region`. +A Datanode stores and processes Region data. A table can contain multiple Regions, but the Datanode does not own table-level routing. Frontend sends data requests by Region, while Metasrv controls Region placement and lifecycle. -![Datanode](/datanode.png) +This boundary lets the same Region server host different storage engines without exposing their implementation to Frontend or Metasrv. + +![Frontend sends Region requests to the Datanode Region server, while Metasrv exchanges lifecycle instructions through the heartbeat task. The Region server uses the local query engine and dispatches requests to the Mito, Metric, or File Region engine.](/datanode-architecture.svg) ## Components -A `Datanode` contains all the components needed for a `region server`. Here we list some of the vital parts: - -- A gRPC service is provided for reading and writing region data, and `Frontend` uses this service - to read and write data from `Datanode`s. -- An HTTP service, through which you can obtain metrics, configuration information, etc., of the current node. -- `Heartbeat Task` is used to send heartbeat to the `Metasrv`. The heartbeat plays a crucial role in the - distributed architecture of GreptimeDB and serves as a basic communication channel for distributed coordination. - The upstream heartbeat messages contain important information such as the workload of a `Region`. If the - `Metasrv `has made scheduling(such as `Region` migration) decisions, it will send instructions to the - `Datanode` via downstream heartbeat messages. -- The `Datanode` does not parse user SQL or perform distributed planning. The user's query requests for one or - more `Table`s will be transformed into `Region` query requests in the `Frontend`. The `Datanode` is responsible - for executing these `Region` query plans with its local query engine. -- A `Region Manager` is used to manage all `Region`s on a `Datanode`. -- GreptimeDB supports a pluggable multi-engine architecture, with existing engines including `File Engine` and - `Mito Engine`. +The main components are: + +- The Region server tracks open Regions and dispatches reads, writes, and lifecycle requests to the engine registered for each Region. +- `Mito` is the primary time-series Region engine. `Metric` maps many logical metric Regions onto shared Mito Regions, and `File` exposes external files through the Region interface. +- The local query engine executes Region query plans. It does not parse client SQL or perform cluster-wide planning. +- The heartbeat task reports node and Region state to Metasrv and receives instructions such as open, close, upgrade, downgrade, and migration steps. +- gRPC carries Region requests to the Datanode. HTTP exposes node diagnostics such as metrics and configuration. + +## Region Request Lifecycle + +For a Mito write, the Region server selects Mito from the Region metadata. Mito appends the mutation to the WAL, applies it to a memtable, and later flushes the memtable to SST files. A Metric write is first rewritten with the logical-table identity and then delegated to its physical Mito Region. + +For a read, the local query engine executes the Region plan against a table provider backed by the Region engine. A Mito scan takes an immutable Region version, reads the relevant memtables and SST files, merges and deduplicates rows, and returns a stream of Arrow record batches. + +Region ownership can change without restarting the Datanode. Metasrv sends lifecycle instructions over the heartbeat stream; the Region server applies them to the engine and reports the new Region role and statistics in subsequent heartbeats. diff --git a/versioned_docs/version-1.1/contributor-guide/datanode/python-scripts.md b/versioned_docs/version-1.1/contributor-guide/datanode/python-scripts.md deleted file mode 100644 index 98909142a6..0000000000 --- a/versioned_docs/version-1.1/contributor-guide/datanode/python-scripts.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -keywords: [Python scripts, data analysis, CPython backend, RustPython interpreter, RecordBatch] -description: Guide on using Python scripts for data analysis in GreptimeDB, including backend options and setup instructions. ---- - -# Python Scripts - -## Introduction - -Python scripts are methods for analyzing data in GreptimeDB, -by running it in the database directly instead of fetching all the data from the database and running it locally. -This approach saves a lot of data transfer costs. -The image below depicts how the script works. -The `RecordBatch` (which is basically a column in a table with type and nullability metadata) -can come from anywhere in the database, -and the returned `RecordBatch` can be annotated in Python grammar to indicate its metadata, -such as type or nullability. -The script will do its best to convert the returned object to a `RecordBatch`, -whether it is a Python list, a `RecordBatch` computed from parameters, -or a constant (which is extended to the same length as the input arguments). - -![Python Coprocessor](/python-coprocessor.png) - -## Two optional backends - -### CPython Backend powered by PyO3 - -This backend is powered by [PyO3](https://pyo3.rs/v0.18.1/), enabling the use of your favourite Python libraries (such as NumPy, Pandas, etc.) and allowing Conda to manage your Python environment. - -But using it also involves some complications. You must set up the correct Python shared library, which can be a bit challenging. In general, you just need to install the `python-dev` package. However, if you are using Homebrew to install Python on macOS, you must create a proper soft link to `Library/Frameworks/Python.framework`. Detailed instructions on using PyO3 crate with different Python Version can be found [here](https://pyo3.rs/v0.18.1/building_and_distribution#configuring-the-python-version) - -### Embedded RustPython Interpreter - -An experiment [python interpreter](https://github.com/RustPython/RustPython) to run -the coprocessor script, it supports Python 3.10 grammar. You can use all the very Python syntax, see [User Guide/Python Coprocessor](/user-guide/python-scripts/overview.md) for more! diff --git a/versioned_docs/version-1.1/contributor-guide/datanode/query-engine.md b/versioned_docs/version-1.1/contributor-guide/datanode/query-engine.md index 5b83a30613..91f307aa59 100644 --- a/versioned_docs/version-1.1/contributor-guide/datanode/query-engine.md +++ b/versioned_docs/version-1.1/contributor-guide/datanode/query-engine.md @@ -7,51 +7,30 @@ description: Overview of GreptimeDB's query engine, its architecture, data repre ## Introduction -GreptimeDB's query engine is built on [Apache DataFusion][1] (subproject under [Apache -Arrow][2]), a brilliant query engine written in Rust. It provides a set of well functional components from -logical plan, physical plan and the execution runtime. Below explains how each component is orchestrated and their positions during execution. +GreptimeDB's query engine is built on [Apache DataFusion][1]. DataFusion supplies the logical and physical plan interfaces, optimizer framework, and execution runtime. GreptimeDB adds planners for its query languages, storage-aware optimizer rules, custom plan nodes, and distributed execution. -![Execution Procedure](/execution-procedure.png) +DDL and other control-plane operations are dispatched by the statement executor. The query engine receives plans for data processing, including the input side of operations such as `INSERT ... SELECT`. -The entry point is the logical plan, which is used as the general intermediate representation of a -query or execution logic etc. Two noticeable sources of logical plan are from: 1. the user query, like -SQL through SQL parser and planner; 2. the Frontend's distributed query, which is explained in details in the following section. +## Query Lifecycle -Next is the physical plan, or the execution plan. Unlike the logical plan which is a big -enumeration containing all the logical plan variants (except the special extension plan node), the -physical plan is in fact a trait that defines a group of methods invoked during -execution. All data processing logics are packed in corresponding structures that -implement the trait. They are the actual operations performed on the data, like -aggregator `MIN` or `AVG`, and table scan `SELECT ... FROM`. +1. The SQL, PromQL, or log-query planner resolves tables through the catalog and produces a DataFusion logical plan. GreptimeDB plan extensions represent operations that DataFusion does not provide directly. +2. DataFusion analyzer and optimizer rules run together with GreptimeDB rules. These rules normalize expressions and types, rewrite time-range operations, push projections and filters toward scans, and introduce distributed plan nodes when required. +3. The physical planner converts the optimized logical plan into streaming operators. GreptimeDB then applies physical rules for scan parallelism, ordering, and distributed execution. +4. Execution pulls Arrow record batches through the physical plan. Storage scans receive the projection and predicates, and downstream operators consume the resulting stream without materializing the complete result first. -The optimization phase which improves execution performance by transforming both logical and physical plans, is now all based on rules. It is also called, "Rule Based Optimization". Some of the rules are DataFusion native and others are customized in Greptime DB. In the future, we plan to add more -rules and leverage the data statistics for Cost Based Optimization/CBO. - -The last phase "execute" is a verb, stands for the procedure that reads data from storage, performs -calculations and generates the expected results. Although it's more abstract than previously mentioned concepts, you can just -simply imagine it as executing a Rust async function. And it's indeed a future (stream). - -`EXPLAIN [VERBOSE] ` is very useful if you want to see how your SQL is represented in the logical or physical plan. +Use [`EXPLAIN`](/reference/sql/explain.md) to inspect the logical and physical plans. `EXPLAIN ANALYZE` also executes the plan and reports runtime metrics. ## Data Representation -GreptimeDB uses [Apache Arrow][2] as the in-memory data representation. It's column-oriented, in -cross-platform format, and also contains many high-performance data operators. These features -make it easy to share data in many different environments and implement calculation logic. +GreptimeDB uses [Apache Arrow][2] record batches as its in-memory data representation. A record batch contains equal-length column arrays and a schema. Query operators exchange streams of these batches, which keeps the execution path columnar from Region scans through result encoding. ## Indexing -In time series data, there are two important dimensions: timestamp and tag columns (or like -primary key in a general relational database). GreptimeDB groups data in time buckets, so it's efficient -to locate and extract data within the expected time range at a very low cost. The mainly used persistent file format [Apache Parquet][3] in GreptimeDB helps a lot -- it -provides multi-level indices and filters that make it easy to prune data during querying. In the future, we -will make more use of this feature, and develop our separated index to handle more complex use cases. +Index construction and persistent index formats belong to the storage engine. The query layer supplies predicates and projections to a scan; Mito then uses time ranges, Parquet statistics, and indexes to avoid reading data that cannot match. See [Data Persistence and Indexing](./data-persistence-indexing.md). ## Distributed Execution -Covered in [Distributed Querying][6]. +In distributed mode, the Frontend plans the cluster-wide query and Datanodes execute Region-local subplans. [`MergeScan`](../frontend/distributed-querying.md) is the boundary between those stages. -[1]: https://github.com/apache/arrow-datafusion +[1]: https://datafusion.apache.org/ [2]: https://arrow.apache.org/ -[3]: https://parquet.apache.org -[6]: ../frontend/distributed-querying.md diff --git a/versioned_docs/version-1.1/contributor-guide/datanode/storage-engine.md b/versioned_docs/version-1.1/contributor-guide/datanode/storage-engine.md index c220c72208..36fef62833 100644 --- a/versioned_docs/version-1.1/contributor-guide/datanode/storage-engine.md +++ b/versioned_docs/version-1.1/contributor-guide/datanode/storage-engine.md @@ -7,7 +7,7 @@ description: Overview of the storage engine in GreptimeDB, its architecture, com ## Introduction -The `storage engine` is responsible for storing the data of the database. Mito, based on [LSMT][1] (Log-structured Merge-tree), is the storage engine we use by default. We have made significant optimizations for handling time-series data scenarios, so mito engine is not suitable for general purposes. +Mito is GreptimeDB's default storage engine. It uses an [LSM tree][1] and is designed for time-series workloads rather than as a general-purpose embedded storage engine. ## Architecture @@ -23,9 +23,9 @@ The architecture is the same as a traditional LSMT engine: media. - Log records of the WAL can be stored on the local disk, or in a remote log service such as Kafka (remote WAL) that implements the `Log Store` API. -- Memtables: - - Data is written into the `active memtable`, aka `mutable memtable` first. - - When a `mutable memtable` is full, it will be changed to a `read-only memtable`, aka `immutable memtable`. +- [Memtables](memtable.md): + - Mito routes rows by time index into mutable memtables. + - A flush freezes the mutable memtables, installs a new mutable set for writes, and writes the frozen memtables to SST files. - SST - The full name of SST, aka SSTable is `Sorted String Table`. - `Immutable memtable` is flushed to persistent storage and produces an SST file. @@ -103,7 +103,9 @@ Each Parquet SST is split into row groups, the unit that Parquet can read or ski Mito supports two SST formats: `flat` and `primary_key`. `flat` is the default for new tables and works well across primary-key cardinalities, including high-cardinality keys. `primary_key` is the legacy format kept for compatibility with older tables. See [SST format](/reference/sql/create.md#create-a-table-with-sst-format) and the [table design guide](/user-guide/deployments-administration/performance-tuning/design-table.md#sst-format) for more details. -SST layout +![The default flat Mito SST layout combines file-level metadata with Parquet row groups containing data columns and merge metadata.](/mito-sst-layout.svg) + +An SST may span more than one compaction time window. ## Scan Pruning diff --git a/versioned_docs/version-1.1/contributor-guide/datanode/wal.md b/versioned_docs/version-1.1/contributor-guide/datanode/wal.md index 4ecb19ef02..8f898d91d6 100644 --- a/versioned_docs/version-1.1/contributor-guide/datanode/wal.md +++ b/versioned_docs/version-1.1/contributor-guide/datanode/wal.md @@ -7,30 +7,26 @@ description: Introduction to Write-Ahead Logging (WAL) in GreptimeDB, its purpos ## Introduction -Our storage engine is inspired by the Log-structured Merge Tree (LSMT). Mutating operations are -applied to a MemTable instead of persisting to disk, which significantly improves performance but -also brings durability-related issues, especially when the Datanode crashes unexpectedly. Similar -to all LSMT-like storage engines, GreptimeDB uses a write-ahead log (WAL) to ensure data durability -and is safe from crashing. +Mito buffers writes in [memtables](memtable.md) before flushing them to SST files. It first appends each Region's mutations to the write-ahead log (WAL), so data that has not reached an SST can be recovered. -WAL is an append-only file group. All `INSERT` and `DELETE` operations are transformed into -operation entries and then appended to WAL. Once operation entries are persisted to the underlying -file, the operation can be further applied to MemTable. +The WAL uses a common log-store abstraction with local raft-engine and remote Kafka providers. -When the Datanode restarts, operation entries in WAL are replayed to reconstruct the correct -in-memory state. +## Write and Recovery Cycle -![WAL in Datanode](/wal.png) +The order of a normal write is: + +1. The Region worker assigns sequence numbers and a WAL entry ID. +2. It appends the mutations to the WAL. If the append fails, the mutations are not applied to the memtable. +3. After the append succeeds, Mito writes the mutations to the memtable and publishes the new committed sequence. +4. A flush writes immutable SST files and persists a manifest edit containing the new files and `flushed_entry_id`. +5. After the manifest edit is durable, WAL entries through `flushed_entry_id` are marked obsolete. The log store may reclaim them later. + +The manifest is the recovery boundary. On a normal reopen, Mito rebuilds the Region from the manifest and replays WAL entries starting at `flushed_entry_id + 1`. Region transitions may supply a later replay checkpoint, but they never replay entries before the persisted flush boundary. ## Namespace -Namespace of WAL is used to separate entries from different tables (different regions). Append and -read operations must provide a Namespace. Currently, region ID is used as the Namespace, because -each region has a MemTable that needs to be reconstructed when Datanode restarts. +WAL entries are isolated by Region, not by table. Each append and read identifies a Region namespace so one Region can be replayed or truncated independently. The local raft-engine provider uses the Region ID as its namespace ID. Kafka keeps Region identity within the provider's topic-backed log. ## Synchronous/Asynchronous flush -By default, appending to WAL is asynchronous, which means the writer will not wait until entries are -flushed to disk. This setting provides higher performance, but may lose data when running host shutdown unexpectedly. In the other hand, synchronous flush provides higher durability at the cost of performance. - -In v0.4 version, the new region worker architecture can use batching to alleviate the overhead of sync flush. +For the local raft-engine provider, `sync_write` controls whether an append waits for the log to be synced to durable storage. It defaults to `false`. Asynchronous writes reduce latency but can lose recently acknowledged entries if the host fails before buffered data is synced. Kafka WAL durability is controlled by its producer and cluster settings instead of this local option. diff --git a/versioned_docs/version-1.1/contributor-guide/flownode/arrangement.md b/versioned_docs/version-1.1/contributor-guide/flownode/arrangement.md index aed75af777..8b472ea316 100644 --- a/versioned_docs/version-1.1/contributor-guide/flownode/arrangement.md +++ b/versioned_docs/version-1.1/contributor-guide/flownode/arrangement.md @@ -5,6 +5,8 @@ description: Details on the arrangement component in Flownode, which stores stat # Arrangement +This page describes state used by Flownode's legacy streaming mode. Batching mode does not use an Arrangement. + Arrangement stores the state in the dataflow's process. It stores the streams of update flows for further querying and updating. The arrangement essentially stores key-value pairs with timestamps to mark their change time. diff --git a/versioned_docs/version-1.1/contributor-guide/flownode/batching_mode.md b/versioned_docs/version-1.1/contributor-guide/flownode/batching_mode.md index 37a695ce9c..aa8099d1ad 100644 --- a/versioned_docs/version-1.1/contributor-guide/flownode/batching_mode.md +++ b/versioned_docs/version-1.1/contributor-guide/flownode/batching_mode.md @@ -9,13 +9,13 @@ This guide provides a brief overview of the batching mode in `flownode`. It's in ## Overview -The batching mode in `flownode` is designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. This is in contrast to the original streaming mode, now deprecated, where data was processed as it arrived. +The batching mode in `flownode` is designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. This is in contrast to the legacy streaming path, which processes data as it arrives and is retained for compatibility but deprecated for new workloads. The core idea is to: 1. Define a `flow` with a SQL query that aggregates data from a source table into a sink table. 2. The query typically includes a time window function (e.g., `date_bin`) on a timestamp column. 3. When new data is inserted into the source table, the system marks the corresponding time windows as "dirty." -4. A background task periodically wakes up, identifies these dirty windows, and re-runs the aggregation query for those specific time ranges. +4. A background task runs on its own cadence, consumes the pending dirty windows at its next evaluation, and re-runs the aggregation query for those time ranges. 5. The results are then inserted into the sink table, effectively updating the aggregated view. ## Architecture @@ -39,15 +39,15 @@ A `BatchingTask` represents a single, independent data flow. Each task is associ - **State (`TaskState`)**: This contains the dynamic, mutable state of the task, most importantly the `DirtyTimeWindows`. - **Execution Loop**: The task runs an infinite loop (`start_executing_loop`) that: 1. Checks for a shutdown signal. - 2. Waits for a scheduled interval or until it's woken up. + 2. Sleeps until its next evaluation time. A task with an evaluation schedule sleeps until the next scheduled time; an adaptive task sleeps for a polling interval derived from the time window size and the minimum refresh duration. 3. Generates a new query plan (`gen_insert_plan`) based on the current set of dirty time windows. 4. Executes the query (`execute_logical_plan`) against the database. 5. Cleans up the processed dirty windows. ### `TaskState` and `DirtyTimeWindows` -- **`TaskState`**: This struct tracks the runtime state of a `BatchingTask`. It includes `dirty_time_windows`, which is crucial for determining what work needs to be done. -- **`DirtyTimeWindows`**: This is a key data structure that keeps track of which time windows have received new data since the last query execution. It stores a set of non-overlapping time ranges. When a task's execution loop runs, it consults this structure to build a `WHERE` clause that filters the source table for only the dirty time windows. +- **`TaskState`**: This struct tracks the runtime state of a `BatchingTask`, including the `dirty_time_windows` that determine its pending work. +- **`DirtyTimeWindows`**: This data structure tracks which time windows have received new data since the last query execution. It stores a set of non-overlapping time ranges. The execution loop uses it to build a `WHERE` clause that selects only the dirty windows from the source table. ### `TimeWindowExpr` @@ -56,15 +56,15 @@ The `TimeWindowExpr` is a helper utility for dealing with time window expression - **Evaluation**: It can take a timestamp and evaluate the time window expression to determine the start and end of the window that the timestamp falls into. - **Window Size**: It can also determine the size (duration) of the time window from the expression. -This is essential for both marking windows as dirty and for generating the correct filter conditions when querying the source table. +The same calculation is used to mark dirty windows and generate the source-table filters. ## Query Execution Flow Here's a simplified step-by-step walkthrough of how a query is executed in batch mode: 1. **Data Ingestion**: New data is written to a source table. -2. **Marking Dirty**: The `BatchingEngine` receives a notification about the new data. It uses the `TimeWindowExpr` associated with each relevant flow to determine which time windows are affected by the new data points. These windows are then added to the `DirtyTimeWindows` set in the corresponding `TaskState`. -3. **Task Wake-up**: The `BatchingTask`'s execution loop wakes up, either due to its periodic schedule or because it was notified of a large backlog of dirty windows. +2. **Marking Dirty**: The `BatchingEngine` receives a notification about the new data. It uses the `TimeWindowExpr` associated with each relevant flow to determine which time windows are affected by the new data points. These windows are then added to the `DirtyTimeWindows` set in the corresponding `TaskState`. Marking a window dirty does not wake the task. +3. **Next Evaluation**: The `BatchingTask`'s execution loop reaches its next evaluation, either at a scheduled time or after its adaptive polling interval, and consumes the pending dirty windows. 4. **Plan Generation**: The task calls `gen_insert_plan`. This method: - Inspects the `DirtyTimeWindows`. - Generates a series of `OR`'d `WHERE` clauses (e.g., `(ts >= 't1' AND ts < 't2') OR (ts >= 't3' AND ts < 't4') ...`) that cover the dirty windows. diff --git a/versioned_docs/version-1.1/contributor-guide/flownode/dataflow.md b/versioned_docs/version-1.1/contributor-guide/flownode/dataflow.md index 000a65edb3..c876054d6d 100644 --- a/versioned_docs/version-1.1/contributor-guide/flownode/dataflow.md +++ b/versioned_docs/version-1.1/contributor-guide/flownode/dataflow.md @@ -1,17 +1,38 @@ --- -keywords: [dataflow module, SQL query transformation, execution plan, DAG, map and reduce operations] -description: Explanation of the dataflow module in Flownode, its operations, internal data handling, and future enhancements. +keywords: [Flownode, batching mode, streaming mode, dataflow, dirty time windows] +description: How Flownode selects and runs its batching and legacy streaming execution paths. --- # Dataflow +Flownode has two internal execution paths: + +- **Batching mode** is the primary path for aggregation and TQL workloads. It evaluates queries over persisted source data and writes materialized results to a sink table. +- **Streaming mode** is the legacy path retained for compatibility and deprecated for new workloads. It incrementally processes rows mirrored from Frontend as they arrive. + +Users do not select the mode directly. When a Flow is created, GreptimeDB chooses the path from the query and source-table properties. Aggregation, `DISTINCT`, and TQL queries use batching mode. Simple non-aggregation queries, and any Flow whose source table has `ttl = 'instant'`, currently use streaming mode. A Flow deferred because its source table does not yet exist starts as a pending batching Flow. + +## Batching mode + +Batching mode reuses GreptimeDB's query engine instead of maintaining an operator graph for every incoming row. For a time-windowed Flow, its main loop is: + +1. A source-table write marks the affected time windows as dirty. +2. A `BatchingTask` runs on its evaluation schedule or adaptive polling cadence and collects the pending dirty windows at that evaluation. Marking a window dirty does not wake the task. +3. The task adds time predicates for those windows to the Flow query and asks Frontend to execute it against the source tables. +4. The query result is inserted into the sink table, updating the materialized result for windows that were evaluated. +5. Successfully processed windows are removed from the dirty set. Failed work remains available for a later evaluation. + +Flows with an evaluation interval but without a time-window expression run the complete query on each scheduled evaluation. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Flownode Batching Mode Developer Guide](./batching_mode.md) for the task and dirty-window components. + +## Streaming mode + The `dataflow` module (see `flow::compute` module) is the core computing module of `flow`. It takes a SQL query and transforms it into flow's internal execution plan. This execution plan is then rendered into an actual dataflow, which is essentially a directed acyclic graph (DAG) of functions with input and output ports. -The dataflow is triggered to run when needed. +New row changes drive the graph incrementally. -Currently, this dataflow only supports `map` and `reduce` operations. Support for `join` operations will be added in the future. +The renderer supports map/filter/project and reduce operations. Join and union plan nodes exist, but their streaming renderers are not implemented. Internally, the dataflow handles data in row format, using a tuple `(row, time, diff)`. Here, `row` represents the actual data being passed, which may contain multiple `Value` objects. `time` is the system time which tracks the progress of the dataflow, and `diff` typically represents the insertion or deletion of the row (+1 or -1). -Therefore, the tuple represents the insert/delete operation of the `row` at a given system `time`. \ No newline at end of file +Therefore, the tuple represents the insert/delete operation of the `row` at a given system `time`. Stateful operators keep indexed traces of these changes in an [Arrangement](./arrangement.md). diff --git a/versioned_docs/version-1.1/contributor-guide/frontend/distributed-querying.md b/versioned_docs/version-1.1/contributor-guide/frontend/distributed-querying.md index 21ee07d7e8..ca3822e113 100644 --- a/versioned_docs/version-1.1/contributor-guide/frontend/distributed-querying.md +++ b/versioned_docs/version-1.1/contributor-guide/frontend/distributed-querying.md @@ -5,29 +5,16 @@ description: Describes the process of distributed querying in GreptimeDB, focusi # Distributed Querying -Most steps of querying in frontend and datanode are identical. The only difference is that -Frontend have a "special" step in planning phase to make the logical query plan distributed. -Let's reference it as "dist planner" in the following text. - -The modified, distributed logical plan has multiple stages, each of them is executed in different -server node. +Frontend and Datanode use the same DataFusion-based query engine. In distributed mode, Frontend adds a planning step that separates work performed by Datanodes from work completed by Frontend. ![Frontend query](/frontend-query.png) ## Dist Planner -Planner will traverse the input logical plan, and split it into multiple stages by the "[commutativity -rule](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md)". +The distributed planner rewrites the logical plan. It pushes compatible operators toward table scans and wraps remote subplans in `MergeScan` nodes. Partition predicates are also used to prune Regions before the remote work is scheduled. -This rule is under heavy development. At present it will consider things like: -- whether the operator itself is commutative -- how the partition rule is configured -- etc... +Whether an operator can be pushed down depends on the plan shape and the operator's properties. Unsupported parts remain on Frontend. The original design and its commutativity rules are described in the [distributed planner RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md). ## Dist Plan -Except the first stage, which have to read data from files in storage. All other stages' leaf node -are actually a gRPC call to its previous stage. - -Sub-plan in a stage is itself a complete logical plan, and can be executed independently without -the follow up stages. The plan is encoded in [substrait format](https://substrait.io). +A remote input is a complete logical subplan, not just a table scan. Frontend serializes the subplan in [Substrait](https://substrait.io) format and sends a Region-specific request to the Datanode that owns the data. The Datanode plans and executes it locally, then streams the result back. Frontend merges the remote streams and executes any operators that were not pushed down. diff --git a/versioned_docs/version-1.1/contributor-guide/frontend/overview.md b/versioned_docs/version-1.1/contributor-guide/frontend/overview.md index 3a2f63e7ae..18cabfc2b8 100644 --- a/versioned_docs/version-1.1/contributor-guide/frontend/overview.md +++ b/versioned_docs/version-1.1/contributor-guide/frontend/overview.md @@ -5,27 +5,47 @@ description: Overview of GreptimeDB's Frontend component - a stateless proxy ser # Frontend -The **Frontend** is a stateless service that serves as the entry point for client requests in GreptimeDB. It provides a unified interface for multiple database protocols and acts as a proxy that forwards read/write requests to appropriate Datanodes in the distributed system. +Frontend is GreptimeDB's stateless request-orchestration service. The server layer terminates protocols and converts wire messages; Frontend supplies the database behavior behind those handlers, including permission checks, statement execution, routing, and distributed query planning. + +Frontend does not store table data. It caches catalog and route metadata obtained from Metasrv, and Metasrv invalidates those caches through heartbeat responses when metadata changes. ## Core Functions -- **Protocol Support**: Multiple database protocols including SQL, PromQL, MySQL, and PostgreSQL. See [Protocols][1] for details -- **Request Routing**: Routes requests to appropriate Datanodes based on metadata -- **Query Distribution**: Splits distributed queries across multiple nodes -- **Response Aggregation**: Combines results from multiple Datanodes -- **Authorization**: Security and access control validation +- Provide query and ingestion behavior for the supported [protocols][1]. +- Resolve catalogs, schemas, tables, and Region routes. +- Validate permissions before executing a request. +- Plan distributed queries and merge results from Datanodes. +- Convert table-level writes and deletes into Region requests. ## Architecture ### Key Components -- **Protocol Handlers**: Handle different database protocols -- **Catalog Manager**: Caches metadata from Metasrv to enable efficient request routing and schema validation -- **Dist Planner**: Converts logical plans to distributed execution plans -- **Request Router**: Determines target Datanodes for each request + +- Protocol handlers adapt SQL, PromQL, gRPC ingestion, and observability protocols to Frontend's internal request interfaces. +- The catalog and partition managers provide table metadata, partition rules, and Region routes. +- The statement executor dispatches queries, DML, and DDL to their respective execution paths. +- The distributed planner replaces table scans with `MergeScan` plans that can run across Datanodes. ### Request Flow -![request flow](/request_flow.png) +The request path depends on the operation. + +#### Queries + +1. A protocol handler creates the query context and performs authentication and permission checks. +2. The language-specific planner produces a logical plan. In distributed mode, the planner uses partition metadata to select Regions and constructs a distributed plan. +3. Frontend sends Region subplans to the owning Datanodes. Datanodes execute them against local Region engines and return streams of Arrow record batches. +4. Frontend runs the remaining operators, merges the streams, and formats the result for the client protocol. + +#### Writes and deletes + +1. Frontend validates the request against the table schema. Protocols that support schema-on-write may create a missing table or add columns before retrying the write. +2. The partition rule assigns rows to Regions. Frontend builds one Region request per target and routes it to the current Region leader. +3. The Datanode's Region server dispatches each request to the Region engine. In standalone mode, the same request is sent to an embedded Region server instead of over RPC. + +#### DDL + +The statement executor converts DDL into a task. In distributed mode, Metasrv runs that task as a persisted procedure, updates metadata, and coordinates Region operations on Datanodes. Standalone mode uses the same statement boundary with local implementations of the metadata and procedure services. ### Deployment diff --git a/versioned_docs/version-1.1/contributor-guide/frontend/table-sharding.md b/versioned_docs/version-1.1/contributor-guide/frontend/table-sharding.md index a60276d14e..beaece40c0 100644 --- a/versioned_docs/version-1.1/contributor-guide/frontend/table-sharding.md +++ b/versioned_docs/version-1.1/contributor-guide/frontend/table-sharding.md @@ -5,21 +5,15 @@ description: Explains how table data in GreptimeDB is sharded and distributed, i # Table Sharding -The sharding of stored data is essential to any distributed database. This document will describe how table's data in GreptimeDB is being sharded, and distributed. +GreptimeDB shards a table into Regions. Partition expressions define which rows belong to each Region, while Region routes define which Datanode currently owns each Region. ## Partition -For the syntax of creating a partitioned table, please refer to the [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) section in the User Guide. +A partition is a logical row set described by an expression over one or more columns. The partition layout must cover the table's input domain so each row has one target Region. See [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) for the SQL syntax and supported expressions. ## Region -The data within a table is logically split after creating partitions. You may ask the question " -how are the data, after being logically partitioned, stored in the GreptimeDB? The answer is in "`Region`"s. - -Each region is corresponding to a partition, and stores the data in the partition. The regions are distributed among -`Datanode`s. `Metasrv` manages the route information that maps regions to Datanodes. -If the partition layout needs to change after table creation, GreptimeDB supports explicit -[repartitioning](/user-guide/deployments-administration/manage-data/repartition.md) through split and merge operations. +Each partition maps to one Region. Region IDs remain the storage and routing identity used by Frontend, Datanode, and Metasrv. Multiple Regions from the same table may be placed on one Datanode. The relationship between partition and region can be viewed as the following diagram: @@ -53,3 +47,14 @@ The relationship between partition and region can be viewed as the following dia │ │ └──────────────────────────────────┘ Could be placed in one Datanode +``` + +## Routing and Pruning + +For writes, Frontend evaluates the partition rule for each row, groups rows by Region, and sends Region requests to the current leaders from the route table. + +For queries, the distributed planner compares query predicates with the partition expressions. It scans only Regions that can satisfy the predicates. If partition metadata is missing or cannot be interpreted safely, the planner falls back to all Regions rather than risk omitting data. + +## Changing the Partition Layout + +[Repartitioning](/user-guide/deployments-administration/manage-data/repartition.md) changes an existing layout through explicit split and merge operations. Metasrv runs the change as a persisted procedure, updates the Region routes and partition expressions, and invalidates stale table-route caches. New requests use the published layout after their Frontend refreshes that metadata. diff --git a/versioned_docs/version-1.1/contributor-guide/getting-started.md b/versioned_docs/version-1.1/contributor-guide/getting-started.md index b17184dfa8..7e8cca3ad8 100644 --- a/versioned_docs/version-1.1/contributor-guide/getting-started.md +++ b/versioned_docs/version-1.1/contributor-guide/getting-started.md @@ -15,14 +15,13 @@ At the moment, GreptimeDB supports Linux (both amd64 and arm64), macOS (both amd ### Build Dependencies -- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) (optional) +- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) (optional; needed to clone the repository, not to build it) - C/C++ Toolchain: provides essential tools for compiling and linking. This is available either as `build-essential` on ubuntu or a similar name on other platforms. -- Rust nightly toolchain ([guide][1]) - - Compile the source code +- [Rustup][1]. The repository pins the required nightly toolchain in `rust-toolchain.toml`. - Protobuf ([guide][2]) - Compile the proto file - Note that the version needs to be >= 3.15. You can check it with `protoc --version` -- Machine: Recommended memory is 16GB or more, or use the [mold](https://github.com/rui314/mold) tool to reduce memory usage during linking. +- Machine: 16GB of memory or more is recommended. On a smaller machine, use [mold](https://github.com/rui314/mold) to reduce memory usage during linking. [1]: [2]: diff --git a/versioned_docs/version-1.1/contributor-guide/how-to/how-to-write-sdk.md b/versioned_docs/version-1.1/contributor-guide/how-to/how-to-write-sdk.md index 40d0bc3713..e306448642 100644 --- a/versioned_docs/version-1.1/contributor-guide/how-to/how-to-write-sdk.md +++ b/versioned_docs/version-1.1/contributor-guide/how-to/how-to-write-sdk.md @@ -1,21 +1,17 @@ --- keywords: [gRPC SDK, GreptimeDatabase, Handle, HandleRequests, GreptimeRequest, GreptimeResponse] -description: Explains how to write a gRPC SDK for GreptimeDB, focusing on the GreptimeDatabase service, its methods, and the structure of requests and responses. +description: Protocol contracts and error-handling requirements for a GreptimeDB gRPC ingestion SDK. --- # How to write a gRPC SDK for GreptimeDB -A GreptimeDB gRPC SDK only needs to handle the writes. The reads are standard SQL and PromQL, can be handled by any JDBC -client or Prometheus client. This is also why GreptimeDB gRPC SDKs are all named -like "`greptimedb-ingester-`". Please make sure your GreptimeDB SDK follow the same naming convention. +GreptimeDB's public gRPC SDKs are ingestion clients. Queries normally use SQL or PromQL through their standard clients. A new SDK should therefore focus on writes and deletes unless it has a separate requirement, and follow the `greptimedb-ingester-` naming convention. See the [gRPC SDK overview](/user-guide/ingest-data/for-iot/grpc-sdks/overview.md) for the user-facing API. ## `GreptimeDatabase` Service -GreptimeDB defines a custom gRPC service called `GreptimeDatabase`. All you need to do in your SDK are implement it. You -can find its Protobuf -definitions [here](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto). +Generate client stubs from the [`GreptimeDatabase` Protobuf definition](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto). Do not maintain a handwritten copy of the messages or service definition. -The service contains two RPC methods: +The service provides a unary method and a client-streaming method: ```protobuf service GreptimeDatabase { @@ -25,13 +21,9 @@ service GreptimeDatabase { } ``` -The `Handle` method is for unary call: when a `GreptimeRequest` is received and processed by a GreptimeDB -server, it responds with a `GreptimeResponse` immediately. +`Handle` returns one response for one request. It is the usual choice for an SDK's insert and delete APIs. -The `HandleRequests` acts in -a "[Client streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc)" style. It ingests a -stream of `GreptimeRequest`, and handles them on the fly. After all the requests have been handled, it returns a -summarized `GreptimeResponse`. Through `HandleRequests`, we can achieve a very high throughput of requests handling. +`HandleRequests` is a [client-streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc). The server returns a cumulative response only after the client closes the request stream. An SDK that exposes streaming must document this acknowledgement boundary and bind the stream to one endpoint. ### `GreptimeRequest` @@ -51,13 +43,13 @@ message GreptimeRequest { } ``` -A `RequestHeader` is needed, it includes some context, authentication and others. The "oneof" field contains the request -to the GreptimeDB server. +A client must populate `RequestHeader` with the database context and authentication expected by the server. Set exactly one request variant. -Note that we have two types of insertions, one is in the form of "column" (the `InsertRequests`), and the other is " -row" (`RowInsertRequests`). It's generally recommended to use the "row" form, since it's more natural for insertions on -a table, and easier to use. However, if there's a need to insert a large number of columns at once, or there're plenty -of "null" values to insert, the "column" form is better to be used. +The message also contains query and DDL variants used by internal callers. The public ingester API should not expose them: `GreptimeDatabase` does not return query result streams. + +GreptimeDB accepts row-oriented `RowInsertRequests` and column-oriented `InsertRequests`. Row-oriented requests are the default for public ingestion APIs. A column-native client may use the column form, but it must keep column lengths consistent and preserve null values, timestamp precision, data types, and column semantic types during conversion. + +Deletes have the same row-oriented and column-oriented distinction. Expose only the forms that the SDK can map without losing type information. ### `GreptimeResponse` @@ -70,8 +62,18 @@ message GreptimeResponse { } ``` -The `ResponseHeader` contains the response's status code, and error message (if there's any). The "oneof" response only -contains the affected rows for now. +On success, the response contains a successful header and `affected_rows`. Treat that value as the number acknowledged by the server, including the cumulative value returned when a request stream closes. + +Request failures are returned as a gRPC status. When present, the trailing metadata key `x-greptime-err-code` carries GreptimeDB's error code, and the status message carries the error text. Preserve the gRPC status and expose the GreptimeDB error code rather than replacing them with a generic SDK error. + +## Retry and Delivery Semantics + +Retries must be bounded and observable. A unary request may be retried only when the failure is classified as retryable and the deadline still permits it. Do not retry cancellation or deadline-expiration errors. + +A lost response does not prove that the server rejected a write. Retrying such a request can insert duplicate rows unless the caller's data model makes the operation idempotent. Document this possibility and return the final error when delivery is ambiguous. + +Do not transparently retry a partially sent `HandleRequests` stream. The server may already have accepted some requests even though the client has not received the cumulative response. Close the failed stream and report the ambiguity to the caller. + +Keep Arrow Flight bulk ingestion separate from the `GreptimeDatabase` RPCs. Its batching and partial-acceptance behavior needs its own API contract. -GreptimeDB has a lot of SDKs now, you can refer to -them [here](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=) for some examples. +Use the existing [GreptimeDB ingester repositories](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=) to compare public API conventions, but derive wire behavior from the current Protobuf definition and server contract. diff --git a/versioned_docs/version-1.1/contributor-guide/metasrv/admin-api.md b/versioned_docs/version-1.1/contributor-guide/metasrv/admin-api.md index b091b48b70..d1f2ee6dbe 100644 --- a/versioned_docs/version-1.1/contributor-guide/metasrv/admin-api.md +++ b/versioned_docs/version-1.1/contributor-guide/metasrv/admin-api.md @@ -1,6 +1,6 @@ --- -keywords: [admin api, health check, leader query, heartbeat, maintenance mode, RESTful API] -description: Details the Admin API for Metasrv, including endpoints for health checks, leader queries, heartbeat data, maintenance mode, and Procedure Manager controls. +keywords: [admin api, health check, leader query, heartbeat, maintenance mode, recovery mode, table id sequence] +description: Details the Metasrv Admin API for status inspection, cluster controls, and metadata recovery. --- # Admin API @@ -9,16 +9,17 @@ description: Details the Admin API for Metasrv, including endpoints for health c Note that all Admin API endpoints in this document listen on Metasrv's `HTTP_PORT`, which defaults to `4000`. ::: -The Admin API provides a simple way to view and manage cluster information, including metasrv health detection, metasrv leader query, datanode heartbeat detection, maintenance mode, and Procedure Manager controls. - -The Admin API is an HTTP service that provides a set of RESTful APIs that can be called through HTTP requests. The Admin API is simple, user-friendly and safe. +The Admin API exposes Metasrv status, cluster controls, and metadata recovery operations over HTTP. It does not provide authentication, and some endpoints change cluster behavior or metadata allocation. Deployments must protect the HTTP port with network-level controls. This page covers the following APIs: - /health - /leader - /heartbeat +- /node-lease - /maintenance - /procedure-manager +- /recovery +- /sequence/table All these APIs are under the parent resource `/admin`. @@ -26,7 +27,7 @@ In the following sections, we assume that your metasrv instance is running on lo ## /health HTTP endpoint -The `/health` endpoint accepts GET HTTP requests and you can use this endpoint to check the health of your metasrv instance. +The `/health` endpoint accepts GET requests and returns `OK` when the HTTP service is running. It does not check whether this Metasrv is the leader or whether external dependencies are available. ### Definition @@ -120,9 +121,17 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100' ] ``` +## /node-lease HTTP endpoint + +The `/node-lease` endpoint returns the current leases recorded for Datanodes. Use it when diagnosing whether Metasrv still considers a Datanode active. + +```bash +curl -X GET http://localhost:4000/admin/node-lease +``` + ## /maintenance HTTP endpoint -Cluster Maintenance Mode is a safety feature in GreptimeDB that temporarily disables automatic cluster management operations. This mode is particularly useful during cluster upgrades, planned downtime, and any operation that might temporarily affect cluster stability. For more details, please refer to [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md). +Maintenance mode temporarily disables automatic cluster management operations during upgrades, planned downtime, or similar work. See [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md) for its effect on the cluster. The `/maintenance` endpoint supports the following HTTP requests: @@ -155,3 +164,39 @@ The response body uses the following format: "status": "running" } ``` + +## /recovery HTTP endpoints + +Recovery mode gates metadata repair endpoints such as manual table ID sequence changes. It is intended for recovery work, not routine maintenance. + +- `GET /admin/recovery/status`: query whether recovery mode is enabled. +- `POST /admin/recovery/enable`: enable recovery mode. +- `POST /admin/recovery/disable`: disable recovery mode. + +The response body uses the following format: + +```json +{ + "enabled": true +} +``` + +Disable recovery mode after the repair is complete. Use [maintenance mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md) instead when the goal is to suspend automatic cluster operations during planned maintenance. + +## /sequence/table HTTP endpoints + +These endpoints inspect or repair the table ID sequence: + +- `GET /admin/sequence/table/next-id`: return the next table ID without allocating it. +- `POST /admin/sequence/table/set-next-id`: advance the next table ID. + +Setting the sequence requires recovery mode. The new value must be greater than the current value; the API cannot move the sequence backwards. Recovery mode is an API precondition, not a DDL barrier. Follow [Manage table ID sequences](/user-guide/deployments-administration/maintenance/sequence-management.md) for the required cluster-wide procedure. + +```bash +curl -X POST \ + -H 'Content-Type: application/json' \ + -d '{"next_table_id": 2048}' \ + http://localhost:4000/admin/sequence/table/set-next-id +``` + +Changing this value affects IDs allocated to future tables. diff --git a/versioned_docs/version-1.1/contributor-guide/metasrv/overview.md b/versioned_docs/version-1.1/contributor-guide/metasrv/overview.md index 7aa052dd7e..6458b6380d 100644 --- a/versioned_docs/version-1.1/contributor-guide/metasrv/overview.md +++ b/versioned_docs/version-1.1/contributor-guide/metasrv/overview.md @@ -1,161 +1,101 @@ --- -keywords: [metasrv, metadata, request-router, load balancing, election, high availability, heartbeat] -description: Provides an overview of the Metasrv service, its components, interactions with the Frontend, architecture, and key functionalities like distributed consensus and heartbeat management. +keywords: [metasrv, metadata, routing, leader election, procedure, heartbeat] +description: Overview of the metadata and coordination mechanisms provided by Metasrv. --- # Metasrv -![meta](/meta.png) - ## What's in Metasrv -- Store metadata (Catalog, Schema, Table, Region, etc.) -- Request-Router. It tells the Frontend where to write and read data. -- Load balancing for Datanode, determines who should handle new table creation requests, more precisely, it makes resource allocation decisions. -- Election & High Availability, GreptimeDB is designed in a Leader-Follower architecture, only Leader nodes can write while Follower nodes can read, the number of Follower nodes is usually >= 1, and Follower nodes need to be able to switch to Leader quickly when Leader is not available. -- Statistical data collection (reported via Heartbeats on each node), such as CPU, Load, number of Tables on the node, average/peak data read/write size, etc., can be used as the basis for distributed scheduling. +Metasrv is the metadata and coordination service in a distributed GreptimeDB cluster. It does not sit on the data path. Its main responsibilities are: + +- storing Catalog, Schema, Table, Region, route, and node metadata; +- choosing Datanodes for new Regions and maintaining table routes; +- electing one Metasrv leader to coordinate metadata changes; +- running recoverable procedures for DDL, Region migration, failover, and repartitioning; +- tracking node leases and Region statistics through heartbeats; +- broadcasting cache invalidations to Frontends, Datanodes, and Flownodes when metadata changes; +- sending Region lifecycle instructions to Datanodes. ## How the Frontend interacts with Metasrv -First, the routing table in Request-Router is in the following structure (note that this is only the logical structure, the actual storage structure varies, for example, endpoints may have dictionary compression). +Frontend obtains table metadata and Region routes from Metasrv and caches them locally. Metadata-changing statements are sent to the Metasrv leader, while reads and writes use the cached routes to reach Datanodes directly. + +The control and data paths are separate: + +```text +Frontend + |-- metadata lookup and DDL ------------> Metasrv leader + `-- Region reads and writes ------------> Datanode + +Metasrv leader + |-- Region lifecycle instructions ------> Datanode + `-- cache invalidations ----------------> Frontend / Datanode / Flownode +Datanode + `-- heartbeat, lease renewal, Region stats -> Metasrv leader ``` - table_A - table_name - table_schema // for physical plan - regions - region_1 - mutate_endpoint - select_endpoint_1, select_endpoint_2 - region_2 - mutate_endpoint - select_endpoint_1, select_endpoint_2, select_endpoint_3 - region_xxx - table_B - ... + +In steady state, a table route records one leader peer and zero or more follower peers for each Region. The leader is the write target. Deployments with read-replica support can route reads to followers: + +```text +Table route + |-- Region 0 + | |-- leader -> Datanode A + | `-- followers -> Datanode B, Datanode C + `-- Region 1 + `-- leader -> Datanode D ``` +Region migration or failover changes peer roles and can temporarily leave a Region without a leader. Frontend refreshes its cached route before sending subsequent reads or writes to the current peers. + ### Create Table -1. The Frontend sends `CREATE TABLE` requests to Metasrv. -2. Plan the number of Regions according to the partition rules contained in the request. -3. Check the global view of resources available to Datanodes (collected by Heartbeats) and assign one node to each region. -4. The Frontend creates the table and stores the `Schema` to Metasrv after successful creation. +1. Frontend submits the DDL request to the Metasrv leader. +2. Metasrv derives Regions from the partition rules and [selects a Datanode for each Region](/contributor-guide/metasrv/selector.md). +3. A persisted procedure creates the Regions and records the table and route metadata. If leadership changes, the procedure can resume from its persisted state. +4. Metasrv notifies Frontends after the metadata change is committed so their caches can be refreshed. ### Insert -1. The Frontend fetches the routes of the specified table from Metasrv. Note that the smallest routing unit is the route of the table (several regions), i.e., it contains the addresses of all regions of this table. -2. The best practice is that the Frontend first fetches the routes from its local cache and forwards the request to the Datanode. If the route is no longer valid, then Datanode is obliged to return an `Invalid Route` error, and the Frontend re-fetches the latest data from Metasrv and updates its cache. Route information does not change frequently, thus, it's sufficient for Frontend uses the Lazy policy to maintain the cache. -3. The Frontend processes a batch of writes that may contain multiple tables and multiple regions, so the Frontend needs to split user requests based on the 'route table'. +Frontend resolves the table route, splits rows according to the partition rules, and sends each Region write to the corresponding Datanode. Route changes cause the cached metadata to be invalidated and fetched again from Metasrv. ### Select -1. As with `Insert`, the Frontend first fetches the route table from the local cache. -2. Unlike `Insert`, for `Select`, the Frontend needs to extract the read-only node (follower) from the route table, then dispatch the request to the leader or follower node depending on the priority. -3. The distributed query engine in the Frontend distributes multiple sub-query tasks based on the routing information and aggregates the query results. +Frontend uses table and Region metadata while planning the query. Predicates on partition columns prune Regions, and the distributed query engine sends work to the Datanodes that own the selected Regions. See [Distributed Querying](../frontend/distributed-querying.md). ## Metasrv Architecture -![metasrv-architecture](/metasrv-architecture.png) - -## Distributed Consensus +The main coordination paths are: + +```text +Leader election + | + v +Metasrv leader +├─ DDL manager -> Procedure manager +├─ Selector -> new Region placement +├─ Heartbeat handler chain -> leases and Region statistics +├─ Region supervisor -> Region migration procedures +├─ Mailbox -> cache invalidations and Region instructions +└─ Metadata managers -> KV backend +``` -As you can see, Metasrv has a dependency on distributed consensus because: +These mechanisms share metadata, but they have different failure boundaries. A process restart may discard caches and leader-local state; metadata and procedure state required for recovery must be durable. -1. First, Metasrv has to elect a leader, Datanode only sends heartbeats to the leader, and we only use a single metasrv node to receive heartbeats, which makes it easy to do some calculations or scheduling accurately and quickly based on global information. As for how the Datanode connects to the leader, this is for MetaClient to decide (using a redirect, Heartbeat requests becomes a gRPC stream, and using redirect will be less error-prone than forwarding), and it is transparent to the Datanode. -2. Second, Metasrv must provide an election API for Datanode to elect "write" and "read-only" nodes and help Datanode achieve high availability. -3. Finally, `Metadata`, `Schema` and other data must be reliably and consistently stored on Metasrv. Therefore, consensus-based algorithms are the ideal approach for storing them. +## Distributed Consensus -For the first version of Metasrv, we choose Etcd as the consensus algorithm component (Metasrv is designed to consider adapting different implementations and even creating a new wheel) for the following reasons: +Metasrv separates leader election from metadata storage. Only the elected Metasrv leader performs coordination and metadata-changing operations. Other Metasrv nodes direct clients to the current leader. -1. Etcd provides exactly the API we need, such as `Watch`, `Election`, `KV`, etc. -2. We only perform two tasks with distributed consensus: elections (using the `Watch` mechanism) and storing (a small amount of metadata), and neither of them requires us to customize our own state machine, nor do we need to customize our own state machine based on raft; the small amount of data also does not require multi-raft-group support. -3. The initial version of Metasrv uses Etcd, which allows us to focus on the capabilities of Metasrv and not spend too much effort on distributed consensus algorithms, which improves the design of the system (avoiding coupling with consensus algorithms) and helps with rapid development at the beginning, as well as allows easy access to good consensus algorithm implementations in the future through good architectural designs. +The key-value backend stores table metadata, routes, procedure state, and other information that must survive a leader change. Metasrv does not use this election to create leader and follower replicas for Datanode Regions; Region availability is managed through heartbeats, Region failure detection, and failover procedures. ## Heartbeat Management -The primary means of communication between Datanode and Metasrv is the Heartbeat Request/Response Stream, and we want this to be the only way to communicate. This idea is inspired by the design of [TiKV PD](https://github.com/tikv/pd), and we have practical experience in [RheaKV](https://github.com/sofastack/sofa-jraft/tree/master/jraft-rheakv/rheakv-pd). The request sends its state, while Metasrv sends different scheduling instructions via Heartbeat Response. - -A heartbeat will probably carry the data listed below, but this is not the final design, and we are still discussing and exploring exactly which data should be mostly collected. - -``` -service Heartbeat { - // Heartbeat, there may be many contents of the heartbeat, such as: - // 1. Metadata to be registered to metasrv and discoverable by other nodes. - // 2. Some performance metrics, such as Load, CPU usage, etc. - // 3. The number of computing tasks being executed. - rpc Heartbeat(stream HeartbeatRequest) returns (stream HeartbeatResponse) {} -} - -message HeartbeatRequest { - RequestHeader header = 1; - - // Self peer - Peer peer = 2; - // Leader node - bool is_leader = 3; - // Actually reported time interval - TimeInterval report_interval = 4; - // Node stat - NodeStat node_stat = 5; - // Region stats in this node - repeated RegionStat region_stats = 6; - // Follower nodes and stats, empty on follower nodes - repeated ReplicaStat replica_stats = 7; -} - -message NodeStat { - // The read capacity units during this period - uint64 rcus = 1; - // The write capacity units during this period - uint64 wcus = 2; - // Table number in this node - uint64 table_num = 3; - // Region number in this node - uint64 region_num = 4; - - double cpu_usage = 5; - double load = 6; - // Read disk I/O in the node - double read_io_rate = 7; - // Write disk I/O in the node - double write_io_rate = 8; - - // Others - map attrs = 100; -} - -message RegionStat { - uint64 region_id = 1; - TableName table_name = 2; - // The read capacity units during this period - uint64 rcus = 3; - // The write capacity units during this period - uint64 wcus = 4; - // Approximate region size - uint64 approximate_size = 5; - // Approximate number of rows - uint64 approximate_rows = 6; - - // Others - map attrs = 100; -} - -message ReplicaStat { - Peer peer = 1; - bool in_sync = 2; - bool is_learner = 3; -} -``` - -## Central Nervous System (CNS) - -We are to build an algorithmic system, which relies on real-time and historical heartbeat data from each node, should make some smarter scheduling decisions and send them to Metasrv's Autoadmin unit, which distributes the scheduling decisions, either by the Datanode itself or more likely by the PaaS platform. - -## Abstraction of Workloads +Datanodes maintain heartbeat streams to the Metasrv leader. Heartbeat requests report node identity, lease information, Region statistics, and other state used for placement and supervision. Responses carry control messages such as Region lifecycle instructions and cache invalidations. -The level of workload abstraction determines the efficiency and quality of the scheduling strategy generated by Metasrv such as resource allocation. +A heartbeat drives two independent mechanisms, and a change to heartbeat timing affects both: -DynamoDB defines RCUs & WCUs (Read Capacity Units / Write Capacity Units), explaining that a RCU is a read request of 4KB data, and a WCU is a write request of 1KB data. When using RCU and WCU to describe workloads, it's easier to achieve performance measurability and get more informative resource preallocation because we can abstract different hardware capabilities as a combination of RCU and WCU. +- **Node lease.** The keep-lease handler renews the sending Datanode's lease. Selectors and the `/node-lease` endpoint use these leases to decide whether a Datanode is still active. +- **Region failure detection.** The Region supervisor keeps a per-Region Phi Accrual detector over heartbeat arrival intervals. Its verdict is independent of lease expiry. -However, GreptimeDB still faces a more complex situation than DynamoDB, in particular, RCU doesn't fit to describe GreptimeDB's read workloads which require a lot of computation. We are working on that. +A failure verdict submits a failover migration only when Region failover is enabled; it is disabled by default and requires remote WAL unless explicitly allowed on local WAL. Maintenance mode also suppresses failover. See [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md) for the prerequisites and how to enable it. diff --git a/versioned_docs/version-1.1/contributor-guide/metasrv/selector.md b/versioned_docs/version-1.1/contributor-guide/metasrv/selector.md index 790ffb849b..23190cd6a9 100644 --- a/versioned_docs/version-1.1/contributor-guide/metasrv/selector.md +++ b/versioned_docs/version-1.1/contributor-guide/metasrv/selector.md @@ -7,32 +7,28 @@ description: Describes the different types of selectors in the Metasrv service, ## Introduction -What is the `Selector`? As its name suggests, it allows users to select specific items from a given `namespace` and `context`. There is a related trait, also named `Selector`, whose definition can be found [below][0]. - -[0]: https://github.com/GreptimeTeam/greptimedb/blob/main/src/meta-srv/src/selector.rs - -There is a specific scenario in `Metasrv` service. When a request to create a table is sent to the `Metasrv` service, it creates a routing table (the details of table creation will not be described here). The `Metasrv` service needs to select the appropriate `Datanode` list when creating a routing table. +When a table is created, Metasrv uses a `Selector` to choose Datanodes for its Regions. Selection uses the current node leases and, depending on the selector, Region statistics. ## Selector Type The `Metasrv` service currently offers the following types of `Selectors`: -### LeasebasedSelector +### LeaseBasedSelector -`LeasebasedSelector` randomly selects from all available (in lease) `Datanode`s, its characteristic is simplicity and fast. +`LeaseBasedSelector` randomly selects from Datanodes with valid leases. ### LoadBasedSelector The `LoadBasedSelector` load value is determined by the number of regions on each `Datanode`, fewer regions indicate lower load, and `LoadBasedSelector` prioritizes selecting low-load `Datanodes`. ### RoundRobinSelector [default] -`RoundRobinSelector` selects `Datanode`s in a round-robin fashion. It is recommended and the default option in most cases. If you're unsure which to choose, it's usually the right choice. +`RoundRobinSelector` selects `Datanode`s in a round-robin fashion. It is the default and recommended choice for most deployments. ## Configuration You can configure the `Selector` by its name when starting the `Metasrv` service. -- LeasebasedSelector: `lease_based` or `LeaseBased` +- LeaseBasedSelector: `lease_based` or `LeaseBased` - LoadBasedSelector: `load_based` or `LoadBased` - RoundRobinSelector: `round_robin` or `RoundRobin` diff --git a/versioned_docs/version-1.1/contributor-guide/overview.md b/versioned_docs/version-1.1/contributor-guide/overview.md index 875b4b1e16..6e6ec669c9 100644 --- a/versioned_docs/version-1.1/contributor-guide/overview.md +++ b/versioned_docs/version-1.1/contributor-guide/overview.md @@ -5,9 +5,7 @@ description: Overview of GreptimeDB's architecture, key components, and how they # Contributor Guide -DeepWiki provides a detailed and clear explanation of GreptimeDB's architecture and implementation. Highly recommended: - -[https://deepwiki.com/GreptimeTeam/greptimedb](https://deepwiki.com/GreptimeTeam/greptimedb) +This guide explains the internal design of GreptimeDB for contributors. Start with [Getting Started](/contributor-guide/getting-started.md) to build and run it from source. Submission requirements, including the CLA, license headers, formatting, and the checks a pull request must pass, are maintained in the source repository's [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md). ## Architecture @@ -18,8 +16,13 @@ For more details on each component, see the following guides: - [frontend][1] - [datanode][2] - [metasrv][3] +- [flownode][4] [1]: /contributor-guide/frontend/overview.md [2]: /contributor-guide/datanode/overview.md [3]: /contributor-guide/metasrv/overview.md +[4]: /contributor-guide/flownode/overview.md + +## Additional reference +[DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb) provides an automatically generated walkthrough of the GreptimeDB repository. It can help when exploring an unfamiliar area, but it is a secondary reference: verify version-sensitive behavior against the source code. diff --git a/versioned_docs/version-1.1/contributor-guide/tests/integration-test.md b/versioned_docs/version-1.1/contributor-guide/tests/integration-test.md index 5d5f6cb1a5..bc52ee4854 100644 --- a/versioned_docs/version-1.1/contributor-guide/tests/integration-test.md +++ b/versioned_docs/version-1.1/contributor-guide/tests/integration-test.md @@ -7,7 +7,14 @@ description: Guide on writing and running integration tests in GreptimeDB, cover ## Introduction -Integration testing is written with Rust test harness (`#[test]`), unlike unit testing, they are placed separately -[here](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration). -It covers scenarios involving multiple components, in which one typical case is HTTP/gRPC-related features. You can check -its [documentation](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md) for more information. +Integration tests cover behavior that crosses crate or service boundaries, such as HTTP and gRPC handling, distributed components, or external storage. They use Rust's test harness and live in the [`tests-integration`](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration) package. + +Run the package with: + +```shell +cargo nextest run -p tests-integration +``` + +Some cases require environment variables or fixtures for external services. Follow the package's [setup instructions](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md) before running those cases. + +Use an integration test when a crate-level test or a Sqlness case cannot exercise the required boundary. Keep isolated logic in unit tests so that failures remain fast to reproduce. diff --git a/versioned_docs/version-1.1/contributor-guide/tests/overview.md b/versioned_docs/version-1.1/contributor-guide/tests/overview.md index feeefe2b6d..07290b79bd 100644 --- a/versioned_docs/version-1.1/contributor-guide/tests/overview.md +++ b/versioned_docs/version-1.1/contributor-guide/tests/overview.md @@ -5,4 +5,12 @@ description: Overview of the testing methods used in GreptimeDB to ensure its be # Tests -Our team has conducted lots of tests to ensure the behaviours of `GreptimeDB` . This chapter will introduce several significant methods used to test `GreptimeDB`, and how to work with them. +Choose the narrowest test that exercises the behavior you changed: + +| Test type | Use it for | Typical command | +| --- | --- | --- | +| [Unit test](unit-test.md) | Logic contained within one crate or component | `cargo nextest run -p ` | +| [Sqlness test](sqlness-test.md) | SQL, protocol, planner, execution, and end-to-end regressions | `cargo sqlness bare -t ` | +| [Integration test](integration-test.md) | Behavior that crosses components or requires external services | `cargo nextest run -p tests-integration` | + +Run `make test` when a change needs the full Rust workspace test suite. diff --git a/versioned_docs/version-1.1/contributor-guide/tests/sqlness-test.md b/versioned_docs/version-1.1/contributor-guide/tests/sqlness-test.md index b5fdcf2b0b..0262ff6851 100644 --- a/versioned_docs/version-1.1/contributor-guide/tests/sqlness-test.md +++ b/versioned_docs/version-1.1/contributor-guide/tests/sqlness-test.md @@ -7,42 +7,34 @@ description: Instructions for running SQL tests in GreptimeDB using the `sqlness ## Introduction -SQL is an important user interface for `GreptimeDB`. We have a separate test suite for it (named `sqlness`). +Sqlness is GreptimeDB's end-to-end regression suite for SQL and protocol behavior. A case sends statements to a running GreptimeDB instance and compares the output with a checked-in result file. ## Sqlness manual ### Case file -Sqlness has two types of file +Each case uses two files: - `.sql`: test input, SQL only - `.result`: expected test output, SQL and its results -The `.result` file is the expected execution output. If you see `.result` files changed, -it means the test gets a different result and indicates it may fail. You should -check the change logs to solve the problem. - -You only need to write test SQL in the `.sql` file, and run the test. +Write the input in the `.sql` file and run the test to generate or update `.result`. Review every result diff: accept it only when the behavior change is intended. ### Case organization -The root dir of input cases is `tests/cases`. It contains several sub-directories stand for different test -modes. E.g., `standalone/` contains all the tests to run under `greptimedb standalone start` mode. +Input cases live under `tests/cases`. The first directory level selects an environment. For example, `standalone/` runs against a standalone GreptimeDB instance. -Under the first level of sub-directory (e.g. the `cases/standalone`), you can organize your cases as you like. -Sqlness walks through every file recursively and runs them. +Within an environment, group a new case with the feature it exercises. Sqlness discovers case files recursively. ## Run the test -Unlike other tests, this harness is in a binary target form. You can run it with +Run the suite with: ```shell -cargo run --bin sqlness-runner bare +cargo sqlness bare ``` -It automatically finishes the following procedures: compile `GreptimeDB`, start it, grab tests and feed it to -the server, then collect and compare the results. You only need to check whether any `.result` files changed. -If not, congratulations, the test is passed 🥳! +The command builds and starts GreptimeDB, runs the selected cases, and compares their output. A changed `.result` file is part of the review, not proof that the new output is correct. ### Run a specific test diff --git a/versioned_docs/version-1.1/contributor-guide/tests/unit-test.md b/versioned_docs/version-1.1/contributor-guide/tests/unit-test.md index 82e30bf5fa..183a72aab4 100644 --- a/versioned_docs/version-1.1/contributor-guide/tests/unit-test.md +++ b/versioned_docs/version-1.1/contributor-guide/tests/unit-test.md @@ -8,25 +8,26 @@ description: Guide on writing and running unit tests in GreptimeDB using Rust's ## Introduction Unit tests are embedded into the codebase, usually placed next to the logic being tested. -They are written using Rust's `#[test]` attribute and can run with `cargo nextest run`. +They are written using Rust's `#[test]` attribute. GreptimeDB uses [`cargo-nextest`](https://nexte.st/) as its primary Rust test runner. -The default test runner ships with `cargo` is not supported in GreptimeDB codebase. It's recommended -to use [`nextest`](https://nexte.st/) instead. You can install it with +Install it with: ```shell cargo install cargo-nextest --locked ``` -And run the tests (here the `--workspace` is not necessary) +Run the package you changed first: ```shell -cargo nextest run +cargo nextest run -p ``` -Notes if your Rust is installed via `rustup`, be sure to install `nextest` with `cargo` rather -than the package manager like `homebrew`. Otherwise it will mess up your local environment. +Use a test name or nextest filter to narrow the run further while developing. Before submitting a change with broad effects, run the full workspace suite: + +```shell +make test +``` ## Coverage -Our continuous integration (CI) jobs have a "coverage checking" step. It will report how many -codes are covered by unit tests. Please add the necessary unit test to your patch. +CI reports unit-test coverage. Add tests for changed behavior and failure cases that could otherwise regress; coverage percentage alone is not the goal. diff --git a/versioned_docs/version-1.1/reference/sql/create.md b/versioned_docs/version-1.1/reference/sql/create.md index 87f3f97cb0..9493db1369 100644 --- a/versioned_docs/version-1.1/reference/sql/create.md +++ b/versioned_docs/version-1.1/reference/sql/create.md @@ -26,7 +26,7 @@ If the `db_name` database already exists, then GreptimeDB has the following beha The database can also carry options similar to the `CREATE TABLE` statement by using the `WITH` keyword. The following options are available for databases: - `ttl` - Time-To-Live for data in all tables within the database (cannot be set to `instant`) -- `memtable.type` - Type of memtable (`time_series`, `partition_tree`) +- `memtable.type` - Type of memtable (`bulk`, `time_series`) - `append_mode` - Whether tables in the database should be append-only (`true`/`false`) - `merge_mode` - Strategy for merging duplicate rows (`last_row`, `last_non_null`) - `skip_wal` - Whether to disable Write-Ahead-Log for tables in the database (`'true'`/`'false'`) @@ -74,7 +74,7 @@ Create a database with multiple options, including append mode and custom memtab ```sql CREATE DATABASE test WITH ( ttl='30d', - 'memtable.type'='partition_tree', + 'memtable.type'='bulk', 'append_mode'='true' ); ``` @@ -154,7 +154,7 @@ Users can add table options by using `WITH`. The valid options contain the follo | `compaction.twcs.trigger_file_num` | Number of files in a specific time window to trigger a compaction | String value, such as '8'. Only available when `compaction.type` is `twcs`. You can refer to this [document](https://cassandra.apache.org/doc/latest/cassandra/managing/operating/compaction/twcs.html) to learn more about the `twcs` compaction strategy. | | `compaction.twcs.time_window` | Compaction time window | String value, such as '1d' for 1 day. The table usually partitions rows into different time windows by their timestamps. Only available when `compaction.type` is `twcs`. | | `compaction.twcs.max_output_file_size` | Maximum allowed output file size for TWCS compaction | String value, such as '1GB', '512MB'. Sets the maximum size for files produced by TWCS compaction. Only available when `compaction.type` is `twcs`. | -| `memtable.type` | Type of the memtable. | String value, supports `time_series`, `partition_tree`. | +| `memtable.type` | Type of the memtable | String value: `bulk` or `time_series`. If omitted, Mito selects the implementation from the SST format; the default flat format uses `bulk`. Setting `bulk` forces `sst_format=flat`, and flat SSTs use the bulk implementation even if `time_series` is specified. The legacy value `partition_tree` is accepted for compatibility and maps to the bulk and flat path. | | `append_mode` | Whether the table is append-only | String value. Default is 'false', which removes duplicate rows by primary keys and timestamps according to the `merge_mode`. Setting it to 'true' to enable append mode and create an append-only table which keeps duplicate rows. | | `merge_mode` | The strategy to merge duplicate rows | String value. Only available when `append_mode` is 'false'. Default is `last_row`, which keeps the last row for the same primary key and timestamp. Setting it to `last_non_null` to keep the last non-null field for the same primary key and timestamp. | | `sst_format` | The format of SST files | String value, supports `primary_key`, `flat`. Default is `flat`. `flat` is recommended for tables which have a large number of unique primary keys. | diff --git a/versioned_docs/version-1.1/user-guide/deployments-administration/configuration.md b/versioned_docs/version-1.1/user-guide/deployments-administration/configuration.md index 40588adc2a..31b63bfba2 100644 --- a/versioned_docs/version-1.1/user-guide/deployments-administration/configuration.md +++ b/versioned_docs/version-1.1/user-guide/deployments-administration/configuration.md @@ -479,20 +479,9 @@ create_on_compaction = "auto" apply_on_query = "auto" mem_threshold_on_create = "64M" intermediate_path = "" - -[region_engine.mito.memtable] -type = "time_series" ``` -The `mito` engine provides an experimental memtable which optimizes for write performance and memory efficiency under large amounts of time-series. Its read performance might not as fast as the default `time_series` memtable. - -```toml -[region_engine.mito.memtable] -type = "partition_tree" -index_max_keys_per_shard = 8192 -data_freeze_threshold = 32768 -fork_dictionary_bytes = "1GiB" -``` +Mito selects the memtable implementation for each Region according to its table options and SST format. When `default_flat_format` is `true`, Regions without an explicit `sst_format` use flat SSTs and the bulk memtable. Configure `memtable.type` as a database or table option; `[region_engine.mito.memtable]` is not an engine setting. See [table options](/reference/sql/create.md#table-options). Available options: @@ -526,7 +515,7 @@ Available options: | `scan_memory_on_exhausted` | String | `fail` | Behavior when scan memory is exhausted. Options: `fail` (fail fast), `wait` or `wait()` (wait for memory). | | `min_compaction_interval` | String | `0m` | Minimum time interval between two compactions. Set to "0m" (default) to allow compactions to run immediately without restriction. | | `schedule_compaction_after_edit` | Bool | `true` | Whether to allow scheduling a compaction after a successful region edit.
Setting this to `true` is a necessary but not sufficient condition for scheduling compaction after a region edit. Other constraints, such as `min_compaction_interval`, may still prevent compaction from being scheduled.
Setting this to `false` guarantees that compaction will not be scheduled after a region edit. | -| `default_flat_format` | Bool | `true` | Whether to enable flat format as the default SST format. | +| `default_flat_format` | Bool | `true` | Whether Regions without an explicit `sst_format` use flat SSTs. Flat SSTs use the bulk memtable. | | `scan_parallelism` | Integer | `0` | (Deprecated, use `max_concurrent_scan_files` instead) Legacy option for scan parallelism. | | `index` | -- | -- | The options for index in Mito engine. | | `index.aux_path` | String | `""` | Auxiliary directory path for the index in the filesystem. This path is used to store intermediate files for creating the index and staging files for searching the index. It defaults to `{data_home}/index_intermediate`. The default name for this directory is `index_intermediate` for backward compatibility. This path contains two subdirectories: `__intm` for storing intermediate files used during index creation, and `staging` for storing staging files used during index searching. | @@ -542,10 +531,6 @@ Available options: | `inverted_index.apply_on_query` | String | `auto` | Whether to apply the index on query
- `auto`: automatically
- `disable`: never | | `inverted_index.mem_threshold_on_create` | String | `64M` | Memory threshold for performing an external sort during index creation.
Setting to empty will disable external sorting, forcing all sorting operations to happen in memory. | | `inverted_index.intermediate_path` | String | `""` | File system path to store intermediate files for external sorting (default `{data_home}/index_intermediate`). | -| `memtable.type` | String | `time_series` | Memtable type.
- `time_series`: time-series memtable
- `partition_tree`: partition tree memtable (experimental) | -| `memtable.index_max_keys_per_shard` | Integer | `8192` | The max number of keys in one shard.
Only available for `partition_tree` memtable. | -| `memtable.data_freeze_threshold` | Integer | `32768` | The max rows of data inside the actively writing buffer in one shard.
Only available for `partition_tree` memtable. | -| `memtable.fork_dictionary_bytes` | String | `1GiB` | Max dictionary bytes.
Only available for `partition_tree` memtable. | The `metric` engine is optimized for handling metrics data with a large number of small tables: diff --git a/versioned_docs/version-1.2/contributor-guide/datanode/data-persistence-indexing.md b/versioned_docs/version-1.2/contributor-guide/datanode/data-persistence-indexing.md index 6403ffc2df..1fa5d6d258 100644 --- a/versioned_docs/version-1.2/contributor-guide/datanode/data-persistence-indexing.md +++ b/versioned_docs/version-1.2/contributor-guide/datanode/data-persistence-indexing.md @@ -5,19 +5,23 @@ description: Explanation of data persistence and indexing in GreptimeDB, includi # Data Persistence and Indexing -Similar to all LSMT-like storage engines, data in MemTables is persisted to durable storage, for example, the local disk file system or object storage service. GreptimeDB adopts [Apache Parquet][1] as its persistent file format. +Like other LSM-tree storage engines, GreptimeDB persists data from memtables to durable storage such as a local filesystem or object storage. It uses [Apache Parquet][1] as the persistent file format. ## SST File Format Parquet is an open source columnar format that provides fast data querying and has already been adopted by many projects, such as Delta Lake. -Parquet has a hierarchical structure like "row groups-columns-data pages". Data in a Parquet file is horizontally partitioned into row groups, in which all values of the same column are stored together to form a data page. Data page is the minimal storage unit. This structure greatly improves performance. +Parquet organizes data as row groups, column chunks, and pages. A row group contains one column chunk for each column, and each column chunk contains one or more pages. Pages are the units of encoding and compression; column chunks are the I/O units for reading selected columns. First, clustering data by column makes file scanning more efficient, especially when only a few columns are queried, which is very common in analytical systems. -Second, data of the same column tends to be homogeneous which helps with compression when apply techniques like dictionary and Run-Length Encoding (RLE). +Second, values within a column tend to be similar, which improves compression with techniques such as dictionary encoding and run-length encoding (RLE). -Parquet file format +The following diagram from the Apache Parquet specification also shows the physical file layout: column chunks are stored by row group, while file metadata and its length are written in the footer. + +Apache Parquet file layout + +*Source: Apache Parquet [FileLayout.gif](https://github.com/apache/parquet-format/blob/master/doc/images/FileLayout.gif). Copyright 2014 The Apache Software Foundation, licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).* ## Data Persistence @@ -27,17 +31,17 @@ When the size of data buffered in MemTables reaches that threshold, GreptimeDB w ## Indexing Data in SST Files -Apache Parquet file format provides inherent statistics in headers of column chunks and data pages, which are used for pruning and skipping. +Parquet stores row-group column statistics such as minimum, maximum, and null count in each column chunk's metadata. Page metadata and optional column indexes can provide finer-grained statistics. -Column chunk header +![A name predicate uses Parquet column statistics to skip one row group while retaining another as a read candidate.](/parquet-row-group-statistics.svg) -For example, in the above Parquet file, if you want to filter rows where `name` = `Emily`, you can easily skip row group 0 because the max value for `name` field is `Charlie`. This statistical information reduces IO operations. +For example, a query filtering for `name` = `Emily` can skip row group 0 because the maximum `name` value is `Charlie`. This avoids reading that row group. ## Index Files -For each SST file, GreptimeDB not only maintains an internal index but also generates a separate file to store the index structures specific to that SST file. +When an SST has one or more configured index outputs, GreptimeDB writes them to a Puffin file associated with that SST. An SST with no applicable index does not need a Puffin file. -The index files utilize the [Puffin][3] format, which offers significant flexibility, allowing for the storage of additional metadata and supporting a broader range of index structures. +Puffin provides a container for index blobs and their metadata, allowing different index structures to share one file. ![Puffin](/puffin.png) @@ -57,13 +61,13 @@ The inverted index enables GreptimeDB to skip data segments that do not meet que ![Inverted index searching](/inverted-index-searching.png) -For instance, the query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches the regex `.*users`, and `status` matches the regex `4...`. It then scans these data segments to produce the final results that meet all conditions, significantly reducing the number of IO operations. +The query above uses the inverted index to identify data segments where `job` equals `apiserver`, `handler` matches `.*users`, and `status` matches `4..`. It scans only those segments before applying the remaining filters. ### Inverted Index Format -![Inverted index format](/inverted-index-format.png) +![An inverted-index blob contains one index per column followed by footer metadata; each column index contains a null bitmap, posting bitmaps, and an FST.](/inverted-index-blob-layout.svg) -GreptimeDB builds inverted indexes by column, with each inverted index consisting of an FST and multiple Bitmaps. +GreptimeDB builds inverted indexes by column. Each column index contains a null bitmap, multiple posting bitmaps, and an FST. The blob footer records the offsets, sizes, and metadata needed to locate and decode the column indexes. The FST (Finite State Transducer) enables GreptimeDB to store mappings from column values to Bitmap positions in a compact format and provides excellent search performance and supports complex search capabilities (such as regular expression matching). The Bitmaps maintain a list of data segment IDs, with each bit representing a data segment. @@ -77,7 +81,7 @@ The number of rows in a data segment is controlled by the engine option `index.i ## Unified Data Access Layer: OpenDAL -GreptimeDB uses [OpenDAL][2] to provide a unified data access layer, thus, the storage engine does not need to interact with different storage APIs, and data can be migrated to cloud-based storage like AWS S3 seamlessly. +GreptimeDB uses [OpenDAL][2] to provide a common access layer for local filesystems and object stores. Changing the configured storage backend does not migrate existing data. [1]: https://parquet.apache.org [2]: https://github.com/datafuselabs/opendal diff --git a/versioned_docs/version-1.2/contributor-guide/datanode/memtable.md b/versioned_docs/version-1.2/contributor-guide/datanode/memtable.md new file mode 100644 index 0000000000..92e3eb58de --- /dev/null +++ b/versioned_docs/version-1.2/contributor-guide/datanode/memtable.md @@ -0,0 +1,100 @@ +--- +keywords: [memtable, Mito engine, write buffer, flush, time partition, BulkMemtable] +description: How Mito organizes mutable Region data in memtables and moves it into SST files. +--- + +# Memtable design + +A memtable is Mito's in-memory write buffer for a Region. It makes writes available to reads before a flush creates SST files. A Region version identifies the memtables and SST files that a scan may read. Together with a committed-sequence fence, it keeps the scan consistent while writes and flushes advance the current version. + +## Write and flush lifecycle + +For a normal WAL-backed write, Mito uses this order: + +```text +write request + | + v +WAL append -> mutable memtable -> publish committed sequence + | + freeze + v + immutable memtable -> SST write -> manifest edit +``` + +The Region worker assigns sequence numbers and a WAL entry ID before appending the mutation to the [write-ahead log](wal.md). If the WAL append fails, Mito does not update the memtable. After the memtable update succeeds, Mito publishes the committed sequence and the rows become visible to new reads. A Region configured with `skip_wal` omits the WAL append, but keeps the same memtable and visibility ordering. + +A flush freezes the mutable memtables and installs a new mutable set before starting the background SST write. New writes therefore continue without changing the frozen data. The flush writes the immutable memtables to SST files, then persists a manifest edit containing the files and the flushed WAL and sequence checkpoints. Only after that edit is durable does Mito remove the flushed memtables from the current Region version. If the flush fails, the immutable memtables remain available for a later attempt. + +## Region versions and time partitions + +Each Region has one mutable `TimePartitions` container, which can hold more than one memtable: + +```text +Region version +├─ mutable TimePartitions +│ ├─ [t0, t1) -> memtable +│ └─ [t1, t2) -> memtable +├─ immutable memtables +└─ SST files +``` + +Mito routes each row to a partition by its time-index value. Partition ranges are half-open and aligned to a fixed duration. The duration follows the Region's compaction time window; Mito uses one day until a compaction window is available. An out-of-order write can create an earlier partition alongside the latest one. + +Freezing a Region freezes all mutable time partitions together. Mito moves their memtables to the immutable list and creates a new `TimePartitions` container. A failed flush can leave more than one generation of immutable memtables, so reads and later flushes must not assume that the list contains a single item. + +## Memtable implementations + +Mito selects a memtable implementation from the Region's SST format, primary-key encoding, and memtable options: + +```text +flat SST format (the default) or sparse primary-key encoding -> BulkMemtable +memtable.type=bulk -> BulkMemtable, and forces flat SST +primary_key SST with dense encoding (legacy) -> a legacy implementation +``` + +With the default engine configuration, a Region without an explicit SST format uses `flat`, so `BulkMemtable` is the normal path and the rest of this page describes it. The rules exist to prevent incompatible combinations: flat format or sparse primary-key encoding requires `BulkMemtable`, and explicitly selecting the bulk implementation forces flat format. + +### BulkMemtable + +`BulkMemtable` stores writes as parts in the flat Arrow layout instead of inserting rows into per-series buffers: + +```text +BulkMemtable +├─ unordered_part +│ └─ small BulkPart batches +└─ parts + ├─ BulkPart (Arrow RecordBatch) + ├─ MultiBulkPart (raw RecordBatches) + └─ EncodedBulkPart (in-memory Parquet) +``` + +Small parts accumulate in `unordered_part`; larger parts enter `parts` directly. Background memtable compaction merge-sorts eligible parts into a `MultiBulkPart` or encodes them as an `EncodedBulkPart`. Scans use part statistics to prune ranges, and flush can write encoded ranges to SST without decoding and encoding the rows again. For the design rationale and performance results, see [Scaling Time Series to Millions of Cardinalities: GreptimeDB's Flat Format](https://greptime.com/blogs/2025-12-22-flat-format). + +### Legacy implementations + +Regions on the legacy `primary_key` SST format with dense primary-key encoding still use `TimeSeriesMemtable`, which groups rows by encoded primary key rather than storing flat parts. A Region with no primary-key columns gets `SimpleBulkMemtable` from the same builder. Both are compatibility code for existing tables and may be removed once the `primary_key` format is retired; new work targets the bulk and flat path. + +The removed `partition_tree` memtable is not a third implementation. The option parser accepts `memtable.type=partition_tree` for compatibility, but it does not recreate that implementation. The Region uses the bulk and flat path. + +## Read snapshots + +A scan obtains the Region version and committed sequence together from `VersionControl`. It selects the version before applying the sequence fence. Reading the sequence separately before the version could pair that sequence with a later version after flush or compaction removes an older input, producing an incomplete snapshot. + +The selected version supplies mutable memtables, immutable memtables, and SST files. Mito first prunes sources by time range, then asks each memtable for ranges using the scan's projection, predicate, and sequence bounds. The scan merges the resulting ranges with SST ranges and applies the same ordering, deletion, and merge semantics across all sources. References held by the scan keep an older memtable alive even after a newer Region version removes it. + +## Memory pressure + +Each memtable tracks its estimated heap allocation through the engine's write-buffer manager. Freezing a memtable removes its allocation from the mutable-memory count, but total usage includes the allocation until all references to that memtable are released. The mutable-memory count therefore tracks data that can still accept writes, while total usage continues to include memory retained by active scans. + +The global write-buffer limit causes workers to select Regions for flush. If memory remains above the configured limits, Mito stalls writes and can reject them at a higher threshold. An optional per-Region limit applies the same pressure to one hot Region without stalling unrelated Regions. Periodic, manual, and Region lifecycle operations can also request a flush. + +## Constraints for changes + +Changes to memtable code must preserve these properties: + +- For a WAL-backed Region, append to the WAL before installing rows in a memtable. Publish the committed sequence only after installation succeeds. +- Keep frozen memtables readable and retryable until the SST files and manifest edit are durable. +- Obtain the Region version and committed sequence from the same `VersionControl` snapshot; never read the sequence separately before the version. +- Preserve the ordering and metadata that scans and flushes need to apply the same deletion, deduplication, and merge rules across memtable and SST ranges. +- Charge allocations to the write-buffer manager and release them only when the underlying memory can no longer be referenced. diff --git a/versioned_docs/version-1.2/contributor-guide/datanode/metric-engine.md b/versioned_docs/version-1.2/contributor-guide/datanode/metric-engine.md index 064872ce14..fde3c178a3 100644 --- a/versioned_docs/version-1.2/contributor-guide/datanode/metric-engine.md +++ b/versioned_docs/version-1.2/contributor-guide/datanode/metric-engine.md @@ -7,9 +7,9 @@ description: Overview of the Metric engine in GreptimeDB, its concepts, architec ## Overview -The `Metric` engine is a component of GreptimeDB, and it's an implementation of the storage engine. It mainly targets scenarios with a large number of small tables for observable metrics. +The `Metric` engine stores workloads with many small metric tables. -Its main feature is to use synthetic physical wide tables to store a large amount of small table data, achieving effects such as reuse of the same column and metadata. This reduces storage overhead for small tables and improves columnar compression efficiency. The concept of a table becomes even more lightweight under the `Metric` engine. +It maps those logical tables onto shared physical wide tables so they can reuse columns and metadata. This reduces per-table storage overhead and improves columnar compression. ## Concepts @@ -19,7 +19,7 @@ The `Metric` engine introduces two new concepts: "logical table" and "physical t A logical table refers to user-defined tables. Just like any other ordinary table, its definition includes the name of the table, column definitions, index definitions etc. All operations such as queries or write-ins by users are based on these logical tables. Users don't need to worry about differences between logical and ordinary tables during usage. -From an implementation standpoint, a logical table is virtual; it doesn't directly read or write physical data but maps read/write requests into corresponding requests for physical tables in order to implement data storage and querying. +A logical table is virtual. The engine maps its read and write requests to the corresponding physical table instead of storing data for it directly. ### Physical Table @@ -29,16 +29,14 @@ A physical table is a table that actually stores data, possessing several physic The main design architecture of the `Metric` engine is as follows: -![Arch](/metric-engine-arch.png) +![Multiple logical tables map through the Metric engine to shared data and metadata Regions managed by Mito.](/metric-engine-architecture.svg) -In the current version implementation, the `Metric` engine reuses the `Mito` engine to achieve storage and query capabilities for physical data. It also provides access to both physical tables and logical tables simultaneously. +The `Metric` engine delegates physical storage and queries to the `Mito` engine. Each physical Region group contains a data Region, which stores rows from its mapped logical tables, and a metadata Region, which stores the logical-table and logical-column mappings. -Regarding partitioning, logical tables have identical partition rules and Region distribution as physical tables. This makes sense because the data of logical tables are directly stored in physical tables, so their partition rules are consistent. +Logical tables associated with the same physical table share its partition layout. During writes, the engine records the logical table identity with each row. During reads, it adds a logical-table filter before scanning the physical Region. -Concerning routing metadata, the routing address of a logical table is a logical address - what its corresponding physical table is - then through this physical table for secondary routing to obtain the real physical address. This indirect routing method can significantly reduce the number of metadata modifications required when Region migration scheduling occurs in Metric engines. +A logical table's route stores only the ID of its physical table; the physical table route resolves that to the Datanodes holding the Regions. Because logical routes do not name peers, migrating a physical Region rewrites one physical route instead of every logical route that maps to it. -Operationally speaking, The `Metric` engine supports standard DML operations (INSERT, DELETE, SELECT) on logical tables. However, it only supports limited operations on physical tables to prevent misoperations - for example, writing directly to a physical table is prohibited as it could affect user's logical table data. Generally speaking, users can consider that they have read-only access to these physical tables. +Logical tables support normal INSERT, DELETE, and SELECT operations. Direct writes to a physical Region are rejected because they would bypass the logical-table mapping; querying a physical table remains supported. -To improve performance during simultaneous DDL (Data Definition Language) operations on many tables, the 'Metric' engine has introduced some batch DDL operations. These batch DDL operations can merge lots of DDL actions into one request thereby reducing queries and modifications times for metadata thus enhancing performance. This feature is particularly beneficial in scenarios such as the automatic creation requests brought about by large amounts of metrics during Prometheus Remote Write cold start-up, as well as the modification requests for numerous route-tables mentioned earlier during migration of many physical regions. - -Apart from physical data regions belonging to physical tables, the 'Metric' engine creates an additional metadata region physically for each individual physical data region used in storing some metadata needed by itself while maintaining mapping and other states. This metadata includes the mapping relationship between logical tables and physical tables, the mapping relationship between logical columns and physical columns etc. +Batch DDL operations reduce metadata work when many logical tables are created or updated together, such as during Prometheus Remote Write auto-creation or physical Region migration. diff --git a/versioned_docs/version-1.2/contributor-guide/datanode/overview.md b/versioned_docs/version-1.2/contributor-guide/datanode/overview.md index d0afe21b34..a21c112faa 100644 --- a/versioned_docs/version-1.2/contributor-guide/datanode/overview.md +++ b/versioned_docs/version-1.2/contributor-guide/datanode/overview.md @@ -7,28 +7,26 @@ description: Overview of Datanode in GreptimeDB, its responsibilities, component ## Introduction -`Datanode` is mainly responsible for storing the actual data for GreptimeDB. As we know, in GreptimeDB, -a `table` can have one or more `Region`s, and `Datanode` is responsible for managing the reading and writing -of these `Region`s. `Datanode` is not aware of `table` and can be considered as a `region server`. Therefore, -`Frontend` and `Metasrv` operate `Datanode` at the granularity of `Region`. +A Datanode stores and processes Region data. A table can contain multiple Regions, but the Datanode does not own table-level routing. Frontend sends data requests by Region, while Metasrv controls Region placement and lifecycle. -![Datanode](/datanode.png) +This boundary lets the same Region server host different storage engines without exposing their implementation to Frontend or Metasrv. + +![Frontend sends Region requests to the Datanode Region server, while Metasrv exchanges lifecycle instructions through the heartbeat task. The Region server uses the local query engine and dispatches requests to the Mito, Metric, or File Region engine.](/datanode-architecture.svg) ## Components -A `Datanode` contains all the components needed for a `region server`. Here we list some of the vital parts: - -- A gRPC service is provided for reading and writing region data, and `Frontend` uses this service - to read and write data from `Datanode`s. -- An HTTP service, through which you can obtain metrics, configuration information, etc., of the current node. -- `Heartbeat Task` is used to send heartbeat to the `Metasrv`. The heartbeat plays a crucial role in the - distributed architecture of GreptimeDB and serves as a basic communication channel for distributed coordination. - The upstream heartbeat messages contain important information such as the workload of a `Region`. If the - `Metasrv `has made scheduling(such as `Region` migration) decisions, it will send instructions to the - `Datanode` via downstream heartbeat messages. -- The `Datanode` does not parse user SQL or perform distributed planning. The user's query requests for one or - more `Table`s will be transformed into `Region` query requests in the `Frontend`. The `Datanode` is responsible - for executing these `Region` query plans with its local query engine. -- A `Region Manager` is used to manage all `Region`s on a `Datanode`. -- GreptimeDB supports a pluggable multi-engine architecture, with existing engines including `File Engine` and - `Mito Engine`. +The main components are: + +- The Region server tracks open Regions and dispatches reads, writes, and lifecycle requests to the engine registered for each Region. +- `Mito` is the primary time-series Region engine. `Metric` maps many logical metric Regions onto shared Mito Regions, and `File` exposes external files through the Region interface. +- The local query engine executes Region query plans. It does not parse client SQL or perform cluster-wide planning. +- The heartbeat task reports node and Region state to Metasrv and receives instructions such as open, close, upgrade, downgrade, and migration steps. +- gRPC carries Region requests to the Datanode. HTTP exposes node diagnostics such as metrics and configuration. + +## Region Request Lifecycle + +For a Mito write, the Region server selects Mito from the Region metadata. Mito appends the mutation to the WAL, applies it to a memtable, and later flushes the memtable to SST files. A Metric write is first rewritten with the logical-table identity and then delegated to its physical Mito Region. + +For a read, the local query engine executes the Region plan against a table provider backed by the Region engine. A Mito scan takes an immutable Region version, reads the relevant memtables and SST files, merges and deduplicates rows, and returns a stream of Arrow record batches. + +Region ownership can change without restarting the Datanode. Metasrv sends lifecycle instructions over the heartbeat stream; the Region server applies them to the engine and reports the new Region role and statistics in subsequent heartbeats. diff --git a/versioned_docs/version-1.2/contributor-guide/datanode/python-scripts.md b/versioned_docs/version-1.2/contributor-guide/datanode/python-scripts.md deleted file mode 100644 index 98909142a6..0000000000 --- a/versioned_docs/version-1.2/contributor-guide/datanode/python-scripts.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -keywords: [Python scripts, data analysis, CPython backend, RustPython interpreter, RecordBatch] -description: Guide on using Python scripts for data analysis in GreptimeDB, including backend options and setup instructions. ---- - -# Python Scripts - -## Introduction - -Python scripts are methods for analyzing data in GreptimeDB, -by running it in the database directly instead of fetching all the data from the database and running it locally. -This approach saves a lot of data transfer costs. -The image below depicts how the script works. -The `RecordBatch` (which is basically a column in a table with type and nullability metadata) -can come from anywhere in the database, -and the returned `RecordBatch` can be annotated in Python grammar to indicate its metadata, -such as type or nullability. -The script will do its best to convert the returned object to a `RecordBatch`, -whether it is a Python list, a `RecordBatch` computed from parameters, -or a constant (which is extended to the same length as the input arguments). - -![Python Coprocessor](/python-coprocessor.png) - -## Two optional backends - -### CPython Backend powered by PyO3 - -This backend is powered by [PyO3](https://pyo3.rs/v0.18.1/), enabling the use of your favourite Python libraries (such as NumPy, Pandas, etc.) and allowing Conda to manage your Python environment. - -But using it also involves some complications. You must set up the correct Python shared library, which can be a bit challenging. In general, you just need to install the `python-dev` package. However, if you are using Homebrew to install Python on macOS, you must create a proper soft link to `Library/Frameworks/Python.framework`. Detailed instructions on using PyO3 crate with different Python Version can be found [here](https://pyo3.rs/v0.18.1/building_and_distribution#configuring-the-python-version) - -### Embedded RustPython Interpreter - -An experiment [python interpreter](https://github.com/RustPython/RustPython) to run -the coprocessor script, it supports Python 3.10 grammar. You can use all the very Python syntax, see [User Guide/Python Coprocessor](/user-guide/python-scripts/overview.md) for more! diff --git a/versioned_docs/version-1.2/contributor-guide/datanode/query-engine.md b/versioned_docs/version-1.2/contributor-guide/datanode/query-engine.md index 5b83a30613..91f307aa59 100644 --- a/versioned_docs/version-1.2/contributor-guide/datanode/query-engine.md +++ b/versioned_docs/version-1.2/contributor-guide/datanode/query-engine.md @@ -7,51 +7,30 @@ description: Overview of GreptimeDB's query engine, its architecture, data repre ## Introduction -GreptimeDB's query engine is built on [Apache DataFusion][1] (subproject under [Apache -Arrow][2]), a brilliant query engine written in Rust. It provides a set of well functional components from -logical plan, physical plan and the execution runtime. Below explains how each component is orchestrated and their positions during execution. +GreptimeDB's query engine is built on [Apache DataFusion][1]. DataFusion supplies the logical and physical plan interfaces, optimizer framework, and execution runtime. GreptimeDB adds planners for its query languages, storage-aware optimizer rules, custom plan nodes, and distributed execution. -![Execution Procedure](/execution-procedure.png) +DDL and other control-plane operations are dispatched by the statement executor. The query engine receives plans for data processing, including the input side of operations such as `INSERT ... SELECT`. -The entry point is the logical plan, which is used as the general intermediate representation of a -query or execution logic etc. Two noticeable sources of logical plan are from: 1. the user query, like -SQL through SQL parser and planner; 2. the Frontend's distributed query, which is explained in details in the following section. +## Query Lifecycle -Next is the physical plan, or the execution plan. Unlike the logical plan which is a big -enumeration containing all the logical plan variants (except the special extension plan node), the -physical plan is in fact a trait that defines a group of methods invoked during -execution. All data processing logics are packed in corresponding structures that -implement the trait. They are the actual operations performed on the data, like -aggregator `MIN` or `AVG`, and table scan `SELECT ... FROM`. +1. The SQL, PromQL, or log-query planner resolves tables through the catalog and produces a DataFusion logical plan. GreptimeDB plan extensions represent operations that DataFusion does not provide directly. +2. DataFusion analyzer and optimizer rules run together with GreptimeDB rules. These rules normalize expressions and types, rewrite time-range operations, push projections and filters toward scans, and introduce distributed plan nodes when required. +3. The physical planner converts the optimized logical plan into streaming operators. GreptimeDB then applies physical rules for scan parallelism, ordering, and distributed execution. +4. Execution pulls Arrow record batches through the physical plan. Storage scans receive the projection and predicates, and downstream operators consume the resulting stream without materializing the complete result first. -The optimization phase which improves execution performance by transforming both logical and physical plans, is now all based on rules. It is also called, "Rule Based Optimization". Some of the rules are DataFusion native and others are customized in Greptime DB. In the future, we plan to add more -rules and leverage the data statistics for Cost Based Optimization/CBO. - -The last phase "execute" is a verb, stands for the procedure that reads data from storage, performs -calculations and generates the expected results. Although it's more abstract than previously mentioned concepts, you can just -simply imagine it as executing a Rust async function. And it's indeed a future (stream). - -`EXPLAIN [VERBOSE] ` is very useful if you want to see how your SQL is represented in the logical or physical plan. +Use [`EXPLAIN`](/reference/sql/explain.md) to inspect the logical and physical plans. `EXPLAIN ANALYZE` also executes the plan and reports runtime metrics. ## Data Representation -GreptimeDB uses [Apache Arrow][2] as the in-memory data representation. It's column-oriented, in -cross-platform format, and also contains many high-performance data operators. These features -make it easy to share data in many different environments and implement calculation logic. +GreptimeDB uses [Apache Arrow][2] record batches as its in-memory data representation. A record batch contains equal-length column arrays and a schema. Query operators exchange streams of these batches, which keeps the execution path columnar from Region scans through result encoding. ## Indexing -In time series data, there are two important dimensions: timestamp and tag columns (or like -primary key in a general relational database). GreptimeDB groups data in time buckets, so it's efficient -to locate and extract data within the expected time range at a very low cost. The mainly used persistent file format [Apache Parquet][3] in GreptimeDB helps a lot -- it -provides multi-level indices and filters that make it easy to prune data during querying. In the future, we -will make more use of this feature, and develop our separated index to handle more complex use cases. +Index construction and persistent index formats belong to the storage engine. The query layer supplies predicates and projections to a scan; Mito then uses time ranges, Parquet statistics, and indexes to avoid reading data that cannot match. See [Data Persistence and Indexing](./data-persistence-indexing.md). ## Distributed Execution -Covered in [Distributed Querying][6]. +In distributed mode, the Frontend plans the cluster-wide query and Datanodes execute Region-local subplans. [`MergeScan`](../frontend/distributed-querying.md) is the boundary between those stages. -[1]: https://github.com/apache/arrow-datafusion +[1]: https://datafusion.apache.org/ [2]: https://arrow.apache.org/ -[3]: https://parquet.apache.org -[6]: ../frontend/distributed-querying.md diff --git a/versioned_docs/version-1.2/contributor-guide/datanode/storage-engine.md b/versioned_docs/version-1.2/contributor-guide/datanode/storage-engine.md index c220c72208..36fef62833 100644 --- a/versioned_docs/version-1.2/contributor-guide/datanode/storage-engine.md +++ b/versioned_docs/version-1.2/contributor-guide/datanode/storage-engine.md @@ -7,7 +7,7 @@ description: Overview of the storage engine in GreptimeDB, its architecture, com ## Introduction -The `storage engine` is responsible for storing the data of the database. Mito, based on [LSMT][1] (Log-structured Merge-tree), is the storage engine we use by default. We have made significant optimizations for handling time-series data scenarios, so mito engine is not suitable for general purposes. +Mito is GreptimeDB's default storage engine. It uses an [LSM tree][1] and is designed for time-series workloads rather than as a general-purpose embedded storage engine. ## Architecture @@ -23,9 +23,9 @@ The architecture is the same as a traditional LSMT engine: media. - Log records of the WAL can be stored on the local disk, or in a remote log service such as Kafka (remote WAL) that implements the `Log Store` API. -- Memtables: - - Data is written into the `active memtable`, aka `mutable memtable` first. - - When a `mutable memtable` is full, it will be changed to a `read-only memtable`, aka `immutable memtable`. +- [Memtables](memtable.md): + - Mito routes rows by time index into mutable memtables. + - A flush freezes the mutable memtables, installs a new mutable set for writes, and writes the frozen memtables to SST files. - SST - The full name of SST, aka SSTable is `Sorted String Table`. - `Immutable memtable` is flushed to persistent storage and produces an SST file. @@ -103,7 +103,9 @@ Each Parquet SST is split into row groups, the unit that Parquet can read or ski Mito supports two SST formats: `flat` and `primary_key`. `flat` is the default for new tables and works well across primary-key cardinalities, including high-cardinality keys. `primary_key` is the legacy format kept for compatibility with older tables. See [SST format](/reference/sql/create.md#create-a-table-with-sst-format) and the [table design guide](/user-guide/deployments-administration/performance-tuning/design-table.md#sst-format) for more details. -SST layout +![The default flat Mito SST layout combines file-level metadata with Parquet row groups containing data columns and merge metadata.](/mito-sst-layout.svg) + +An SST may span more than one compaction time window. ## Scan Pruning diff --git a/versioned_docs/version-1.2/contributor-guide/datanode/wal.md b/versioned_docs/version-1.2/contributor-guide/datanode/wal.md index 4ecb19ef02..8f898d91d6 100644 --- a/versioned_docs/version-1.2/contributor-guide/datanode/wal.md +++ b/versioned_docs/version-1.2/contributor-guide/datanode/wal.md @@ -7,30 +7,26 @@ description: Introduction to Write-Ahead Logging (WAL) in GreptimeDB, its purpos ## Introduction -Our storage engine is inspired by the Log-structured Merge Tree (LSMT). Mutating operations are -applied to a MemTable instead of persisting to disk, which significantly improves performance but -also brings durability-related issues, especially when the Datanode crashes unexpectedly. Similar -to all LSMT-like storage engines, GreptimeDB uses a write-ahead log (WAL) to ensure data durability -and is safe from crashing. +Mito buffers writes in [memtables](memtable.md) before flushing them to SST files. It first appends each Region's mutations to the write-ahead log (WAL), so data that has not reached an SST can be recovered. -WAL is an append-only file group. All `INSERT` and `DELETE` operations are transformed into -operation entries and then appended to WAL. Once operation entries are persisted to the underlying -file, the operation can be further applied to MemTable. +The WAL uses a common log-store abstraction with local raft-engine and remote Kafka providers. -When the Datanode restarts, operation entries in WAL are replayed to reconstruct the correct -in-memory state. +## Write and Recovery Cycle -![WAL in Datanode](/wal.png) +The order of a normal write is: + +1. The Region worker assigns sequence numbers and a WAL entry ID. +2. It appends the mutations to the WAL. If the append fails, the mutations are not applied to the memtable. +3. After the append succeeds, Mito writes the mutations to the memtable and publishes the new committed sequence. +4. A flush writes immutable SST files and persists a manifest edit containing the new files and `flushed_entry_id`. +5. After the manifest edit is durable, WAL entries through `flushed_entry_id` are marked obsolete. The log store may reclaim them later. + +The manifest is the recovery boundary. On a normal reopen, Mito rebuilds the Region from the manifest and replays WAL entries starting at `flushed_entry_id + 1`. Region transitions may supply a later replay checkpoint, but they never replay entries before the persisted flush boundary. ## Namespace -Namespace of WAL is used to separate entries from different tables (different regions). Append and -read operations must provide a Namespace. Currently, region ID is used as the Namespace, because -each region has a MemTable that needs to be reconstructed when Datanode restarts. +WAL entries are isolated by Region, not by table. Each append and read identifies a Region namespace so one Region can be replayed or truncated independently. The local raft-engine provider uses the Region ID as its namespace ID. Kafka keeps Region identity within the provider's topic-backed log. ## Synchronous/Asynchronous flush -By default, appending to WAL is asynchronous, which means the writer will not wait until entries are -flushed to disk. This setting provides higher performance, but may lose data when running host shutdown unexpectedly. In the other hand, synchronous flush provides higher durability at the cost of performance. - -In v0.4 version, the new region worker architecture can use batching to alleviate the overhead of sync flush. +For the local raft-engine provider, `sync_write` controls whether an append waits for the log to be synced to durable storage. It defaults to `false`. Asynchronous writes reduce latency but can lose recently acknowledged entries if the host fails before buffered data is synced. Kafka WAL durability is controlled by its producer and cluster settings instead of this local option. diff --git a/versioned_docs/version-1.2/contributor-guide/flownode/arrangement.md b/versioned_docs/version-1.2/contributor-guide/flownode/arrangement.md index aed75af777..8b472ea316 100644 --- a/versioned_docs/version-1.2/contributor-guide/flownode/arrangement.md +++ b/versioned_docs/version-1.2/contributor-guide/flownode/arrangement.md @@ -5,6 +5,8 @@ description: Details on the arrangement component in Flownode, which stores stat # Arrangement +This page describes state used by Flownode's legacy streaming mode. Batching mode does not use an Arrangement. + Arrangement stores the state in the dataflow's process. It stores the streams of update flows for further querying and updating. The arrangement essentially stores key-value pairs with timestamps to mark their change time. diff --git a/versioned_docs/version-1.2/contributor-guide/flownode/batching_mode.md b/versioned_docs/version-1.2/contributor-guide/flownode/batching_mode.md index 37a695ce9c..aa8099d1ad 100644 --- a/versioned_docs/version-1.2/contributor-guide/flownode/batching_mode.md +++ b/versioned_docs/version-1.2/contributor-guide/flownode/batching_mode.md @@ -9,13 +9,13 @@ This guide provides a brief overview of the batching mode in `flownode`. It's in ## Overview -The batching mode in `flownode` is designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. This is in contrast to the original streaming mode, now deprecated, where data was processed as it arrived. +The batching mode in `flownode` is designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. This is in contrast to the legacy streaming path, which processes data as it arrives and is retained for compatibility but deprecated for new workloads. The core idea is to: 1. Define a `flow` with a SQL query that aggregates data from a source table into a sink table. 2. The query typically includes a time window function (e.g., `date_bin`) on a timestamp column. 3. When new data is inserted into the source table, the system marks the corresponding time windows as "dirty." -4. A background task periodically wakes up, identifies these dirty windows, and re-runs the aggregation query for those specific time ranges. +4. A background task runs on its own cadence, consumes the pending dirty windows at its next evaluation, and re-runs the aggregation query for those time ranges. 5. The results are then inserted into the sink table, effectively updating the aggregated view. ## Architecture @@ -39,15 +39,15 @@ A `BatchingTask` represents a single, independent data flow. Each task is associ - **State (`TaskState`)**: This contains the dynamic, mutable state of the task, most importantly the `DirtyTimeWindows`. - **Execution Loop**: The task runs an infinite loop (`start_executing_loop`) that: 1. Checks for a shutdown signal. - 2. Waits for a scheduled interval or until it's woken up. + 2. Sleeps until its next evaluation time. A task with an evaluation schedule sleeps until the next scheduled time; an adaptive task sleeps for a polling interval derived from the time window size and the minimum refresh duration. 3. Generates a new query plan (`gen_insert_plan`) based on the current set of dirty time windows. 4. Executes the query (`execute_logical_plan`) against the database. 5. Cleans up the processed dirty windows. ### `TaskState` and `DirtyTimeWindows` -- **`TaskState`**: This struct tracks the runtime state of a `BatchingTask`. It includes `dirty_time_windows`, which is crucial for determining what work needs to be done. -- **`DirtyTimeWindows`**: This is a key data structure that keeps track of which time windows have received new data since the last query execution. It stores a set of non-overlapping time ranges. When a task's execution loop runs, it consults this structure to build a `WHERE` clause that filters the source table for only the dirty time windows. +- **`TaskState`**: This struct tracks the runtime state of a `BatchingTask`, including the `dirty_time_windows` that determine its pending work. +- **`DirtyTimeWindows`**: This data structure tracks which time windows have received new data since the last query execution. It stores a set of non-overlapping time ranges. The execution loop uses it to build a `WHERE` clause that selects only the dirty windows from the source table. ### `TimeWindowExpr` @@ -56,15 +56,15 @@ The `TimeWindowExpr` is a helper utility for dealing with time window expression - **Evaluation**: It can take a timestamp and evaluate the time window expression to determine the start and end of the window that the timestamp falls into. - **Window Size**: It can also determine the size (duration) of the time window from the expression. -This is essential for both marking windows as dirty and for generating the correct filter conditions when querying the source table. +The same calculation is used to mark dirty windows and generate the source-table filters. ## Query Execution Flow Here's a simplified step-by-step walkthrough of how a query is executed in batch mode: 1. **Data Ingestion**: New data is written to a source table. -2. **Marking Dirty**: The `BatchingEngine` receives a notification about the new data. It uses the `TimeWindowExpr` associated with each relevant flow to determine which time windows are affected by the new data points. These windows are then added to the `DirtyTimeWindows` set in the corresponding `TaskState`. -3. **Task Wake-up**: The `BatchingTask`'s execution loop wakes up, either due to its periodic schedule or because it was notified of a large backlog of dirty windows. +2. **Marking Dirty**: The `BatchingEngine` receives a notification about the new data. It uses the `TimeWindowExpr` associated with each relevant flow to determine which time windows are affected by the new data points. These windows are then added to the `DirtyTimeWindows` set in the corresponding `TaskState`. Marking a window dirty does not wake the task. +3. **Next Evaluation**: The `BatchingTask`'s execution loop reaches its next evaluation, either at a scheduled time or after its adaptive polling interval, and consumes the pending dirty windows. 4. **Plan Generation**: The task calls `gen_insert_plan`. This method: - Inspects the `DirtyTimeWindows`. - Generates a series of `OR`'d `WHERE` clauses (e.g., `(ts >= 't1' AND ts < 't2') OR (ts >= 't3' AND ts < 't4') ...`) that cover the dirty windows. diff --git a/versioned_docs/version-1.2/contributor-guide/flownode/dataflow.md b/versioned_docs/version-1.2/contributor-guide/flownode/dataflow.md index 000a65edb3..c876054d6d 100644 --- a/versioned_docs/version-1.2/contributor-guide/flownode/dataflow.md +++ b/versioned_docs/version-1.2/contributor-guide/flownode/dataflow.md @@ -1,17 +1,38 @@ --- -keywords: [dataflow module, SQL query transformation, execution plan, DAG, map and reduce operations] -description: Explanation of the dataflow module in Flownode, its operations, internal data handling, and future enhancements. +keywords: [Flownode, batching mode, streaming mode, dataflow, dirty time windows] +description: How Flownode selects and runs its batching and legacy streaming execution paths. --- # Dataflow +Flownode has two internal execution paths: + +- **Batching mode** is the primary path for aggregation and TQL workloads. It evaluates queries over persisted source data and writes materialized results to a sink table. +- **Streaming mode** is the legacy path retained for compatibility and deprecated for new workloads. It incrementally processes rows mirrored from Frontend as they arrive. + +Users do not select the mode directly. When a Flow is created, GreptimeDB chooses the path from the query and source-table properties. Aggregation, `DISTINCT`, and TQL queries use batching mode. Simple non-aggregation queries, and any Flow whose source table has `ttl = 'instant'`, currently use streaming mode. A Flow deferred because its source table does not yet exist starts as a pending batching Flow. + +## Batching mode + +Batching mode reuses GreptimeDB's query engine instead of maintaining an operator graph for every incoming row. For a time-windowed Flow, its main loop is: + +1. A source-table write marks the affected time windows as dirty. +2. A `BatchingTask` runs on its evaluation schedule or adaptive polling cadence and collects the pending dirty windows at that evaluation. Marking a window dirty does not wake the task. +3. The task adds time predicates for those windows to the Flow query and asks Frontend to execute it against the source tables. +4. The query result is inserted into the sink table, updating the materialized result for windows that were evaluated. +5. Successfully processed windows are removed from the dirty set. Failed work remains available for a later evaluation. + +Flows with an evaluation interval but without a time-window expression run the complete query on each scheduled evaluation. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Flownode Batching Mode Developer Guide](./batching_mode.md) for the task and dirty-window components. + +## Streaming mode + The `dataflow` module (see `flow::compute` module) is the core computing module of `flow`. It takes a SQL query and transforms it into flow's internal execution plan. This execution plan is then rendered into an actual dataflow, which is essentially a directed acyclic graph (DAG) of functions with input and output ports. -The dataflow is triggered to run when needed. +New row changes drive the graph incrementally. -Currently, this dataflow only supports `map` and `reduce` operations. Support for `join` operations will be added in the future. +The renderer supports map/filter/project and reduce operations. Join and union plan nodes exist, but their streaming renderers are not implemented. Internally, the dataflow handles data in row format, using a tuple `(row, time, diff)`. Here, `row` represents the actual data being passed, which may contain multiple `Value` objects. `time` is the system time which tracks the progress of the dataflow, and `diff` typically represents the insertion or deletion of the row (+1 or -1). -Therefore, the tuple represents the insert/delete operation of the `row` at a given system `time`. \ No newline at end of file +Therefore, the tuple represents the insert/delete operation of the `row` at a given system `time`. Stateful operators keep indexed traces of these changes in an [Arrangement](./arrangement.md). diff --git a/versioned_docs/version-1.2/contributor-guide/frontend/distributed-querying.md b/versioned_docs/version-1.2/contributor-guide/frontend/distributed-querying.md index 21ee07d7e8..ca3822e113 100644 --- a/versioned_docs/version-1.2/contributor-guide/frontend/distributed-querying.md +++ b/versioned_docs/version-1.2/contributor-guide/frontend/distributed-querying.md @@ -5,29 +5,16 @@ description: Describes the process of distributed querying in GreptimeDB, focusi # Distributed Querying -Most steps of querying in frontend and datanode are identical. The only difference is that -Frontend have a "special" step in planning phase to make the logical query plan distributed. -Let's reference it as "dist planner" in the following text. - -The modified, distributed logical plan has multiple stages, each of them is executed in different -server node. +Frontend and Datanode use the same DataFusion-based query engine. In distributed mode, Frontend adds a planning step that separates work performed by Datanodes from work completed by Frontend. ![Frontend query](/frontend-query.png) ## Dist Planner -Planner will traverse the input logical plan, and split it into multiple stages by the "[commutativity -rule](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md)". +The distributed planner rewrites the logical plan. It pushes compatible operators toward table scans and wraps remote subplans in `MergeScan` nodes. Partition predicates are also used to prune Regions before the remote work is scheduled. -This rule is under heavy development. At present it will consider things like: -- whether the operator itself is commutative -- how the partition rule is configured -- etc... +Whether an operator can be pushed down depends on the plan shape and the operator's properties. Unsupported parts remain on Frontend. The original design and its commutativity rules are described in the [distributed planner RFC](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-05-09-distributed-planner.md). ## Dist Plan -Except the first stage, which have to read data from files in storage. All other stages' leaf node -are actually a gRPC call to its previous stage. - -Sub-plan in a stage is itself a complete logical plan, and can be executed independently without -the follow up stages. The plan is encoded in [substrait format](https://substrait.io). +A remote input is a complete logical subplan, not just a table scan. Frontend serializes the subplan in [Substrait](https://substrait.io) format and sends a Region-specific request to the Datanode that owns the data. The Datanode plans and executes it locally, then streams the result back. Frontend merges the remote streams and executes any operators that were not pushed down. diff --git a/versioned_docs/version-1.2/contributor-guide/frontend/overview.md b/versioned_docs/version-1.2/contributor-guide/frontend/overview.md index 3a2f63e7ae..18cabfc2b8 100644 --- a/versioned_docs/version-1.2/contributor-guide/frontend/overview.md +++ b/versioned_docs/version-1.2/contributor-guide/frontend/overview.md @@ -5,27 +5,47 @@ description: Overview of GreptimeDB's Frontend component - a stateless proxy ser # Frontend -The **Frontend** is a stateless service that serves as the entry point for client requests in GreptimeDB. It provides a unified interface for multiple database protocols and acts as a proxy that forwards read/write requests to appropriate Datanodes in the distributed system. +Frontend is GreptimeDB's stateless request-orchestration service. The server layer terminates protocols and converts wire messages; Frontend supplies the database behavior behind those handlers, including permission checks, statement execution, routing, and distributed query planning. + +Frontend does not store table data. It caches catalog and route metadata obtained from Metasrv, and Metasrv invalidates those caches through heartbeat responses when metadata changes. ## Core Functions -- **Protocol Support**: Multiple database protocols including SQL, PromQL, MySQL, and PostgreSQL. See [Protocols][1] for details -- **Request Routing**: Routes requests to appropriate Datanodes based on metadata -- **Query Distribution**: Splits distributed queries across multiple nodes -- **Response Aggregation**: Combines results from multiple Datanodes -- **Authorization**: Security and access control validation +- Provide query and ingestion behavior for the supported [protocols][1]. +- Resolve catalogs, schemas, tables, and Region routes. +- Validate permissions before executing a request. +- Plan distributed queries and merge results from Datanodes. +- Convert table-level writes and deletes into Region requests. ## Architecture ### Key Components -- **Protocol Handlers**: Handle different database protocols -- **Catalog Manager**: Caches metadata from Metasrv to enable efficient request routing and schema validation -- **Dist Planner**: Converts logical plans to distributed execution plans -- **Request Router**: Determines target Datanodes for each request + +- Protocol handlers adapt SQL, PromQL, gRPC ingestion, and observability protocols to Frontend's internal request interfaces. +- The catalog and partition managers provide table metadata, partition rules, and Region routes. +- The statement executor dispatches queries, DML, and DDL to their respective execution paths. +- The distributed planner replaces table scans with `MergeScan` plans that can run across Datanodes. ### Request Flow -![request flow](/request_flow.png) +The request path depends on the operation. + +#### Queries + +1. A protocol handler creates the query context and performs authentication and permission checks. +2. The language-specific planner produces a logical plan. In distributed mode, the planner uses partition metadata to select Regions and constructs a distributed plan. +3. Frontend sends Region subplans to the owning Datanodes. Datanodes execute them against local Region engines and return streams of Arrow record batches. +4. Frontend runs the remaining operators, merges the streams, and formats the result for the client protocol. + +#### Writes and deletes + +1. Frontend validates the request against the table schema. Protocols that support schema-on-write may create a missing table or add columns before retrying the write. +2. The partition rule assigns rows to Regions. Frontend builds one Region request per target and routes it to the current Region leader. +3. The Datanode's Region server dispatches each request to the Region engine. In standalone mode, the same request is sent to an embedded Region server instead of over RPC. + +#### DDL + +The statement executor converts DDL into a task. In distributed mode, Metasrv runs that task as a persisted procedure, updates metadata, and coordinates Region operations on Datanodes. Standalone mode uses the same statement boundary with local implementations of the metadata and procedure services. ### Deployment diff --git a/versioned_docs/version-1.2/contributor-guide/frontend/table-sharding.md b/versioned_docs/version-1.2/contributor-guide/frontend/table-sharding.md index a60276d14e..beaece40c0 100644 --- a/versioned_docs/version-1.2/contributor-guide/frontend/table-sharding.md +++ b/versioned_docs/version-1.2/contributor-guide/frontend/table-sharding.md @@ -5,21 +5,15 @@ description: Explains how table data in GreptimeDB is sharded and distributed, i # Table Sharding -The sharding of stored data is essential to any distributed database. This document will describe how table's data in GreptimeDB is being sharded, and distributed. +GreptimeDB shards a table into Regions. Partition expressions define which rows belong to each Region, while Region routes define which Datanode currently owns each Region. ## Partition -For the syntax of creating a partitioned table, please refer to the [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) section in the User Guide. +A partition is a logical row set described by an expression over one or more columns. The partition layout must cover the table's input domain so each row has one target Region. See [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md) for the SQL syntax and supported expressions. ## Region -The data within a table is logically split after creating partitions. You may ask the question " -how are the data, after being logically partitioned, stored in the GreptimeDB? The answer is in "`Region`"s. - -Each region is corresponding to a partition, and stores the data in the partition. The regions are distributed among -`Datanode`s. `Metasrv` manages the route information that maps regions to Datanodes. -If the partition layout needs to change after table creation, GreptimeDB supports explicit -[repartitioning](/user-guide/deployments-administration/manage-data/repartition.md) through split and merge operations. +Each partition maps to one Region. Region IDs remain the storage and routing identity used by Frontend, Datanode, and Metasrv. Multiple Regions from the same table may be placed on one Datanode. The relationship between partition and region can be viewed as the following diagram: @@ -53,3 +47,14 @@ The relationship between partition and region can be viewed as the following dia │ │ └──────────────────────────────────┘ Could be placed in one Datanode +``` + +## Routing and Pruning + +For writes, Frontend evaluates the partition rule for each row, groups rows by Region, and sends Region requests to the current leaders from the route table. + +For queries, the distributed planner compares query predicates with the partition expressions. It scans only Regions that can satisfy the predicates. If partition metadata is missing or cannot be interpreted safely, the planner falls back to all Regions rather than risk omitting data. + +## Changing the Partition Layout + +[Repartitioning](/user-guide/deployments-administration/manage-data/repartition.md) changes an existing layout through explicit split and merge operations. Metasrv runs the change as a persisted procedure, updates the Region routes and partition expressions, and invalidates stale table-route caches. New requests use the published layout after their Frontend refreshes that metadata. diff --git a/versioned_docs/version-1.2/contributor-guide/getting-started.md b/versioned_docs/version-1.2/contributor-guide/getting-started.md index b17184dfa8..7e8cca3ad8 100644 --- a/versioned_docs/version-1.2/contributor-guide/getting-started.md +++ b/versioned_docs/version-1.2/contributor-guide/getting-started.md @@ -15,14 +15,13 @@ At the moment, GreptimeDB supports Linux (both amd64 and arm64), macOS (both amd ### Build Dependencies -- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) (optional) +- [Git](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) (optional; needed to clone the repository, not to build it) - C/C++ Toolchain: provides essential tools for compiling and linking. This is available either as `build-essential` on ubuntu or a similar name on other platforms. -- Rust nightly toolchain ([guide][1]) - - Compile the source code +- [Rustup][1]. The repository pins the required nightly toolchain in `rust-toolchain.toml`. - Protobuf ([guide][2]) - Compile the proto file - Note that the version needs to be >= 3.15. You can check it with `protoc --version` -- Machine: Recommended memory is 16GB or more, or use the [mold](https://github.com/rui314/mold) tool to reduce memory usage during linking. +- Machine: 16GB of memory or more is recommended. On a smaller machine, use [mold](https://github.com/rui314/mold) to reduce memory usage during linking. [1]: [2]: diff --git a/versioned_docs/version-1.2/contributor-guide/how-to/how-to-write-sdk.md b/versioned_docs/version-1.2/contributor-guide/how-to/how-to-write-sdk.md index 40d0bc3713..0fd47f94e4 100644 --- a/versioned_docs/version-1.2/contributor-guide/how-to/how-to-write-sdk.md +++ b/versioned_docs/version-1.2/contributor-guide/how-to/how-to-write-sdk.md @@ -1,21 +1,17 @@ --- keywords: [gRPC SDK, GreptimeDatabase, Handle, HandleRequests, GreptimeRequest, GreptimeResponse] -description: Explains how to write a gRPC SDK for GreptimeDB, focusing on the GreptimeDatabase service, its methods, and the structure of requests and responses. +description: Protocol contracts and error-handling requirements for a GreptimeDB gRPC ingestion SDK. --- # How to write a gRPC SDK for GreptimeDB -A GreptimeDB gRPC SDK only needs to handle the writes. The reads are standard SQL and PromQL, can be handled by any JDBC -client or Prometheus client. This is also why GreptimeDB gRPC SDKs are all named -like "`greptimedb-ingester-`". Please make sure your GreptimeDB SDK follow the same naming convention. +GreptimeDB's public gRPC SDKs are ingestion clients. Queries normally use SQL or PromQL through their standard clients. A new SDK should therefore focus on writes and deletes unless it has a separate requirement, and follow the `greptimedb-ingester-` naming convention. See the [gRPC SDK overview](/user-guide/ingest-data/for-iot/grpc-sdks/overview.md) for the user-facing API. ## `GreptimeDatabase` Service -GreptimeDB defines a custom gRPC service called `GreptimeDatabase`. All you need to do in your SDK are implement it. You -can find its Protobuf -definitions [here](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto). +Generate client stubs from the [`GreptimeDatabase` Protobuf definition](https://github.com/GreptimeTeam/greptime-proto/blob/main/proto/greptime/v1/database.proto). Do not maintain a handwritten copy of the messages or service definition. -The service contains two RPC methods: +The service provides a unary method and a client-streaming method: ```protobuf service GreptimeDatabase { @@ -25,13 +21,9 @@ service GreptimeDatabase { } ``` -The `Handle` method is for unary call: when a `GreptimeRequest` is received and processed by a GreptimeDB -server, it responds with a `GreptimeResponse` immediately. +`Handle` returns one response for one request. It is the usual choice for an SDK's insert and delete APIs. -The `HandleRequests` acts in -a "[Client streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc)" style. It ingests a -stream of `GreptimeRequest`, and handles them on the fly. After all the requests have been handled, it returns a -summarized `GreptimeResponse`. Through `HandleRequests`, we can achieve a very high throughput of requests handling. +`HandleRequests` is a [client-streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#client-streaming-rpc). The server returns a cumulative response only after the client closes the request stream. An SDK that exposes streaming must document this acknowledgement boundary and bind the stream to one endpoint. ### `GreptimeRequest` @@ -51,13 +43,13 @@ message GreptimeRequest { } ``` -A `RequestHeader` is needed, it includes some context, authentication and others. The "oneof" field contains the request -to the GreptimeDB server. +A client must populate `RequestHeader` with the database context and authentication expected by the server. Set exactly one request variant. -Note that we have two types of insertions, one is in the form of "column" (the `InsertRequests`), and the other is " -row" (`RowInsertRequests`). It's generally recommended to use the "row" form, since it's more natural for insertions on -a table, and easier to use. However, if there's a need to insert a large number of columns at once, or there're plenty -of "null" values to insert, the "column" form is better to be used. +The message also contains query and DDL variants used by internal callers. The public ingester API should not expose them: `GreptimeDatabase` does not return query result streams. + +GreptimeDB accepts row-oriented `RowInsertRequests` and column-oriented `InsertRequests`. Row-oriented requests are the default for public ingestion APIs. A column-native client may use the column form, but it must keep column lengths consistent and preserve null values, timestamp precision, data types, and column semantic types during conversion. + +Deletes have the same row-oriented and column-oriented distinction. Expose only the forms that the SDK can map without losing type information. ### `GreptimeResponse` @@ -70,8 +62,18 @@ message GreptimeResponse { } ``` -The `ResponseHeader` contains the response's status code, and error message (if there's any). The "oneof" response only -contains the affected rows for now. +On success, the response contains a successful header and `affected_rows`. Treat that value as the number acknowledged by the server, including the cumulative value returned when a request stream closes. + +Request failures are returned as a gRPC status. When present, the trailing metadata keys `x-greptime-err-code` and `x-greptime-err-retry-hint` carry GreptimeDB's error code and retry classification. Preserve the gRPC status and expose the GreptimeDB metadata rather than replacing them with a generic SDK error. + +## Retry and Delivery Semantics + +Retries must be bounded and observable. A unary request may be retried only when the failure is classified as retryable and the deadline still permits it. Do not retry cancellation or deadline-expiration errors. + +A lost response does not prove that the server rejected a write. Retrying such a request can insert duplicate rows unless the caller's data model makes the operation idempotent. Document this possibility and return the final error when delivery is ambiguous. + +Do not transparently retry a partially sent `HandleRequests` stream. The server may already have accepted some requests even though the client has not received the cumulative response. Close the failed stream and report the ambiguity to the caller. + +Keep Arrow Flight bulk ingestion separate from the `GreptimeDatabase` RPCs. Its batching and partial-acceptance behavior needs its own API contract. -GreptimeDB has a lot of SDKs now, you can refer to -them [here](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=) for some examples. +Use the existing [GreptimeDB ingester repositories](https://github.com/GreptimeTeam?q=ingester&type=all&language=&sort=) to compare public API conventions, but derive wire behavior from the current Protobuf definition and server contract. diff --git a/versioned_docs/version-1.2/contributor-guide/metasrv/admin-api.md b/versioned_docs/version-1.2/contributor-guide/metasrv/admin-api.md index b091b48b70..d1f2ee6dbe 100644 --- a/versioned_docs/version-1.2/contributor-guide/metasrv/admin-api.md +++ b/versioned_docs/version-1.2/contributor-guide/metasrv/admin-api.md @@ -1,6 +1,6 @@ --- -keywords: [admin api, health check, leader query, heartbeat, maintenance mode, RESTful API] -description: Details the Admin API for Metasrv, including endpoints for health checks, leader queries, heartbeat data, maintenance mode, and Procedure Manager controls. +keywords: [admin api, health check, leader query, heartbeat, maintenance mode, recovery mode, table id sequence] +description: Details the Metasrv Admin API for status inspection, cluster controls, and metadata recovery. --- # Admin API @@ -9,16 +9,17 @@ description: Details the Admin API for Metasrv, including endpoints for health c Note that all Admin API endpoints in this document listen on Metasrv's `HTTP_PORT`, which defaults to `4000`. ::: -The Admin API provides a simple way to view and manage cluster information, including metasrv health detection, metasrv leader query, datanode heartbeat detection, maintenance mode, and Procedure Manager controls. - -The Admin API is an HTTP service that provides a set of RESTful APIs that can be called through HTTP requests. The Admin API is simple, user-friendly and safe. +The Admin API exposes Metasrv status, cluster controls, and metadata recovery operations over HTTP. It does not provide authentication, and some endpoints change cluster behavior or metadata allocation. Deployments must protect the HTTP port with network-level controls. This page covers the following APIs: - /health - /leader - /heartbeat +- /node-lease - /maintenance - /procedure-manager +- /recovery +- /sequence/table All these APIs are under the parent resource `/admin`. @@ -26,7 +27,7 @@ In the following sections, we assume that your metasrv instance is running on lo ## /health HTTP endpoint -The `/health` endpoint accepts GET HTTP requests and you can use this endpoint to check the health of your metasrv instance. +The `/health` endpoint accepts GET requests and returns `OK` when the HTTP service is running. It does not check whether this Metasrv is the leader or whether external dependencies are available. ### Definition @@ -120,9 +121,17 @@ curl -X GET 'http://localhost:4000/admin/heartbeat?addr=127.0.0.1:4100' ] ``` +## /node-lease HTTP endpoint + +The `/node-lease` endpoint returns the current leases recorded for Datanodes. Use it when diagnosing whether Metasrv still considers a Datanode active. + +```bash +curl -X GET http://localhost:4000/admin/node-lease +``` + ## /maintenance HTTP endpoint -Cluster Maintenance Mode is a safety feature in GreptimeDB that temporarily disables automatic cluster management operations. This mode is particularly useful during cluster upgrades, planned downtime, and any operation that might temporarily affect cluster stability. For more details, please refer to [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md). +Maintenance mode temporarily disables automatic cluster management operations during upgrades, planned downtime, or similar work. See [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md) for its effect on the cluster. The `/maintenance` endpoint supports the following HTTP requests: @@ -155,3 +164,39 @@ The response body uses the following format: "status": "running" } ``` + +## /recovery HTTP endpoints + +Recovery mode gates metadata repair endpoints such as manual table ID sequence changes. It is intended for recovery work, not routine maintenance. + +- `GET /admin/recovery/status`: query whether recovery mode is enabled. +- `POST /admin/recovery/enable`: enable recovery mode. +- `POST /admin/recovery/disable`: disable recovery mode. + +The response body uses the following format: + +```json +{ + "enabled": true +} +``` + +Disable recovery mode after the repair is complete. Use [maintenance mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md) instead when the goal is to suspend automatic cluster operations during planned maintenance. + +## /sequence/table HTTP endpoints + +These endpoints inspect or repair the table ID sequence: + +- `GET /admin/sequence/table/next-id`: return the next table ID without allocating it. +- `POST /admin/sequence/table/set-next-id`: advance the next table ID. + +Setting the sequence requires recovery mode. The new value must be greater than the current value; the API cannot move the sequence backwards. Recovery mode is an API precondition, not a DDL barrier. Follow [Manage table ID sequences](/user-guide/deployments-administration/maintenance/sequence-management.md) for the required cluster-wide procedure. + +```bash +curl -X POST \ + -H 'Content-Type: application/json' \ + -d '{"next_table_id": 2048}' \ + http://localhost:4000/admin/sequence/table/set-next-id +``` + +Changing this value affects IDs allocated to future tables. diff --git a/versioned_docs/version-1.2/contributor-guide/metasrv/overview.md b/versioned_docs/version-1.2/contributor-guide/metasrv/overview.md index 7aa052dd7e..6458b6380d 100644 --- a/versioned_docs/version-1.2/contributor-guide/metasrv/overview.md +++ b/versioned_docs/version-1.2/contributor-guide/metasrv/overview.md @@ -1,161 +1,101 @@ --- -keywords: [metasrv, metadata, request-router, load balancing, election, high availability, heartbeat] -description: Provides an overview of the Metasrv service, its components, interactions with the Frontend, architecture, and key functionalities like distributed consensus and heartbeat management. +keywords: [metasrv, metadata, routing, leader election, procedure, heartbeat] +description: Overview of the metadata and coordination mechanisms provided by Metasrv. --- # Metasrv -![meta](/meta.png) - ## What's in Metasrv -- Store metadata (Catalog, Schema, Table, Region, etc.) -- Request-Router. It tells the Frontend where to write and read data. -- Load balancing for Datanode, determines who should handle new table creation requests, more precisely, it makes resource allocation decisions. -- Election & High Availability, GreptimeDB is designed in a Leader-Follower architecture, only Leader nodes can write while Follower nodes can read, the number of Follower nodes is usually >= 1, and Follower nodes need to be able to switch to Leader quickly when Leader is not available. -- Statistical data collection (reported via Heartbeats on each node), such as CPU, Load, number of Tables on the node, average/peak data read/write size, etc., can be used as the basis for distributed scheduling. +Metasrv is the metadata and coordination service in a distributed GreptimeDB cluster. It does not sit on the data path. Its main responsibilities are: + +- storing Catalog, Schema, Table, Region, route, and node metadata; +- choosing Datanodes for new Regions and maintaining table routes; +- electing one Metasrv leader to coordinate metadata changes; +- running recoverable procedures for DDL, Region migration, failover, and repartitioning; +- tracking node leases and Region statistics through heartbeats; +- broadcasting cache invalidations to Frontends, Datanodes, and Flownodes when metadata changes; +- sending Region lifecycle instructions to Datanodes. ## How the Frontend interacts with Metasrv -First, the routing table in Request-Router is in the following structure (note that this is only the logical structure, the actual storage structure varies, for example, endpoints may have dictionary compression). +Frontend obtains table metadata and Region routes from Metasrv and caches them locally. Metadata-changing statements are sent to the Metasrv leader, while reads and writes use the cached routes to reach Datanodes directly. + +The control and data paths are separate: + +```text +Frontend + |-- metadata lookup and DDL ------------> Metasrv leader + `-- Region reads and writes ------------> Datanode + +Metasrv leader + |-- Region lifecycle instructions ------> Datanode + `-- cache invalidations ----------------> Frontend / Datanode / Flownode +Datanode + `-- heartbeat, lease renewal, Region stats -> Metasrv leader ``` - table_A - table_name - table_schema // for physical plan - regions - region_1 - mutate_endpoint - select_endpoint_1, select_endpoint_2 - region_2 - mutate_endpoint - select_endpoint_1, select_endpoint_2, select_endpoint_3 - region_xxx - table_B - ... + +In steady state, a table route records one leader peer and zero or more follower peers for each Region. The leader is the write target. Deployments with read-replica support can route reads to followers: + +```text +Table route + |-- Region 0 + | |-- leader -> Datanode A + | `-- followers -> Datanode B, Datanode C + `-- Region 1 + `-- leader -> Datanode D ``` +Region migration or failover changes peer roles and can temporarily leave a Region without a leader. Frontend refreshes its cached route before sending subsequent reads or writes to the current peers. + ### Create Table -1. The Frontend sends `CREATE TABLE` requests to Metasrv. -2. Plan the number of Regions according to the partition rules contained in the request. -3. Check the global view of resources available to Datanodes (collected by Heartbeats) and assign one node to each region. -4. The Frontend creates the table and stores the `Schema` to Metasrv after successful creation. +1. Frontend submits the DDL request to the Metasrv leader. +2. Metasrv derives Regions from the partition rules and [selects a Datanode for each Region](/contributor-guide/metasrv/selector.md). +3. A persisted procedure creates the Regions and records the table and route metadata. If leadership changes, the procedure can resume from its persisted state. +4. Metasrv notifies Frontends after the metadata change is committed so their caches can be refreshed. ### Insert -1. The Frontend fetches the routes of the specified table from Metasrv. Note that the smallest routing unit is the route of the table (several regions), i.e., it contains the addresses of all regions of this table. -2. The best practice is that the Frontend first fetches the routes from its local cache and forwards the request to the Datanode. If the route is no longer valid, then Datanode is obliged to return an `Invalid Route` error, and the Frontend re-fetches the latest data from Metasrv and updates its cache. Route information does not change frequently, thus, it's sufficient for Frontend uses the Lazy policy to maintain the cache. -3. The Frontend processes a batch of writes that may contain multiple tables and multiple regions, so the Frontend needs to split user requests based on the 'route table'. +Frontend resolves the table route, splits rows according to the partition rules, and sends each Region write to the corresponding Datanode. Route changes cause the cached metadata to be invalidated and fetched again from Metasrv. ### Select -1. As with `Insert`, the Frontend first fetches the route table from the local cache. -2. Unlike `Insert`, for `Select`, the Frontend needs to extract the read-only node (follower) from the route table, then dispatch the request to the leader or follower node depending on the priority. -3. The distributed query engine in the Frontend distributes multiple sub-query tasks based on the routing information and aggregates the query results. +Frontend uses table and Region metadata while planning the query. Predicates on partition columns prune Regions, and the distributed query engine sends work to the Datanodes that own the selected Regions. See [Distributed Querying](../frontend/distributed-querying.md). ## Metasrv Architecture -![metasrv-architecture](/metasrv-architecture.png) - -## Distributed Consensus +The main coordination paths are: + +```text +Leader election + | + v +Metasrv leader +├─ DDL manager -> Procedure manager +├─ Selector -> new Region placement +├─ Heartbeat handler chain -> leases and Region statistics +├─ Region supervisor -> Region migration procedures +├─ Mailbox -> cache invalidations and Region instructions +└─ Metadata managers -> KV backend +``` -As you can see, Metasrv has a dependency on distributed consensus because: +These mechanisms share metadata, but they have different failure boundaries. A process restart may discard caches and leader-local state; metadata and procedure state required for recovery must be durable. -1. First, Metasrv has to elect a leader, Datanode only sends heartbeats to the leader, and we only use a single metasrv node to receive heartbeats, which makes it easy to do some calculations or scheduling accurately and quickly based on global information. As for how the Datanode connects to the leader, this is for MetaClient to decide (using a redirect, Heartbeat requests becomes a gRPC stream, and using redirect will be less error-prone than forwarding), and it is transparent to the Datanode. -2. Second, Metasrv must provide an election API for Datanode to elect "write" and "read-only" nodes and help Datanode achieve high availability. -3. Finally, `Metadata`, `Schema` and other data must be reliably and consistently stored on Metasrv. Therefore, consensus-based algorithms are the ideal approach for storing them. +## Distributed Consensus -For the first version of Metasrv, we choose Etcd as the consensus algorithm component (Metasrv is designed to consider adapting different implementations and even creating a new wheel) for the following reasons: +Metasrv separates leader election from metadata storage. Only the elected Metasrv leader performs coordination and metadata-changing operations. Other Metasrv nodes direct clients to the current leader. -1. Etcd provides exactly the API we need, such as `Watch`, `Election`, `KV`, etc. -2. We only perform two tasks with distributed consensus: elections (using the `Watch` mechanism) and storing (a small amount of metadata), and neither of them requires us to customize our own state machine, nor do we need to customize our own state machine based on raft; the small amount of data also does not require multi-raft-group support. -3. The initial version of Metasrv uses Etcd, which allows us to focus on the capabilities of Metasrv and not spend too much effort on distributed consensus algorithms, which improves the design of the system (avoiding coupling with consensus algorithms) and helps with rapid development at the beginning, as well as allows easy access to good consensus algorithm implementations in the future through good architectural designs. +The key-value backend stores table metadata, routes, procedure state, and other information that must survive a leader change. Metasrv does not use this election to create leader and follower replicas for Datanode Regions; Region availability is managed through heartbeats, Region failure detection, and failover procedures. ## Heartbeat Management -The primary means of communication between Datanode and Metasrv is the Heartbeat Request/Response Stream, and we want this to be the only way to communicate. This idea is inspired by the design of [TiKV PD](https://github.com/tikv/pd), and we have practical experience in [RheaKV](https://github.com/sofastack/sofa-jraft/tree/master/jraft-rheakv/rheakv-pd). The request sends its state, while Metasrv sends different scheduling instructions via Heartbeat Response. - -A heartbeat will probably carry the data listed below, but this is not the final design, and we are still discussing and exploring exactly which data should be mostly collected. - -``` -service Heartbeat { - // Heartbeat, there may be many contents of the heartbeat, such as: - // 1. Metadata to be registered to metasrv and discoverable by other nodes. - // 2. Some performance metrics, such as Load, CPU usage, etc. - // 3. The number of computing tasks being executed. - rpc Heartbeat(stream HeartbeatRequest) returns (stream HeartbeatResponse) {} -} - -message HeartbeatRequest { - RequestHeader header = 1; - - // Self peer - Peer peer = 2; - // Leader node - bool is_leader = 3; - // Actually reported time interval - TimeInterval report_interval = 4; - // Node stat - NodeStat node_stat = 5; - // Region stats in this node - repeated RegionStat region_stats = 6; - // Follower nodes and stats, empty on follower nodes - repeated ReplicaStat replica_stats = 7; -} - -message NodeStat { - // The read capacity units during this period - uint64 rcus = 1; - // The write capacity units during this period - uint64 wcus = 2; - // Table number in this node - uint64 table_num = 3; - // Region number in this node - uint64 region_num = 4; - - double cpu_usage = 5; - double load = 6; - // Read disk I/O in the node - double read_io_rate = 7; - // Write disk I/O in the node - double write_io_rate = 8; - - // Others - map attrs = 100; -} - -message RegionStat { - uint64 region_id = 1; - TableName table_name = 2; - // The read capacity units during this period - uint64 rcus = 3; - // The write capacity units during this period - uint64 wcus = 4; - // Approximate region size - uint64 approximate_size = 5; - // Approximate number of rows - uint64 approximate_rows = 6; - - // Others - map attrs = 100; -} - -message ReplicaStat { - Peer peer = 1; - bool in_sync = 2; - bool is_learner = 3; -} -``` - -## Central Nervous System (CNS) - -We are to build an algorithmic system, which relies on real-time and historical heartbeat data from each node, should make some smarter scheduling decisions and send them to Metasrv's Autoadmin unit, which distributes the scheduling decisions, either by the Datanode itself or more likely by the PaaS platform. - -## Abstraction of Workloads +Datanodes maintain heartbeat streams to the Metasrv leader. Heartbeat requests report node identity, lease information, Region statistics, and other state used for placement and supervision. Responses carry control messages such as Region lifecycle instructions and cache invalidations. -The level of workload abstraction determines the efficiency and quality of the scheduling strategy generated by Metasrv such as resource allocation. +A heartbeat drives two independent mechanisms, and a change to heartbeat timing affects both: -DynamoDB defines RCUs & WCUs (Read Capacity Units / Write Capacity Units), explaining that a RCU is a read request of 4KB data, and a WCU is a write request of 1KB data. When using RCU and WCU to describe workloads, it's easier to achieve performance measurability and get more informative resource preallocation because we can abstract different hardware capabilities as a combination of RCU and WCU. +- **Node lease.** The keep-lease handler renews the sending Datanode's lease. Selectors and the `/node-lease` endpoint use these leases to decide whether a Datanode is still active. +- **Region failure detection.** The Region supervisor keeps a per-Region Phi Accrual detector over heartbeat arrival intervals. Its verdict is independent of lease expiry. -However, GreptimeDB still faces a more complex situation than DynamoDB, in particular, RCU doesn't fit to describe GreptimeDB's read workloads which require a lot of computation. We are working on that. +A failure verdict submits a failover migration only when Region failover is enabled; it is disabled by default and requires remote WAL unless explicitly allowed on local WAL. Maintenance mode also suppresses failover. See [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md) for the prerequisites and how to enable it. diff --git a/versioned_docs/version-1.2/contributor-guide/metasrv/selector.md b/versioned_docs/version-1.2/contributor-guide/metasrv/selector.md index 790ffb849b..23190cd6a9 100644 --- a/versioned_docs/version-1.2/contributor-guide/metasrv/selector.md +++ b/versioned_docs/version-1.2/contributor-guide/metasrv/selector.md @@ -7,32 +7,28 @@ description: Describes the different types of selectors in the Metasrv service, ## Introduction -What is the `Selector`? As its name suggests, it allows users to select specific items from a given `namespace` and `context`. There is a related trait, also named `Selector`, whose definition can be found [below][0]. - -[0]: https://github.com/GreptimeTeam/greptimedb/blob/main/src/meta-srv/src/selector.rs - -There is a specific scenario in `Metasrv` service. When a request to create a table is sent to the `Metasrv` service, it creates a routing table (the details of table creation will not be described here). The `Metasrv` service needs to select the appropriate `Datanode` list when creating a routing table. +When a table is created, Metasrv uses a `Selector` to choose Datanodes for its Regions. Selection uses the current node leases and, depending on the selector, Region statistics. ## Selector Type The `Metasrv` service currently offers the following types of `Selectors`: -### LeasebasedSelector +### LeaseBasedSelector -`LeasebasedSelector` randomly selects from all available (in lease) `Datanode`s, its characteristic is simplicity and fast. +`LeaseBasedSelector` randomly selects from Datanodes with valid leases. ### LoadBasedSelector The `LoadBasedSelector` load value is determined by the number of regions on each `Datanode`, fewer regions indicate lower load, and `LoadBasedSelector` prioritizes selecting low-load `Datanodes`. ### RoundRobinSelector [default] -`RoundRobinSelector` selects `Datanode`s in a round-robin fashion. It is recommended and the default option in most cases. If you're unsure which to choose, it's usually the right choice. +`RoundRobinSelector` selects `Datanode`s in a round-robin fashion. It is the default and recommended choice for most deployments. ## Configuration You can configure the `Selector` by its name when starting the `Metasrv` service. -- LeasebasedSelector: `lease_based` or `LeaseBased` +- LeaseBasedSelector: `lease_based` or `LeaseBased` - LoadBasedSelector: `load_based` or `LoadBased` - RoundRobinSelector: `round_robin` or `RoundRobin` diff --git a/versioned_docs/version-1.2/contributor-guide/overview.md b/versioned_docs/version-1.2/contributor-guide/overview.md index 875b4b1e16..6e6ec669c9 100644 --- a/versioned_docs/version-1.2/contributor-guide/overview.md +++ b/versioned_docs/version-1.2/contributor-guide/overview.md @@ -5,9 +5,7 @@ description: Overview of GreptimeDB's architecture, key components, and how they # Contributor Guide -DeepWiki provides a detailed and clear explanation of GreptimeDB's architecture and implementation. Highly recommended: - -[https://deepwiki.com/GreptimeTeam/greptimedb](https://deepwiki.com/GreptimeTeam/greptimedb) +This guide explains the internal design of GreptimeDB for contributors. Start with [Getting Started](/contributor-guide/getting-started.md) to build and run it from source. Submission requirements, including the CLA, license headers, formatting, and the checks a pull request must pass, are maintained in the source repository's [CONTRIBUTING.md](https://github.com/GreptimeTeam/greptimedb/blob/main/CONTRIBUTING.md). ## Architecture @@ -18,8 +16,13 @@ For more details on each component, see the following guides: - [frontend][1] - [datanode][2] - [metasrv][3] +- [flownode][4] [1]: /contributor-guide/frontend/overview.md [2]: /contributor-guide/datanode/overview.md [3]: /contributor-guide/metasrv/overview.md +[4]: /contributor-guide/flownode/overview.md + +## Additional reference +[DeepWiki](https://deepwiki.com/GreptimeTeam/greptimedb) provides an automatically generated walkthrough of the GreptimeDB repository. It can help when exploring an unfamiliar area, but it is a secondary reference: verify version-sensitive behavior against the source code. diff --git a/versioned_docs/version-1.2/contributor-guide/tests/integration-test.md b/versioned_docs/version-1.2/contributor-guide/tests/integration-test.md index 5d5f6cb1a5..bc52ee4854 100644 --- a/versioned_docs/version-1.2/contributor-guide/tests/integration-test.md +++ b/versioned_docs/version-1.2/contributor-guide/tests/integration-test.md @@ -7,7 +7,14 @@ description: Guide on writing and running integration tests in GreptimeDB, cover ## Introduction -Integration testing is written with Rust test harness (`#[test]`), unlike unit testing, they are placed separately -[here](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration). -It covers scenarios involving multiple components, in which one typical case is HTTP/gRPC-related features. You can check -its [documentation](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md) for more information. +Integration tests cover behavior that crosses crate or service boundaries, such as HTTP and gRPC handling, distributed components, or external storage. They use Rust's test harness and live in the [`tests-integration`](https://github.com/GreptimeTeam/greptimedb/tree/main/tests-integration) package. + +Run the package with: + +```shell +cargo nextest run -p tests-integration +``` + +Some cases require environment variables or fixtures for external services. Follow the package's [setup instructions](https://github.com/GreptimeTeam/greptimedb/blob/main/tests-integration/README.md) before running those cases. + +Use an integration test when a crate-level test or a Sqlness case cannot exercise the required boundary. Keep isolated logic in unit tests so that failures remain fast to reproduce. diff --git a/versioned_docs/version-1.2/contributor-guide/tests/overview.md b/versioned_docs/version-1.2/contributor-guide/tests/overview.md index feeefe2b6d..6fe48b2c32 100644 --- a/versioned_docs/version-1.2/contributor-guide/tests/overview.md +++ b/versioned_docs/version-1.2/contributor-guide/tests/overview.md @@ -5,4 +5,13 @@ description: Overview of the testing methods used in GreptimeDB to ensure its be # Tests -Our team has conducted lots of tests to ensure the behaviours of `GreptimeDB` . This chapter will introduce several significant methods used to test `GreptimeDB`, and how to work with them. +Choose the narrowest test that exercises the behavior you changed: + +| Test type | Use it for | Typical command | +| --- | --- | --- | +| [Unit test](unit-test.md) | Logic contained within one crate or component | `cargo nextest run -p ` | +| [Sqlness test](sqlness-test.md) | SQL, protocol, planner, execution, and end-to-end regressions | `cargo sqlness bare -t ` | +| [Integration test](integration-test.md) | Behavior that crosses components or requires external services | `cargo nextest run -p tests-integration` | +| [Compatibility test](https://github.com/GreptimeTeam/greptimedb/blob/main/tests/compatibility/README.md) | Reading data or metadata written by an older release | `cargo sqlness compat --from-version ` | + +Run `make test` when a change needs the full Rust workspace test suite. Changes to persisted metadata, WAL records, SST files, or wire formats should also include a compatibility test when an older release may have produced the input. diff --git a/versioned_docs/version-1.2/contributor-guide/tests/sqlness-test.md b/versioned_docs/version-1.2/contributor-guide/tests/sqlness-test.md index b5fdcf2b0b..0262ff6851 100644 --- a/versioned_docs/version-1.2/contributor-guide/tests/sqlness-test.md +++ b/versioned_docs/version-1.2/contributor-guide/tests/sqlness-test.md @@ -7,42 +7,34 @@ description: Instructions for running SQL tests in GreptimeDB using the `sqlness ## Introduction -SQL is an important user interface for `GreptimeDB`. We have a separate test suite for it (named `sqlness`). +Sqlness is GreptimeDB's end-to-end regression suite for SQL and protocol behavior. A case sends statements to a running GreptimeDB instance and compares the output with a checked-in result file. ## Sqlness manual ### Case file -Sqlness has two types of file +Each case uses two files: - `.sql`: test input, SQL only - `.result`: expected test output, SQL and its results -The `.result` file is the expected execution output. If you see `.result` files changed, -it means the test gets a different result and indicates it may fail. You should -check the change logs to solve the problem. - -You only need to write test SQL in the `.sql` file, and run the test. +Write the input in the `.sql` file and run the test to generate or update `.result`. Review every result diff: accept it only when the behavior change is intended. ### Case organization -The root dir of input cases is `tests/cases`. It contains several sub-directories stand for different test -modes. E.g., `standalone/` contains all the tests to run under `greptimedb standalone start` mode. +Input cases live under `tests/cases`. The first directory level selects an environment. For example, `standalone/` runs against a standalone GreptimeDB instance. -Under the first level of sub-directory (e.g. the `cases/standalone`), you can organize your cases as you like. -Sqlness walks through every file recursively and runs them. +Within an environment, group a new case with the feature it exercises. Sqlness discovers case files recursively. ## Run the test -Unlike other tests, this harness is in a binary target form. You can run it with +Run the suite with: ```shell -cargo run --bin sqlness-runner bare +cargo sqlness bare ``` -It automatically finishes the following procedures: compile `GreptimeDB`, start it, grab tests and feed it to -the server, then collect and compare the results. You only need to check whether any `.result` files changed. -If not, congratulations, the test is passed 🥳! +The command builds and starts GreptimeDB, runs the selected cases, and compares their output. A changed `.result` file is part of the review, not proof that the new output is correct. ### Run a specific test diff --git a/versioned_docs/version-1.2/contributor-guide/tests/unit-test.md b/versioned_docs/version-1.2/contributor-guide/tests/unit-test.md index 82e30bf5fa..183a72aab4 100644 --- a/versioned_docs/version-1.2/contributor-guide/tests/unit-test.md +++ b/versioned_docs/version-1.2/contributor-guide/tests/unit-test.md @@ -8,25 +8,26 @@ description: Guide on writing and running unit tests in GreptimeDB using Rust's ## Introduction Unit tests are embedded into the codebase, usually placed next to the logic being tested. -They are written using Rust's `#[test]` attribute and can run with `cargo nextest run`. +They are written using Rust's `#[test]` attribute. GreptimeDB uses [`cargo-nextest`](https://nexte.st/) as its primary Rust test runner. -The default test runner ships with `cargo` is not supported in GreptimeDB codebase. It's recommended -to use [`nextest`](https://nexte.st/) instead. You can install it with +Install it with: ```shell cargo install cargo-nextest --locked ``` -And run the tests (here the `--workspace` is not necessary) +Run the package you changed first: ```shell -cargo nextest run +cargo nextest run -p ``` -Notes if your Rust is installed via `rustup`, be sure to install `nextest` with `cargo` rather -than the package manager like `homebrew`. Otherwise it will mess up your local environment. +Use a test name or nextest filter to narrow the run further while developing. Before submitting a change with broad effects, run the full workspace suite: + +```shell +make test +``` ## Coverage -Our continuous integration (CI) jobs have a "coverage checking" step. It will report how many -codes are covered by unit tests. Please add the necessary unit test to your patch. +CI reports unit-test coverage. Add tests for changed behavior and failure cases that could otherwise regress; coverage percentage alone is not the goal. diff --git a/versioned_docs/version-1.2/reference/sql/create.md b/versioned_docs/version-1.2/reference/sql/create.md index 4eccb986d0..29a9cf666f 100644 --- a/versioned_docs/version-1.2/reference/sql/create.md +++ b/versioned_docs/version-1.2/reference/sql/create.md @@ -26,7 +26,7 @@ If the `db_name` database already exists, then GreptimeDB has the following beha The database can also carry options similar to the `CREATE TABLE` statement by using the `WITH` keyword. The following options are available for databases: - `ttl` - Time-To-Live for data in all tables within the database (cannot be set to `instant`) -- `memtable.type` - Type of memtable (`time_series`, `partition_tree`) +- `memtable.type` - Type of memtable (`bulk`, `time_series`) - `append_mode` - Whether tables in the database should be append-only (`true`/`false`) - `merge_mode` - Strategy for merging duplicate rows (`last_row`, `last_non_null`) - `skip_wal` - Whether to disable Write-Ahead-Log for tables in the database (`'true'`/`'false'`) @@ -74,7 +74,7 @@ Create a database with multiple options, including append mode and custom memtab ```sql CREATE DATABASE test WITH ( ttl='30d', - 'memtable.type'='partition_tree', + 'memtable.type'='bulk', 'append_mode'='true' ); ``` @@ -154,7 +154,7 @@ Users can add table options by using `WITH`. The valid options contain the follo | `compaction.twcs.trigger_file_num` | Number of files in a specific time window to trigger a compaction | String value, such as '8'. Only available when `compaction.type` is `twcs`. You can refer to this [document](https://cassandra.apache.org/doc/latest/cassandra/managing/operating/compaction/twcs.html) to learn more about the `twcs` compaction strategy. | | `compaction.twcs.time_window` | Compaction time window | String value, such as '1d' for 1 day. The table usually partitions rows into different time windows by their timestamps. Only available when `compaction.type` is `twcs`. | | `compaction.twcs.max_output_file_size` | Maximum allowed output file size for TWCS compaction | String value, such as '1GB', '512MB'. Sets the maximum size for files produced by TWCS compaction. Only available when `compaction.type` is `twcs`. | -| `memtable.type` | Type of the memtable. | String value, supports `time_series`, `partition_tree`. | +| `memtable.type` | Type of the memtable | String value: `bulk` or `time_series`. If omitted, Mito selects the implementation from the SST format; the default flat format uses `bulk`. Setting `bulk` forces `sst_format=flat`, and flat SSTs use the bulk implementation even if `time_series` is specified. The legacy value `partition_tree` is accepted for compatibility and maps to the bulk and flat path. | | `append_mode` | Whether the table is append-only | String value. Default is 'false', which removes duplicate rows by primary keys and timestamps according to the `merge_mode`. Setting it to 'true' to enable append mode and create an append-only table which keeps duplicate rows. | | `merge_mode` | The strategy to merge duplicate rows | String value. Only available when `append_mode` is 'false'. Default is `last_row`, which keeps the last row for the same primary key and timestamp. Setting it to `last_non_null` to keep the last non-null field for the same primary key and timestamp. | | `sst_format` | The format of SST files | String value, supports `primary_key`, `flat`. Default is `flat`. `flat` is recommended for tables which have a large number of unique primary keys. | diff --git a/versioned_docs/version-1.2/user-guide/deployments-administration/configuration.md b/versioned_docs/version-1.2/user-guide/deployments-administration/configuration.md index 1895b44b83..09cc360a7f 100644 --- a/versioned_docs/version-1.2/user-guide/deployments-administration/configuration.md +++ b/versioned_docs/version-1.2/user-guide/deployments-administration/configuration.md @@ -596,20 +596,9 @@ create_on_compaction = "auto" apply_on_query = "auto" mem_threshold_on_create = "64M" intermediate_path = "" - -[region_engine.mito.memtable] -type = "time_series" ``` -The `mito` engine provides an experimental memtable which optimizes for write performance and memory efficiency under large amounts of time-series. Its read performance might not as fast as the default `time_series` memtable. - -```toml -[region_engine.mito.memtable] -type = "partition_tree" -index_max_keys_per_shard = 8192 -data_freeze_threshold = 32768 -fork_dictionary_bytes = "1GiB" -``` +Mito selects the memtable implementation for each Region according to its table options and SST format. When `default_flat_format` is `true`, Regions without an explicit `sst_format` use flat SSTs and the bulk memtable. Configure `memtable.type` as a database or table option; `[region_engine.mito.memtable]` is not an engine setting. See [table options](/reference/sql/create.md#table-options). Available options: @@ -644,7 +633,7 @@ Available options: | `scan_memory_on_exhausted` | String | `fail` | Behavior when scan memory is exhausted. Options: `fail` (fail fast), `wait` or `wait()` (wait for memory). | | `min_compaction_interval` | String | `0m` | Minimum time interval between two compactions. Set to "0m" (default) to allow compactions to run immediately without restriction. | | `schedule_compaction_after_edit` | Bool | `true` | Whether to allow scheduling a compaction after a successful region edit.
Setting this to `true` is a necessary but not sufficient condition for scheduling compaction after a region edit. Other constraints, such as `min_compaction_interval`, may still prevent compaction from being scheduled.
Setting this to `false` guarantees that compaction will not be scheduled after a region edit. | -| `default_flat_format` | Bool | `true` | Whether to enable flat format as the default SST format. | +| `default_flat_format` | Bool | `true` | Whether Regions without an explicit `sst_format` use flat SSTs. Flat SSTs use the bulk memtable. | | `scan_parallelism` | Integer | `0` | (Deprecated, use `max_concurrent_scan_files` instead) Legacy option for scan parallelism. | | `index` | -- | -- | The options for index in Mito engine. | | `index.aux_path` | String | `""` | Auxiliary directory path for the index in the filesystem. This path is used to store intermediate files for creating the index and staging files for searching the index. It defaults to `{data_home}/index_intermediate`. The default name for this directory is `index_intermediate` for backward compatibility. This path contains two subdirectories: `__intm` for storing intermediate files used during index creation, and `staging` for storing staging files used during index searching. | @@ -660,10 +649,6 @@ Available options: | `inverted_index.apply_on_query` | String | `auto` | Whether to apply the index on query
- `auto`: automatically
- `disable`: never | | `inverted_index.mem_threshold_on_create` | String | `64M` | Memory threshold for performing an external sort during index creation.
Setting to empty will disable external sorting, forcing all sorting operations to happen in memory. | | `inverted_index.intermediate_path` | String | `""` | File system path to store intermediate files for external sorting (default `{data_home}/index_intermediate`). | -| `memtable.type` | String | `time_series` | Memtable type.
- `time_series`: time-series memtable
- `partition_tree`: partition tree memtable (experimental) | -| `memtable.index_max_keys_per_shard` | Integer | `8192` | The max number of keys in one shard.
Only available for `partition_tree` memtable. | -| `memtable.data_freeze_threshold` | Integer | `32768` | The max rows of data inside the actively writing buffer in one shard.
Only available for `partition_tree` memtable. | -| `memtable.fork_dictionary_bytes` | String | `1GiB` | Max dictionary bytes.
Only available for `partition_tree` memtable. | The `metric` engine is optimized for handling metrics data with a large number of small tables. diff --git a/versioned_sidebars/version-1.1-sidebars.json b/versioned_sidebars/version-1.1-sidebars.json index f75b74f205..74da2398f0 100644 --- a/versioned_sidebars/version-1.1-sidebars.json +++ b/versioned_sidebars/version-1.1-sidebars.json @@ -732,6 +732,7 @@ "label": "Overview" }, "contributor-guide/datanode/storage-engine", + "contributor-guide/datanode/memtable", "contributor-guide/datanode/query-engine", "contributor-guide/datanode/data-persistence-indexing", "contributor-guide/datanode/wal", diff --git a/versioned_sidebars/version-1.2-sidebars.json b/versioned_sidebars/version-1.2-sidebars.json index 983fd9a25b..ecaeb12fdc 100644 --- a/versioned_sidebars/version-1.2-sidebars.json +++ b/versioned_sidebars/version-1.2-sidebars.json @@ -759,6 +759,7 @@ "label": "Overview" }, "contributor-guide/datanode/storage-engine", + "contributor-guide/datanode/memtable", "contributor-guide/datanode/query-engine", "contributor-guide/datanode/data-persistence-indexing", "contributor-guide/datanode/wal",