From 88610d5962d960367d6ea85bb64848d11867d771 Mon Sep 17 00:00:00 2001 From: Dennis Zhuang Date: Wed, 2 Sep 2026 22:43:20 +0800 Subject: [PATCH 1/3] docs: shorten the batching mode title and redraw its diagram Rename the page to Batching Mode. The old heading also drove the sidebar label, which was too long for the flownode section. Cross-references in dataflow.md and overview.md follow the new name. Replace batching_mode_arch.png with an inline mermaid diagram, matching the plugin added in #2801 and the usage in enterprise/iceberg.md. The rendered diagram goes from 633x1582 to 387x608, and mermaid picks up the site's light and dark themes. The old image also had errors. It drew BatchingEngine and Database as decision diamonds, and its Notify edge implied that a source-table write drives the task, which #2784 corrected: marking a window dirty does not wake the task. The new diagram names Frontend rather than Database, because the task encodes an INSERT INTO sink SELECT plan and sends it through FrontendClient, and adds the source-table scan that Frontend performs while executing that plan. --- docs/contributor-guide/flownode/batching_mode.md | 13 +++++++++++-- docs/contributor-guide/flownode/dataflow.md | 2 +- docs/contributor-guide/flownode/overview.md | 2 +- .../contributor-guide/flownode/batching_mode.md | 13 +++++++++++-- .../current/contributor-guide/flownode/dataflow.md | 2 +- .../current/contributor-guide/flownode/overview.md | 2 +- .../contributor-guide/flownode/batching_mode.md | 13 +++++++++++-- .../contributor-guide/flownode/dataflow.md | 2 +- .../contributor-guide/flownode/overview.md | 2 +- .../contributor-guide/flownode/batching_mode.md | 13 +++++++++++-- .../contributor-guide/flownode/dataflow.md | 2 +- .../contributor-guide/flownode/overview.md | 2 +- .../contributor-guide/flownode/batching_mode.md | 13 +++++++++++-- .../contributor-guide/flownode/dataflow.md | 2 +- .../contributor-guide/flownode/overview.md | 2 +- .../contributor-guide/flownode/batching_mode.md | 13 +++++++++++-- .../contributor-guide/flownode/dataflow.md | 2 +- .../contributor-guide/flownode/overview.md | 2 +- .../contributor-guide/flownode/batching_mode.md | 13 +++++++++++-- .../contributor-guide/flownode/dataflow.md | 2 +- .../contributor-guide/flownode/overview.md | 2 +- .../contributor-guide/flownode/batching_mode.md | 13 +++++++++++-- .../contributor-guide/flownode/dataflow.md | 2 +- .../contributor-guide/flownode/overview.md | 2 +- 24 files changed, 104 insertions(+), 32 deletions(-) diff --git a/docs/contributor-guide/flownode/batching_mode.md b/docs/contributor-guide/flownode/batching_mode.md index aa8099d1ad..422c918491 100644 --- a/docs/contributor-guide/flownode/batching_mode.md +++ b/docs/contributor-guide/flownode/batching_mode.md @@ -3,7 +3,7 @@ keywords: [batching mode, flow management, Flownode components, Flownode limitat description: Overview of Flownode's batching mode, the active execution mode for continuous data aggregation, including its architecture and query execution flow. --- -# Flownode Batching Mode Developer Guide +# Batching Mode This guide provides a brief overview of the batching mode in `flownode`. It's intended for developers who want to understand the internal workings of this mode. @@ -22,7 +22,16 @@ The core idea is to: The batching mode consists of several key components that work together to achieve this continuous aggregation. As shown in the diagram below: -![batching mode architecture](/batching_mode_arch.png) +```mermaid +flowchart TB + SRC["Source table"] -->|"write"| ENG["BatchingEngine"] + ENG -->|"mark affected windows"| DTW("DirtyTimeWindows
pending ranges per task") + DTW -->|"read at next evaluation"| RUN["BatchingTask
schedule or polling cadence"] + RUN -.->|"clear processed"| DTW + RUN -->|"INSERT INTO sink SELECT
with time predicates"| FE["Frontend"] + FE -->|"scan"| SRC + FE -->|"upsert by time window"| SINK["Sink table"] +``` ### `BatchingEngine` diff --git a/docs/contributor-guide/flownode/dataflow.md b/docs/contributor-guide/flownode/dataflow.md index c876054d6d..479ea3c74a 100644 --- a/docs/contributor-guide/flownode/dataflow.md +++ b/docs/contributor-guide/flownode/dataflow.md @@ -22,7 +22,7 @@ Batching mode reuses GreptimeDB's query engine instead of maintaining an operato 4. The query result is inserted into the sink table, updating the materialized result for windows that were evaluated. 5. Successfully processed windows are removed from the dirty set. Failed work remains available for a later evaluation. -Flows with an evaluation interval but without a time-window expression run the complete query on each scheduled evaluation. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Flownode Batching Mode Developer Guide](./batching_mode.md) for the task and dirty-window components. +Flows with an evaluation interval but without a time-window expression run the complete query on each scheduled evaluation. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Batching Mode](./batching_mode.md) for the task and dirty-window components. ## Streaming mode diff --git a/docs/contributor-guide/flownode/overview.md b/docs/contributor-guide/flownode/overview.md index d1317bfc1c..f7da792f3e 100644 --- a/docs/contributor-guide/flownode/overview.md +++ b/docs/contributor-guide/flownode/overview.md @@ -14,7 +14,7 @@ description: Overview of Flownode, a component providing Flow computation capabi `Flownode` support both `standalone` and `distributed` mode. In `standalone` mode, `Flownode` runs in the same process as the database. In `distributed` mode, `Flownode` runs in a separate process and communicates with the database through the network. There are two execution modes for a flow: -- **Batching Mode**: The active mode for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. Aggregation and TQL queries use this mode. For more details, see the [Batching Mode Developer Guide](./batching_mode.md). +- **Batching Mode**: The active mode for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. Aggregation and TQL queries use this mode. For more details, see [Batching Mode](./batching_mode.md). - **Streaming Mode (deprecated)**: The original mode where data is processed as it arrives. It is kept for legacy compatibility and is not recommended for new workloads. ## Components diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md index 8e2ecdb80b..4acf86c3c9 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md @@ -3,7 +3,7 @@ keywords: [批处理模式, flow 管理, Flownode 组件, Flownode 限制, 持 description: Flownode 批处理模式概述,这是持续数据聚合当前使用的执行模式,包括其架构和查询执行流程。 --- -# Flownode 批处理模式开发者指南 +# 批处理模式 本指南简要概述了 `flownode` 中的批处理模式。它旨在帮助希望了解此模式内部工作原理的开发人员。 @@ -22,7 +22,16 @@ description: Flownode 批处理模式概述,这是持续数据聚合当前使 批处理模式由几个协同工作的关键组件组成,以实现这种持续聚合。如下图所示: -![batching mode architecture](/batching_mode_arch.png) +```mermaid +flowchart TB + SRC["源表"] -->|"写入"| ENG["BatchingEngine"] + ENG -->|"标记受影响的时间窗口"| DTW("DirtyTimeWindows
每个任务的待处理区间") + DTW -->|"在下次求值时读取"| RUN["BatchingTask
按调度或轮询节奏运行"] + RUN -.->|"清除已处理窗口"| DTW + RUN -->|"INSERT INTO sink SELECT
带时间谓词"| FE["Frontend"] + FE -->|"扫描"| SRC + FE -->|"按时间窗口 upsert"| SINK["目标表"] +``` ### `BatchingEngine` diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md index 95def5e844..159b8a5f26 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md @@ -22,7 +22,7 @@ Batching mode 复用 GreptimeDB 的查询引擎,不需要为每一行输入维 4. 查询结果写入 sink table,更新已重新计算窗口对应的物化结果。 5. 成功处理的窗口从 dirty set 中移除;执行失败的工作仍可在后续调度中处理。 -设置了 evaluation interval、但查询中没有时间窗口表达式的 Flow,会在每次调度时执行完整查询。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见 [Flownode 批处理模式开发者指南](./batching_mode.md)。 +设置了 evaluation interval、但查询中没有时间窗口表达式的 Flow,会在每次调度时执行完整查询。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见[批处理模式](./batching_mode.md)。 ## Streaming mode diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/overview.md index a093e564b4..da9753cbfd 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/overview.md @@ -13,7 +13,7 @@ description: Flownode 概览,一个为数据库提供 Flow 计算能力的组 `Flownode` 支持 `standalone`(单机)和 `distributed`(分布式)两种模式。在 `standalone` 模式下,`Flownode` 与数据库运行在同一进程中。在 `distributed` 模式下,`Flownode` 运行在单独的进程中,并通过网络与数据库通信。 一个 flow 有两种执行模式: -- **批处理模式 (Batching Mode)**: 持续数据聚合当前使用的模式。它在离散的、微小的时间窗口上周期性地执行用户定义的 SQL 查询。聚合和 TQL 查询使用此模式。更多详情,请参阅[批处理模式开发者指南](./batching_mode.md)。 +- **批处理模式 (Batching Mode)**: 持续数据聚合当前使用的模式。它在离散的、微小的时间窗口上周期性地执行用户定义的 SQL 查询。聚合和 TQL 查询使用此模式。更多详情,请参阅[批处理模式](./batching_mode.md)。 - **流处理模式 (Streaming Mode,已废弃)**: 原始的模式,数据在到达时即被处理。该模式保留用于兼容旧 workload,不推荐新 workload 使用。 ## 组件 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/batching_mode.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/batching_mode.md index f83c4e5da1..421121b69e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/batching_mode.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/batching_mode.md @@ -3,7 +3,7 @@ keywords: [流处理, flow 管理, Flownode 组件, Flownode 限制, 批处理 description: Flownode 批处理模式概述,一个为数据库提供持续数据聚合能力的组件,包括其架构和查询执行流程。 --- -# Flownode 批处理模式开发者指南 +# 批处理模式 本指南简要概述了 `flownode` 中的批处理模式。它旨在帮助希望了解此模式内部工作原理的开发人员。 @@ -22,7 +22,16 @@ description: Flownode 批处理模式概述,一个为数据库提供持续数 批处理模式由几个协同工作的关键组件组成,以实现这种持续聚合。如下图所示: -![batching mode architecture](/batching_mode_arch.png) +```mermaid +flowchart TB + SRC["源表"] -->|"写入"| ENG["BatchingEngine"] + ENG -->|"标记受影响的时间窗口"| DTW("DirtyTimeWindows
每个任务的待处理区间") + DTW -->|"在下次求值时读取"| RUN["BatchingTask
按调度或轮询节奏运行"] + RUN -.->|"清除已处理窗口"| DTW + RUN -->|"INSERT INTO sink SELECT
带时间谓词"| FE["Frontend"] + FE -->|"扫描"| SRC + FE -->|"按时间窗口 upsert"| SINK["目标表"] +``` ### `BatchingEngine` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/dataflow.md index ee17b9c785..59555f8bcf 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/dataflow.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/dataflow.md @@ -22,7 +22,7 @@ Batching mode 复用 GreptimeDB 的查询引擎,不需要为每一行输入维 4. 查询结果写入 sink table,更新已重新计算窗口对应的物化结果。 5. 成功处理的窗口从 dirty set 中移除;执行失败的工作仍可在后续调度中处理。 -设置了 evaluation interval、但查询中没有时间窗口表达式的 Flow,会在每次调度时执行完整查询。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见 [Flownode 批处理模式开发者指南](./batching_mode.md)。 +设置了 evaluation interval、但查询中没有时间窗口表达式的 Flow,会在每次调度时执行完整查询。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见[批处理模式](./batching_mode.md)。 ## Streaming mode diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/overview.md index b9759e9f42..851ccf3dec 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/overview.md @@ -14,7 +14,7 @@ description: Flownode 概览,一个为数据库提供流处理能力的组件 一个 flow 有两种执行模式: - **流处理模式 (Streaming Mode)**: 原始的模式,数据在到达时即被处理。 -- **批处理模式 (Batching Mode)**: 一种为持续数据聚合设计的较新模式。它在离散的、微小的时间窗口上周期性地执行用户定义的 SQL 查询。目前所有的聚合查询都使用此模式。更多详情,请参阅[批处理模式开发者指南](./batching_mode.md)。 +- **批处理模式 (Batching Mode)**: 一种为持续数据聚合设计的较新模式。它在离散的、微小的时间窗口上周期性地执行用户定义的 SQL 查询。目前所有的聚合查询都使用此模式。更多详情,请参阅[批处理模式](./batching_mode.md)。 ## 组件 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/batching_mode.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/batching_mode.md index 8e2ecdb80b..4acf86c3c9 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/batching_mode.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/batching_mode.md @@ -3,7 +3,7 @@ keywords: [批处理模式, flow 管理, Flownode 组件, Flownode 限制, 持 description: Flownode 批处理模式概述,这是持续数据聚合当前使用的执行模式,包括其架构和查询执行流程。 --- -# Flownode 批处理模式开发者指南 +# 批处理模式 本指南简要概述了 `flownode` 中的批处理模式。它旨在帮助希望了解此模式内部工作原理的开发人员。 @@ -22,7 +22,16 @@ description: Flownode 批处理模式概述,这是持续数据聚合当前使 批处理模式由几个协同工作的关键组件组成,以实现这种持续聚合。如下图所示: -![batching mode architecture](/batching_mode_arch.png) +```mermaid +flowchart TB + SRC["源表"] -->|"写入"| ENG["BatchingEngine"] + ENG -->|"标记受影响的时间窗口"| DTW("DirtyTimeWindows
每个任务的待处理区间") + DTW -->|"在下次求值时读取"| RUN["BatchingTask
按调度或轮询节奏运行"] + RUN -.->|"清除已处理窗口"| DTW + RUN -->|"INSERT INTO sink SELECT
带时间谓词"| FE["Frontend"] + FE -->|"扫描"| SRC + FE -->|"按时间窗口 upsert"| SINK["目标表"] +``` ### `BatchingEngine` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/dataflow.md index 95def5e844..159b8a5f26 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/dataflow.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/dataflow.md @@ -22,7 +22,7 @@ Batching mode 复用 GreptimeDB 的查询引擎,不需要为每一行输入维 4. 查询结果写入 sink table,更新已重新计算窗口对应的物化结果。 5. 成功处理的窗口从 dirty set 中移除;执行失败的工作仍可在后续调度中处理。 -设置了 evaluation interval、但查询中没有时间窗口表达式的 Flow,会在每次调度时执行完整查询。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见 [Flownode 批处理模式开发者指南](./batching_mode.md)。 +设置了 evaluation interval、但查询中没有时间窗口表达式的 Flow,会在每次调度时执行完整查询。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见[批处理模式](./batching_mode.md)。 ## Streaming mode diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/overview.md index a093e564b4..da9753cbfd 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/overview.md @@ -13,7 +13,7 @@ description: Flownode 概览,一个为数据库提供 Flow 计算能力的组 `Flownode` 支持 `standalone`(单机)和 `distributed`(分布式)两种模式。在 `standalone` 模式下,`Flownode` 与数据库运行在同一进程中。在 `distributed` 模式下,`Flownode` 运行在单独的进程中,并通过网络与数据库通信。 一个 flow 有两种执行模式: -- **批处理模式 (Batching Mode)**: 持续数据聚合当前使用的模式。它在离散的、微小的时间窗口上周期性地执行用户定义的 SQL 查询。聚合和 TQL 查询使用此模式。更多详情,请参阅[批处理模式开发者指南](./batching_mode.md)。 +- **批处理模式 (Batching Mode)**: 持续数据聚合当前使用的模式。它在离散的、微小的时间窗口上周期性地执行用户定义的 SQL 查询。聚合和 TQL 查询使用此模式。更多详情,请参阅[批处理模式](./batching_mode.md)。 - **流处理模式 (Streaming Mode,已废弃)**: 原始的模式,数据在到达时即被处理。该模式保留用于兼容旧 workload,不推荐新 workload 使用。 ## 组件 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/batching_mode.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/batching_mode.md index 8e2ecdb80b..4acf86c3c9 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/batching_mode.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/batching_mode.md @@ -3,7 +3,7 @@ keywords: [批处理模式, flow 管理, Flownode 组件, Flownode 限制, 持 description: Flownode 批处理模式概述,这是持续数据聚合当前使用的执行模式,包括其架构和查询执行流程。 --- -# Flownode 批处理模式开发者指南 +# 批处理模式 本指南简要概述了 `flownode` 中的批处理模式。它旨在帮助希望了解此模式内部工作原理的开发人员。 @@ -22,7 +22,16 @@ description: Flownode 批处理模式概述,这是持续数据聚合当前使 批处理模式由几个协同工作的关键组件组成,以实现这种持续聚合。如下图所示: -![batching mode architecture](/batching_mode_arch.png) +```mermaid +flowchart TB + SRC["源表"] -->|"写入"| ENG["BatchingEngine"] + ENG -->|"标记受影响的时间窗口"| DTW("DirtyTimeWindows
每个任务的待处理区间") + DTW -->|"在下次求值时读取"| RUN["BatchingTask
按调度或轮询节奏运行"] + RUN -.->|"清除已处理窗口"| DTW + RUN -->|"INSERT INTO sink SELECT
带时间谓词"| FE["Frontend"] + FE -->|"扫描"| SRC + FE -->|"按时间窗口 upsert"| SINK["目标表"] +``` ### `BatchingEngine` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/dataflow.md index 95def5e844..159b8a5f26 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/dataflow.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/dataflow.md @@ -22,7 +22,7 @@ Batching mode 复用 GreptimeDB 的查询引擎,不需要为每一行输入维 4. 查询结果写入 sink table,更新已重新计算窗口对应的物化结果。 5. 成功处理的窗口从 dirty set 中移除;执行失败的工作仍可在后续调度中处理。 -设置了 evaluation interval、但查询中没有时间窗口表达式的 Flow,会在每次调度时执行完整查询。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见 [Flownode 批处理模式开发者指南](./batching_mode.md)。 +设置了 evaluation interval、但查询中没有时间窗口表达式的 Flow,会在每次调度时执行完整查询。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见[批处理模式](./batching_mode.md)。 ## Streaming mode diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/overview.md index a093e564b4..da9753cbfd 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/overview.md @@ -13,7 +13,7 @@ description: Flownode 概览,一个为数据库提供 Flow 计算能力的组 `Flownode` 支持 `standalone`(单机)和 `distributed`(分布式)两种模式。在 `standalone` 模式下,`Flownode` 与数据库运行在同一进程中。在 `distributed` 模式下,`Flownode` 运行在单独的进程中,并通过网络与数据库通信。 一个 flow 有两种执行模式: -- **批处理模式 (Batching Mode)**: 持续数据聚合当前使用的模式。它在离散的、微小的时间窗口上周期性地执行用户定义的 SQL 查询。聚合和 TQL 查询使用此模式。更多详情,请参阅[批处理模式开发者指南](./batching_mode.md)。 +- **批处理模式 (Batching Mode)**: 持续数据聚合当前使用的模式。它在离散的、微小的时间窗口上周期性地执行用户定义的 SQL 查询。聚合和 TQL 查询使用此模式。更多详情,请参阅[批处理模式](./batching_mode.md)。 - **流处理模式 (Streaming Mode,已废弃)**: 原始的模式,数据在到达时即被处理。该模式保留用于兼容旧 workload,不推荐新 workload 使用。 ## 组件 diff --git a/versioned_docs/version-1.0/contributor-guide/flownode/batching_mode.md b/versioned_docs/version-1.0/contributor-guide/flownode/batching_mode.md index d90dc87bfd..7131a44678 100644 --- a/versioned_docs/version-1.0/contributor-guide/flownode/batching_mode.md +++ b/versioned_docs/version-1.0/contributor-guide/flownode/batching_mode.md @@ -3,7 +3,7 @@ keywords: [streaming process, flow management, Flownode components, Flownode lim description: Overview of Flownode's batching mode, a component providing continuous data aggregation capabilities to the database, including its architecture and query execution flow. --- -# Flownode Batching Mode Developer Guide +# Batching Mode This guide provides a brief overview of the batching mode in `flownode`. It's intended for developers who want to understand the internal workings of this mode. @@ -22,7 +22,16 @@ The core idea is to: The batching mode consists of several key components that work together to achieve this continuous aggregation. As shown in the diagram below: -![batching mode architecture](/batching_mode_arch.png) +```mermaid +flowchart TB + SRC["Source table"] -->|"write"| ENG["BatchingEngine"] + ENG -->|"mark affected windows"| DTW("DirtyTimeWindows
pending ranges per task") + DTW -->|"read at next evaluation"| RUN["BatchingTask
schedule or polling cadence"] + RUN -.->|"clear processed"| DTW + RUN -->|"INSERT INTO sink SELECT
with time predicates"| FE["Frontend"] + FE -->|"scan"| SRC + FE -->|"upsert by time window"| SINK["Sink table"] +``` ### `BatchingEngine` diff --git a/versioned_docs/version-1.0/contributor-guide/flownode/dataflow.md b/versioned_docs/version-1.0/contributor-guide/flownode/dataflow.md index 8000f14166..08b21b4f4b 100644 --- a/versioned_docs/version-1.0/contributor-guide/flownode/dataflow.md +++ b/versioned_docs/version-1.0/contributor-guide/flownode/dataflow.md @@ -22,7 +22,7 @@ Batching mode reuses GreptimeDB's query engine instead of maintaining an operato 4. The query result is inserted into the sink table, updating the materialized result for windows that were evaluated. 5. Successfully processed windows are removed from the dirty set. Failed work remains available for a later evaluation. -Flows with an evaluation interval but without a time-window expression run the complete query on each scheduled evaluation. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Flownode Batching Mode Developer Guide](./batching_mode.md) for the task and dirty-window components. +Flows with an evaluation interval but without a time-window expression run the complete query on each scheduled evaluation. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Batching Mode](./batching_mode.md) for the task and dirty-window components. ## Streaming mode diff --git a/versioned_docs/version-1.0/contributor-guide/flownode/overview.md b/versioned_docs/version-1.0/contributor-guide/flownode/overview.md index 1a0be544d2..0e784478ca 100644 --- a/versioned_docs/version-1.0/contributor-guide/flownode/overview.md +++ b/versioned_docs/version-1.0/contributor-guide/flownode/overview.md @@ -15,7 +15,7 @@ description: Overview of Flownode, a component providing streaming process capab There are two execution modes for a flow: - **Streaming Mode**: The original mode where data is processed as it arrives. -- **Batching Mode**: A newer mode designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. All aggregation queries now use this mode. For more details, see the [Batching Mode Developer Guide](./batching_mode.md). +- **Batching Mode**: A newer mode designed for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. All aggregation queries now use this mode. For more details, see [Batching Mode](./batching_mode.md). ## Components diff --git a/versioned_docs/version-1.1/contributor-guide/flownode/batching_mode.md b/versioned_docs/version-1.1/contributor-guide/flownode/batching_mode.md index aa8099d1ad..422c918491 100644 --- a/versioned_docs/version-1.1/contributor-guide/flownode/batching_mode.md +++ b/versioned_docs/version-1.1/contributor-guide/flownode/batching_mode.md @@ -3,7 +3,7 @@ keywords: [batching mode, flow management, Flownode components, Flownode limitat description: Overview of Flownode's batching mode, the active execution mode for continuous data aggregation, including its architecture and query execution flow. --- -# Flownode Batching Mode Developer Guide +# Batching Mode This guide provides a brief overview of the batching mode in `flownode`. It's intended for developers who want to understand the internal workings of this mode. @@ -22,7 +22,16 @@ The core idea is to: The batching mode consists of several key components that work together to achieve this continuous aggregation. As shown in the diagram below: -![batching mode architecture](/batching_mode_arch.png) +```mermaid +flowchart TB + SRC["Source table"] -->|"write"| ENG["BatchingEngine"] + ENG -->|"mark affected windows"| DTW("DirtyTimeWindows
pending ranges per task") + DTW -->|"read at next evaluation"| RUN["BatchingTask
schedule or polling cadence"] + RUN -.->|"clear processed"| DTW + RUN -->|"INSERT INTO sink SELECT
with time predicates"| FE["Frontend"] + FE -->|"scan"| SRC + FE -->|"upsert by time window"| SINK["Sink table"] +``` ### `BatchingEngine` diff --git a/versioned_docs/version-1.1/contributor-guide/flownode/dataflow.md b/versioned_docs/version-1.1/contributor-guide/flownode/dataflow.md index c876054d6d..479ea3c74a 100644 --- a/versioned_docs/version-1.1/contributor-guide/flownode/dataflow.md +++ b/versioned_docs/version-1.1/contributor-guide/flownode/dataflow.md @@ -22,7 +22,7 @@ Batching mode reuses GreptimeDB's query engine instead of maintaining an operato 4. The query result is inserted into the sink table, updating the materialized result for windows that were evaluated. 5. Successfully processed windows are removed from the dirty set. Failed work remains available for a later evaluation. -Flows with an evaluation interval but without a time-window expression run the complete query on each scheduled evaluation. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Flownode Batching Mode Developer Guide](./batching_mode.md) for the task and dirty-window components. +Flows with an evaluation interval but without a time-window expression run the complete query on each scheduled evaluation. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Batching Mode](./batching_mode.md) for the task and dirty-window components. ## Streaming mode diff --git a/versioned_docs/version-1.1/contributor-guide/flownode/overview.md b/versioned_docs/version-1.1/contributor-guide/flownode/overview.md index d1317bfc1c..f7da792f3e 100644 --- a/versioned_docs/version-1.1/contributor-guide/flownode/overview.md +++ b/versioned_docs/version-1.1/contributor-guide/flownode/overview.md @@ -14,7 +14,7 @@ description: Overview of Flownode, a component providing Flow computation capabi `Flownode` support both `standalone` and `distributed` mode. In `standalone` mode, `Flownode` runs in the same process as the database. In `distributed` mode, `Flownode` runs in a separate process and communicates with the database through the network. There are two execution modes for a flow: -- **Batching Mode**: The active mode for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. Aggregation and TQL queries use this mode. For more details, see the [Batching Mode Developer Guide](./batching_mode.md). +- **Batching Mode**: The active mode for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. Aggregation and TQL queries use this mode. For more details, see [Batching Mode](./batching_mode.md). - **Streaming Mode (deprecated)**: The original mode where data is processed as it arrives. It is kept for legacy compatibility and is not recommended for new workloads. ## Components diff --git a/versioned_docs/version-1.2/contributor-guide/flownode/batching_mode.md b/versioned_docs/version-1.2/contributor-guide/flownode/batching_mode.md index aa8099d1ad..422c918491 100644 --- a/versioned_docs/version-1.2/contributor-guide/flownode/batching_mode.md +++ b/versioned_docs/version-1.2/contributor-guide/flownode/batching_mode.md @@ -3,7 +3,7 @@ keywords: [batching mode, flow management, Flownode components, Flownode limitat description: Overview of Flownode's batching mode, the active execution mode for continuous data aggregation, including its architecture and query execution flow. --- -# Flownode Batching Mode Developer Guide +# Batching Mode This guide provides a brief overview of the batching mode in `flownode`. It's intended for developers who want to understand the internal workings of this mode. @@ -22,7 +22,16 @@ The core idea is to: The batching mode consists of several key components that work together to achieve this continuous aggregation. As shown in the diagram below: -![batching mode architecture](/batching_mode_arch.png) +```mermaid +flowchart TB + SRC["Source table"] -->|"write"| ENG["BatchingEngine"] + ENG -->|"mark affected windows"| DTW("DirtyTimeWindows
pending ranges per task") + DTW -->|"read at next evaluation"| RUN["BatchingTask
schedule or polling cadence"] + RUN -.->|"clear processed"| DTW + RUN -->|"INSERT INTO sink SELECT
with time predicates"| FE["Frontend"] + FE -->|"scan"| SRC + FE -->|"upsert by time window"| SINK["Sink table"] +``` ### `BatchingEngine` diff --git a/versioned_docs/version-1.2/contributor-guide/flownode/dataflow.md b/versioned_docs/version-1.2/contributor-guide/flownode/dataflow.md index c876054d6d..479ea3c74a 100644 --- a/versioned_docs/version-1.2/contributor-guide/flownode/dataflow.md +++ b/versioned_docs/version-1.2/contributor-guide/flownode/dataflow.md @@ -22,7 +22,7 @@ Batching mode reuses GreptimeDB's query engine instead of maintaining an operato 4. The query result is inserted into the sink table, updating the materialized result for windows that were evaluated. 5. Successfully processed windows are removed from the dirty set. Failed work remains available for a later evaluation. -Flows with an evaluation interval but without a time-window expression run the complete query on each scheduled evaluation. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Flownode Batching Mode Developer Guide](./batching_mode.md) for the task and dirty-window components. +Flows with an evaluation interval but without a time-window expression run the complete query on each scheduled evaluation. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Batching Mode](./batching_mode.md) for the task and dirty-window components. ## Streaming mode diff --git a/versioned_docs/version-1.2/contributor-guide/flownode/overview.md b/versioned_docs/version-1.2/contributor-guide/flownode/overview.md index d1317bfc1c..f7da792f3e 100644 --- a/versioned_docs/version-1.2/contributor-guide/flownode/overview.md +++ b/versioned_docs/version-1.2/contributor-guide/flownode/overview.md @@ -14,7 +14,7 @@ description: Overview of Flownode, a component providing Flow computation capabi `Flownode` support both `standalone` and `distributed` mode. In `standalone` mode, `Flownode` runs in the same process as the database. In `distributed` mode, `Flownode` runs in a separate process and communicates with the database through the network. There are two execution modes for a flow: -- **Batching Mode**: The active mode for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. Aggregation and TQL queries use this mode. For more details, see the [Batching Mode Developer Guide](./batching_mode.md). +- **Batching Mode**: The active mode for continuous data aggregation. It periodically executes a user-defined SQL query over small, discrete time windows. Aggregation and TQL queries use this mode. For more details, see [Batching Mode](./batching_mode.md). - **Streaming Mode (deprecated)**: The original mode where data is processed as it arrives. It is kept for legacy compatibility and is not recommended for new workloads. ## Components From fc068228c0c57997b2d49ce7b94194116d7c4269 Mon Sep 17 00:00:00 2001 From: Dennis Zhuang Date: Thu, 3 Sep 2026 11:24:41 +0800 Subject: [PATCH 2/3] docs: diagram how a Flow picks its execution mode The prose listed the rules without their precedence, so a reader could conclude that an aggregating Flow on a source table with ttl = 'instant' runs in batching mode. determine_flow_type checks the source tables first: a missing source table needs defer_on_missing_source and yields a pending batching Flow, an instant-TTL source forces streaming, and only then are TQL and Aggregate/Distinct considered. Replace the paragraph with a flowchart in that order, and state the instant-TTL consequence explicitly. v1.0 gets a shorter diagram: defer_on_missing_source does not exist in v1.0.2, where an instant-TTL source returns streaming directly from the source-table loop. --- docs/contributor-guide/flownode/dataflow.md | 18 +++++++++++++++++- .../contributor-guide/flownode/dataflow.md | 18 +++++++++++++++++- .../contributor-guide/flownode/dataflow.md | 15 ++++++++++++++- .../contributor-guide/flownode/dataflow.md | 18 +++++++++++++++++- .../contributor-guide/flownode/dataflow.md | 18 +++++++++++++++++- .../contributor-guide/flownode/dataflow.md | 15 ++++++++++++++- .../contributor-guide/flownode/dataflow.md | 18 +++++++++++++++++- .../contributor-guide/flownode/dataflow.md | 18 +++++++++++++++++- 8 files changed, 130 insertions(+), 8 deletions(-) diff --git a/docs/contributor-guide/flownode/dataflow.md b/docs/contributor-guide/flownode/dataflow.md index 479ea3c74a..4aa4443d0f 100644 --- a/docs/contributor-guide/flownode/dataflow.md +++ b/docs/contributor-guide/flownode/dataflow.md @@ -10,7 +10,23 @@ Flownode has two internal execution paths: - **Batching mode** is the primary path for aggregation and TQL workloads. It evaluates queries over persisted source data and writes materialized results to a sink table. - **Streaming mode** is the legacy path retained for compatibility and deprecated for new workloads. It incrementally processes rows mirrored from Frontend as they arrive. -Users do not select the mode directly. When a Flow is created, GreptimeDB chooses the path from the query and source-table properties. Aggregation, `DISTINCT`, and TQL queries use batching mode. Simple non-aggregation queries, and any Flow whose source table has `ttl = 'instant'`, currently use streaming mode. A Flow deferred because its source table does not yet exist starts as a pending batching Flow. +Users do not select the mode directly. When a Flow is created, GreptimeDB derives the path from the source tables and then the query, in this order: + +```mermaid +flowchart LR + MISS{"any source table
missing?"} + MISS -->|"yes, deferred"| PEND["pending
batching Flow"] + MISS -->|"yes, otherwise"| ERR["rejected"] + MISS -->|"no"| TTL{"any source table
ttl = 'instant'?"} + TTL -->|"yes"| STREAM["Streaming mode"] + TTL -->|"no"| TQL{"TQL query?"} + TQL -->|"yes"| BATCH["Batching mode"] + TQL -->|"no"| AGG{"plan has Aggregate
or Distinct?"} + AGG -->|"yes"| BATCH + AGG -->|"no"| STREAM +``` + +The source-table checks come first, so a Flow that aggregates a source table with `ttl = 'instant'` runs in streaming mode rather than batching mode. A Flow whose source table does not exist yet is rejected unless it is created with `WITH (defer_on_missing_source = true)`, which produces a pending batching Flow. ## Batching mode diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md index 159b8a5f26..645cf0d699 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md @@ -10,7 +10,23 @@ Flownode 内部有两条执行路径: - **Batching mode** 是聚合和 TQL workload 的主要执行路径。它查询已经持久化的 source 数据,并将物化结果写入 sink table。 - **Streaming mode** 是为兼容已有 workload 而保留的旧执行路径,不推荐新 workload 使用。Frontend 会把新到达的行同步给它进行增量处理。 -用户不能直接选择执行模式。创建 Flow 时,GreptimeDB 根据查询和 source table 的属性选择执行路径。聚合、`DISTINCT` 和 TQL 查询使用 batching mode;简单的非聚合查询,以及任何 source table 使用 `ttl = 'instant'` 的 Flow,目前仍使用 streaming mode。如果 source table 尚不存在并选择延迟创建,Flow 会先成为 pending batching Flow。 +用户不能直接选择执行模式。创建 Flow 时,GreptimeDB 先根据 source table、再根据查询判定执行路径,顺序如下: + +```mermaid +flowchart LR + MISS{"存在缺失的源表?"} + MISS -->|"是,已声明延迟创建"| PEND["pending
batching Flow"] + MISS -->|"是,其余情况"| ERR["拒绝创建"] + MISS -->|"否"| TTL{"存在 ttl = 'instant'
的源表?"} + TTL -->|"是"| STREAM["Streaming mode"] + TTL -->|"否"| TQL{"TQL 查询?"} + TQL -->|"是"| BATCH["Batching mode"] + TQL -->|"否"| AGG{"计划含 Aggregate
或 Distinct?"} + AGG -->|"是"| BATCH + AGG -->|"否"| STREAM +``` + +source table 的判定在前,因此聚合 `ttl = 'instant'` source table 的 Flow 走的是 streaming mode,而不是 batching mode。source table 尚不存在时,Flow 会被拒绝创建,除非建表时声明 `WITH (defer_on_missing_source = true)`,此时会得到一个 pending batching Flow。 ## Batching mode diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/dataflow.md index 59555f8bcf..8136036bef 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/dataflow.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/dataflow.md @@ -10,7 +10,20 @@ Flownode 内部有两条执行路径: - **Batching mode** 是聚合和 TQL workload 的主要执行路径。它查询已经持久化的 source 数据,并将物化结果写入 sink table。 - **Streaming mode** 是为兼容已有 workload 而保留的旧执行路径,不推荐新 workload 使用。Frontend 会把新到达的行同步给它进行增量处理。 -用户不能直接选择执行模式。创建 Flow 时,GreptimeDB 根据查询和 source table 的属性选择执行路径。聚合、`DISTINCT` 和 TQL 查询使用 batching mode;简单的非聚合查询,以及任何 source table 使用 `ttl = 'instant'` 的 Flow,目前仍使用 streaming mode。 +用户不能直接选择执行模式。创建 Flow 时,GreptimeDB 先根据 source table、再根据查询判定执行路径,顺序如下: + +```mermaid +flowchart LR + TTL{"存在 ttl = 'instant'
的源表?"} + TTL -->|"是"| STREAM["Streaming mode"] + TTL -->|"否"| TQL{"TQL 查询?"} + TQL -->|"是"| BATCH["Batching mode"] + TQL -->|"否"| AGG{"计划含 Aggregate
或 Distinct?"} + AGG -->|"是"| BATCH + AGG -->|"否"| STREAM +``` + +source table 的判定在前,因此聚合 `ttl = 'instant'` source table 的 Flow 走的是 streaming mode,而不是 batching mode。 ## Batching mode diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/dataflow.md index 159b8a5f26..645cf0d699 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/dataflow.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/dataflow.md @@ -10,7 +10,23 @@ Flownode 内部有两条执行路径: - **Batching mode** 是聚合和 TQL workload 的主要执行路径。它查询已经持久化的 source 数据,并将物化结果写入 sink table。 - **Streaming mode** 是为兼容已有 workload 而保留的旧执行路径,不推荐新 workload 使用。Frontend 会把新到达的行同步给它进行增量处理。 -用户不能直接选择执行模式。创建 Flow 时,GreptimeDB 根据查询和 source table 的属性选择执行路径。聚合、`DISTINCT` 和 TQL 查询使用 batching mode;简单的非聚合查询,以及任何 source table 使用 `ttl = 'instant'` 的 Flow,目前仍使用 streaming mode。如果 source table 尚不存在并选择延迟创建,Flow 会先成为 pending batching Flow。 +用户不能直接选择执行模式。创建 Flow 时,GreptimeDB 先根据 source table、再根据查询判定执行路径,顺序如下: + +```mermaid +flowchart LR + MISS{"存在缺失的源表?"} + MISS -->|"是,已声明延迟创建"| PEND["pending
batching Flow"] + MISS -->|"是,其余情况"| ERR["拒绝创建"] + MISS -->|"否"| TTL{"存在 ttl = 'instant'
的源表?"} + TTL -->|"是"| STREAM["Streaming mode"] + TTL -->|"否"| TQL{"TQL 查询?"} + TQL -->|"是"| BATCH["Batching mode"] + TQL -->|"否"| AGG{"计划含 Aggregate
或 Distinct?"} + AGG -->|"是"| BATCH + AGG -->|"否"| STREAM +``` + +source table 的判定在前,因此聚合 `ttl = 'instant'` source table 的 Flow 走的是 streaming mode,而不是 batching mode。source table 尚不存在时,Flow 会被拒绝创建,除非建表时声明 `WITH (defer_on_missing_source = true)`,此时会得到一个 pending batching Flow。 ## Batching mode diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/dataflow.md index 159b8a5f26..645cf0d699 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/dataflow.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/dataflow.md @@ -10,7 +10,23 @@ Flownode 内部有两条执行路径: - **Batching mode** 是聚合和 TQL workload 的主要执行路径。它查询已经持久化的 source 数据,并将物化结果写入 sink table。 - **Streaming mode** 是为兼容已有 workload 而保留的旧执行路径,不推荐新 workload 使用。Frontend 会把新到达的行同步给它进行增量处理。 -用户不能直接选择执行模式。创建 Flow 时,GreptimeDB 根据查询和 source table 的属性选择执行路径。聚合、`DISTINCT` 和 TQL 查询使用 batching mode;简单的非聚合查询,以及任何 source table 使用 `ttl = 'instant'` 的 Flow,目前仍使用 streaming mode。如果 source table 尚不存在并选择延迟创建,Flow 会先成为 pending batching Flow。 +用户不能直接选择执行模式。创建 Flow 时,GreptimeDB 先根据 source table、再根据查询判定执行路径,顺序如下: + +```mermaid +flowchart LR + MISS{"存在缺失的源表?"} + MISS -->|"是,已声明延迟创建"| PEND["pending
batching Flow"] + MISS -->|"是,其余情况"| ERR["拒绝创建"] + MISS -->|"否"| TTL{"存在 ttl = 'instant'
的源表?"} + TTL -->|"是"| STREAM["Streaming mode"] + TTL -->|"否"| TQL{"TQL 查询?"} + TQL -->|"是"| BATCH["Batching mode"] + TQL -->|"否"| AGG{"计划含 Aggregate
或 Distinct?"} + AGG -->|"是"| BATCH + AGG -->|"否"| STREAM +``` + +source table 的判定在前,因此聚合 `ttl = 'instant'` source table 的 Flow 走的是 streaming mode,而不是 batching mode。source table 尚不存在时,Flow 会被拒绝创建,除非建表时声明 `WITH (defer_on_missing_source = true)`,此时会得到一个 pending batching Flow。 ## Batching mode diff --git a/versioned_docs/version-1.0/contributor-guide/flownode/dataflow.md b/versioned_docs/version-1.0/contributor-guide/flownode/dataflow.md index 08b21b4f4b..76ce721fad 100644 --- a/versioned_docs/version-1.0/contributor-guide/flownode/dataflow.md +++ b/versioned_docs/version-1.0/contributor-guide/flownode/dataflow.md @@ -10,7 +10,20 @@ Flownode has two internal execution paths: - **Batching mode** is the primary path for aggregation and TQL workloads. It evaluates queries over persisted source data and writes materialized results to a sink table. - **Streaming mode** is the legacy path retained for compatibility and deprecated for new workloads. It incrementally processes rows mirrored from Frontend as they arrive. -Users do not select the mode directly. When a Flow is created, GreptimeDB chooses the path from the query and source-table properties. Aggregation, `DISTINCT`, and TQL queries use batching mode. Simple non-aggregation queries, and any Flow whose source table has `ttl = 'instant'`, currently use streaming mode. +Users do not select the mode directly. When a Flow is created, GreptimeDB derives the path from the source tables and then the query, in this order: + +```mermaid +flowchart LR + TTL{"any source table
ttl = 'instant'?"} + TTL -->|"yes"| STREAM["Streaming mode"] + TTL -->|"no"| TQL{"TQL query?"} + TQL -->|"yes"| BATCH["Batching mode"] + TQL -->|"no"| AGG{"plan has Aggregate
or Distinct?"} + AGG -->|"yes"| BATCH + AGG -->|"no"| STREAM +``` + +The source-table check comes first, so a Flow that aggregates a source table with `ttl = 'instant'` runs in streaming mode rather than batching mode. ## Batching mode diff --git a/versioned_docs/version-1.1/contributor-guide/flownode/dataflow.md b/versioned_docs/version-1.1/contributor-guide/flownode/dataflow.md index 479ea3c74a..4aa4443d0f 100644 --- a/versioned_docs/version-1.1/contributor-guide/flownode/dataflow.md +++ b/versioned_docs/version-1.1/contributor-guide/flownode/dataflow.md @@ -10,7 +10,23 @@ Flownode has two internal execution paths: - **Batching mode** is the primary path for aggregation and TQL workloads. It evaluates queries over persisted source data and writes materialized results to a sink table. - **Streaming mode** is the legacy path retained for compatibility and deprecated for new workloads. It incrementally processes rows mirrored from Frontend as they arrive. -Users do not select the mode directly. When a Flow is created, GreptimeDB chooses the path from the query and source-table properties. Aggregation, `DISTINCT`, and TQL queries use batching mode. Simple non-aggregation queries, and any Flow whose source table has `ttl = 'instant'`, currently use streaming mode. A Flow deferred because its source table does not yet exist starts as a pending batching Flow. +Users do not select the mode directly. When a Flow is created, GreptimeDB derives the path from the source tables and then the query, in this order: + +```mermaid +flowchart LR + MISS{"any source table
missing?"} + MISS -->|"yes, deferred"| PEND["pending
batching Flow"] + MISS -->|"yes, otherwise"| ERR["rejected"] + MISS -->|"no"| TTL{"any source table
ttl = 'instant'?"} + TTL -->|"yes"| STREAM["Streaming mode"] + TTL -->|"no"| TQL{"TQL query?"} + TQL -->|"yes"| BATCH["Batching mode"] + TQL -->|"no"| AGG{"plan has Aggregate
or Distinct?"} + AGG -->|"yes"| BATCH + AGG -->|"no"| STREAM +``` + +The source-table checks come first, so a Flow that aggregates a source table with `ttl = 'instant'` runs in streaming mode rather than batching mode. A Flow whose source table does not exist yet is rejected unless it is created with `WITH (defer_on_missing_source = true)`, which produces a pending batching Flow. ## Batching mode diff --git a/versioned_docs/version-1.2/contributor-guide/flownode/dataflow.md b/versioned_docs/version-1.2/contributor-guide/flownode/dataflow.md index 479ea3c74a..4aa4443d0f 100644 --- a/versioned_docs/version-1.2/contributor-guide/flownode/dataflow.md +++ b/versioned_docs/version-1.2/contributor-guide/flownode/dataflow.md @@ -10,7 +10,23 @@ Flownode has two internal execution paths: - **Batching mode** is the primary path for aggregation and TQL workloads. It evaluates queries over persisted source data and writes materialized results to a sink table. - **Streaming mode** is the legacy path retained for compatibility and deprecated for new workloads. It incrementally processes rows mirrored from Frontend as they arrive. -Users do not select the mode directly. When a Flow is created, GreptimeDB chooses the path from the query and source-table properties. Aggregation, `DISTINCT`, and TQL queries use batching mode. Simple non-aggregation queries, and any Flow whose source table has `ttl = 'instant'`, currently use streaming mode. A Flow deferred because its source table does not yet exist starts as a pending batching Flow. +Users do not select the mode directly. When a Flow is created, GreptimeDB derives the path from the source tables and then the query, in this order: + +```mermaid +flowchart LR + MISS{"any source table
missing?"} + MISS -->|"yes, deferred"| PEND["pending
batching Flow"] + MISS -->|"yes, otherwise"| ERR["rejected"] + MISS -->|"no"| TTL{"any source table
ttl = 'instant'?"} + TTL -->|"yes"| STREAM["Streaming mode"] + TTL -->|"no"| TQL{"TQL query?"} + TQL -->|"yes"| BATCH["Batching mode"] + TQL -->|"no"| AGG{"plan has Aggregate
or Distinct?"} + AGG -->|"yes"| BATCH + AGG -->|"no"| STREAM +``` + +The source-table checks come first, so a Flow that aggregates a source table with `ttl = 'instant'` runs in streaming mode rather than batching mode. A Flow whose source table does not exist yet is rejected unless it is created with `WITH (defer_on_missing_source = true)`, which produces a pending batching Flow. ## Batching mode From 5748d9a839b584a35d34e0b18b4eeaa58f6194c9 Mon Sep 17 00:00:00 2001 From: Dennis Zhuang Date: Thu, 3 Sep 2026 18:42:55 +0800 Subject: [PATCH 3/3] docs: separate the unfiltered full-query path in batching mode gen_query_with_time_window has two paths. Windowed SQL is pruned by dirty-window time predicates; TQL and evaluation-interval SQL whose plan cannot be safely pruned run an unfiltered full snapshot, where the dirty set is only a scheduling signal. The diagram showed only the first, and the prose omitted TQL entirely. Give the diagram two labelled edges into Frontend, and rewrite the prose to name both callers. Diagram height is unchanged at 608px. v1.0 keeps its own wording: there TQL always runs the full query, while no-window SQL clears the dirty set first and skips the evaluation when nothing is dirty. --- docs/contributor-guide/flownode/batching_mode.md | 3 ++- docs/contributor-guide/flownode/dataflow.md | 2 +- .../current/contributor-guide/flownode/batching_mode.md | 3 ++- .../current/contributor-guide/flownode/dataflow.md | 2 +- .../version-1.0/contributor-guide/flownode/batching_mode.md | 3 ++- .../version-1.0/contributor-guide/flownode/dataflow.md | 2 +- .../version-1.1/contributor-guide/flownode/batching_mode.md | 3 ++- .../version-1.1/contributor-guide/flownode/dataflow.md | 2 +- .../version-1.2/contributor-guide/flownode/batching_mode.md | 3 ++- .../version-1.2/contributor-guide/flownode/dataflow.md | 2 +- .../version-1.0/contributor-guide/flownode/batching_mode.md | 3 ++- .../version-1.0/contributor-guide/flownode/dataflow.md | 2 +- .../version-1.1/contributor-guide/flownode/batching_mode.md | 3 ++- .../version-1.1/contributor-guide/flownode/dataflow.md | 2 +- .../version-1.2/contributor-guide/flownode/batching_mode.md | 3 ++- .../version-1.2/contributor-guide/flownode/dataflow.md | 2 +- 16 files changed, 24 insertions(+), 16 deletions(-) diff --git a/docs/contributor-guide/flownode/batching_mode.md b/docs/contributor-guide/flownode/batching_mode.md index 422c918491..0384282e2d 100644 --- a/docs/contributor-guide/flownode/batching_mode.md +++ b/docs/contributor-guide/flownode/batching_mode.md @@ -28,7 +28,8 @@ flowchart TB ENG -->|"mark affected windows"| DTW("DirtyTimeWindows
pending ranges per task") DTW -->|"read at next evaluation"| RUN["BatchingTask
schedule or polling cadence"] RUN -.->|"clear processed"| DTW - RUN -->|"INSERT INTO sink SELECT
with time predicates"| FE["Frontend"] + RUN -->|"windowed: INSERT INTO sink SELECT
with time predicates"| FE["Frontend"] + RUN -->|"TQL or unprunable plan:
unfiltered full query"| FE FE -->|"scan"| SRC FE -->|"upsert by time window"| SINK["Sink table"] ``` diff --git a/docs/contributor-guide/flownode/dataflow.md b/docs/contributor-guide/flownode/dataflow.md index 4aa4443d0f..fa719bbe7b 100644 --- a/docs/contributor-guide/flownode/dataflow.md +++ b/docs/contributor-guide/flownode/dataflow.md @@ -38,7 +38,7 @@ Batching mode reuses GreptimeDB's query engine instead of maintaining an operato 4. The query result is inserted into the sink table, updating the materialized result for windows that were evaluated. 5. Successfully processed windows are removed from the dirty set. Failed work remains available for a later evaluation. -Flows with an evaluation interval but without a time-window expression run the complete query on each scheduled evaluation. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Batching Mode](./batching_mode.md) for the task and dirty-window components. +TQL Flows, and evaluation-interval Flows whose plan cannot be pruned safely by dirty windows, run the complete query rather than a time-filtered one. On that path the dirty set is only a scheduling signal. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Batching Mode](./batching_mode.md) for the task and dirty-window components. ## Streaming mode diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md index 4acf86c3c9..7d320eb57e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/batching_mode.md @@ -28,7 +28,8 @@ flowchart TB ENG -->|"标记受影响的时间窗口"| DTW("DirtyTimeWindows
每个任务的待处理区间") DTW -->|"在下次求值时读取"| RUN["BatchingTask
按调度或轮询节奏运行"] RUN -.->|"清除已处理窗口"| DTW - RUN -->|"INSERT INTO sink SELECT
带时间谓词"| FE["Frontend"] + RUN -->|"按窗口:INSERT INTO sink SELECT
带时间谓词"| FE["Frontend"] + RUN -->|"TQL 或无法裁剪的计划:
非过滤全量查询"| FE FE -->|"扫描"| SRC FE -->|"按时间窗口 upsert"| SINK["目标表"] ``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md index 645cf0d699..23ceb49c62 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor-guide/flownode/dataflow.md @@ -38,7 +38,7 @@ Batching mode 复用 GreptimeDB 的查询引擎,不需要为每一行输入维 4. 查询结果写入 sink table,更新已重新计算窗口对应的物化结果。 5. 成功处理的窗口从 dirty set 中移除;执行失败的工作仍可在后续调度中处理。 -设置了 evaluation interval、但查询中没有时间窗口表达式的 Flow,会在每次调度时执行完整查询。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见[批处理模式](./batching_mode.md)。 +TQL Flow,以及计划无法按 dirty window 安全裁剪的 evaluation interval Flow,执行的是完整查询,而不是按时间过滤的查询;这条路径上 dirty set 只作为调度信号。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见[批处理模式](./batching_mode.md)。 ## Streaming mode diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/batching_mode.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/batching_mode.md index 421121b69e..f7cb260785 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/batching_mode.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/batching_mode.md @@ -28,7 +28,8 @@ flowchart TB ENG -->|"标记受影响的时间窗口"| DTW("DirtyTimeWindows
每个任务的待处理区间") DTW -->|"在下次求值时读取"| RUN["BatchingTask
按调度或轮询节奏运行"] RUN -.->|"清除已处理窗口"| DTW - RUN -->|"INSERT INTO sink SELECT
带时间谓词"| FE["Frontend"] + RUN -->|"按窗口:INSERT INTO sink SELECT
带时间谓词"| FE["Frontend"] + RUN -->|"TQL 或无法裁剪的计划:
非过滤全量查询"| FE FE -->|"扫描"| SRC FE -->|"按时间窗口 upsert"| SINK["目标表"] ``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/dataflow.md index 8136036bef..efec67883b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/dataflow.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.0/contributor-guide/flownode/dataflow.md @@ -35,7 +35,7 @@ Batching mode 复用 GreptimeDB 的查询引擎,不需要为每一行输入维 4. 查询结果写入 sink table,更新已重新计算窗口对应的物化结果。 5. 成功处理的窗口从 dirty set 中移除;执行失败的工作仍可在后续调度中处理。 -设置了 evaluation interval、但查询中没有时间窗口表达式的 Flow,会在每次调度时执行完整查询。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见[批处理模式](./batching_mode.md)。 +TQL Flow,以及设置了 evaluation interval、但查询中没有时间窗口表达式的 SQL Flow,执行的是完整查询,而不是按时间过滤的查询;其中 SQL 的那种情况在没有 dirty 数据时会跳过该次调度。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见[批处理模式](./batching_mode.md)。 ## Streaming mode diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/batching_mode.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/batching_mode.md index 4acf86c3c9..7d320eb57e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/batching_mode.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/batching_mode.md @@ -28,7 +28,8 @@ flowchart TB ENG -->|"标记受影响的时间窗口"| DTW("DirtyTimeWindows
每个任务的待处理区间") DTW -->|"在下次求值时读取"| RUN["BatchingTask
按调度或轮询节奏运行"] RUN -.->|"清除已处理窗口"| DTW - RUN -->|"INSERT INTO sink SELECT
带时间谓词"| FE["Frontend"] + RUN -->|"按窗口:INSERT INTO sink SELECT
带时间谓词"| FE["Frontend"] + RUN -->|"TQL 或无法裁剪的计划:
非过滤全量查询"| FE FE -->|"扫描"| SRC FE -->|"按时间窗口 upsert"| SINK["目标表"] ``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/dataflow.md index 645cf0d699..23ceb49c62 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/dataflow.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.1/contributor-guide/flownode/dataflow.md @@ -38,7 +38,7 @@ Batching mode 复用 GreptimeDB 的查询引擎,不需要为每一行输入维 4. 查询结果写入 sink table,更新已重新计算窗口对应的物化结果。 5. 成功处理的窗口从 dirty set 中移除;执行失败的工作仍可在后续调度中处理。 -设置了 evaluation interval、但查询中没有时间窗口表达式的 Flow,会在每次调度时执行完整查询。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见[批处理模式](./batching_mode.md)。 +TQL Flow,以及计划无法按 dirty window 安全裁剪的 evaluation interval Flow,执行的是完整查询,而不是按时间过滤的查询;这条路径上 dirty set 只作为调度信号。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见[批处理模式](./batching_mode.md)。 ## Streaming mode diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/batching_mode.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/batching_mode.md index 4acf86c3c9..7d320eb57e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/batching_mode.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/batching_mode.md @@ -28,7 +28,8 @@ flowchart TB ENG -->|"标记受影响的时间窗口"| DTW("DirtyTimeWindows
每个任务的待处理区间") DTW -->|"在下次求值时读取"| RUN["BatchingTask
按调度或轮询节奏运行"] RUN -.->|"清除已处理窗口"| DTW - RUN -->|"INSERT INTO sink SELECT
带时间谓词"| FE["Frontend"] + RUN -->|"按窗口:INSERT INTO sink SELECT
带时间谓词"| FE["Frontend"] + RUN -->|"TQL 或无法裁剪的计划:
非过滤全量查询"| FE FE -->|"扫描"| SRC FE -->|"按时间窗口 upsert"| SINK["目标表"] ``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/dataflow.md b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/dataflow.md index 645cf0d699..23ceb49c62 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/dataflow.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-1.2/contributor-guide/flownode/dataflow.md @@ -38,7 +38,7 @@ Batching mode 复用 GreptimeDB 的查询引擎,不需要为每一行输入维 4. 查询结果写入 sink table,更新已重新计算窗口对应的物化结果。 5. 成功处理的窗口从 dirty set 中移除;执行失败的工作仍可在后续调度中处理。 -设置了 evaluation interval、但查询中没有时间窗口表达式的 Flow,会在每次调度时执行完整查询。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见[批处理模式](./batching_mode.md)。 +TQL Flow,以及计划无法按 dirty window 安全裁剪的 evaluation interval Flow,执行的是完整查询,而不是按时间过滤的查询;这条路径上 dirty set 只作为调度信号。这条路径还可以使用 streaming renderer 尚未实现的查询引擎能力。任务和 dirty window 组件的进一步说明见[批处理模式](./batching_mode.md)。 ## Streaming mode diff --git a/versioned_docs/version-1.0/contributor-guide/flownode/batching_mode.md b/versioned_docs/version-1.0/contributor-guide/flownode/batching_mode.md index 7131a44678..d9cec42441 100644 --- a/versioned_docs/version-1.0/contributor-guide/flownode/batching_mode.md +++ b/versioned_docs/version-1.0/contributor-guide/flownode/batching_mode.md @@ -28,7 +28,8 @@ flowchart TB ENG -->|"mark affected windows"| DTW("DirtyTimeWindows
pending ranges per task") DTW -->|"read at next evaluation"| RUN["BatchingTask
schedule or polling cadence"] RUN -.->|"clear processed"| DTW - RUN -->|"INSERT INTO sink SELECT
with time predicates"| FE["Frontend"] + RUN -->|"windowed: INSERT INTO sink SELECT
with time predicates"| FE["Frontend"] + RUN -->|"TQL or unprunable plan:
unfiltered full query"| FE FE -->|"scan"| SRC FE -->|"upsert by time window"| SINK["Sink table"] ``` diff --git a/versioned_docs/version-1.0/contributor-guide/flownode/dataflow.md b/versioned_docs/version-1.0/contributor-guide/flownode/dataflow.md index 76ce721fad..f8e603fdfe 100644 --- a/versioned_docs/version-1.0/contributor-guide/flownode/dataflow.md +++ b/versioned_docs/version-1.0/contributor-guide/flownode/dataflow.md @@ -35,7 +35,7 @@ Batching mode reuses GreptimeDB's query engine instead of maintaining an operato 4. The query result is inserted into the sink table, updating the materialized result for windows that were evaluated. 5. Successfully processed windows are removed from the dirty set. Failed work remains available for a later evaluation. -Flows with an evaluation interval but without a time-window expression run the complete query on each scheduled evaluation. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Batching Mode](./batching_mode.md) for the task and dirty-window components. +TQL Flows, and SQL Flows with an evaluation interval but without a time-window expression, run the complete query rather than a time-filtered one. For the SQL case, an evaluation that finds nothing dirty is skipped. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Batching Mode](./batching_mode.md) for the task and dirty-window components. ## Streaming mode diff --git a/versioned_docs/version-1.1/contributor-guide/flownode/batching_mode.md b/versioned_docs/version-1.1/contributor-guide/flownode/batching_mode.md index 422c918491..0384282e2d 100644 --- a/versioned_docs/version-1.1/contributor-guide/flownode/batching_mode.md +++ b/versioned_docs/version-1.1/contributor-guide/flownode/batching_mode.md @@ -28,7 +28,8 @@ flowchart TB ENG -->|"mark affected windows"| DTW("DirtyTimeWindows
pending ranges per task") DTW -->|"read at next evaluation"| RUN["BatchingTask
schedule or polling cadence"] RUN -.->|"clear processed"| DTW - RUN -->|"INSERT INTO sink SELECT
with time predicates"| FE["Frontend"] + RUN -->|"windowed: INSERT INTO sink SELECT
with time predicates"| FE["Frontend"] + RUN -->|"TQL or unprunable plan:
unfiltered full query"| FE FE -->|"scan"| SRC FE -->|"upsert by time window"| SINK["Sink table"] ``` diff --git a/versioned_docs/version-1.1/contributor-guide/flownode/dataflow.md b/versioned_docs/version-1.1/contributor-guide/flownode/dataflow.md index 4aa4443d0f..fa719bbe7b 100644 --- a/versioned_docs/version-1.1/contributor-guide/flownode/dataflow.md +++ b/versioned_docs/version-1.1/contributor-guide/flownode/dataflow.md @@ -38,7 +38,7 @@ Batching mode reuses GreptimeDB's query engine instead of maintaining an operato 4. The query result is inserted into the sink table, updating the materialized result for windows that were evaluated. 5. Successfully processed windows are removed from the dirty set. Failed work remains available for a later evaluation. -Flows with an evaluation interval but without a time-window expression run the complete query on each scheduled evaluation. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Batching Mode](./batching_mode.md) for the task and dirty-window components. +TQL Flows, and evaluation-interval Flows whose plan cannot be pruned safely by dirty windows, run the complete query rather than a time-filtered one. On that path the dirty set is only a scheduling signal. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Batching Mode](./batching_mode.md) for the task and dirty-window components. ## Streaming mode diff --git a/versioned_docs/version-1.2/contributor-guide/flownode/batching_mode.md b/versioned_docs/version-1.2/contributor-guide/flownode/batching_mode.md index 422c918491..0384282e2d 100644 --- a/versioned_docs/version-1.2/contributor-guide/flownode/batching_mode.md +++ b/versioned_docs/version-1.2/contributor-guide/flownode/batching_mode.md @@ -28,7 +28,8 @@ flowchart TB ENG -->|"mark affected windows"| DTW("DirtyTimeWindows
pending ranges per task") DTW -->|"read at next evaluation"| RUN["BatchingTask
schedule or polling cadence"] RUN -.->|"clear processed"| DTW - RUN -->|"INSERT INTO sink SELECT
with time predicates"| FE["Frontend"] + RUN -->|"windowed: INSERT INTO sink SELECT
with time predicates"| FE["Frontend"] + RUN -->|"TQL or unprunable plan:
unfiltered full query"| FE FE -->|"scan"| SRC FE -->|"upsert by time window"| SINK["Sink table"] ``` diff --git a/versioned_docs/version-1.2/contributor-guide/flownode/dataflow.md b/versioned_docs/version-1.2/contributor-guide/flownode/dataflow.md index 4aa4443d0f..fa719bbe7b 100644 --- a/versioned_docs/version-1.2/contributor-guide/flownode/dataflow.md +++ b/versioned_docs/version-1.2/contributor-guide/flownode/dataflow.md @@ -38,7 +38,7 @@ Batching mode reuses GreptimeDB's query engine instead of maintaining an operato 4. The query result is inserted into the sink table, updating the materialized result for windows that were evaluated. 5. Successfully processed windows are removed from the dirty set. Failed work remains available for a later evaluation. -Flows with an evaluation interval but without a time-window expression run the complete query on each scheduled evaluation. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Batching Mode](./batching_mode.md) for the task and dirty-window components. +TQL Flows, and evaluation-interval Flows whose plan cannot be pruned safely by dirty windows, run the complete query rather than a time-filtered one. On that path the dirty set is only a scheduling signal. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Batching Mode](./batching_mode.md) for the task and dirty-window components. ## Streaming mode