Skip to content

docs: remove json path #2797

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sunng87 wants to merge 2 commits into
base: main
Choose a base branch
from
+22 −248
Open

docs: remove json path #2797

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
371ee37
doc: pipeline json type
sunng87 Aug 28, 2026
b6e1f9f
doc: remove deprecated json_path processor from pipeline reference
sunng87 Aug 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
135 changes: 11 additions & 124 deletions docs/reference/pipeline/pipeline-config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,6 @@ We currently provide the following built-in Processors:
- `regex`: performs regular expression matching on log data fields.
- `urlencoding`: performs URL encoding/decoding on log data fields.
- `csv`: parses CSV data fields in logs.
- `json_path`: extracts fields from JSON data. (**deprecated**, please use `vrl` instead)
- `json_parse`: parse a field into JSON object.
- `simple_extract`: extracts fields from JSON data using simple key.
- `digest`: extracts the template from a log message by removing variable content.
Expand Down Expand Up @@ -441,127 +440,6 @@ In the above example, the configuration of the `csv` processor includes the foll
- `trim`: Whether to trim whitespace. Defaults to `false`.
- `ignore_missing`: Ignores the case when the field is missing. Defaults to `false`. If the field is missing and this configuration is set to `false`, an exception will be thrown.

### `json_path` (deprecated)

:::danger Deprecated Feature
The `json_path` processor is deprecated. The `vrl` processor covers its use cases and provides more flexible JSON field extraction, so use `vrl` in new pipelines.
:::

The `json_path` processor is used to extract fields from JSON data. Here's an example configuration:

```yaml
processors:
- json_path:
fields:
- complex_object
json_path: "$.shop.orders[?(@.active)].id"
ignore_missing: true
result_index: 1
```

In the above example, the configuration of the `json_path` processor includes the following fields:

- `fields`: A list of field names to be extracted.
- `json_path`: The JSON path to extract.
- `ignore_missing`: Ignores the case when the field is missing. Defaults to `false`. If the field is missing and this configuration is set to `false`, an exception will be thrown.
- `result_index`: Specifies the index of the value in the extracted array to be used as the result value. By default, all values are included. The extracted value of the processor is an array containing all the values of the path. If an index is specified, the corresponding value in the extracted array will be used as the final result.

#### JSON path syntax

The JSON path syntax is based on the [jsonpath-rust](https://github.com/besok/jsonpath-rust) library.

At this stage we only recommend using some simple field extraction operations to facilitate the extraction of nested fields to the top level.

#### `json_path` example

For example, given the following log data:

```json
{
"product_object": {
"hello": "world"
},
"product_array": [
"hello",
"world"
],
"complex_object": {
"shop": {
"orders": [
{
"id": 1,
"active": true
},
{
"id": 2
},
{
"id": 3
},
{
"id": 4,
"active": true
}
]
}
}
}
```

Using the following configuration:

```yaml
processors:
- json_path:
fields:
- product_object, object_target
json_path: "$.hello"
result_index: 0
- json_path:
fields:
- product_array, array_target
json_path: "$.[1]"
result_index: 0
- json_path:
fields:
- complex_object, complex_target_1
json_path: "$.shop.orders[?(@.active)].id"
- json_path:
fields:
- complex_target_1, complex_target_2
json_path: "$.[1]"
result_index: 0
- json_path:
fields:
- complex_object, complex_target_3
json_path: "$.shop.orders[?(@.active)].id"
result_index: 1
transform:
- fields:
- object_target
- array_target
type: string
- fields:
- complex_target_3
- complex_target_2
type: uint32
- fields:
- complex_target_1
type: json
```

The result will be:

```json
{
"object_target": "world",
"array_target": "world",
"complex_target_3": 4,
"complex_target_2": 4,
"complex_target_1": [1, 4]
}
```

### `json_parse`

`json_parse`, as its name suggests, parses a string field into a JSON object. Here's an example configuration:
Expand Down Expand Up @@ -596,9 +474,14 @@ processors:
- json_parse:
fields:
- product_object

transform:
- fields:
- product_object
type: json
```

The result will be:
After the `json_parse` processor runs, the `product_object` field in the context becomes a JSON object:

```json
{
Expand All @@ -608,9 +491,11 @@ The result will be:
}
```

The `transform` section then maps the parsed object to the `json` type, so it is stored as a JSON column named `product_object` in the table.

### `simple_extract`

While `json_path` processor is capable of extracting fields from JSON objects using complex expressions, it's relatively slow and costly. The `simple_extract` processor offers a simple way to extract fields by using just key names. Here's an example configuration:
The `simple_extract` processor offers a simple way to extract fields from JSON objects using just key names. Here's an example configuration:

```yaml
processors:
Expand Down Expand Up @@ -1052,9 +937,11 @@ GreptimeDB currently provides the following built-in transformation types:
- `int8`, `int16`, `int32`, `int64`: Integer types.
- `uint8`, `uint16`, `uint32`, `uint64`: Unsigned integer types.
- `float32`, `float64`: Floating-point types.
- `boolean`: Boolean type.
- `string`: String type.
- `time`: Time type, which will be converted to GreptimeDB `timestamp(9)` type.
- `epoch`: Timestamp type, which will be converted to GreptimeDB `timestamp(n)` type. The value of `n` depends on the precision of the epoch. When the precision is `s`, `n` is 0; when the precision is `ms`, `n` is 3; when the precision is `us`, `n` is 6; when the precision is `ns`, `n` is 9.
- `json`: JSON type. The field value must be a JSON object or array, such as one produced by the [`json_parse`](#json_parse) processor, and it is stored as a JSON column in the table. Scalar values (for example, strings, numbers) cannot be converted to the `json` type.

If a field obtains an illegal value during the transformation process, the Pipeline will throw an exception. For example, when converting a string `abc` to an integer, an exception will be thrown because the string is not a valid integer.

Expand Down
135 changes: 11 additions & 124 deletions ...zh/docusaurus-plugin-content-docs/current/reference/pipeline/pipeline-config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,6 @@ Processor 由一个 name 和多个配置组成，不同类型的 Processor 配
- `regex`: 对 log 数据字段进行正则匹配。
- `urlencoding`: 对 log 数据字段进行 URL 编解码。
- `csv`: 对 log 数据字段进行 CSV 解析。
- `json_path`: 从 JSON 数据中提取字段。（**已废弃**，请使用 `vrl` ）
- `json_parse`: 将一个字段解析成 JSON 对象。
- `simple_extract`: 使用简单的 key 从 JSON 数据中提取字段。
- `digest`: 提取日志消息模板。
Expand Down Expand Up @@ -460,127 +459,6 @@ processors:
- `trim`: 是否去除空格。默认为 `false`。
- `ignore_missing`: 忽略字段不存在的情况。默认为 `false`。如果字段不存在，并且此配置为 false，则会抛出异常。

### `json_path`（废弃）

:::danger 废弃特性
`json_path` 处理器已废弃。`vrl` 覆盖了它的使用场景，并提供更灵活的 JSON 字段提取能力，因此新 pipeline 应使用 `vrl`。
:::

`json_path` 处理器用于从 JSON 数据中提取字段。以下是一个配置示例：

```yaml
processors:
- json_path:
fields:
- complex_object
json_path: "$.shop.orders[?(@.active)].id"
ignore_missing: true
result_index: 1
```

在上述示例中，`json_path` processor 的配置包括以下字段：

- `fields`：要提取的字段名称列表。
- `json_path`：要提取的 JSON 路径。
- `ignore_missing`：忽略字段缺失的情况。默认为 `false`。如果字段缺失且此配置设置为 `false`，将抛出异常。
- `result_index`：指定提取数组中要用作结果值的下标。默认情况下，包含所有值。Processor 提取的结果值是包含 path 中所有值的数组。如果指定了索引，将使用提取数组中对应的下标的值作为最终结果。

#### JSON 路径语法

JSON 路径语法基于 [jsonpath-rust](https://github.com/besok/jsonpath-rust) 库。

在此阶段，我们仅推荐使用一些简单的字段提取操作，以便将嵌套字段提取到顶层。

#### `json_path` 示例

例如，给定以下日志数据：

```json
{
"product_object": {
"hello": "world"
},
"product_array": [
"hello",
"world"
],
"complex_object": {
"shop": {
"orders": [
{
"id": 1,
"active": true
},
{
"id": 2
},
{
"id": 3
},
{
"id": 4,
"active": true
}
]
}
}
}
```

使用以下配置：

```yaml
processors:
- json_path:
fields:
- product_object, object_target
json_path: "$.hello"
result_index: 0
- json_path:
fields:
- product_array, array_target
json_path: "$.[1]"
result_index: 0
- json_path:
fields:
- complex_object, complex_target_1
json_path: "$.shop.orders[?(@.active)].id"
- json_path:
fields:
- complex_target_1, complex_target_2
json_path: "$.[1]"
result_index: 0
- json_path:
fields:
- complex_object, complex_target_3
json_path: "$.shop.orders[?(@.active)].id"
result_index: 1
transform:
- fields:
- object_target
- array_target
type: string
- fields:
- complex_target_3
- complex_target_2
type: uint32
- fields:
- complex_target_1
type: json
```

结果将是：

```json
{
"object_target": "world",
"array_target": "world",
"complex_target_3": 4,
"complex_target_2": 4,
"complex_target_1": [1, 4]
}
```

### `json_parse`

`json_parse`，如其名字所示，将一个字符串解析成一个 JSON 对象。以下是一份示例配置:
Expand Down Expand Up @@ -615,9 +493,14 @@ processors:
- json_parse:
fields:
- product_object

transform:
- fields:
- product_object
type: json
```

结果将是：
`json_parse` 处理器执行后，上下文中的 `product_object` 字段将变为一个 JSON 对象：

```json
{
Expand All @@ -627,9 +510,11 @@ processors:
}
```

随后 `transform` 部分将该解析后的对象映射为 `json` 类型，因此它会被存储为表中名为 `product_object` 的 JSON 列。

### `simple_extract`

虽然 `json_path` 处理器能够使用复杂表达式从 JSON 对象中提取字段，但它相对较慢且成本较高。`simple_extract` 处理器提供了一种简单的方法，仅使用键名来提取字段。以下是示例配置：
`simple_extract` 处理器提供了一种简单的方法，仅使用键名从 JSON 对象中提取字段。以下是示例配置：

```yaml
processors:
Expand Down Expand Up @@ -1070,9 +955,11 @@ GreptimeDB 目前内置了以下几种转换类型：
- `int8`, `int16`, `int32`, `int64`: 整数类型。
- `uint8`, `uint16`, `uint32`, `uint64`: 无符号整数类型。
- `float32`, `float64`: 浮点数类型。
- `boolean`: 布尔类型。
- `string`: 字符串类型。
- `time`: 时间类型。将被转换为 GreptimeDB `timestamp(9)` 类型。
- `epoch`: 时间戳类型。将被转换为 GreptimeDB `timestamp(n)` 类型。n 为时间戳精度，n 的值视 epoch 精度而定。当精度为 `s` 时，n 为 0；当精度为 `ms` 时，n 为 3；当精度为 `us` 时，n 为 6；当精度为 `ns` 时，n 为 9。
- `json`: JSON 类型。字段值必须是 JSON 对象或数组（例如由 [`json_parse`](#json_parse) 处理器生成），将被存储为表中的 JSON 列。标量值（例如字符串、数字）无法转换为 `json` 类型。

如果字段在转换过程中获得了非法值，Pipeline 将会抛出异常。例如将一个字符串 `abc` 转换为整数时，由于该字符串不是一个合法的整数，Pipeline 将会抛出异常。

Expand Down
Loading