Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 18 additions & 10 deletions docs/content.zh/docs/deployment/advanced/logging.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,31 +91,39 @@ Flink 附带了 [Log4j API bridge](https://logging.apache.org/log4j/log4j-2.2/lo

<a name="configuring-log4j1"></a>

## 配置 Log4j1
## 配置 Log4j 1 (reload4j)

要将 Flink 与 [Log4j 1](https://logging.apache.org/log4j/1.2/) 一起使用,必须确保:
- Classpath 中不存在 `org.apache.logging.log4j:log4j-core`、`org.apache.logging.log4j:log4j-slf4j-impl` 和 `org.apache.logging.log4j:log4j-1.2-api`;
- 且 Classpath 中存在 `log4j:log4j`、`org.slf4j:slf4j-log4j12`、`org.apache.logging.log4j:log4j-to-slf4j` 和 `org.apache.logging.log4j:log4j-api`。
要将 Flink 与 [reload4j](https://reload4j.qos.ch/) 一起使用,必须确保:
- Classpath 中不存在 `org.apache.logging.log4j:log4j-core`、`org.apache.logging.log4j:log4j-slf4j2-impl` 和 `org.apache.logging.log4j:log4j-1.2-api`;
- 且 Classpath 中存在 `ch.qos.reload4j:reload4j`、`org.slf4j:slf4j-reload4j`、`org.apache.logging.log4j:log4j-to-slf4j` 和 `org.apache.logging.log4j:log4j-api`。

如果在 IDE 中使用 Log4j 1,则必须在 pom 文件中使用上述 Classpath 中应该存在的 jars 依赖项来替换 Classpath 中不应该存在的 jars 依赖项,并尽可能的排除那些传递依赖于 Classpath 中不存在 jars 的依赖项。
如果在 IDE 中使用 reload4j,则必须在 pom 文件中使用上述 Classpath 中应该存在的 jars 依赖项来替换 Classpath 中不应该存在的 jars 依赖项,并尽可能的排除那些传递依赖于 Classpath 中不存在 jars 的依赖项。

对于 Flink 发行版,这意味着你必须
- 从 `lib` 目录中移除 `log4j-core`,`log4j-slf4j-impl` 和 `log4j-1.2-api` jars;
- 往 `lib` 目录中添加 `log4j`,`slf4j-log4j12` 和 `log4j-to-slf4j` jars;
- 从 `lib` 目录中移除 `log4j-core`,`log4j-slf4j2-impl` 和 `log4j-1.2-api` jars;
- 往 `lib` 目录中添加 `reload4j`,`slf4j-reload4j` 和 `log4j-to-slf4j` jars;
- 用适配的 Log4j1 版本替换 `conf` 目录中的所有 log4j 配置文件。

{{< hint warning >}}

[Reload4j](https://reload4j.qos.ch/) 是 Log4j 1.2.17 的一个持续维护的分支,保留了 `org.apache.log4j` 类和 Log4j 1 的配置格式。
不能使用原来的 `log4j:log4j`,因为它的 SLF4J binding(`org.slf4j:slf4j-log4j12`)只适用于 SLF4J 1.7,Flink 自带的 SLF4J 2 会直接忽略它。
请使用与 Flink 的 `slf4j-api`(2.x)版本一致的 `org.slf4j:slf4j-reload4j`。

{{< /hint >}}

<a name="configuring-logback"></a>

## 配置 logback

要将 Flink 与 [logback](https://logback.qos.ch/) 一起使用,必须确保:
- Classpath 中不存在 `org.apache.logging.log4j:log4j-slf4j-impl`;
- Classpath 中不存在 `org.apache.logging.log4j:log4j-slf4j2-impl`;
- Classpath 中存在 `ch.qos.logback:logback-core` 和 `ch.qos.logback:logback-classic`。

如果在 IDE 中使用 logback,则必须在 pom 文件中使用上述 Classpath 中应该存在的 jars 依赖项来替换 Classpath 中不应该存在的 jars 依赖项,并尽可能的排除那些传递依赖于 Classpath 中不存在 jars 的依赖项。

对于 Flink 发行版,这意味着你必须
- 从 `lib` 目录中移除 `log4j-slf4j-impl` jars;
- 从 `lib` 目录中移除 `log4j-slf4j2-impl` jars;
- 向 `lib` 目录中添加 `logback-core` 和 `logback-classic` jars。

Flink 发行版在 `conf` 目录中附带了以下 logback 配置文件,如果启用了 logback,则会自动使用这些文件:
Expand All @@ -125,7 +133,7 @@ Flink 发行版在 `conf` 目录中附带了以下 logback 配置文件,如果

{{< hint warning >}}

Logback 1.3+ 需要 SLF4J 2,目前不支持
Flink 使用 SLF4J 2,因此需要 logback 1.3 或更高版本

{{< /hint >}}

Expand Down
4 changes: 2 additions & 2 deletions docs/content.zh/docs/dev/configuration/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ ext {
javaVersion = '1.8'
flinkVersion = '{{< version >}}'
scalaBinaryVersion = '{{< scala_version >}}'
slf4jVersion = '1.7.36'
slf4jVersion = '2.0.19'
log4jVersion = '2.26.1'
}
sourceCompatibility = javaVersion
Expand Down Expand Up @@ -146,7 +146,7 @@ dependencies {
// connectors. These must be in the flinkShadowJar configuration!
// --------------------------------------------------------------
//flinkShadowJar "org.apache.flink:flink-connector-kafka:${flinkVersion}"
runtimeOnly "org.apache.logging.log4j:log4j-slf4j-impl:${log4jVersion}"
runtimeOnly "org.apache.logging.log4j:log4j-slf4j2-impl:${log4jVersion}"
runtimeOnly "org.apache.logging.log4j:log4j-api:${log4jVersion}"
runtimeOnly "org.apache.logging.log4j:log4j-core:${log4jVersion}"
// Add test dependencies here.
Expand Down
24 changes: 15 additions & 9 deletions docs/content/docs/deployment/advanced/logging.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,29 +98,35 @@ Flink ships with the [Log4j API bridge](https://logging.apache.org/log4j/log4j-2

If you have custom Log4j 1 properties files or code that relies on Log4j 1, please check out the official Log4j [compatibility](https://logging.apache.org/log4j/2.x/manual/compatibility.html) and [migration](https://logging.apache.org/log4j/2.x/manual/migration.html) guides.

## Configuring Log4j1
## Configuring Log4j 1 (reload4j) <a name="configuring-log4j1" />

To use Flink with [Log4j 1](https://logging.apache.org/log4j/1.2/) you must ensure that:
- `org.apache.logging.log4j:log4j-core`, `org.apache.logging.log4j:log4j-slf4j-impl` and `org.apache.logging.log4j:log4j-1.2-api` are not on the classpath,
- `log4j:log4j`, `org.slf4j:slf4j-log4j12`, `org.apache.logging.log4j:log4j-to-slf4j` and `org.apache.logging.log4j:log4j-api` are on the classpath.
To use Flink with [reload4j](https://reload4j.qos.ch/) you must ensure that:
- `org.apache.logging.log4j:log4j-core`, `org.apache.logging.log4j:log4j-slf4j2-impl` and `org.apache.logging.log4j:log4j-1.2-api` are not on the classpath,
- `ch.qos.reload4j:reload4j`, `org.slf4j:slf4j-reload4j`, `org.apache.logging.log4j:log4j-to-slf4j` and `org.apache.logging.log4j:log4j-api` are on the classpath.

In the IDE this means you have to replace such dependencies defined in your pom, and possibly add exclusions on dependencies that transitively depend on them.

For Flink distributions this means you have to
- remove the `log4j-core`, `log4j-slf4j-impl` and `log4j-1.2-api` jars from the `lib` directory,
- add the `log4j`, `slf4j-log4j12` and `log4j-to-slf4j` jars to the `lib` directory,
- remove the `log4j-core`, `log4j-slf4j2-impl` and `log4j-1.2-api` jars from the `lib` directory,
- add the `reload4j`, `slf4j-reload4j` and `log4j-to-slf4j` jars to the `lib` directory,
- replace all log4j properties files in the `conf` directory with Log4j1-compliant versions.

{{< hint warning >}}
[Reload4j](https://reload4j.qos.ch/) is a maintained fork of Log4j 1.2.17; it keeps the `org.apache.log4j` classes and the Log4j 1 configuration format.
The original `log4j:log4j` artifact cannot be used, because its SLF4J binding (`org.slf4j:slf4j-log4j12`) exists for SLF4J 1.7 only and is ignored by the SLF4J 2 version Flink ships with.
Use an `org.slf4j:slf4j-reload4j` version matching Flink's `slf4j-api` (2.x).
{{< /hint >}}

## Configuring logback

To use Flink with [logback](https://logback.qos.ch/) you must ensure that:
- `org.apache.logging.log4j:log4j-slf4j-impl` is not on the classpath,
- `org.apache.logging.log4j:log4j-slf4j2-impl` is not on the classpath,
- `ch.qos.logback:logback-core` and `ch.qos.logback:logback-classic` are on the classpath.

In the IDE this means you have to replace such dependencies defined in your pom, and possibly add exclusions on dependencies that transitively depend on them.

For Flink distributions this means you have to
- remove the `log4j-slf4j-impl` jar from the `lib` directory,
- remove the `log4j-slf4j2-impl` jar from the `lib` directory,
- add the `logback-core`, and `logback-classic` jars to the `lib` directory.

The Flink distribution ships with the following logback configuration files in the `conf` directory, which are used automatically if logback is enabled:
Expand All @@ -129,7 +135,7 @@ The Flink distribution ships with the following logback configuration files in t
- `logback.xml`: used for command line interface and Job-/TaskManagers by default

{{< hint warning >}}
Logback 1.3+ requires SLF4J 2, which is currently not supported.
Flink uses SLF4J 2, which requires logback 1.3 or later.
{{< /hint >}}

## Best practices for developers
Expand Down
4 changes: 2 additions & 2 deletions docs/content/docs/dev/configuration/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ ext {
javaVersion = '1.8'
flinkVersion = '{{< version >}}'
scalaBinaryVersion = '{{< scala_version >}}'
slf4jVersion = '1.7.36'
slf4jVersion = '2.0.19'
log4jVersion = '2.26.1'
}
sourceCompatibility = javaVersion
Expand Down Expand Up @@ -148,7 +148,7 @@ dependencies {
// connectors. These must be in the flinkShadowJar configuration!
// --------------------------------------------------------------
//flinkShadowJar "org.apache.flink:flink-connector-kafka:${flinkVersion}"
runtimeOnly "org.apache.logging.log4j:log4j-slf4j-impl:${log4jVersion}"
runtimeOnly "org.apache.logging.log4j:log4j-slf4j2-impl:${log4jVersion}"
runtimeOnly "org.apache.logging.log4j:log4j-api:${log4jVersion}"
runtimeOnly "org.apache.logging.log4j:log4j-core:${log4jVersion}"
// Add test dependencies here.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,10 +50,9 @@ void tearDown() throws Exception {
}

/**
* The {@link MDC#setContextMap(Map)} method in Logback 1.2 does not accept nulls, unlike Log4j
* and Logback 1.3.2. BasicMDCAdapter is used to replicate this behavior for testing without
* bundling Logback into flink-core. See https://issues.apache.org/jira/browse/FLINK-36227 for
* details.
* Restoring an absent (null) MDC context must not fail, regardless of the backend in use.
* Historically, {@link MDC#setContextMap(Map)} in Logback 1.2 rejected nulls, unlike Log4j and
* Logback 1.3.2 (see https://issues.apache.org/jira/browse/FLINK-36227).
*/
@Test
void testContextRestorationWorksWithNullContext() {
Expand All @@ -67,13 +66,13 @@ void testContextRestorationWorksWithNullContext() {
}

private MDCAdapter getCurrentMDCAdapter() throws Exception {
Field adapterField = MDC.class.getDeclaredField("mdcAdapter");
Field adapterField = MDC.class.getDeclaredField("MDC_ADAPTER");
adapterField.setAccessible(true);
return (MDCAdapter) adapterField.get(null);
}

private void setMDCAdapter(MDCAdapter adapter) throws Exception {
Field adapterField = MDC.class.getDeclaredField("mdcAdapter");
Field adapterField = MDC.class.getDeclaredField("MDC_ADAPTER");
adapterField.setAccessible(true);
adapterField.set(null, adapter);
}
Expand Down
2 changes: 1 addition & 1 deletion flink-dist/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ under the License.

<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<artifactId>log4j-slf4j2-impl</artifactId>
<scope>compile</scope>
</dependency>

Expand Down
2 changes: 1 addition & 1 deletion flink-dist/src/main/assemblies/bin.xml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ under the License.
<includes>
<include>org.apache.logging.log4j:log4j-api</include>
<include>org.apache.logging.log4j:log4j-core</include>
<include>org.apache.logging.log4j:log4j-slf4j-impl</include>
<include>org.apache.logging.log4j:log4j-slf4j2-impl</include>
<include>org.apache.logging.log4j:log4j-1.2-api</include>
<include>org.apache.logging.log4j:log4j-layout-template-json</include>
</includes>
Expand Down
2 changes: 1 addition & 1 deletion flink-dist/src/main/resources/META-INF/NOTICE
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ See bundled license files for details.
This project bundles the following dependencies under the MIT/X11 license.
See bundled license files for details.

- org.slf4j:slf4j-api:1.7.36
- org.slf4j:slf4j-api:2.0.19

This project bundles the following dependencies under the CDDL 1.1 license.
See bundled license files for details.
Expand Down
2 changes: 1 addition & 1 deletion flink-docs/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ under the License.

<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<artifactId>log4j-slf4j2-impl</artifactId>
<scope>compile</scope>
</dependency>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ under the License.
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<artifactId>log4j-slf4j2-impl</artifactId>
<version>${log4j.version}</version>
</dependency>
<dependency>
Expand Down
2 changes: 1 addition & 1 deletion flink-examples/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ under the License.

<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<artifactId>log4j-slf4j2-impl</artifactId>
<scope>compile</scope>
</dependency>

Expand Down
2 changes: 1 addition & 1 deletion flink-python/src/main/resources/META-INF/NOTICE
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ See bundled license files for details.
- net.razorvine:pyrolite:4.13
- org.checkerframework:checker-qual:3.42.0
- io.github.classgraph:classgraph:4.8.162
- org.slf4j:slf4j-api:1.7.36
- org.slf4j:slf4j-api:2.0.19
- args4j:args4j:2.33

The bundled Apache Beam dependencies bundle the following dependencies under the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ under the License.
<!-- These dependencies are excluded from the application JAR by default. -->
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<artifactId>log4j-slf4j2-impl</artifactId>
<version>${log4j.version}</version>
<scope>runtime</scope>
</dependency>
Expand Down
2 changes: 1 addition & 1 deletion flink-test-utils-parent/flink-test-utils-junit/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ under the License.

<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<artifactId>log4j-slf4j2-impl</artifactId>
<scope>compile</scope>
</dependency>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ under the License.
<!-- These dependencies are excluded from the application JAR by default. -->
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<artifactId>log4j-slf4j2-impl</artifactId>
<version>${log4j.version}</version>
<scope>runtime</scope>
</dependency>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ under the License.
<!-- Add logging framework for console output when running in the IDE. -->
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<artifactId>log4j-slf4j2-impl</artifactId>
<version>${log4j.version}</version>
<scope>runtime</scope>
</dependency>
Expand Down
8 changes: 4 additions & 4 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ under the License.
<flink.markBundledAsOptional>true</flink.markBundledAsOptional>
<source.java.version>11</source.java.version>
<target.java.version>17</target.java.version>
<slf4j.version>1.7.36</slf4j.version>
<slf4j.version>2.0.19</slf4j.version>
<log4j.version>2.26.1</log4j.version>
<!-- Overwrite default values from parent pom.
IntelliJ IDEA is (sometimes?) using those values to choose target language level
Expand Down Expand Up @@ -305,7 +305,7 @@ under the License.

<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<artifactId>log4j-slf4j2-impl</artifactId>
<scope>test</scope>
</dependency>

Expand Down Expand Up @@ -538,7 +538,7 @@ under the License.

<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<artifactId>log4j-slf4j2-impl</artifactId>
<version>${log4j.version}</version>
</dependency>

Expand Down Expand Up @@ -2285,7 +2285,7 @@ under the License.
<dependency>log4j:log4j</dependency>
<dependency>org.slf4j:slf4j-log4j12</dependency>
<!-- log4j2 -->
<dependency>org.apache.logging.log4j:log4j-slf4j-impl</dependency>
<dependency>org.apache.logging.log4j:log4j-slf4j2-impl</dependency>
<dependency>org.apache.logging.log4j:log4j-api</dependency>
<dependency>org.apache.logging.log4j:log4j-core</dependency>
<dependency>org.apache.logging.log4j:log4j-1.2-api</dependency>
Expand Down
2 changes: 1 addition & 1 deletion tools/ci/flink-ci-tools/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ under the License.
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<artifactId>log4j-slf4j2-impl</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
Expand Down
2 changes: 1 addition & 1 deletion tools/releasing/NOTICE-binary_PREAMBLE.txt
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ This project bundles the following dependencies under the Apache Software Licens

- org.apache.logging.log4j:log4j-api:2.26.1
- org.apache.logging.log4j:log4j-core:2.26.1
- org.apache.logging.log4j:log4j-slf4j-impl:2.26.1
- org.apache.logging.log4j:log4j-slf4j2-impl:2.26.1
- org.apache.logging.log4j:log4j-1.2-api:2.26.1
- org.apache.logging.log4j:log4j-layout-template-json:2.26.1

Expand Down