document add_docker_metadata fields and fix labels.dedot default#6573
document add_docker_metadata fields and fix labels.dedot default#6573alexandra5000 wants to merge 2 commits into
Conversation
Elastic Docs AI PR menuCheck the box to run an AI review for this pull request.
Powered by GitHub Agentic Workflows and docs-actions. For more information, reach out to the docs team. |
🔍 Preview links for changed docs |
✅ Vale Linting ResultsNo issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
|
@alexandra5000 could we have a technical review from an SME on the subject? |
|
@vishaangelova yup, I reached out to @belimawr on Slack, I'm waiting for his review |
| | `container.id` | Container ID | | ||
| | `container.name` | Container name | | ||
| | `container.image.name` | Container image name | | ||
| | `container.labels.<key>` | One field per Docker label. By default, dots (`.`) in label keys are replaced with underscores (`_`). Refer to [`labels.dedot`](#labels-dedot-behavior) for more information. | |
There was a problem hiding this comment.
I'm not sure labels.dedot is the correct text for this link. I followed the link expecting to see the documentation for lablels.dedot, but landed on a section about "Working with labels".
I'd change it to
| | `container.labels.<key>` | One field per Docker label. By default, dots (`.`) in label keys are replaced with underscores (`_`). Refer to [`labels.dedot`](#labels-dedot-behavior) for more information. | | |
| | `container.labels.<key>` | One field per Docker label. By default, dots (`.`) in label keys are replaced with underscores (`_`). Refer to [working with labels](#labels-dedot-behavior) section for more information. | |
| Creating a field with a name that is dynamically determined at runtime (for example, using the value of a label as a field name) is not supported by this processor or by `add_fields`. To react to label values, use [conditions-based autodiscover templates](/reference/fleet/conditions-based-autodiscover.md) instead. | ||
| :::: | ||
|
|
||
| The Docker provider exposes the same labels as variables for configuration templating (`${docker.container.labels.com.docker.compose.service}`). The provider preserves dots in the key for variable references, regardless of the `labels.dedot` setting. For more information, refer to [Docker provider](/reference/fleet/docker-provider.md). |
There was a problem hiding this comment.
Maybe this is my engineer brain who knows the internals of Filebeat talking, take this with a grain of salt or just ignore it ;)
I don't like the 'preserves' verb here.... Variable resolution is not related to the fields defined in the events, but this paragraph makes it sound like otherwise.
| The Docker provider exposes the same labels as variables for configuration templating (`${docker.container.labels.com.docker.compose.service}`). The provider preserves dots in the key for variable references, regardless of the `labels.dedot` setting. For more information, refer to [Docker provider](/reference/fleet/docker-provider.md). | |
| The Docker provider exposes the same labels as variables for configuration templating (`${docker.container.labels.com.docker.compose.service}`). The provider always references keys in the dotted format. For more information, refer to [Docker provider](/reference/fleet/docker-provider.md). |
The Docker provider runs way before any processor, it uses its variables to render an input configuration, once the input is running, any defined processor will enrich/modify the ingested events, that's when the add_docker_meteadata processor actually runs.
| # Docker provider [docker-provider] | ||
|
|
||
| Provides inventory information from Docker. The available dynamic variables are: | ||
| Provides inventory information from Docker. {{agent}} uses the Docker provider to automatically discover containers and build input configurations. It is also responsible for enriching events with container metadata through the [`add_docker_metadata` processor](/reference/fleet/add_docker_metadata-processor.md), which it applies by default to all inputs it generates. |
There was a problem hiding this comment.
The Docker provider and the add_docker_metadata processor are totally independent pieces of software. They even run on different process.
Expanding on what I mentioned before: The Docker provider runs inside Elastic Agent, it is responsible for discovering containers and "rendering" (technically the provider variables are used to render the input configuration, but let's keep it simple) the input configuration. Then the Elastic Agent send this configuration to a Beat or Otel collector, both are different process. Once the Beat/Otel collector starts running, it will instantiate its own (if configured to do so), add_docker_metadata processor to enrich the events it is ingesting.
What also happens, is that the input configuration uses the add_fields processor with values resolved from the Docker provider to enrich the final event, here is an example of an input generated by the Docker integration, note that there is no add_docker_metadata processor but the labels are added via add_fields processor.
- data_stream:
namespace: default
id: filestream-docker-e20c8b8c-5558-4d91-9d07-9029aa798f01-docker-30f4b47c10cdb2a794c3f2cf996e2f837cdb60cf071c4b0e21cf74bc1fe80877
meta:
package:
name: docker
policy_template: docker
release: ga
version: 2.15.1
name: docker-1
original_id: filestream-docker-e20c8b8c-5558-4d91-9d07-9029aa798f01
package_policy_id: e20c8b8c-5558-4d91-9d07-9029aa798f01
policy:
revision: 2
processors:
- add_fields:
fields:
id: 30f4b47c10cdb2a794c3f2cf996e2f837cdb60cf071c4b0e21cf74bc1fe80877
image.name: mingrammer/flog
labels:
a_doted_label: it works
foo: bar
name: flog
target: container
revision: 1
streams:
- data_stream:
dataset: docker.container_logs
elasticsearch:
dynamic_dataset: true
dynamic_namespace: true
type: logs
id: docker-container-logs-flog-30f4b47c10cdb2a794c3f2cf996e2f837cdb60cf071c4b0e21cf74bc1fe80877
parsers:
- container:
format: docker
stream: all
paths:
- /var/lib/docker/containers/30f4b47c10cdb2a794c3f2cf996e2f837cdb60cf071c4b0e21cf74bc1fe80877/*-json.log
type: filestream
Last, but not least, Elastic Agent always adds (well, it's hardcoded inside Beats) the add_docker_metadata processor to all Beats, but the user has no control over its configurations. I believe this will change in 9.5.0.
Given all that, here is how I'd change this paragraph:
| Provides inventory information from Docker. {{agent}} uses the Docker provider to automatically discover containers and build input configurations. It is also responsible for enriching events with container metadata through the [`add_docker_metadata` processor](/reference/fleet/add_docker_metadata-processor.md), which it applies by default to all inputs it generates. | |
| Provides inventory information from Docker. {{agent}} uses the Docker provider to automatically discover containers and build input configurations. |
|
|
||
| Label keys are available as variables using their original names, including dots. For example, for a container with the label `com.docker.compose.service=redis`, you can reference it as `${docker.container.labels.com.docker.compose.service}`. | ||
|
|
||
| However, when the Docker provider enriches events using the `add_docker_metadata` processor, dots in label keys are replaced with underscores by default (`labels.dedot: true`). This means that in the resulting {{es}} document, the same label is stored as `container.labels.com_docker_compose_service`. |
There was a problem hiding this comment.
I'm not sure this behaviour can be configured in the Docker provider. I'd also remove all references to the add_docker_metadata processor.
| However, when the Docker provider enriches events using the `add_docker_metadata` processor, dots in label keys are replaced with underscores by default (`labels.dedot: true`). This means that in the resulting {{es}} document, the same label is stored as `container.labels.com_docker_compose_service`. | |
| However, when the Docker provider enriches events, dots in label keys are replaced with underscores. This means that in the resulting {{es}} document, the same label is stored as `container.labels.com_docker_compose_service`. |
| | Variable reference in the configuration template (`${...}`) | Dots preserved — `docker.container.labels.com.docker.compose.service` | | ||
| | Field in the event/{{es}} document | Dots replaced by `_` — `container.labels.com_docker_compose_service` | | ||
|
|
||
| For more details on the label behavior, refer to [Working with labels](/reference/fleet/add_docker_metadata-processor.md#labels-dedot-behavior) in the `add_docker_metadata` processor documentation. |
There was a problem hiding this comment.
I'm not sure how much of this can be influenced. I can do some more testing/investigation later.
| For more details on the label behavior, refer to [Working with labels](/reference/fleet/add_docker_metadata-processor.md#labels-dedot-behavior) in the `add_docker_metadata` processor documentation. |
3 participants
Summary
Addresses issue #1468 — users couldn't find what metadata fields
add_docker_metadataactually adds, and were getting confused by the undocumented label key transformation.add_docker_metadata-processor.md: replaces the "Container ID / Name / Image / Labels" bullet list with a proper field table showing actual field names (container.id,container.name,container.image.name,container.labels.<key>). Fixes thelabels.dedotdefault fromfalse→true(source code hasDeDot=true). Adds a "Working with labels" section with a Compose/Swarm example and a note that dynamic field names are not supported.docker-provider.md: adds a Provider configuration section (host,ssl,cleanup_timeout), a variables table, and a "Working with labels" subsection explaining the two label representations. Adds cross-links to the processor doc.🤖 Generated with Claude Code