From e7292ab5e1f154c8d3776e450fc02167ce6fa03f Mon Sep 17 00:00:00 2001 From: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> Date: Tue, 18 Aug 2026 15:13:34 -0500 Subject: [PATCH 1/5] docs(observability): document metadata promotion Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> --- .../observability/opentelemetry.mdx | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/docs/configure-plugins/observability/opentelemetry.mdx b/docs/configure-plugins/observability/opentelemetry.mdx index e9af34819..2bf8d0b0a 100644 --- a/docs/configure-plugins/observability/opentelemetry.mdx +++ b/docs/configure-plugins/observability/opentelemetry.mdx @@ -91,6 +91,7 @@ type = "gen_ai" endpoint = "http://localhost:4318/v1/traces" transport = "http_binary" service_name = "agent-service" +promote_metadata_prefixes = ["nv."] max_queue_size = 4096 max_export_batch_size = 512 scheduled_delay_millis = 1000 @@ -143,6 +144,34 @@ to the other exporters. | `mark_projection` | `inherit` | Mark representation for `full` and `openinference`: `inherit`, `event`, or `tool`. | | `mark_exclude_names` | `["llm.chunk"]` | Mark names excluded from `full` and `openinference` projection. | | `attribute_mappings` | `[]` | `{ key, alias }` copies applied by `full` and `openinference` projection. | +| `promote_metadata_prefixes` | `[]` | Literal Event metadata prefixes copied to top-level span attributes. | + +## Event Metadata Promotion + +Set `promote_metadata_prefixes` on a trace endpoint to copy selected keys from +the final sanitized Event metadata into that endpoint's OpenTelemetry output. +Prefixes use the Event metadata key syntax: ASCII letters, numbers, underscores, +and hyphens in segments separated by single dots. A trailing dot is optional. +For example, `"nv."`, `"nv_"`, and `"nv-"` are valid literal prefixes. +Leading or repeated dots, whitespace, other punctuation, and glob expressions +such as `"nv.*"` are rejected. + +Scope metadata is promoted when the scope-end Event completes the span, so each +promoted key has one final value. Mark metadata is promoted to the attributes +of the projected span event or tool span. The `gen_ai` projection continues to +omit Marks. + +Promotion supports strings, booleans, signed 64-bit integers, floating-point +numbers, and homogeneous arrays of those primitive types. Nulls, empty arrays, +nested arrays, mixed-type arrays, objects, and unsigned integers larger than +the signed 64-bit range are omitted. Relay records a bounded runtime diagnostic +containing the key and reason, but it does not record the rejected value or stop +trace export. + +Projection-owned attributes and configured attribute-mapping aliases take +precedence over promoted metadata with the same key. Promotion does not modify +the Event or ATOF payload. Use `resource_attributes` instead for static values +that must be attached to every span from an endpoint. ## Log and Metric Endpoint Resolution From 2901e8e1d7b6b6dfe9ac6a4c90db505aa1857295 Mon Sep 17 00:00:00 2001 From: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> Date: Tue, 18 Aug 2026 18:22:49 -0500 Subject: [PATCH 2/5] docs(observability): clarify literal metadata prefixes Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> --- .../observability/opentelemetry.mdx | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/docs/configure-plugins/observability/opentelemetry.mdx b/docs/configure-plugins/observability/opentelemetry.mdx index 2bf8d0b0a..2144e3b4b 100644 --- a/docs/configure-plugins/observability/opentelemetry.mdx +++ b/docs/configure-plugins/observability/opentelemetry.mdx @@ -156,6 +156,12 @@ For example, `"nv."`, `"nv_"`, and `"nv-"` are valid literal prefixes. Leading or repeated dots, whitespace, other punctuation, and glob expressions such as `"nv.*"` are rejected. +Matching is case-sensitive and compares the beginning of each key literally. +Relay does not infer a dot or metadata-key segment boundary. For example, +`"nv."` selects keys that begin with `nv.`, `"nv_"` selects keys that begin +with `nv_`, and `"user"` selects keys such as `username` and `user_api_key`. +Configure the narrowest prefix that selects the metadata you intend to export. + Scope metadata is promoted when the scope-end Event completes the span, so each promoted key has one final value. Mark metadata is promoted to the attributes of the projected span event or tool span. The `gen_ai` projection continues to @@ -169,9 +175,15 @@ containing the key and reason, but it does not record the rejected value or stop trace export. Projection-owned attributes and configured attribute-mapping aliases take -precedence over promoted metadata with the same key. Promotion does not modify -the Event or ATOF payload. Use `resource_attributes` instead for static values -that must be attached to every span from an endpoint. +precedence over promoted metadata with the same key. Relay also omits selected +keys in namespaces owned by Relay or supported semantic projections: +`nemo_relay.`, `gen_ai.`, `error.`, `exception.`, `input.`, `output.`, `llm.`, +`openinference.`, `server.`, `service.`, `session.`, `tool.`, `tool_call.`, and +`user.`. Relay omits the bare `metadata` key as well. Rejected keys produce a +rate-limited operational diagnostic without dropping the Event or span. + +Promotion does not modify the Event or ATOF payload. Use `resource_attributes` +instead for static values that must be attached to every span from an endpoint. ## Log and Metric Endpoint Resolution From 136c5bb2a2b5425ee5c5fa1aa27f69abb4309f80 Mon Sep 17 00:00:00 2001 From: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> Date: Tue, 18 Aug 2026 23:03:11 -0500 Subject: [PATCH 3/5] docs(observability): align metadata promotion contract Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> --- .../observability/opentelemetry.mdx | 32 ++++++++++++------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/docs/configure-plugins/observability/opentelemetry.mdx b/docs/configure-plugins/observability/opentelemetry.mdx index 2144e3b4b..8cfdf0a54 100644 --- a/docs/configure-plugins/observability/opentelemetry.mdx +++ b/docs/configure-plugins/observability/opentelemetry.mdx @@ -150,8 +150,18 @@ to the other exporters. Set `promote_metadata_prefixes` on a trace endpoint to copy selected keys from the final sanitized Event metadata into that endpoint's OpenTelemetry output. -Prefixes use the Event metadata key syntax: ASCII letters, numbers, underscores, -and hyphens in segments separated by single dots. A trailing dot is optional. +The injection and promotion stages use the following contract: + +| Behavior | Contract | +|---|---| +| Exact metadata keys | ASCII letters, numbers, underscores, and hyphens in nonempty segments separated by single dots. | +| Promotion prefixes | The same key syntax with an optional trailing dot. Matching is literal and case-sensitive. | +| Values | Strings, booleans, signed 64-bit integers, floating-point numbers, empty arrays, and homogeneous arrays containing one supported primitive type. | +| Rejected values | Nulls, objects, nested arrays, mixed-type arrays, and unsigned integers larger than `i64::MAX`. | +| Injector conflicts | Existing Event metadata wins. Otherwise, Relay orders injectors by ascending priority and registration name, and the first successful insertion wins. | +| Scope lifecycle | Relay applies injector precedence independently to Scope-start and Scope-end Events. A Scope-end key is authoritative when Relay constructs the final span; when Scope-end omits the key, the Scope-start value remains. | +| OpenTelemetry collisions | Projection-owned attributes and configured attribute-mapping aliases win over promoted metadata. | + For example, `"nv."`, `"nv_"`, and `"nv-"` are valid literal prefixes. Leading or repeated dots, whitespace, other punctuation, and glob expressions such as `"nv.*"` are rejected. @@ -162,17 +172,17 @@ Relay does not infer a dot or metadata-key segment boundary. For example, with `nv_`, and `"user"` selects keys such as `username` and `user_api_key`. Configure the narrowest prefix that selects the metadata you intend to export. -Scope metadata is promoted when the scope-end Event completes the span, so each -promoted key has one final value. Mark metadata is promoted to the attributes -of the projected span event or tool span. The `gen_ai` projection continues to -omit Marks. +Scope-start and Scope-end are separate Event records, and Relay runs the +ordered injector chain independently for each record. When Scope-end completes +the span, a metadata key present on that Event replaces the corresponding +promoted Scope-start value. Mark metadata is promoted to the attributes of the +projected span event or tool span. The `gen_ai` projection continues to omit +Marks. Promotion supports strings, booleans, signed 64-bit integers, floating-point -numbers, and homogeneous arrays of those primitive types. Nulls, empty arrays, -nested arrays, mixed-type arrays, objects, and unsigned integers larger than -the signed 64-bit range are omitted. Relay records a bounded runtime diagnostic -containing the key and reason, but it does not record the rejected value or stop -trace export. +numbers, empty arrays, and homogeneous arrays of those primitive types. Relay +omits rejected values and records a bounded runtime diagnostic containing the +key and reason, but it does not record the rejected value or stop trace export. Projection-owned attributes and configured attribute-mapping aliases take precedence over promoted metadata with the same key. Relay also omits selected From 2fafe0ffa9f565e8f15d717e78b43a69dd45a673 Mon Sep 17 00:00:00 2001 From: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> Date: Wed, 19 Aug 2026 11:14:19 -0500 Subject: [PATCH 4/5] docs(observability): clarify metadata promotion Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> --- .../observability/opentelemetry.mdx | 26 +++++++++---------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/docs/configure-plugins/observability/opentelemetry.mdx b/docs/configure-plugins/observability/opentelemetry.mdx index 8cfdf0a54..53e28a587 100644 --- a/docs/configure-plugins/observability/opentelemetry.mdx +++ b/docs/configure-plugins/observability/opentelemetry.mdx @@ -91,7 +91,7 @@ type = "gen_ai" endpoint = "http://localhost:4318/v1/traces" transport = "http_binary" service_name = "agent-service" -promote_metadata_prefixes = ["nv."] +promote_metadata_prefixes = ["app."] max_queue_size = 4096 max_export_batch_size = 512 scheduled_delay_millis = 1000 @@ -144,36 +144,34 @@ to the other exporters. | `mark_projection` | `inherit` | Mark representation for `full` and `openinference`: `inherit`, `event`, or `tool`. | | `mark_exclude_names` | `["llm.chunk"]` | Mark names excluded from `full` and `openinference` projection. | | `attribute_mappings` | `[]` | `{ key, alias }` copies applied by `full` and `openinference` projection. | -| `promote_metadata_prefixes` | `[]` | Literal Event metadata prefixes copied to top-level span attributes. | +| `promote_metadata_prefixes` | `[]` | Literal prefixes that select sanitized Event metadata to copy to top-level span attributes. | ## Event Metadata Promotion Set `promote_metadata_prefixes` on a trace endpoint to copy selected keys from the final sanitized Event metadata into that endpoint's OpenTelemetry output. -The injection and promotion stages use the following contract: +The setting defaults to an empty list, so Relay does not promote metadata unless +you configure at least one prefix. | Behavior | Contract | |---|---| -| Exact metadata keys | ASCII letters, numbers, underscores, and hyphens in nonempty segments separated by single dots. | -| Promotion prefixes | The same key syntax with an optional trailing dot. Matching is literal and case-sensitive. | -| Values | Strings, booleans, signed 64-bit integers, floating-point numbers, empty arrays, and homogeneous arrays containing one supported primitive type. | +| Promotion prefixes | ASCII letters, numbers, underscores, and hyphens in nonempty segments separated by single dots, with an optional trailing dot. Matching is literal and case-sensitive. | +| Supported values | Strings, booleans, signed 64-bit integers, floating-point numbers, empty arrays, and homogeneous arrays containing one supported primitive type. | | Rejected values | Nulls, objects, nested arrays, mixed-type arrays, and unsigned integers larger than `i64::MAX`. | -| Injector conflicts | Existing Event metadata wins. Otherwise, Relay orders injectors by ascending priority and registration name, and the first successful insertion wins. | -| Scope lifecycle | Relay applies injector precedence independently to Scope-start and Scope-end Events. A Scope-end key is authoritative when Relay constructs the final span; when Scope-end omits the key, the Scope-start value remains. | +| Scope lifecycle | A Scope-end key is authoritative when Relay constructs the final span. When Scope-end omits the key, the Scope-start value remains. | | OpenTelemetry collisions | Projection-owned attributes and configured attribute-mapping aliases win over promoted metadata. | -For example, `"nv."`, `"nv_"`, and `"nv-"` are valid literal prefixes. +For example, `"app."`, `"app_"`, and `"app-"` are valid literal prefixes. Leading or repeated dots, whitespace, other punctuation, and glob expressions -such as `"nv.*"` are rejected. +such as `"app.*"` are rejected. Matching is case-sensitive and compares the beginning of each key literally. Relay does not infer a dot or metadata-key segment boundary. For example, -`"nv."` selects keys that begin with `nv.`, `"nv_"` selects keys that begin -with `nv_`, and `"user"` selects keys such as `username` and `user_api_key`. +`"app."` selects `app.name` and `app.version`, but not `app_name`. The broader +`"app"` prefix selects all three keys. Configure the narrowest prefix that selects the metadata you intend to export. -Scope-start and Scope-end are separate Event records, and Relay runs the -ordered injector chain independently for each record. When Scope-end completes +Scope-start and Scope-end are separate Event records. When Scope-end completes the span, a metadata key present on that Event replaces the corresponding promoted Scope-start value. Mark metadata is promoted to the attributes of the projected span event or tool span. The `gen_ai` projection continues to omit From 6e8a6db69a9d7860c9a6241382e23c3a8d6d8aba Mon Sep 17 00:00:00 2001 From: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> Date: Wed, 19 Aug 2026 12:05:36 -0500 Subject: [PATCH 5/5] docs(observability): clarify promotion constraints Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> --- docs/configure-plugins/observability/opentelemetry.mdx | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/configure-plugins/observability/opentelemetry.mdx b/docs/configure-plugins/observability/opentelemetry.mdx index 53e28a587..c7f2ec911 100644 --- a/docs/configure-plugins/observability/opentelemetry.mdx +++ b/docs/configure-plugins/observability/opentelemetry.mdx @@ -157,9 +157,9 @@ you configure at least one prefix. |---|---| | Promotion prefixes | ASCII letters, numbers, underscores, and hyphens in nonempty segments separated by single dots, with an optional trailing dot. Matching is literal and case-sensitive. | | Supported values | Strings, booleans, signed 64-bit integers, floating-point numbers, empty arrays, and homogeneous arrays containing one supported primitive type. | -| Rejected values | Nulls, objects, nested arrays, mixed-type arrays, and unsigned integers larger than `i64::MAX`. | +| Rejected values | Nulls, objects, nested arrays, mixed-type arrays, and integers outside the signed 64-bit range. | | Scope lifecycle | A Scope-end key is authoritative when Relay constructs the final span. When Scope-end omits the key, the Scope-start value remains. | -| OpenTelemetry collisions | Projection-owned attributes and configured attribute-mapping aliases win over promoted metadata. | +| OpenTelemetry collisions | Projection-owned attributes always win. For `full` and `openinference`, configured attribute-mapping aliases also win over promoted metadata. | For example, `"app."`, `"app_"`, and `"app-"` are valid literal prefixes. Leading or repeated dots, whitespace, other punctuation, and glob expressions @@ -182,8 +182,9 @@ numbers, empty arrays, and homogeneous arrays of those primitive types. Relay omits rejected values and records a bounded runtime diagnostic containing the key and reason, but it does not record the rejected value or stop trace export. -Projection-owned attributes and configured attribute-mapping aliases take -precedence over promoted metadata with the same key. Relay also omits selected +Projection-owned attributes take precedence over promoted metadata with the +same key. For `full` and `openinference`, configured attribute-mapping aliases +also take precedence. Relay also omits selected keys in namespaces owned by Relay or supported semantic projections: `nemo_relay.`, `gen_ai.`, `error.`, `exception.`, `input.`, `output.`, `llm.`, `openinference.`, `server.`, `service.`, `session.`, `tool.`, `tool_call.`, and