From 8b009f171dd30979f2b48749a56e71e80781b242 Mon Sep 17 00:00:00 2001 From: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> Date: Mon, 17 Aug 2026 19:03:17 -0500 Subject: [PATCH 01/12] docs: document Event metadata injection Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> --- docs/about-nemo-relay/concepts/middleware.mdx | 110 ++++++++++++++---- 1 file changed, 88 insertions(+), 22 deletions(-) diff --git a/docs/about-nemo-relay/concepts/middleware.mdx b/docs/about-nemo-relay/concepts/middleware.mdx index 8fd94c1ef..0d8fd8c03 100644 --- a/docs/about-nemo-relay/concepts/middleware.mdx +++ b/docs/about-nemo-relay/concepts/middleware.mdx @@ -9,12 +9,13 @@ import { MermaidStyles } from "@/components/MermaidStyles"; SPDX-License-Identifier: Apache-2.0 */} This page explains the runtime behavior that runs around managed tool and LLM -calls and sanitizes emitted mark and scope events. +calls, injects Event metadata, and sanitizes emitted mark and scope events. ## What Middleware Is -Middleware controls or transforms tool and LLM execution and sanitizes emitted -events. NeMo Relay applies each middleware type at a specific lifecycle point. +Middleware controls or transforms tool and LLM execution and enriches or +sanitizes emitted events. NeMo Relay applies each middleware type at a specific +lifecycle point. Middleware is organized by lifecycle meaning rather than as one undifferentiated hook system. @@ -27,8 +28,9 @@ registrations accept callbacks that return a value or an awaitable when invoked through an asynchronous Relay API or queued event publication. Worker and native-plugin middleware can also complete asynchronously. Within each middleware chain, Relay awaits entries sequentially in priority order so later -callbacks observe earlier middleware output. Payload and event sanitizer chains -run on the queued publication path and do not delay managed execution. +callbacks observe earlier middleware output. Payload sanitizers, Event metadata +injectors, and Event sanitizer chains run on the queued publication path and do +not delay managed execution. The experimental raw C FFI and Go binding retain synchronous middleware callbacks. Relay invokes each callback on a native thread and waits for it to @@ -49,10 +51,11 @@ sanitization and publication rather than awaiting it. Manual lifecycle APIs synchronous and create or close their handle immediately. -Event sanitizers, conditional-execution guardrails, request intercepts, -execution intercepts, and subscribers are not re-entrant. These callbacks must -not invoke another NeMo Relay API that runs middleware, flushes subscriber -delivery, waits on an exporter, or clears plugins. Scope APIs remain supported: +Event metadata injectors, Event sanitizers, conditional-execution guardrails, +request intercepts, execution intercepts, and subscribers are not re-entrant. +These callbacks must not invoke another NeMo Relay API that runs middleware, +flushes subscriber delivery, waits on an exporter, or clears plugins. Scope +APIs remain supported: callbacks may create, push, or pop scopes at any nesting level and may replace the active scope stack with an arbitrary stack. Emitting a new event is the only supported operation that can enqueue additional callback work; Relay @@ -83,8 +86,8 @@ separately from this application-visible result. ## Registration Levels -Middleware and subscribers can be registered at different levels depending on their -lifetime and visibility. +Middleware and subscribers can be registered at different levels depending on +their lifetime and visibility. ### Global Registrations @@ -107,10 +110,11 @@ everything in application code. ## Middleware Families -NeMo Relay has two major middleware families with three distinct purposes: +NeMo Relay has three major middleware families with four distinct purposes: - **Intercepts** change the real request or callback execution path. - **Conditional-execution guardrails** decide whether the real work runs. +- **Event metadata injectors** add flat metadata to emitted events. - **Sanitize guardrails** change emitted observability without changing the real request or result. @@ -122,6 +126,8 @@ Choose the middleware type that matches the behavior you need: rejected. - Use a **request intercept** when the real request must change before the call. - Use an **execution intercept** when code must run before or after the callback. +- Use an **Event metadata injector** when every emitted Event should receive + additional flat metadata before sanitization. - Use a **sanitize guardrail** when only subscribers and exporters should see rewritten data. - Use a **mark or scope event sanitizer** when the sensitive fields are in @@ -144,6 +150,10 @@ execution intercept cannot undo work from a `next` continuation that it already invoked. This fail-closed contract is the same whether the callback completes directly or asynchronously. +Event metadata injectors fail open. If an injector fails, panics, or returns an +invalid attribute set, Relay logs the failure, omits all metadata from that +injector for the current Event, and continues with the original Event. + ## Intercepts Intercepts are middleware that change the real request or execution path. @@ -177,6 +187,57 @@ LLM streaming has a stream execution path for wrappers that need to run around chunk delivery and finalization rather than only around a single response object. +## Event Metadata Injection + +Event metadata injectors add OTel-compatible values to the existing Event +`metadata` object. They run on every delivered Event and receive the current +Event as immutable context, so a callback can inspect the Event kind and name +before deciding what to return. + +The v1 contract accepts keys made of ASCII letter, number, underscore, or +hyphen segments separated by optional single dots. Empty keys, whitespace, +unsupported punctuation, leading or trailing dots, and repeated dots are +invalid. Relay stores accepted keys literally; dots do not create nested +objects. Values may be strings, numbers, booleans, or homogeneous lists of +those primitive types. Nested objects, `null`, and mixed-type lists are rejected +as one atomic callback result. + +Injection is insert-only. Existing Event metadata is never overwritten. +Injectors run in ascending numeric priority and then by registration name, so +the first injector to insert a missing key wins. A later injector that returns +the same key cannot replace it. + +The following Rust registration adds one string to matching Events: + +```rust +use std::collections::BTreeMap; +use std::sync::Arc; + +use nemo_relay::api::registry::register_event_metadata_injector; +use serde_json::json; + +register_event_metadata_injector( + "machine-profile", + 10, + Arc::new(|event| { + Box::pin(async move { + let attributes = if event.name().starts_with("tool") { + BTreeMap::from([("nv.machine.profile".into(), json!("dgx"))]) + } else { + BTreeMap::new() + }; + Ok(attributes) + }) + }), +)?; +``` + +Register injectors globally, on an owning scope, or through a plugin +registration context. Injectors execute on the queued serial publication path, +before Event sanitizers, so they do not delay the managed tool or LLM callback. +Sanitizers retain the final decision to preserve, rewrite, or remove injected +metadata before subscriber and exporter delivery. + ## Guardrails Guardrails are middleware that block execution or sanitize observability payloads. @@ -229,16 +290,17 @@ arguments passed to the callback or the real value returned to the caller. ## Queued Event Publication Scope operations, marks, and manual or managed tool/LLM lifecycle calls do not -await observability sanitizers. At emission time Relay snapshots the event-only -payload, visible sanitizer chains, and subscribers, then places the work on a -serial dispatcher. The dispatcher awaits the specialized tool or LLM payload -sanitizers, then the event sanitizers, and publishes the event later in FIFO -order. +await observability middleware. At emission time Relay snapshots the event-only +payload, visible Event metadata injectors, sanitizer chains, and subscribers, +then places the work on a serial dispatcher. The dispatcher awaits specialized +tool or LLM payload sanitizers, applies Event metadata injectors, runs Event +sanitizers, and publishes the Event later in FIFO order. Subscriber and exporter delivery is therefore delayed, while start/end/mark order is preserved. Closing a scope or deregistering middleware after emission -does not affect queued snapshots. Sanitizer failures fail closed: Relay records -the callback failure and withholds the governed observability payload. +does not affect queued snapshots. Injector failures fail open and omit that +injector's output. Sanitizer failures fail closed: Relay records the callback +failure and withholds the governed observability payload. ## Managed Execution Order @@ -273,10 +335,13 @@ flowchart LR subgraph PublicationPath[Serial Publication Path] RequestSanitizers[Request Sanitizers] + StartInjection[Event Metadata Injectors] StartEvent[Scope-Start Sanitizers and Deliver Start] ResponseSanitizers[Response Sanitizers] + EndInjection[Event Metadata Injectors] EndEvent[Scope-End Sanitizers and Deliver End] - RequestSanitizers --> StartEvent --> ResponseSanitizers --> EndEvent + RequestSanitizers --> StartInjection --> StartEvent + StartEvent --> ResponseSanitizers --> EndInjection --> EndEvent end QueueStart -.-> RequestSanitizers @@ -292,8 +357,8 @@ The phases run as follows: 3. **Response phase:** Relay snapshots and enqueues the end event's observability copy, then returns the real result. 4. **Publication path:** The serial dispatcher runs request or response - sanitizers, then the matching scope-event sanitizers, and finally delivers - the event to subscribers and exporters. + sanitizers, applies Event metadata injectors, runs the matching scope-event + sanitizers, and finally delivers the Event to subscribers and exporters. The start event is submitted before execution begins, but its sanitizers run later on the publication path and may overlap application execution. The serial @@ -303,6 +368,7 @@ for queued sanitization and delivery when a caller needs that barrier. This ordering preserves the distinction between the families: - Use an intercept to change real execution. +- Use an Event metadata injector to add observability context. - Use a sanitize guardrail to change only emitted observability. ### Rejection Path From 59f8ffb2ed44db9092501eac0d179d1307c10210 Mon Sep 17 00:00:00 2001 From: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> Date: Wed, 19 Aug 2026 01:07:44 -0500 Subject: [PATCH 02/12] docs: clarify event metadata injector semantics Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> --- docs/about-nemo-relay/concepts/middleware.mdx | 24 ++++++++++--------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/docs/about-nemo-relay/concepts/middleware.mdx b/docs/about-nemo-relay/concepts/middleware.mdx index 0d8fd8c03..a323314e5 100644 --- a/docs/about-nemo-relay/concepts/middleware.mdx +++ b/docs/about-nemo-relay/concepts/middleware.mdx @@ -110,7 +110,7 @@ everything in application code. ## Middleware Families -NeMo Relay has three major middleware families with four distinct purposes: +NeMo Relay provides four middleware purposes: - **Intercepts** change the real request or callback execution path. - **Conditional-execution guardrails** decide whether the real work runs. @@ -150,9 +150,10 @@ execution intercept cannot undo work from a `next` continuation that it already invoked. This fail-closed contract is the same whether the callback completes directly or asynchronously. -Event metadata injectors fail open. If an injector fails, panics, or returns an -invalid attribute set, Relay logs the failure, omits all metadata from that -injector for the current Event, and continues with the original Event. +Event metadata injectors fail open. If an injector fails, panics, or returns +invalid attributes, Relay logs the failure, ignores only that injector's output +for the current Event, and continues with later injectors using the Event as it +stood before the failed injector. ## Intercepts @@ -194,13 +195,14 @@ Event metadata injectors add OTel-compatible values to the existing Event Event as immutable context, so a callback can inspect the Event kind and name before deciding what to return. -The v1 contract accepts keys made of ASCII letter, number, underscore, or -hyphen segments separated by optional single dots. Empty keys, whitespace, -unsupported punctuation, leading or trailing dots, and repeated dots are -invalid. Relay stores accepted keys literally; dots do not create nested -objects. Values may be strings, numbers, booleans, or homogeneous lists of -those primitive types. Nested objects, `null`, and mixed-type lists are rejected -as one atomic callback result. +The v1 contract accepts keys made of non-empty segments containing only ASCII +letters, digits, underscores, or hyphens. Segments may be separated by single +dots, so keys such as `region.zone.name` are valid. Leading dots, trailing +dots, consecutive dots, whitespace, and unsupported punctuation are invalid. +Relay stores accepted keys literally; dots do not create nested objects. Values +may be strings, numbers, booleans, or homogeneous lists of those primitive +types. Nested objects, `null`, and mixed-type lists are rejected as one atomic +callback result. Injection is insert-only. Existing Event metadata is never overwritten. Injectors run in ascending numeric priority and then by registration name, so From 78d2e87d2b52dcda3c2e16c4700279f7605fb511 Mon Sep 17 00:00:00 2001 From: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> Date: Wed, 19 Aug 2026 01:47:01 -0500 Subject: [PATCH 03/12] docs: use neutral metadata injection example Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> --- docs/about-nemo-relay/concepts/middleware.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/about-nemo-relay/concepts/middleware.mdx b/docs/about-nemo-relay/concepts/middleware.mdx index a323314e5..69d0d64b5 100644 --- a/docs/about-nemo-relay/concepts/middleware.mdx +++ b/docs/about-nemo-relay/concepts/middleware.mdx @@ -219,12 +219,12 @@ use nemo_relay::api::registry::register_event_metadata_injector; use serde_json::json; register_event_metadata_injector( - "machine-profile", + "tool-category", 10, Arc::new(|event| { Box::pin(async move { let attributes = if event.name().starts_with("tool") { - BTreeMap::from([("nv.machine.profile".into(), json!("dgx"))]) + BTreeMap::from([("example.tool.category".into(), json!("catalog"))]) } else { BTreeMap::new() }; From 26616cf5f635cdb4f91b815725b1633f96551d79 Mon Sep 17 00:00:00 2001 From: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> Date: Wed, 19 Aug 2026 10:34:06 -0500 Subject: [PATCH 04/12] docs: clarify Event metadata injection lifecycle Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> --- docs/about-nemo-relay/concepts/middleware.mdx | 90 +++++++------------ 1 file changed, 34 insertions(+), 56 deletions(-) diff --git a/docs/about-nemo-relay/concepts/middleware.mdx b/docs/about-nemo-relay/concepts/middleware.mdx index 69d0d64b5..97e3db197 100644 --- a/docs/about-nemo-relay/concepts/middleware.mdx +++ b/docs/about-nemo-relay/concepts/middleware.mdx @@ -13,9 +13,9 @@ calls, injects Event metadata, and sanitizes emitted mark and scope events. ## What Middleware Is -Middleware controls or transforms tool and LLM execution and enriches or -sanitizes emitted events. NeMo Relay applies each middleware type at a specific -lifecycle point. +Middleware can control or transform tool and LLM execution. It can also enrich +or sanitize emitted Events. NeMo Relay applies each middleware type at a +specific lifecycle point. Middleware is organized by lifecycle meaning rather than as one undifferentiated hook system. @@ -28,9 +28,9 @@ registrations accept callbacks that return a value or an awaitable when invoked through an asynchronous Relay API or queued event publication. Worker and native-plugin middleware can also complete asynchronously. Within each middleware chain, Relay awaits entries sequentially in priority order so later -callbacks observe earlier middleware output. Payload sanitizers, Event metadata -injectors, and Event sanitizer chains run on the queued publication path and do -not delay managed execution. +callbacks observe earlier middleware output. Event payload transformations, +Event metadata injectors, and Event sanitizer chains run on the queued +publication path and do not delay managed execution. The experimental raw C FFI and Go binding retain synchronous middleware callbacks. Relay invokes each callback on a native thread and waits for it to @@ -190,50 +190,25 @@ object. ## Event Metadata Injection -Event metadata injectors add OTel-compatible values to the existing Event -`metadata` object. They run on every delivered Event and receive the current -Event as immutable context, so a callback can inspect the Event kind and name -before deciding what to return. +Event metadata injectors add values to the existing Event `metadata` object. +They run on every delivered Event and receive the current Event as immutable +context, so a callback can inspect the Event kind and name before deciding what +to return. -The v1 contract accepts keys made of non-empty segments containing only ASCII +Event metadata keys contain one or more non-empty segments made of ASCII letters, digits, underscores, or hyphens. Segments may be separated by single -dots, so keys such as `region.zone.name` are valid. Leading dots, trailing -dots, consecutive dots, whitespace, and unsupported punctuation are invalid. -Relay stores accepted keys literally; dots do not create nested objects. Values -may be strings, numbers, booleans, or homogeneous lists of those primitive -types. Nested objects, `null`, and mixed-type lists are rejected as one atomic -callback result. +dots, so keys such as `region.zone.name` are valid. Leading dots, trailing dots, +consecutive dots, whitespace, and unsupported punctuation are invalid. Relay +stores accepted keys literally; dots do not create nested objects. Values may +be strings, numbers, booleans, or homogeneous lists of those primitive types. +Nested objects, `null`, and mixed-type lists are rejected as one atomic callback +result. Injection is insert-only. Existing Event metadata is never overwritten. Injectors run in ascending numeric priority and then by registration name, so the first injector to insert a missing key wins. A later injector that returns the same key cannot replace it. -The following Rust registration adds one string to matching Events: - -```rust -use std::collections::BTreeMap; -use std::sync::Arc; - -use nemo_relay::api::registry::register_event_metadata_injector; -use serde_json::json; - -register_event_metadata_injector( - "tool-category", - 10, - Arc::new(|event| { - Box::pin(async move { - let attributes = if event.name().starts_with("tool") { - BTreeMap::from([("example.tool.category".into(), json!("catalog"))]) - } else { - BTreeMap::new() - }; - Ok(attributes) - }) - }), -)?; -``` - Register injectors globally, on an owning scope, or through a plugin registration context. Injectors execute on the queued serial publication path, before Event sanitizers, so they do not delay the managed tool or LLM callback. @@ -294,9 +269,9 @@ arguments passed to the callback or the real value returned to the caller. Scope operations, marks, and manual or managed tool/LLM lifecycle calls do not await observability middleware. At emission time Relay snapshots the event-only payload, visible Event metadata injectors, sanitizer chains, and subscribers, -then places the work on a serial dispatcher. The dispatcher awaits specialized -tool or LLM payload sanitizers, applies Event metadata injectors, runs Event -sanitizers, and publishes the Event later in FIFO order. +then places the work on a serial dispatcher. The dispatcher completes the +internal tool or LLM Event payload transformation, applies Event metadata +injectors, runs Event sanitizers, and publishes the Event later in FIFO order. Subscriber and exporter delivery is therefore delayed, while start/end/mark order is preserved. Closing a scope or deregistering middleware after emission @@ -336,18 +311,20 @@ flowchart LR Callback --> QueueEnd subgraph PublicationPath[Serial Publication Path] - RequestSanitizers[Request Sanitizers] + RequestTransform[Request Event Payload Transform] StartInjection[Event Metadata Injectors] - StartEvent[Scope-Start Sanitizers and Deliver Start] - ResponseSanitizers[Response Sanitizers] + StartSanitizers[Scope-Start Event Sanitizers] + DeliverStart[Deliver Start Event] + ResponseTransform[Response Event Payload Transform] EndInjection[Event Metadata Injectors] - EndEvent[Scope-End Sanitizers and Deliver End] - RequestSanitizers --> StartInjection --> StartEvent - StartEvent --> ResponseSanitizers --> EndInjection --> EndEvent + EndSanitizers[Scope-End Event Sanitizers] + DeliverEnd[Deliver End Event] + RequestTransform --> StartInjection --> StartSanitizers --> DeliverStart + DeliverStart --> ResponseTransform --> EndInjection --> EndSanitizers --> DeliverEnd end - QueueStart -.-> RequestSanitizers - QueueEnd -.-> ResponseSanitizers + QueueStart -.-> RequestTransform + QueueEnd -.-> ResponseTransform ``` The phases run as follows: @@ -358,9 +335,10 @@ The phases run as follows: 2. **Execution phase:** Execution intercepts wrap or replace the real callback. 3. **Response phase:** Relay snapshots and enqueues the end event's observability copy, then returns the real result. -4. **Publication path:** The serial dispatcher runs request or response - sanitizers, applies Event metadata injectors, runs the matching scope-event - sanitizers, and finally delivers the Event to subscribers and exporters. +4. **Publication path:** The serial dispatcher completes the internal request or + response Event payload transformation, applies Event metadata injectors, runs + the matching scope-Event sanitizers, and finally delivers the Event to + subscribers and exporters. The start event is submitted before execution begins, but its sanitizers run later on the publication path and may overlap application execution. The serial From e5ce788257c21b0f29cecdf1134c5995d561a644 Mon Sep 17 00:00:00 2001 From: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> Date: Wed, 19 Aug 2026 10:45:26 -0500 Subject: [PATCH 05/12] docs: polish Event metadata injection concepts Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> --- docs/about-nemo-relay/concepts/middleware.mdx | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/about-nemo-relay/concepts/middleware.mdx b/docs/about-nemo-relay/concepts/middleware.mdx index 97e3db197..e88a3de32 100644 --- a/docs/about-nemo-relay/concepts/middleware.mdx +++ b/docs/about-nemo-relay/concepts/middleware.mdx @@ -9,13 +9,13 @@ import { MermaidStyles } from "@/components/MermaidStyles"; SPDX-License-Identifier: Apache-2.0 */} This page explains the runtime behavior that runs around managed tool and LLM -calls, injects Event metadata, and sanitizes emitted mark and scope events. +calls, injects `Event` metadata, and sanitizes emitted mark and scope events. ## What Middleware Is Middleware can control or transform tool and LLM execution. It can also enrich -or sanitize emitted Events. NeMo Relay applies each middleware type at a -specific lifecycle point. +or sanitize emitted `Event` objects. NeMo Relay applies each middleware type at +a specific lifecycle point. Middleware is organized by lifecycle meaning rather than as one undifferentiated hook system. @@ -126,7 +126,7 @@ Choose the middleware type that matches the behavior you need: rejected. - Use a **request intercept** when the real request must change before the call. - Use an **execution intercept** when code must run before or after the callback. -- Use an **Event metadata injector** when every emitted Event should receive +- Use an **Event metadata injector** when every emitted `Event` should receive additional flat metadata before sanitization. - Use a **sanitize guardrail** when only subscribers and exporters should see rewritten data. @@ -152,8 +152,8 @@ directly or asynchronously. Event metadata injectors fail open. If an injector fails, panics, or returns invalid attributes, Relay logs the failure, ignores only that injector's output -for the current Event, and continues with later injectors using the Event as it -stood before the failed injector. +for the current `Event`, and continues with later injectors using the `Event` as +it stood before the failed injector. ## Intercepts @@ -286,6 +286,8 @@ execution, and response phases. +The following diagram shows the managed execution and serial publication paths. + ```mermaid flowchart LR subgraph RequestPhase[Request Phase] From b250c974aea66cf9290468d07d36ec41b5fdb653 Mon Sep 17 00:00:00 2001 From: Eric Evans II <194135482+ericevans-nv@users.noreply.github.com> Date: Wed, 19 Aug 2026 11:51:22 -0500 Subject: [PATCH 06/12] Update docs/about-nemo-relay/concepts/middleware.mdx Co-authored-by: Will Killian <2007799+willkill07@users.noreply.github.com> Signed-off-by: Eric Evans II <194135482+ericevans-nv@users.noreply.github.com> --- docs/about-nemo-relay/concepts/middleware.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/about-nemo-relay/concepts/middleware.mdx b/docs/about-nemo-relay/concepts/middleware.mdx index e88a3de32..0a825126d 100644 --- a/docs/about-nemo-relay/concepts/middleware.mdx +++ b/docs/about-nemo-relay/concepts/middleware.mdx @@ -9,7 +9,7 @@ import { MermaidStyles } from "@/components/MermaidStyles"; SPDX-License-Identifier: Apache-2.0 */} This page explains the runtime behavior that runs around managed tool and LLM -calls, injects `Event` metadata, and sanitizes emitted mark and scope events. +calls, injects metadata into events, and sanitizes emitted mark and scope events. ## What Middleware Is From e156f3db72571d88709ff043d0bdeed3a990b4f2 Mon Sep 17 00:00:00 2001 From: Eric Evans II <194135482+ericevans-nv@users.noreply.github.com> Date: Wed, 19 Aug 2026 11:51:32 -0500 Subject: [PATCH 07/12] Update docs/about-nemo-relay/concepts/middleware.mdx Co-authored-by: Will Killian <2007799+willkill07@users.noreply.github.com> Signed-off-by: Eric Evans II <194135482+ericevans-nv@users.noreply.github.com> --- docs/about-nemo-relay/concepts/middleware.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/about-nemo-relay/concepts/middleware.mdx b/docs/about-nemo-relay/concepts/middleware.mdx index 0a825126d..b0893992d 100644 --- a/docs/about-nemo-relay/concepts/middleware.mdx +++ b/docs/about-nemo-relay/concepts/middleware.mdx @@ -350,7 +350,7 @@ for queued sanitization and delivery when a caller needs that barrier. This ordering preserves the distinction between the families: - Use an intercept to change real execution. -- Use an Event metadata injector to add observability context. +- Use an event metadata injector to add observability context. - Use a sanitize guardrail to change only emitted observability. ### Rejection Path From f0cd7ceeef846d929e5c5cda30073fd01f75a849 Mon Sep 17 00:00:00 2001 From: Eric Evans II <194135482+ericevans-nv@users.noreply.github.com> Date: Wed, 19 Aug 2026 11:51:46 -0500 Subject: [PATCH 08/12] Update docs/about-nemo-relay/concepts/middleware.mdx Co-authored-by: Will Killian <2007799+willkill07@users.noreply.github.com> Signed-off-by: Eric Evans II <194135482+ericevans-nv@users.noreply.github.com> --- docs/about-nemo-relay/concepts/middleware.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/about-nemo-relay/concepts/middleware.mdx b/docs/about-nemo-relay/concepts/middleware.mdx index b0893992d..067bb6c1f 100644 --- a/docs/about-nemo-relay/concepts/middleware.mdx +++ b/docs/about-nemo-relay/concepts/middleware.mdx @@ -51,7 +51,7 @@ sanitization and publication rather than awaiting it. Manual lifecycle APIs synchronous and create or close their handle immediately. -Event metadata injectors, Event sanitizers, conditional-execution guardrails, +Event metadata injectors, event sanitizers, conditional-execution guardrails, request intercepts, execution intercepts, and subscribers are not re-entrant. These callbacks must not invoke another NeMo Relay API that runs middleware, flushes subscriber delivery, waits on an exporter, or clears plugins. Scope From f221978bbae76dfcdc0f956b503d648b88c0ea64 Mon Sep 17 00:00:00 2001 From: Eric Evans II <194135482+ericevans-nv@users.noreply.github.com> Date: Wed, 19 Aug 2026 11:51:56 -0500 Subject: [PATCH 09/12] Update docs/about-nemo-relay/concepts/middleware.mdx Co-authored-by: Will Killian <2007799+willkill07@users.noreply.github.com> Signed-off-by: Eric Evans II <194135482+ericevans-nv@users.noreply.github.com> --- docs/about-nemo-relay/concepts/middleware.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/about-nemo-relay/concepts/middleware.mdx b/docs/about-nemo-relay/concepts/middleware.mdx index 067bb6c1f..b99dcffea 100644 --- a/docs/about-nemo-relay/concepts/middleware.mdx +++ b/docs/about-nemo-relay/concepts/middleware.mdx @@ -29,7 +29,7 @@ through an asynchronous Relay API or queued event publication. Worker and native-plugin middleware can also complete asynchronously. Within each middleware chain, Relay awaits entries sequentially in priority order so later callbacks observe earlier middleware output. Event payload transformations, -Event metadata injectors, and Event sanitizer chains run on the queued +event metadata injectors, and event sanitizer chains run on the queued publication path and do not delay managed execution. The experimental raw C FFI and Go binding retain synchronous middleware From 391caccc631568a6f78276ecf27fb271d6b7beb0 Mon Sep 17 00:00:00 2001 From: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> Date: Wed, 19 Aug 2026 12:12:46 -0500 Subject: [PATCH 10/12] docs: use sentence casing for event terminology Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> --- docs/about-nemo-relay/concepts/middleware.mdx | 28 +++++++++---------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/about-nemo-relay/concepts/middleware.mdx b/docs/about-nemo-relay/concepts/middleware.mdx index b99dcffea..c48a28607 100644 --- a/docs/about-nemo-relay/concepts/middleware.mdx +++ b/docs/about-nemo-relay/concepts/middleware.mdx @@ -14,8 +14,8 @@ calls, injects metadata into events, and sanitizes emitted mark and scope events ## What Middleware Is Middleware can control or transform tool and LLM execution. It can also enrich -or sanitize emitted `Event` objects. NeMo Relay applies each middleware type at -a specific lifecycle point. +or sanitize emitted events. NeMo Relay applies each middleware type at a +specific lifecycle point. Middleware is organized by lifecycle meaning rather than as one undifferentiated hook system. @@ -126,7 +126,7 @@ Choose the middleware type that matches the behavior you need: rejected. - Use a **request intercept** when the real request must change before the call. - Use an **execution intercept** when code must run before or after the callback. -- Use an **Event metadata injector** when every emitted `Event` should receive +- Use an **event metadata injector** when every emitted event should receive additional flat metadata before sanitization. - Use a **sanitize guardrail** when only subscribers and exporters should see rewritten data. @@ -152,7 +152,7 @@ directly or asynchronously. Event metadata injectors fail open. If an injector fails, panics, or returns invalid attributes, Relay logs the failure, ignores only that injector's output -for the current `Event`, and continues with later injectors using the `Event` as +for the current event, and continues with later injectors using the event as it stood before the failed injector. ## Intercepts @@ -190,9 +190,9 @@ object. ## Event Metadata Injection -Event metadata injectors add values to the existing Event `metadata` object. -They run on every delivered Event and receive the current Event as immutable -context, so a callback can inspect the Event kind and name before deciding what +Event metadata injectors add values to the existing event `metadata` object. +They run on every delivered event and receive the current event as immutable +context, so a callback can inspect the event kind and name before deciding what to return. Event metadata keys contain one or more non-empty segments made of ASCII @@ -204,14 +204,14 @@ be strings, numbers, booleans, or homogeneous lists of those primitive types. Nested objects, `null`, and mixed-type lists are rejected as one atomic callback result. -Injection is insert-only. Existing Event metadata is never overwritten. +Injection is insert-only. Existing event metadata is never overwritten. Injectors run in ascending numeric priority and then by registration name, so the first injector to insert a missing key wins. A later injector that returns the same key cannot replace it. Register injectors globally, on an owning scope, or through a plugin registration context. Injectors execute on the queued serial publication path, -before Event sanitizers, so they do not delay the managed tool or LLM callback. +before event sanitizers, so they do not delay the managed tool or LLM callback. Sanitizers retain the final decision to preserve, rewrite, or remove injected metadata before subscriber and exporter delivery. @@ -268,10 +268,10 @@ arguments passed to the callback or the real value returned to the caller. Scope operations, marks, and manual or managed tool/LLM lifecycle calls do not await observability middleware. At emission time Relay snapshots the event-only -payload, visible Event metadata injectors, sanitizer chains, and subscribers, +payload, visible event metadata injectors, sanitizer chains, and subscribers, then places the work on a serial dispatcher. The dispatcher completes the -internal tool or LLM Event payload transformation, applies Event metadata -injectors, runs Event sanitizers, and publishes the Event later in FIFO order. +internal tool or LLM event payload transformation, applies event metadata +injectors, runs event sanitizers, and publishes the event later in FIFO order. Subscriber and exporter delivery is therefore delayed, while start/end/mark order is preserved. Closing a scope or deregistering middleware after emission @@ -338,8 +338,8 @@ The phases run as follows: 3. **Response phase:** Relay snapshots and enqueues the end event's observability copy, then returns the real result. 4. **Publication path:** The serial dispatcher completes the internal request or - response Event payload transformation, applies Event metadata injectors, runs - the matching scope-Event sanitizers, and finally delivers the Event to + response event payload transformation, applies event metadata injectors, runs + the matching scope-event sanitizers, and finally delivers the event to subscribers and exporters. The start event is submitted before execution begins, but its sanitizers run From c45df566b9669603b6de0be3fcc141b047d36baa Mon Sep 17 00:00:00 2001 From: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> Date: Thu, 20 Aug 2026 14:24:23 -0500 Subject: [PATCH 11/12] docs: narrow middleware warning scope Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> --- docs/about-nemo-relay/concepts/middleware.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/about-nemo-relay/concepts/middleware.mdx b/docs/about-nemo-relay/concepts/middleware.mdx index c48a28607..d604391b5 100644 --- a/docs/about-nemo-relay/concepts/middleware.mdx +++ b/docs/about-nemo-relay/concepts/middleware.mdx @@ -64,6 +64,7 @@ An event sanitizer must await child tasks whose emissions belong to its reserved FIFO position. Relay cancels detached async sanitizer tasks after the sanitizer returns. Detached blocking work runs without that sanitizer publication context and must not emit events for the completed publication. + An execution intercept may invoke the `next` continuation supplied to that callback. This is the only supported way for an intercept to enter the @@ -82,7 +83,6 @@ application-owned `result` and an optional opaque `annotation`. A forwarding intercept must preserve both fields in its `ToolExecutionInterceptOutcome` or deliberately replace or remove the annotation. Relay retains pending marks separately from this application-visible result. - ## Registration Levels From 7389dd3d36476e4b5cc6ad3c265e284b6b274a53 Mon Sep 17 00:00:00 2001 From: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> Date: Thu, 20 Aug 2026 14:53:33 -0500 Subject: [PATCH 12/12] docs: address middleware review feedback Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com> --- docs/about-nemo-relay/concepts/middleware.mdx | 31 ++++++++++--------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/docs/about-nemo-relay/concepts/middleware.mdx b/docs/about-nemo-relay/concepts/middleware.mdx index d604391b5..bde15a9f9 100644 --- a/docs/about-nemo-relay/concepts/middleware.mdx +++ b/docs/about-nemo-relay/concepts/middleware.mdx @@ -55,11 +55,11 @@ Event metadata injectors, event sanitizers, conditional-execution guardrails, request intercepts, execution intercepts, and subscribers are not re-entrant. These callbacks must not invoke another NeMo Relay API that runs middleware, flushes subscriber delivery, waits on an exporter, or clears plugins. Scope -APIs remain supported: -callbacks may create, push, or pop scopes at any nesting level and may replace -the active scope stack with an arbitrary stack. Emitting a new event is the -only supported operation that can enqueue additional callback work; Relay -queues that event for later publication instead of recursively dispatching it. +APIs remain supported. Callbacks may create, push, or pop scopes at any nesting +level and may replace the active scope stack with an arbitrary stack. Emitting +a new event is the only supported operation that can enqueue additional +callback work; Relay queues that event for later publication instead of +recursively dispatching it. An event sanitizer must await child tasks whose emissions belong to its reserved FIFO position. Relay cancels detached async sanitizer tasks after the sanitizer returns. Detached blocking work runs without that sanitizer @@ -150,10 +150,10 @@ execution intercept cannot undo work from a `next` continuation that it already invoked. This fail-closed contract is the same whether the callback completes directly or asynchronously. -Event metadata injectors fail open. If an injector fails, panics, or returns -invalid attributes, Relay logs the failure, ignores only that injector's output -for the current event, and continues with later injectors using the event as -it stood before the failed injector. +Event metadata injectors fail open. If an event metadata injector fails, panics, +or returns invalid attributes, Relay logs the failure, ignores only that +injector's output for the current event, and continues with later injectors +using the event as it stood before the failed injector. ## Intercepts @@ -199,12 +199,12 @@ Event metadata keys contain one or more non-empty segments made of ASCII letters, digits, underscores, or hyphens. Segments may be separated by single dots, so keys such as `region.zone.name` are valid. Leading dots, trailing dots, consecutive dots, whitespace, and unsupported punctuation are invalid. Relay -stores accepted keys literally; dots do not create nested objects. Values may -be strings, numbers, booleans, or homogeneous lists of those primitive types. -Nested objects, `null`, and mixed-type lists are rejected as one atomic callback -result. +stores accepted keys literally, which is why dots do not create nested objects. +Values may be strings, numbers, booleans, or homogeneous lists of those +primitive types. Nested objects, `null`, and mixed-type lists are rejected as +one atomic callback result. -Injection is insert-only. Existing event metadata is never overwritten. +Injection is insert-only and existing event metadata is never overwritten. Injectors run in ascending numeric priority and then by registration name, so the first injector to insert a missing key wins. A later injector that returns the same key cannot replace it. @@ -277,7 +277,8 @@ Subscriber and exporter delivery is therefore delayed, while start/end/mark order is preserved. Closing a scope or deregistering middleware after emission does not affect queued snapshots. Injector failures fail open and omit that injector's output. Sanitizer failures fail closed: Relay records the callback -failure and withholds the governed observability payload. +failure, clears the governed observability fields, and continues publication +with the event shell. ## Managed Execution Order