Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/otel-conformance-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,13 +61,13 @@ jobs:
actions: write
contents: read
id-token: write
uses: aws/aws-durable-execution-conformance-tests/.github/workflows/opentelemetry-orchestrator.yml@f9998f305f1e26423baf0c58148a3bd69120d5ef
uses: aws/aws-durable-execution-conformance-tests/.github/workflows/opentelemetry-orchestrator.yml@02d6dca971a38c13d94d6233d12f687e55b2a572
with:
language: java
resource_prefix: j
sdk_repository: aws/aws-durable-execution-sdk-java
sdk_ref: ${{ github.event.pull_request.head.sha || github.sha }}
conformance_test_ref: ${{ inputs.conformance_test_ref || '91740c98b496409fa9f1bb8e8e6c329ca8b0185f' }}
conformance_test_ref: ${{ inputs.conformance_test_ref || '02d6dca971a38c13d94d6233d12f687e55b2a572' }}
checkout_sdk: true
# Build the handlers from this repo's checked-out module instead of the conformance repo's
# bundled examples/java. Path is relative to the conformance workspace where the SDK is
Expand Down
59 changes: 38 additions & 21 deletions otel-plugin/README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# AWS Durable Execution SDK - OpenTelemetry Plugin

OpenTelemetry instrumentation plugin for the AWS Lambda Durable Execution SDK for Java. Emits a deterministic Workflow trace for durable-execution correlation while keeping each Invocation span in the ambient Lambda trace.
OpenTelemetry instrumentation plugin for the AWS Lambda Durable Execution SDK for Java. Anchors every durable execution on one trace so the Workflow span and its per-invocation spans stay correlated, joining the propagated backend trace when one is present.

## Features

- **Deterministic Workflow Traces**: Workflow trace IDs are derived from the execution start time and ARN; stable span IDs are derived from the ARN
- **Ambient Invocation Traces**: Invocation spans inherit the active Lambda/X-Ray context, or receive a fresh provider-generated root trace ID
- **Backend-parented execution trace**: The Workflow span parents onto the execution ancestor resolved at invocation start — a propagated remote context, or a synthetic execution root — for one trace ID that is stable across all invocations, plus a stable span ID derived from the ARN
- **Ambient Invocation Traces**: Invocation spans inherit the active Lambda/X-Ray context, or join the execution ancestor so they stay on the execution trace
- **Scoped ID Generation**: Unrelated instrumentation scopes retain their provider's normal root trace ID generation
- **Span-per-Operation**: Each durable operation (step, wait, map, etc.) gets its own span with accurate timing
- **Attempt Spans**: Each user function execution (step attempt, child context run) gets a span, including retries
Expand Down Expand Up @@ -92,7 +92,7 @@ Build the plugin layer ZIP with the OTel plugin JAR at `java/lib/aws-durable-exe

### 2. AWS X-Ray Active Tracing

Enable active tracing on your Lambda function so the `_X_AMZN_TRACE_ID` environment variable is populated at invocation time. The plugin uses this header to parent Invocation spans to the ambient Lambda/X-Ray trace. The Workflow trace remains independent and deterministic.
Enable active tracing on your Lambda function so the `_X_AMZN_TRACE_ID` environment variable is populated at invocation time. The plugin uses this header both to parent Invocation spans to the ambient Lambda/X-Ray trace and to anchor the execution trace on the propagated context when it carries a complete parent and an explicit sampling decision.

**AWS Console:** Lambda > Configuration > Monitoring and operations tools > Active tracing > Enable

Expand Down Expand Up @@ -157,29 +157,46 @@ The function's execution role needs the `AWSXRayDaemonWriteAccess` managed polic

## Trace Structure

With `InvocationOtelPlugin`, the plugin creates two correlated traces:
The whole execution shares one trace, anchored at the execution ancestor resolved at invocation start. When the backend propagates a valid remote server span (`Root` and `Parent`), that span is the ancestor and the Workflow and Invocation spans nest under it, alongside the ambient Lambda spans on the same trace:

```
Workflow trace:
Workflow (deterministic trace/span IDs, exported once)

Ambient invocation trace:
Lambda/X-Ray parent
└── Invocation
├── fetch-data
│ └── fetch-data attempt 1
├── cool-down
└── process
└── process attempt 1
Remote backend server span (Root / Parent)
├── Workflow (stable span ID, exported once)
├── Ambient Lambda span 1
│ └── Invocation 1
├── Ambient Lambda span 2
│ └── Invocation 2
└── Invocation N (direct child when no same-trace ambient span exists)
```

- **Workflow span** — one logical root per durable execution with a deterministic, X-Ray-compatible trace ID derived from the execution start time and ARN, plus a stable span ID derived from the ARN. Exported only on the terminal invocation (SUCCEEDED/FAILED).
- **Invocation span** — one per Lambda invocation, parented to ambient context when available
When no valid remote parent can be constructed, a synthetic execution root anchors the trace instead and both spans parent onto it:

```
Synthetic execution root
├── Workflow
├── Invocation 1
├── Invocation 2
└── Invocation N
```

- **Execution ancestor** — the common parent both the Workflow and Invocation spans resolve onto. A valid remote server span (`Root` and `Parent`) is used directly, whether or not `Sampled` is present; only when a valid remote parent cannot be constructed does a synthetic execution root take its place. It is a non-recording context, not an exported span.
- **Workflow span** — one logical span per durable execution, joining the execution trace with a stable span ID derived from the ARN. Exported only on the terminal invocation (SUCCEEDED/FAILED).
- **Invocation span** — one per Lambda invocation, parented to the ambient span only when it is on the execution trace, otherwise to the execution ancestor
- **Operation span** — one per durable operation, named after your step/wait names
- **Attempt span** — one per user function execution (retries produce additional attempt spans)

Operation and attempt spans link to the Workflow span. `ExecutionOtelPlugin` reverses that relationship: operations are children of Workflow and link to the current Invocation span.

### Sampling

The sampling decision follows this precedence, highest first:

1. **Backend decision** — `Sampled=1` / `Sampled=0` in the propagated header is authoritative and always preserved.
2. **Configured sampler** — when the header carries no usable `Sampled` value, the configured sampler decides: the customer's sampler when one is set, otherwise the ADOT/community default. It is evaluated with the real trace ID, span name, and attributes, so a trace-ID-ratio sampler produces a stable decision across reinvocations (the trace ID is stable). The sampler is obtained in whichever way is available:
- **Application-owned provider** (for example a `SdkTracerProvider` passed to the plugin) — read directly from the provider.
- **Java-agent path** — the agent's provider is behind a classloader boundary and cannot be read directly, but the settings that configured it are process-global. The plugin rebuilds the equivalent sampler from `OTEL_TRACES_SAMPLER` / `OTEL_TRACES_SAMPLER_ARG` (or the `otel.traces.sampler` / `otel.traces.sampler.arg` system properties). All locally reproducible kinds are honored: `always_on`, `always_off`, `traceidratio`, and their `parentbased_*` variants.
3. **Default to sampled** — used only when no sampler decision can be obtained: nothing is configured, or the configured sampler cannot be reproduced locally. This covers remote samplers (`xray`, `jaeger_remote`, `parentbased_jaeger_remote`), which fetch their decision from a backend, and custom samplers registered inside the agent (by SPI name, a sampler customizer, or declarative config). Defaulting to sampled is deliberate and safe: dropping would be authoritative under a parent-based sampler and would silently discard the whole execution trace, and it matches the OTel default sampler (`parentbased_always_on`) for root spans. For precise control in these cases, set an explicit `Sampled` value upstream (for example by enabling X-Ray active tracing) or use an application-owned tracer provider.

## Span Attributes

### Invocation Span
Expand Down Expand Up @@ -305,7 +322,7 @@ The plugin's spans do not appear as nested subsegments of the Lambda platform se

### Workflow Span

The Workflow span appears in a separate deterministic trace because it uses `setNoParent()`. Invocation spans remain in the ambient Lambda/X-Ray trace. Links correlate durable operations with the other trace.
The Workflow span joins the execution trace by parenting onto the execution ancestor: the propagated remote server span when one is valid, otherwise a synthetic execution root. Either way it shares the execution trace ID and keeps its stable, ARN-derived span ID.

## Verification

Expand All @@ -314,8 +331,8 @@ After deploying your function with the plugin configured:
1. **Invoke your durable function** — trigger at least one execution that includes multiple steps or a wait/resume cycle.

2. **Check CloudWatch console** — Navigate to CloudWatch > Traces. Enable "Group by nodes" to see:
- A deterministic Workflow trace covering the entire execution
- Ambient Lambda traces containing one Invocation span per Lambda invocation
- One execution trace covering the whole execution, with the Workflow span and each Invocation span sharing its trace ID
- One Invocation span per Lambda invocation
- Child spans for each durable operation (named after your step names)
- Links between durable Workflow/operation spans and Invocation spans

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
// SPDX-License-Identifier: Apache-2.0
package software.amazon.lambda.durable.otel;

import io.opentelemetry.sdk.trace.samplers.Sampler;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
* Reconstructs the configured trace sampler from OpenTelemetry autoconfiguration settings.
*
* <p>On the Java-agent path the agent-configured tracer provider is not usable from the application classloader (its
* {@code SdkTracerProvider} type has a different class identity), so the sampler object cannot be read directly. The
* autoconfiguration <em>settings</em> that produced it, however, are process-global and readable: the OpenTelemetry
* Java agent is configured entirely through the zero-code autoconfigure module via the {@code otel.traces.sampler} /
* {@code otel.traces.sampler.arg} system properties or the {@code OTEL_TRACES_SAMPLER} /
* {@code OTEL_TRACES_SAMPLER_ARG} environment variables. This resolver reads those and rebuilds the equivalent
* {@link Sampler} using the SDK's own factories, so the reconstructed sampler matches the agent's semantics and can be
* evaluated with the real trace ID, span name, and attributes.
*
* <p>Only the locally reproducible sampler kinds are rebuilt: {@code always_on}, {@code always_off},
* {@code traceidratio}, and their {@code parentbased_*} variants. Remote samplers ({@code xray}, {@code jaeger_remote},
* {@code parentbased_jaeger_remote}) depend on runtime state fetched from a backend and cannot be reproduced locally;
* custom samplers registered by name or by a customizer SPI live in the agent and are likewise out of reach. For those,
* and when nothing is configured, this returns {@code null} so the caller applies its documented default.
*/
final class ConfiguredSamplerResolver {

private static final Logger logger = LoggerFactory.getLogger(ConfiguredSamplerResolver.class);

private static final String SAMPLER_PROPERTY = "otel.traces.sampler";
private static final String SAMPLER_ARG_PROPERTY = "otel.traces.sampler.arg";
private static final String SAMPLER_ENV = "OTEL_TRACES_SAMPLER";
private static final String SAMPLER_ARG_ENV = "OTEL_TRACES_SAMPLER_ARG";

// Per the OpenTelemetry spec, traceidratio defaults to sampling everything when no ratio arg is provided.
private static final double DEFAULT_RATIO = 1.0d;

private ConfiguredSamplerResolver() {}

/**
* The sampler described by the OTel autoconfiguration settings, or {@code null} when nothing is configured or the
* configured sampler cannot be reproduced locally (remote or custom samplers).
*/
static Sampler resolve() {
return resolve(configValue(SAMPLER_PROPERTY, SAMPLER_ENV), configValue(SAMPLER_ARG_PROPERTY, SAMPLER_ARG_ENV));
}

/** Package-visible for testing: builds the sampler from explicit values without reading the environment. */
static Sampler resolve(String samplerName, String samplerArg) {
if (samplerName == null || samplerName.isBlank()) {
return null;
}
return switch (samplerName.trim()) {
case "always_on" -> Sampler.alwaysOn();
case "always_off" -> Sampler.alwaysOff();
case "traceidratio" -> Sampler.traceIdRatioBased(ratio(samplerArg));
case "parentbased_always_on" -> Sampler.parentBased(Sampler.alwaysOn());
case "parentbased_always_off" -> Sampler.parentBased(Sampler.alwaysOff());
case "parentbased_traceidratio" -> Sampler.parentBased(Sampler.traceIdRatioBased(ratio(samplerArg)));
// Remote samplers (xray, jaeger_remote, parentbased_jaeger_remote) and any custom/unknown name cannot be
// reproduced from configuration alone.
default -> {
logger.debug(
"Configured sampler '{}' cannot be reproduced locally; using the default decision.",
samplerName);
yield null;
}
};
}

/**
* Parses and normalizes the ratio argument to a value {@code Sampler.traceIdRatioBased} accepts, i.e. within
* {@code [0.0, 1.0]}. {@code Double.parseDouble} would otherwise accept negatives, values above 1, NaN, and
* infinities, and passing those to the sampler throws {@code IllegalArgumentException} — an exception that, during
* {@code onInvocationStart}, is swallowed by the plugin runner and silently disables telemetry for the invocation.
* Following the spec's documented clamping, a finite value is clamped into range; an unparseable or non-finite
* value falls back to the default (sample everything). Every non-exact value is logged.
*/
private static double ratio(String samplerArg) {
if (samplerArg == null || samplerArg.isBlank()) {
return DEFAULT_RATIO;
}
double parsed;
try {
parsed = Double.parseDouble(samplerArg.trim());
} catch (NumberFormatException e) {
Comment thread
ayushiahjolia marked this conversation as resolved.
logger.debug(
"Unparseable {} value '{}'; defaulting the sampling ratio to {}.",
SAMPLER_ARG_ENV,
samplerArg,
DEFAULT_RATIO);
return DEFAULT_RATIO;
}
if (!Double.isFinite(parsed)) {
logger.debug(
"Non-finite {} value '{}'; defaulting the sampling ratio to {}.",
SAMPLER_ARG_ENV,
samplerArg,
DEFAULT_RATIO);
return DEFAULT_RATIO;
}
if (parsed < 0.0d) {
logger.debug("{} value '{}' is below 0; clamping the sampling ratio to 0.", SAMPLER_ARG_ENV, samplerArg);
return 0.0d;
}
if (parsed > 1.0d) {
logger.debug("{} value '{}' is above 1; clamping the sampling ratio to 1.", SAMPLER_ARG_ENV, samplerArg);
return 1.0d;
}
return parsed;
}

/** Reads a setting, preferring the system property (matching OTel autoconfigure precedence) then the env var. */
private static String configValue(String systemProperty, String environmentVariable) {
var value = System.getProperty(systemProperty);
if (value == null || value.isBlank()) {
value = System.getenv(environmentVariable);
}
return value;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,33 @@
package software.amazon.lambda.durable.otel;

/**
* Extracts trace context from the Lambda runtime environment.
* Extracts the durable execution's propagated trace context from the Lambda runtime environment.
*
* <p>Implementations read trace context from various sources (X-Ray trace header, W3C traceparent, etc.) and return an
* {@link ExtractedContext} containing the trace ID and optional parent span ID.
*
* <p>Plugins use a valid ambient OpenTelemetry span as the invocation parent when one is available. This extractor is
* consulted only when no ambient span context is available, providing fallback propagation context from the runtime
* environment.
* <p><strong>When it is called:</strong> the plugin invokes {@link #extract()} once at the start of every invocation,
* unconditionally — including when an ambient OpenTelemetry span is active. The extracted context is the durable
* execution's identity and is resolved with the following precedence:
*
* <ol>
* <li>a valid extracted backend context anchors the execution trace (this is what makes the durable spans share one
* stable trace across reinvocations, so it takes precedence over the per-invocation ambient span);
* <li>otherwise the execution is anchored on a deterministic synthetic root derived from the execution ARN;
* <li>the ambient span is never adopted as the execution trace. It is used only to parent the Invocation span when it
* is already on the resolved trace, and otherwise correlated with a span link.
* </ol>
*
* <p><strong>Implementation contract:</strong> because {@code extract()} runs on every invocation, implementations must
* be side-effect-free (or idempotent) and cheap, and must return the durable execution's own context — returning stale
* or unrelated context will displace the correct execution trace.
*/
@FunctionalInterface
public interface ContextExtractor {

/**
* Extracts fallback trace context from the runtime environment.
* Extracts the durable execution's propagated trace context from the runtime environment. Called once per
* invocation, unconditionally.
*
* @return the extracted context, or {@code null} if no context is available
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,22 @@ String generateTraceIdForExecution(String arn, Instant executionStartTime) {
}

String generateWorkflowSpanId(String arn) {
var seed = "workflow:" + (arn != null ? arn : "");
return deterministicSpanId("workflow:" + (arn != null ? arn : ""));
}

/**
* Generates the deterministic span ID for the synthetic execution root from the execution ARN, using a seed
* namespace distinct from the Workflow and operation span IDs. Stable across reinvocations so the synthetic root is
* the same common ancestor every invocation.
*
* @param arn the durable execution ARN
* @return a deterministic 16-char hex span ID
*/
String generateExecutionRootSpanId(String arn) {
return deterministicSpanId("execution-root:" + (arn != null ? arn : ""));
}

private static String deterministicSpanId(String seed) {
var spanId = sha256(seed).substring(0, 16);
if (spanId.equals("0000000000000000")) {
spanId = "0000000000000001";
Expand Down
Loading
Loading