Feat/runtime tracing - Runtime observability: tracing foundation and propagation - #49
Open
gibbsie wants to merge 25 commits into
Open
Feat/runtime tracing - Runtime observability: tracing foundation and propagation#49gibbsie wants to merge 25 commits into
gibbsie wants to merge 25 commits into
Conversation
added 2 commits
July 28, 2026 14:33
…ll stacks Every application Lambda across the eight Lambda-bearing stacks now runs with active tracing applied by a single aspect, and — the real fix — carries the X-Ray write policy that was missing: backend and arbiter functions already had tracing configured but their segments were silently denied at the API, so no traces ever arrived. Shared AWS clients are instrumented at their two factory points, giving DynamoDB and event-bus subsegments across every consumer without per-handler edits; arbiter Python entries patch botocore once, idempotently and safely without a daemon, covering the model-invocation call sites. The GraphQL API's tracing flag is pinned by test. A tracing-only template diff gate proves the pinned backend stack's delta is exclusively the write-policy additions. The committed baseline is deliberately NOT regenerated: reproduction showed regeneration destroys the relocation history the resolver-parity and privilege-equivalence rails depend on, and the existing rails pass unchanged — documented in the gate runner. Framework singletons (log-retention, bucket deployment providers) are excluded from tracing by documented design. Container-internal intake spans stay deferred pending a platform verification spike; HTTP APIs do not support X-Ray and are stated as such.
Event emissions on both languages now carry trace context additively — event detail on the bus, message attributes and body on the queue dispatch path, and the worker's return leg — and every consumer entry parses it when present, annotates its segment with the searchable correlation pair, and emits the correlation-id/trace-id pair in structured logs (keys omitted entirely when no segment exists; absent context never fails a consumer, property-tested per hop). Queue hops ride the native trace header; bus fan-out hops stitch by annotation with the key contract pinned by literal-value tests in both languages — the trace viewer story consumes that contract. The intake container ships an honest self-contained no-op helper: its deployment asset excludes the shared module and it runs a different telemetry stack, so the helper degrades cleanly today and activates if the tracing dependency is ever added — documented in the new tracing runbook alongside the operator filter expressions. The scheduled reconciler is documented as a non-consumer rather than given a fabricated extraction point.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| patch.object(index, 'load_file_from_s3_into_tmp'), \ | ||
| patch('subprocess.run', return_value=mock_result): | ||
| record = {'body': json.dumps(NODE_MESSAGE), 'messageId': 'm1'} | ||
| result = index.lambda_handler({'Records': [record]}, {}) |
| except Exception: # noqa: BLE001 — tracing activation must never break dispatch | ||
| logger.exception("Failed to activate X-Ray tracing (patch_all); continuing untraced.") | ||
| finally: | ||
| _configured = True |
added 2 commits
July 29, 2026 01:25
The arbiter job installed an inline package list that drifted from the requirements chain — the tracing dependency never reached CI, failing thirteen tests. The job now installs from the arbiter requirements files plus a new dev requirements file with pinned test dependencies, mirroring the intake service convention; a fresh environment reproduces the suite exactly as CI runs it.
Side-effect tracing imports take the analyzer-accepted aliased form with ordering comments preserved; the worker drops a redundant module import and an unused helper from both import and fallback shim; test imports standardize to module style; an unused destructured binding is skipped. Findings the analyzer got wrong are documented for dismissal rather than applied — the worker handler genuinely returns the partial-batch response its test asserts.
| # client(s) below and before `import events` (which constructs its own | ||
| # EventBridge client at module scope) so patch_all() instruments botocore | ||
| # ahead of any client creation. | ||
| import common.tracing as tracing # import activates tracing as a side effect |
added 12 commits
July 29, 2026 01:50
Three routes on the telemetry API turn X-Ray traces into waterfall-ready JSON: execution and conversation lookups resolve the requested resource to its owning organization in our own tables and refuse before any trace call when the caller's claim does not match; raw trace-id lookup is admin-only because no ownership entry key exists for it. Trace summaries are found by exact match on the pinned correlation annotation — identifiers are allowlist-validated before filter construction, rejecting injection attempts — and segment documents parse defensively, never throwing on malformed input. Non-admin responses drop raw metadata bags by an egress allowlist. The handler role is read-only by test: zero write actions, zero trace-write permissions. A freshness window distinguishes still-indexing from genuinely empty.
A new observability view renders traces as a waterfall: a collapsible span tree with hand-positioned duration bars, a time ruler, and fault-over-error-over-throttle badges, tested down to concrete bar percentages. One click reaches it from a workflow execution's detail sheet or a project conversation. States are honest: still-indexing (with retry) is distinct from empty, unauthorized surfaces its reason, and an unconfigured API renders unavailable with zero network calls. Raw trace-id lookup appears only for administrators, matching the API's authorization matrix, which is documented in the tracing runbook alongside the residual-risk statement; a new observability doc covers the user-facing flow.
A property test in CI surfaced a real defect: the event publisher's success log passed an interpolated template as the console format argument, so an event type containing a printf specifier could coerce the payload object and throw mid-publish. Logging now uses constant format strings with values as arguments across the emitting and consuming call sites, the discovering counterexample is pinned as a regression test proven to fail on the pre-fix code, and the property itself stays unpinned — randomized seeds are what caught this.
Workflow node lifecycles now publish duration and queue-wait metrics, dispatch and worker-start timestamps ride the node contract additively (absent timestamps skip the metric and never disturb execution), and both Lambda tiers report container cold starts exactly once per container via a module-scope marker. Metric names, units and dimensions are declared once per language and pinned by verbatim tests, because the dashboard work consumes them as a contract. Emission is best-effort throughout: a metrics failure never fails a node or a resolver. Execution inspection now shows this run's node duration percentiles, computed client-side from data already fetched. The intake agent runtime is honestly out of scope for cold-start measurement: its telemetry seam is per-turn with no container-lifecycle entry point, so no signal is fabricated.
One dashboard answers whether the platform is healthy: API, workflow, cost-reconciliation, governance and dead-letter sections across twenty-eight widgets, sourced from the pinned metric contracts rather than retyped names. Six alarms cover node failures, queue-wait breaches, GraphQL server errors, dead-letter arrivals, a stalled reconciler and cost-estimate drift — each with a deliberate missing-data stance (sporadic traffic treated as not-breaching; reconciler absence treated as breaching, because silence there IS the failure) and every one wired to the existing alarm topic, adding no new notification plumbing. Thresholds are commented as dev-calibrated with a documented tuning path. The observability doc gains the dashboard and alarm reference plus a runnable fault-injection procedure so a reviewer can prove an alarm fires.
Governance findings now carry the active trace identifier, stamped at write time between evaluation and the ledger write so the fail-closed ordering is untouched: when no trace context exists — or the lookup raises — the finding is written exactly as before, the decision is unchanged, and dispatch is unaffected, proven by property and non-regression tests across all enforcement modes. The write-once condition and retention are byte-for-byte unchanged; the field is additive and nullable end to end. The decision tracer offers a runtime-trace link and the waterfall gains a governance-decisions panel, so 'why was this allowed' and 'what actually ran' are one click apart. An investigation established that the orchestration identifier and the workflow execution identifier are independently generated, so the stamped trace id is the only guaranteed join: identifier-based fallbacks are implemented but labelled best-effort in the UI and the observability doc — an empty result is explicitly not proof that nothing was governed. Findings written before this change degrade with an explanatory disabled state rather than a broken link.
A workflow run can now be bundled into a single versioned artifact — agent configuration, workflow and specification versions, per-node inputs and outputs, governance findings with their trace ids, and token usage — delivered as a short-lived presigned download from an encrypted, public-access-blocked bucket with a seven-day lifecycle. Access is ownership-gated for organisation members and refused before any data read or object write when the caller's organisation does not match. Secret safety is enforced, not assumed: every string at every nesting depth passes a shared redaction module covering eleven credential and personal-data classes, each with positive and near-miss coverage, and a fail-closed gate then rescans the fully serialised bundle and refuses to publish on any hit. A mutation-kill test proves the gate is load-bearing by disabling redaction and requiring both the survival assertion to fail and the gate to throw. Two real defects surfaced during the work: a stateful-regex false negative in the scanner, and a download URL that signed a write instead of a read — the latter would have handed clients a window to overwrite a sanitised artifact. Honest scope: per-node tool results are modelled as explicitly partial because no queryable tool-execution record exists yet, and they are never reconstructed from logs; conversation-keyed assembly is not implemented and its route reports not-found while remaining ownership-gated.
Conversation runs can now be bundled too, reusing the execution path's envelope, deep sanitisation, fail-closed secret gate, ownership gating and short-lived read-only download without changes to the handler or infrastructure. Messages come from the conversation store in chronological order and token totals aggregate from the cost ledger's project index. Sections with no data source are explicitly partial with stated provenance rather than invented: governance findings cannot be joined because nothing ties a project to an orchestration identifier, and tool results remain blocked on a queryable tool-execution record. Nothing is reconstructed from logs.
Every run now carries one correlation identifier, minted on the server at each outermost entry point — chat, task submission, workflow execution start, app invocation and intake turns — and threaded additively through orchestration creation, node dispatch, events, execution rows, cost rows and governance findings. Client-supplied values are never trusted: each entry point strips an inbound identifier and mints its own, proven per entry by test, so a caller cannot forge a correlation key into another tenant's write-once audit trail. Durability is enforced at build time, not by convention: the shared dispatch context makes the identifier a required field, so a future entry point that forgets to thread it fails type-checking — demonstrated by a type error on omission. A test enumerates the entry points, and an unstamped-dispatch metric (using the pinned metric-name constants) backstops at runtime. The governance stamp keeps the ledger's guarantees intact: it is best-effort inside the existing fail-closed ordering, a finding written without an identifier is byte-identical to before (property-tested), and a lookup that returns nothing or raises never alters a decision — verified across permissive, shadow and strict modes, including that a strict denial keeps an identical payload. Write-once findings are not backfilled; pre-identifier rows stay excluded from identifier joins and the interfaces keep saying so.
The trace viewer now correlates on the run identifier when the execution carries one and reports which key linked the result, falling back to the previous correlation key for runs that predate it. Conversation replay bundles join governance findings for real when the run identifier is present — the previously always-partial findings section becomes a genuine array — and keeps its honest partial shape with stated provenance when it is not. The decision trace gains an execution pivot so a finding resolves to the run that produced it. That pivot originally read a single item before applying its filter, which would have found a match only by luck; it is now a paginated scan with an explicit item cap that stops on first match, logs when a cap truncation makes an empty answer non-authoritative, and refuses rows belonging to another organisation. Its regression tests were confirmed to fail against the previous implementation. Documentation across the tracing, observability, replay and cost references now separates what is guaranteed when a run identifier is present from what remains best-effort for older data, and names the two deferred indexes that would turn identifier-only lookup into a guarantee. No wording implies that an empty result means nothing was governed.
The secret-pattern fixtures live in a helper module beside the tests that use them, so Jest was collecting it as a suite containing no tests and failing the run despite every test passing. Helper modules under the test directory are now excluded, matching the existing fixtures convention.
| global _is_cold_start | ||
| if not _is_cold_start: | ||
| return | ||
| _is_cold_start = False |
added 9 commits
July 30, 2026 12:45
Drops an unused mock import and standardises two test modules that imported the executor both as a module and by name onto a single style. Behaviour is unchanged and all three files pass in a dependency-complete environment.
Backlog renumbering shifted the evaluation-epic story identifiers by one, leaving this document pointing at the wrong stories for eval-fixture promotion and baseline comparison. Each reference was re-resolved against the backlog by title rather than by arithmetic, and identifiers belonging to other epics were left untouched.
…aemon The intake container could never stitch its hop: the helper looked for an X-Ray segment, but that runtime has no daemon and nothing injects a trace header, so the context was always empty — and a note in the backlog wrongly claimed adding the X-Ray dependency would activate it. Research proved otherwise, so nothing was installed. The helper now reads the OpenTelemetry span context that the runtime's own instrumentation already provides and renders the identifiers in the platform-compatible form, keeping the previous source as a harmless fallback. Absent or invalid context, and a missing telemetry package, still yield nothing without raising, and the published event still omits the field entirely. Documents a constraint that would otherwise be discovered the hard way: the managed agent observability surface requires account-level transaction search, which redirects trace storage and makes the query APIs this platform's waterfall viewer depends on return nothing — blinding it for every function in every stack. The runbook now states the two coherent choices, the runtime permissions and toggle the agent still needs for its telemetry to appear at all, the sanctioned way to opt out of the managed pipeline, and the session identifier semantics worth joining on.
- TRACE_BACKEND env selects xray (default) or aws/spans Logs Insights path; frontend contract unchanged - Scoped logs:StartQuery IAM; xray:Get* kept for reversible cutover - AgentCore runtime exec-role observability grants - Empty LANGFUSE_* pins removed (ADOT default-on) - Runbook cutover procedure Finding: 5106aa74 | Decision: dc270923 | Task: c7a4bf52
- rewrite stale intake trace-context caveat (live OTel span source since bb35989) + unblock dev probe 4 - document traceContext + runId event-envelope fields in EVENTBRIDGE_CATALOG - add runId server-minting contract + UnstampedDispatch guard to runbook - README docs index (+OBSERVABILITY, TRACING_RUNBOOK, REPLAY_PACKAGE, COST_QUERY) + Runtime Observability capability - ARCHITECTURE TelemetryStack surface + active-tracing aspect pattern - CORS FRONTEND_ORIGIN-placeholder troubleshooting notes (browser NetworkError, curl unaffected) - cutover parenthetical re deploy prerequisite Audit-Task: 6b47bc12
…ceholder CORS
- deploy.sh auto-resolves citadel-frontend-${ENV} FrontendUrl when FRONTEND_ORIGIN unset (bootstrap-safe warn+proceed, trailing slash stripped)
- extracted env>context>placeholder resolver backend/lib/frontend-origin.ts
- CDK Annotations warning with remediation command when placeholder ships outside local/test (warning not error: telemetry deploys before frontend on fresh accounts)
- .env.example documents the var
- corrected inaccurate fails-fast comment
- 21 new tests incl. AllowOrigins-never-placeholder template pin
Finding: d7d3dd61 | Task: 67695642
… ARN - telemetry-stack lib test suite now scaffolds dist/lambda stub via existing scaffold-stub-assets helper (CannotFindAsset on clean CI checkout; previously passed only when a build or sibling suite ran first) - extend existing IAM5 NagSuppression appliesTo with the exact composed aws/spans log-group ARN literal — trailing :* is the log-stream suffix inherent to log-group ARNs, no IAM permission changed - both verified under verbatim ci.yml commands in clean state (5319 tests, cdk synth --all 9 stacks, zero nag errors) Task: 51d3e9bb | Regressions-from: 6e87494
- remove backend-stack duplicate of citadel-appsync-5xx-${env} — telemetry-stack SLO suite owns it (strict superset: threshold 5 vs 10, DatapointsToAlarm 1, SNS action, runbook description)
- duplicate physical name caused CloudFormation EarlyValidation ResourceExistenceCheck rejection of the telemetry changeset, aborting deploys before frontend
- new guard test asserts no AlarmName repeats across synthesized stack templates (proven to bite pre-fix via stash)
- redeploy note: backend deletes old alarm then telemetry recreates it in the same run (brief 5xx-alarm gap accepted)
Task: 00189cd6 | Blocked-deploy-of: 6fecfff telemetry changeset
…osite - Metrics Insights WHERE supports only =/!= — LIKE citadel-%dlq% rejected server-side at alarm CREATE (invisible to synth/CI), rolling back the telemetry stack - DlqNotEmptyAlarm now sums ApproximateNumberOfMessagesVisible over the 6 explicitly-named citadel DLQs (3-way inventory match: code, templates, live account) - Same fix applied to the matching dashboard widget - SEARCH() widget expressions audited — valid grammar, untouched - All 16 telemetry expressions validated against live CloudWatch via read-only get-metric-data, zero ValidationException - New drift-guard test pins DLQ inventory to alarm dimensions (proven to bite) - Note: composite currently reads 2.0 — alarm will fire on deploy, two real messages sit in dev DLQs Task: 363bd66f | Deploy-blocker: attempt-2 2026-07-31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Runtime observability: (23 commits; 177 files, +25,489/−3,241). The original roadmap has fully landed, plus follow-ons that the work surfaced: run identity, replay packages, a Transaction-Search-ready query backend, and deployment CI hardening.
Landed
Active tracing with authorized delivery (6dbce75). A single CDK aspect applies active tracing to all 83 application Lambdas across the eight Lambda-bearing stacks. The headline discovery: backend and arbiter functions already had tracing configured but their roles lacked the X-Ray write policy — every trace segment this platform ever generated was silently denied at the API. The 23 policy additions are the real fix; traces appear in dev immediately on deploy with no configuration. Shared AWS clients are instrumented at their two factory points (DynamoDB and event-bus subsegments across every consumer, no per-handler edits); arbiter Python entries patch botocore once — idempotent and safe without a daemon — covering the model-invocation call sites. A tracing-only template diff gate proves the pinned backend stack's delta was exclusively the policy fix. Honest limits stated: HTTP APIs do not support X-Ray; intake-container internal spans are deferred pending a platform spike.
Trace context propagation across async hops (db0e88a). Event emissions in both languages carry trace context additively (bus detail, queue message attributes and body, and the worker's return leg) and every consumer entry parses it, annotates its segment with a searchable correlation pair, and logs the correlation-id/trace-id pair (keys omitted when no segment exists; absent context never fails a consumer, property-tested per hop). Queue hops ride the native trace header; bus fan-out stitches by annotation, with the key contract pinned by literal-value tests in both languages. New
docs/TRACING_RUNBOOK.mdcarries the operator filter expressions. The intake container ships an honest self-contained no-op helper (its asset excludes the shared module and it runs a different telemetry stack); the scheduled reconciler is documented as a non-consumer rather than given a fabricated hook.Waterfall trace viewer (2c89844, 630e88d). Three ownership-gated routes turn X-Ray traces into waterfall JSON — org membership is checked against our own tables before any trace call; raw trace-id lookup is admin-only; responses are field-allowlisted. The new observability view renders the collapsible waterfall with honest indexing/empty/unauthorized states and deep links from executions and conversations. Authorization matrix and residual-risk statement in the tracing runbook.
Node latency, queue-wait and cold-start metrics (b065101, ff99e14). Per-node telemetry (duration, queue wait, cold start, retries) emitted from the step runner and worker wrapper under a pinned metric-name contract shared across Python and TypeScript, and surfaced in the execution detail sheet. Event logging hardened against format-specifier injection along the way — found by an unpinned property test, pinned with a counterexample proven to bite pre-fix.
Platform health dashboard + SLO alarms (6fecfff).
citadel-platform-health-<env>dashboard and six on-call-ready alarms (workflow node failure and queue-wait, AppSync 5xx, DLQ depth, cost-reconciler staleness, cost drift) wired to the existing alarm topic.Governance decision ↔ runtime trace linking (a8e5d90). Decisions record the trace they were made inside of — write-once, never retro-stamped — and the tracer UI deep-links into the waterfall, with a copyable fallback for non-admins and a bounded runId pivot (cap 1000, cross-org rows skipped, truncation logged).
Server-minted run identity (8ebef1d, 5690480). A runId is minted server-side at every entry point (client-supplied values stripped), threaded through dispatch, trace annotations and the cost ledger, guarded by an UnstampedDispatch metric; query surfaces join on it (run_id-primary, correlation_id fallback).
Sanitised replay packages (06d733c, 4e86927). Downloadable execution and conversation replay envelopes behind a fail-closed secret gate — proven non-vacuous by a mutation-kill test — with three-layer cross-org protection, a dedicated SSE bucket, and ≤300s URLs. The tool-execution ledger gap is documented honestly rather than papered over.
Intake trace context from the live span (bb35989). The intake container now sources trace context from the live OTel span — no daemon, no new dependency — so conversation flows stitch into the same trace graph. This turns the earlier honest no-op helper into the real thing.
Transaction-Search-ready trace backend (6e87494). AgentCore Observability requires CloudWatch Transaction Search, and enabling it blinds the X-Ray query APIs the viewer uses — a documented mutual exclusion. The trace query handler is now dual-backend behind
TRACE_BACKEND(defaultxray, zero behavior change today): the spans path queries theaws/spanslog group via Logs Insights and shapes results into the identical response contract, so cutover is an env flip per the runbook. Allaws/spansfield assumptions are isolated in one module and marked unverified until the account switch makes a real sample available.Deployment + CI hardening (81b8f18, be374e7, 6134933).
deploy.shnow resolvesFRONTEND_ORIGINfrom the frontend stack output instead of silently shipping a placeholder CORS origin that blocked every browser call to the cost and trace APIs in dev (synth warns loudly if the placeholder would ship). CI regressions fixed: order-dependent test asset scaffolding and a nag suppression for the scopedaws/spansgrant. Documentation synced end to end — runbook, observability guide, event catalog (traceContext/runIdenvelope fields), architecture, README index.Testing
Full backend Jest 5,319 passed / 0 failed, verified clean-state and CI-equivalent (no prior build, no ambient env); arbiter pytest 625+ across scoped suites; intake service 340+; synth of all 9 stacks with zero cdk-nag errors; the services template diff for the runtime grants proven scoped to intent; the seven restructuring gates green and untouched. Regression pins proven to bite pre-fix (format-specifier hardening, replay secret gate mutation-kill).
The port preserves the frontend contract byte-for-byte — pinned annotation and IAM tests unchanged.
Reviewer note
TRACE_BACKENDdefaults toxray, so the spans path is inert until the documented cutover (docs/TRACING_RUNBOOK.md): enable account-wide → verify theaws/spansschema against a real sample → flip the env → cleanup removesxray:Get*.xray:Get*cleanup, trace-context propagation into agent invoke, runId as AgentCore session id tool-execution ledger for full replay fidelity.