Feat/runtime tracing - Runtime observability: tracing foundation and propagation

.github/workflows/ci.yml

@@ -222,7 +222,9 @@ jobs:
 
       - name: Install dependencies
         run: |
-          pip install pytest hypothesis boto3 strands-agents strands-agents-tools
+          pip install boto3 strands-agents strands-agents-tools
+          pip install -r arbiter/requirements.txt
+          pip install -r arbiter/requirements-dev.txt
 
       - name: Run tests
         run: pytest arbiter/ -v --tb=short

.gitignore

@@ -15,6 +15,7 @@ env/
 ENV/
 .venv
 .venv-test/
+.venv-check/
 
 # E2E test artifacts
 scripts/*

README.md

@@ -49,6 +49,8 @@ State is event-driven and eventually consistent: EventBridge carries `citadel.*`
 
 **13 Integration Types.** Seven SaaS connectors (Confluence, Jira, ServiceNow, Slack, Microsoft, PagerDuty, Zendesk), three AgentCore types (AWS Lambda for custom logic, AWS Services via Smithy, external MCP servers), and three legacy connectors (SharePoint, Salesforce, GitHub — partially implemented). Credentials are vended through scoped, short-lived STS sessions and stored in Secrets Manager.
 
+**Runtime Observability.** A one-click waterfall trace viewer shows the nested X-Ray span tree behind any workflow execution or agent conversation, with per-span durations and fault/error/throttle badges, deep-linked from execution detail sheets and conversation views. The viewer never fakes data: honest, explicit states cover loading, X-Ray's eventual-availability indexing window, a genuinely empty trace, cross-org unauthorized access, and an unconfigured deployment. A platform-health CloudWatch dashboard and six SLO alarms (workflow node failure/queue-wait, AppSync 5xx, DLQ depth, cost-reconciler staleness, cost drift) surface on-call-ready signals, and governance decisions link bidirectionally to the runtime trace they were made inside of. A server-minted `runId` (never client-supplied) correlates a flow across every hop — dispatch, trace annotations, and the cost ledger — and downloadable, sanitised replay packages let an execution or conversation be re-examined offline. See [docs/OBSERVABILITY.md](docs/OBSERVABILITY.md).
+
 **AI-Accelerated Modernization Governance.** A first-class governance engine that makes the agent system *accountable*:
 - Architecture Decision Records with locking and controlled reopen attempts, execution specifications with an approval lifecycle, interrogation rounds (with encrypted transcripts), agent design assessments, and program reviews against a structured checklist.
 - A constitutional rule hierarchy, case law, authority units, and composition contracts that constrain what agents are permitted to do.
@@ -114,6 +116,10 @@ Prerequisites: AWS CLI, Node.js 24+, Python 3.14+, CDK 2.100+, and Finch (or Doc
 - **[docs/DEPLOYMENT.md](docs/DEPLOYMENT.md)** - Complete deployment guide
 - **[docs/QUICK_START.md](docs/QUICK_START.md)** - 5-minute quick start
 - **[docs/EVENTBRIDGE_CATALOG.md](docs/EVENTBRIDGE_CATALOG.md)** - EventBridge event catalog (all event types and schemas)
+- **[docs/OBSERVABILITY.md](docs/OBSERVABILITY.md)** - Waterfall trace viewer, platform-health dashboard + SLO alarms, decision↔trace linking
+- **[docs/TRACING_RUNBOOK.md](docs/TRACING_RUNBOOK.md)** - Cross-service trace propagation contract, Transaction Search constraint, and cutover procedure
+- **[docs/REPLAY_PACKAGE.md](docs/REPLAY_PACKAGE.md)** - Sanitised execution/conversation replay package envelope and eval-ingestion contract
+- **[docs/COST_QUERY.md](docs/COST_QUERY.md)** - Cost query API and budget alerts
 - **[docs/RESOLVER_GUIDE.md](docs/RESOLVER_GUIDE.md)** - Lambda resolver development guide
 - **[docs/ADAPTER_GUIDE.md](docs/ADAPTER_GUIDE.md)** - Adapter development guide (adding datastores/integrations)
 - **[docs/AGENT_APPS.md](docs/AGENT_APPS.md)** - Agent Apps platform architecture

arbiter/activator/requirements.txt

@@ -1,3 +1,4 @@
 boto3>=1.42.0
+aws-xray-sdk==2.15.0
 strands-agents==1.30.0
 strands-agents-tools==0.8.5

arbiter/common/__tests__/test_metrics_constants.py

@@ -0,0 +1,28 @@
+"""Literal-value pin tests for arbiter/common/metrics_constants.py.
+
+The downstream dashboards story depends on these exact strings — a rename
+here is a breaking change for that story, so every literal is pinned by an
+explicit equality assertion (not just "is a string" / "is truthy").
+"""
+from common import metrics_constants as mc
+
+
+def test_namespace_is_pinned():
+    assert mc.METRIC_NAMESPACE == 'Citadel/Workflows'
+
+
+def test_metric_names_are_pinned():
+    assert mc.METRIC_NODE_DURATION_MS == 'NodeDurationMs'
+    assert mc.METRIC_NODE_FAILURE == 'NodeFailure'
+    assert mc.METRIC_NODE_QUEUE_WAIT_MS == 'NodeQueueWaitMs'
+    assert mc.METRIC_NODE_COLD_START == 'NodeColdStart'
+
+
+def test_units_are_pinned():
+    assert mc.UNIT_MILLISECONDS == 'Milliseconds'
+    assert mc.UNIT_COUNT == 'Count'
+
+
+def test_dimension_keys_are_pinned():
+    assert mc.DIMENSION_WORKFLOW_ID == 'WorkflowId'
+    assert mc.DIMENSION_AGENT_ID == 'AgentId'

arbiter/common/__tests__/test_metrics_constants_unstamped_dispatch.py

@@ -0,0 +1,15 @@
+"""Literal-value pin test for the new UnstampedDispatch WARN metric name
+(Pass 1, decision f1cbd5ef — runtime backstop layer). Pinned per the
+project's "do NOT retype metric names" lesson: this constant is imported
+everywhere the metric is emitted, never hand-typed as a string literal.
+"""
+from common import metrics_constants as mc
+
+
+def test_unstamped_dispatch_metric_name_is_pinned():
+    assert mc.METRIC_UNSTAMPED_DISPATCH == 'UnstampedDispatch'
+
+
+def test_shares_existing_namespace_and_count_unit_convention():
+    assert mc.METRIC_NAMESPACE == 'Citadel/Workflows'
+    assert mc.UNIT_COUNT == 'Count'

arbiter/common/__tests__/test_run_id.py

@@ -0,0 +1,43 @@
+"""Tests for arbiter/common/run_id.py — run_id mint format/uniqueness and
+the build_dispatch_context required-argument guard (Pass 1, decision
+f1cbd5ef).
+"""
+import re
+
+import pytest
+
+from common import run_id as run_id_mod
+
+_RUN_ID_RE = re.compile(
+    r"^run-[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$",
+    re.IGNORECASE,
+)
+
+
+def test_mint_run_id_produces_run_prefixed_uuid_format():
+    value = run_id_mod.mint_run_id()
+    assert _RUN_ID_RE.match(value)
+
+
+def test_mint_run_id_is_greppable_via_run_prefix():
+    value = run_id_mod.mint_run_id()
+    assert value.startswith("run-")
+
+
+def test_mint_run_id_produces_unique_values():
+    values = {run_id_mod.mint_run_id() for _ in range(100)}
+    assert len(values) == 100
+
+
+def test_build_dispatch_context_includes_required_run_id():
+    ctx = run_id_mod.build_dispatch_context(run_id="run-abc", extra="value")
+    assert ctx["run_id"] == "run-abc"
+    assert ctx["extra"] == "value"
+
+
+def test_build_dispatch_context_requires_run_id_keyword():
+    """Build-time-equivalent guard for Python: omitting run_id raises
+    TypeError immediately (no default), mirroring the TS DispatchContext
+    type's required `runId` field."""
+    with pytest.raises(TypeError):
+        run_id_mod.build_dispatch_context(extra="value")  # type: ignore[call-arg]