[dolho]

Summary

Every OTEL-enabled agent's server stopped starting on its next boot. startup.sh upgraded the agent-server dependencies unpinned on every container start, so it pulled FastAPI 0.142. That release auto-configures OpenTelemetry from the standard OTEL_* env and refuses the grpc protocol Trinity injects for Claude Code. The container stayed Up while nothing listened on :8000.

Three layers:

1. The agent server opts out of FastAPI auto-telemetry. agent_server/fastapi_options.py passes telemetry={"auto_configure": False}, but only when the installed FastAPI has that parameter, so a template that pins an older FastAPI still boots. The OTEL_* env belongs to Claude Code, not the server. With this layer, any FastAPI version is safe for the server.
2. No unpinned upgrade, in two files with two jobs:
   • agent-image-requirements.txt holds exact pins, used by the image build, so rebuilds are reproducible (the Default model claude-opus-5-5 fails on cached Claude Code 2.1.278, and the rejection is misclassified as auth (seat rotation + API-key fallback) #3012 lesson).
   • agent-server-requirements.txt holds floors for the server's own packages, used by startup.sh's per-boot repair. It installs what is missing or older than the floor and leaves a newer version alone. The agent server and template code share ~/.local, so a template's own upgrades survive a restart, and the repair no longer touches rich/cryptography.
   • Floors are tested to sit at or below the pins, so a fresh container's repair does nothing.
   • Dependabot now watches /docker/base-image, the pins' bump channel.
3. OTEL_SDK_DISABLED=true is set wherever Claude Code's OTEL env is:
   • at create (crud.py), DB recovery (lifecycle.py) and system-agent create;
   • and on the recreate path (recreate_container_with_updated_config, which replays the old Config.Env) via setdefault, so an operator's explicit value wins. This covers existing agents on restart, drift self-heal or a rebuild pass before the base image is rebuilt.
   • Claude Code does not read the flag (0 references in the 2.1.281 binary), so its grpc metrics keep flowing.

Verification

• Reproduced first: FastAPI 0.142.1 with OTEL_EXPORTER_OTLP_PROTOCOL=grpc exits with FastAPI automatic telemetry requires the OTLP http/protobuf protocol.

• Lifespan startup on 0.142.1 with the grpc env: fails without the opt-out; completes with it; also completes with OTEL_SDK_DISABLED=true alone.

• Rebuilt image, real boots with ENABLE_AGENT_UI=true and the grpc env:

  Case	FastAPI after restart	Health
  Fresh container	0.141.1	200
  Template installed 0.142.1	0.142.1 (not downgraded)	200
  Template installed 0.115.6	raised past the floor	200

• Tests: tests/unit/test_3106_agent_server_fastapi_telemetry.py has 14 tests, including behavioural recreate-path tests through the real lifecycle seam. The neighbouring suites (recreate / env / base-image / startup): 855 passed, 1 skipped.

Side effects

• OTEL_SDK_DISABLED switches off any OpenTelemetry SDK in the container. Nothing that ships uses one (no OTel packages in the image), but a customer template exporting its own traces would stop.
• A below-floor repair installs the latest release of that package rather than the image pin. That's safe for FastAPI because of layer 1.

Operator note

After deploying, rebuild the base image (./scripts/deploy/build-base-image.sh). Until then, existing agents are protected by layer 3 on their next recreate.

Follow-up

A CI step that boots agent_server.main:app inside the built image with the grpc env. The test venv pins FastAPI 0.115, so unit tests cannot run the 0.142 boot.

Fixes #3106

🤖 Generated with Claude Code

[dolho]

/review Report

Branch: fix/3106-agent-server-fastapi-telemetry → dev (merge-base 3a719da27, dev has not moved)
Files Changed: 9 (+213/−18), plus a learnings fragment added by this review
Scope: CLEAN. All three fixes from #3106 are delivered, with nothing unrelated in the diff.
Plan Completion: from the issue's suggested fix. (a) opt-out: DONE. (b) pins: DONE. (c) OTEL_SDK_DISABLED injection: DONE at create/recovery/system-agent. Regression test "boots the agent server with grpc against the latest allowed FastAPI": CHANGED (see I4).

Execution coverage

changed symbol / file	executed by	live consumer	verdict
agent_server/fastapi_options.py::fastapi_app_options	test_opt_out_is_passed…, test_nothing_is_passed…, test_the_installed_fastapi_accepts…	agent_server/main.py:60	✅ executed
FastAPI(**fastapi_app_options(FastAPI)) in main.py	AST test test_main_builds_the_app_with_the_options	the agent container	🛡 guard (CI's FastAPI is 0.115; see I4)
agent-server-requirements.txt	pin/format tests	Dockerfile:218-220, startup.sh:616	🛡 guard
Dockerfile / startup.sh pip lines	source-text tests over the scripts	image build / container boot	🛡 guard (CI cannot build the image)
OTEL_SDK_DISABLED at 3 sites	AST scan over every function assigning OTEL_EXPORTER_OTLP_PROTOCOL	FastAPI inside the agent container	🛡 guard + manual verification

Fix mutations (8 of 8 red). Each change reverted one at a time on a scratch copy:

Mutation	Test that went red
opt-out dropped from main.py	test_main_builds…
helper returns {}	test_opt_out…
boot back to unpinned --upgrade	test_the_boot_repair…
image back to unpinned	test_the_image_installs…
recovery loses the flag	test_every_otel_injection…
system agent loses the flag	test_every_otel_injection…
pin moved to fastapi==0.142.1	test_fastapi_stays_below…
loose uvicorn>= pin	test_every_agent_server_dependency…

Manual verification (from the PR, re-read here):

• On FastAPI 0.142.1 with OTEL_EXPORTER_OTLP_PROTOCOL=grpc, lifespan startup fails without the opt-out and completes with it. It also completes with the env flag alone.
• An image built from the branch serves on the pin. It still serves after a template installs 0.142.1, and the boot repair restores the pin.

Critical Findings

None.

Informational Findings

[I1] Incomplete fix (4.14): OTEL_SDK_DISABLED does not reach existing agents that get recreated (Confidence: 9/10)
File: src/backend/services/agent_service/lifecycle.py:914 (recreate_container_with_updated_config)

• Issue: the flag is added at create (crud.py), DB recovery (recreate_missing_container, lifecycle.py:1565) and system-agent create. A recreate copies the old container's Config.Env (docstring: "NOT applied at the Config.Env copy point"), so a restart, drift self-heal or rebuild pass on an existing agent gets no flag.
• Scope of the gap: such agents are protected only by layers (a)/(b), i.e. once the base image is rebuilt. The PR body's "Agents recreated from an older image are covered by layer 3" overclaims.
• Suggestion: either correct the PR body and operator note (rebuild the base image; layer 3 covers new and DB-recovered agents only), or add one derived mutation in the recreate path next to the other ~20: when OTEL_EXPORTER_OTLP_PROTOCOL is present and OTEL_SDK_DISABLED is not, set it. The second closes the "backend updated, image not yet rebuilt" window.

[I2] Behaviour change: the exact pins now downgrade packages a template installed (Confidence: 8/10)
File: docker/base-image/startup.sh:616, docker/base-image/agent-server-requirements.txt

• Issue: the agent server and template code share ~/.local. The old boot step only ever moved packages up; the new one restores exact versions on every boot. A template that needs, for example, a newer pydantic, httpx or fastapi for its own code gets rolled back each start. rich and cryptography are now in the boot set too, and the old boot step never touched them.
• Suggestion: use floors (fastapi>=0.141.1, …), keep no --upgrade, and limit the boot repair to the server-critical packages (drop rich/cryptography from the boot set). With (a) in place a newer FastAPI no longer breaks the server, so floors keep the outage fixed without the downgrade. Alternatively, keep exact pins and state the trade-off in the PR.

[I3] OTEL_SDK_DISABLED also disables any OpenTelemetry SDK that template code runs (Confidence: 7/10)
File: crud.py, lifecycle.py, system_agent_service.py (new env_vars['OTEL_SDK_DISABLED'])

• Issue: the variable is global to the container. Verified safe for what ships: no opentelemetry Python packages are in the image, and the Claude Code 2.1.281 binary has 0 references to OTEL_SDK_DISABLED (it reads OTEL_METRICS_EXPORTER 8×, OTEL_EXPORTER_OTLP_PROTOCOL 9×). A customer template that exports its own OTel traces would silently stop, though.
• Suggestion: mention it in the operator note. Once the image fix is widely deployed, consider whether layer 3 is still needed.

[I4] The regression test the issue asks for runs only by hand (Confidence: 6/10)
File: tests/unit/test_3106_agent_server_fastapi_telemetry.py

• Issue: the test venv pins FastAPI 0.115 (tests/requirements-test.txt:51), so CI never boots the server under ≥ 0.142 with the grpc env. The helper is tested against both constructor shapes and the wiring by AST, but real-FastAPI behaviour was verified manually.
• Suggestion: a follow-up CI job, or a step in the base-image build workflow, that runs lifespan startup of agent_server.main:app inside the built image with OTEL_EXPORTER_OTLP_PROTOCOL=grpc.

[I5] Nothing will bump the new pins (Confidence: 8/10)
File: .github/dependabot.yml (pip ecosystem covers /src/backend only)

• Issue: exact pins on cryptography, starlette and fastapi will now go stale past security releases with no signal.
• Suggestion: add a pip entry for /docker/base-image. Dependabot reads *requirements*.txt, so the file is picked up once the directory is listed.

Clean Categories

• SQL / data safety: no DB code changed.
• Concurrency: no shared state touched.
• Auth boundaries: no endpoints changed.
• Credential exposure: the only new env value is the literal true; no secrets in the diff.
• Enterprise disclosure (4.5): no public docs changed apart from a learnings fragment, which passes the guard pattern.
• Frontend: not touched.
• Enum completeness: all three writers of OTEL_EXPORTER_OTLP_PROTOCOL found by AST and covered; template_service._PLATFORM_INJECTED_PREFIXES already includes OTEL_, so the new var is treated as platform-injected.
• Other FastAPI( constructions in the image: only agent_server/main.py:60.
• Boot fragility: startup.sh has no set -e, so a failed pip step stays non-fatal, as before. With the pins satisfied, the boot pip call needs no network, whereas the old --upgrade always did.
• Doc staleness: docs/memory/architecture/agent-runtime.md does not describe the boot upgrade; nothing is stale.

Summary

• Critical: 0.
• Informational: 5. I1 (PR body overclaim / recreate path) and I2 (downgrade behaviour) are worth settling before merge; I5 is a one-line follow-up.
• Scope: clean. Learnings fragment added (docs/memory/learnings/2026-09-30-an-unpinned-install-…md); this class also bit in Default model claude-opus-5-5 fails on cached Claude Code 2.1.278, and the rejection is misclassified as auth (seat rotation + API-key fallback) #3012.

[dolho]

/review Report — re-run after fixes (c8a1b0da5)

Branch: fix/3106-agent-server-fastapi-telemetry → dev (merge-base 3a719da27; dev has not moved)
Files Changed: 12 (+370/−18)
Scope: CLEAN
Plan Completion: from the issue's suggested fix. (a) DONE. (b) DONE, CHANGED: exact pins at build, floors at boot (see I2 below). (c) DONE at all four env writers. Regression test in CI: follow-up (unchanged, I4).

Previous findings

#	Finding	Status
I1	OTEL_SDK_DISABLED never reached existing agents on recreate	✅ Fixed. lifecycle.py recreate path: if 'OTEL_EXPORTER_OTLP_PROTOCOL' in env_vars: env_vars.setdefault('OTEL_SDK_DISABLED', 'true'). Three behavioural tests through the real recreate_container_with_updated_config seam: an OTEL agent gains it; a non-OTEL agent is untouched; an explicit operator value wins. The PR body overclaim is corrected.
I2	Exact pins downgraded template packages at boot	✅ Fixed. Split into agent-image-requirements.txt (exact, build only) and agent-server-requirements.txt (floors, boot only, server packages only; rich/cryptography removed from the boot set). Floors are tested to sit at or below the pins.
I3	OTEL_SDK_DISABLED disables any OTel SDK in template code	ℹ️ Accepted, and now stated in the PR's Side effects.
I4	No CI boot of the server under FastAPI ≥ 0.142	ℹ️ Follow-up, now listed in the PR.
I5	No bump channel for the pins	✅ Fixed. Dependabot pip entry for /docker/base-image; the config validation check passed on this PR.

Execution coverage

changed symbol / file	executed by	live consumer	verdict
fastapi_app_options	3 tests (both constructor shapes + installed FastAPI)	agent_server/main.py:60	✅ executed
recreate-path setdefault	3 behavioural tests via recreate_container_with_updated_config	every restart / drift self-heal / rebuild recreate	✅ executed
OTEL_SDK_DISABLED at create / recovery / system agent	AST scan of every OTEL_EXPORTER_OTLP_PROTOCOL writer	FastAPI in the agent container	🛡 guard + manual verification
agent-image-requirements.txt / agent-server-requirements.txt	pin, floor, coverage and floor ≤ pin tests	Dockerfile build line, startup.sh:616	🛡 guard
Dockerfile / startup.sh pip lines	source-text tests	image build / container boot	🛡 guard + real boots (below)
.github/dependabot.yml	GitHub's config validation check	Dependabot	🛡 guard

Fix mutations: 14 of 14 red, each reverted on a scratch copy and the tree restored:

Area	Mutations
Opt-out	dropped from main.py; helper returns {}
Boot repair	back to unpinned --upgrade; switched to the exact pins (would downgrade)
Image build	back to unpinned
Env flag at create sites	DB recovery loses it; system agent loses it
Recreate path	loses the flag; overwrites an operator value; sets it on non-OTEL agents
Pin files	image pin moved to 0.142.1; a loose image pin; a floor above its pin; boot set grows past the server packages

Real boots from an image rebuilt at this commit, with ENABLE_AGENT_UI=true and OTEL_EXPORTER_OTLP_PROTOCOL=grpc:

Case	FastAPI after restart	Health
Fresh container	0.141.1 (repair did nothing)	200
Template installed 0.142.1	0.142.1 (not downgraded)	200
Template installed 0.115.6	raised to 0.142.1	200

Neighbouring suites (recreate / env / base-image / startup): 855 passed, 1 skipped.

Critical Findings

None.

Informational Findings

[I6] A below-floor boot repair installs the latest release, not the image pin (Confidence: 7/10)
File: docker/base-image/startup.sh:616

• Issue: when a template pushes a server package below its floor, the repair takes PyPI's latest (observed: 0.115.6 → 0.142.1), so a new upstream release can still enter a container on this path.
• Why it's acceptable here: it's much narrower than before (only below-floor, only server packages), and a newer FastAPI is safe because of the opt-out.
• Suggestion (optional): pass the exact pins as an upper bound on this path only, e.g. -c with a generated <= constraints file, if you want no unvetted release on any path. Stated in the PR's Side effects.

[I7] Dependabot's handling of the floor file is unverified (Confidence: 5/10, appendix-level)

• Dependabot pip reads *requirements*.txt in the listed directory, so both files are in scope. For the floor file it may widen or leave the >= ranges rather than bump them. Harmless either way, because the floors only need to stay at or below the exact pins, which the tests enforce.

Clean Categories

• SQL / data safety, auth boundaries, frontend: not touched.
• Concurrency: the recreate mutation is a local dict edit before the container handoff; no shared state.
• Credential exposure: the only new env value is the literal true; gitleaks passed.
• Enterprise disclosure: none.
• Enum completeness: all four agent-env writers covered; template_service._PLATFORM_INJECTED_PREFIXES already includes OTEL_.
• Boot fragility: startup.sh has no set -e, so the repair stays non-fatal. With floors satisfied, the boot pip call needs no network.
• Doc staleness: none; the learnings fragment is present.

CI at posting time

Passed: prod-image-smoke, backend boots without enterprise submodule, schema-parity, gitleaks, the Dependabot config check, guard, journey-impact, and the compose checks. Pending: pytest (head), journey-smoke, verify-non-root, CodeQL.

Summary

• Critical: 0.
• Informational: 2 new (I6 optional, I7 appendix). All actionable prior findings are fixed and mutation-proven.
• Verdict: ready to merge once the pending CI checks pass.

[vybe]

merge-train: batch validated on train/20260930-1240 (#3118 green on Tier 1 + Tier 2)