observability: Tracy zones behind MORPH_ENABLE_TRACY (prefixed MORPH_ZONE macros) and a capture job that proves they fire #843

Scope, rewritten at triage: Part 1 only

This ticket is Part 1, the compile-time Tracy zones, together with the capture job that proves those zones fire. Part 2 (the Tracy sink for morph::observe and the statistics collector) is blocked on two spec decisions and has moved to #860. Part 3's further benchmarks are optional follow-ups.

To implement:

1. MORPH_ENABLE_TRACY (default OFF). First try find_package(Tracy CONFIG QUIET), then fall back to CPM wolfpld/tracy at v0.13.1, the tag Lightweight pins, so that a process ends up with exactly one TracyClient. The define and the Tracy::TracyClient link are INTERFACE.
2. include/morph/core/profiler.hpp with prefixed macros: MORPH_ZONE, MORPH_ZONE_TEXT, MORPH_PLOT, MORPH_THREAD_NAME, MORPH_MESSAGE. They forward to Tracy when it is on and are argument-consuming stubs when it is off. Never define Tracy's bare names; the reason is below.
3. Zones on the hot paths listed below, one zone per phase per thread. Link the phases by writing Context::requestId as zone text. Name the threads: pool workers, the I/O loop, and the TimeoutScheduler.
4. Document the ODR rule: every translation unit must agree on MORPH_TRACY_ENABLED.
5. A nightly tracy-capture job that runs morph_bench under capture and asserts named zones have non-zero counts. It must be shown to fail when a zone is removed.

Premise re-checked on master @ 85a5280e: no Tracy instrumentation in include/morph or src.

Done when: a Tracy-ON build shows the named zones in a capture, a Tracy-OFF build compiles with no warnings and no Tracy dependency, and the capture job goes red with a zone removed.

Summary

morph has no profiler instrumentation. It has a runtime metrics/trace seam (morph::observe) and two opt-in benchmarks, but nothing connects them to a profiler, and nothing records benchmark results over time. Lightweight, which morph already uses as a dependency, has a design worth copying: compile-time Tracy macros that become no-ops when disabled, plus a runtime statistics collector that also sends its samples to Tracy when both are on. This issue proposes the same for morph, in three parts that can land separately.

What exists today

Verification status: everything below was read from the code on master @ 9d001052, from Lightweight's cached source (.cache/cpm/lightweight/650f0137…), and from Lightweight's upstream docs/statistics.md and .github/workflows/build.yml. Nothing was built or profiled for this issue.

Tracy: none.

$ grep -rniI "tracy\|ZoneScoped\|FrameMark" --exclude-dir=build --exclude-dir=.cache --exclude-dir=.git .
docs/superpowers/specs/2026-09-14-runner-fleet-org-wide-design.md:443:- fastcached `package-linux` and Lightweight `tracy_capture` — excluded at

Runtime metrics/tracing: morph::observe (include/morph/core/observability.hpp, docs/spec/core/observability.md).

• A process-wide MetricSink and TraceSink (beginSpan/endSpan).
• When no sink is installed, each call site costs one relaxed atomic load.
• It is fed from RemoteServer/LocalBackend dispatch, register and deregister, SyncWorker, ReconnectCoordinator, and the offline queues' overflow.
• The spec says outright: "No sampling/aggregation/histograms in morph" and "there is no bundled backend."

Benchmarks: two targets, opt-in (MORPH_BUILD_LOAD_TESTS=ON, tests/bench/).

• morph_bench measures RemoteServer dispatch latency and throughput against an echo model. It takes best, median and worst over N trials, gates on the best trial, and writes bench_dispatch_latency.json / .jsonl.
• morph_bench_alloc counts allocations per local execute round trip. It is gated in ctest as bench.alloc_budget.
• No workflow archives the JSON, although the benchmark's own header says it is written "so CI can archive successive runs and diff them for regressions":

  $ grep -n "bench\|jsonl" .github/workflows/*.yml
  .github/workflows/ci.yml:2282:  # harnesses, the vetted-HMAC adapters, the soak/bench targets) were never
  

  Not verified: which CI jobs actually run the bench ctest label, as opposed to only building the targets.
• Nothing measures the codec (ActionTraits<A>::toJson/fromJson), the wire envelope, a SocketBackend loopback round trip, Completion chaining, strand fan-out across many models, journal append, or schema generation.

How Lightweight does it

1. LIGHTWEIGHT_ENABLE_TRACY (default OFF). It tries find_package(Tracy CONFIG) first, then falls back to CPM wolfpld/tracy v0.13.1 with TRACY_ENABLE and TRACY_ON_DEMAND. The library target gets PUBLIC LIGHTWEIGHT_TRACY_ENABLED and PUBLIC Tracy::TracyClient. There is also a vcpkg feature, lightweight[tracy].
2. TracyProfiler.hpp includes <tracy/Tracy.hpp> when Tracy is enabled. Otherwise it defines no-op stubs for Tracy's own macro names (ZoneScoped, ZoneScopedN, ZoneText, TracyPlot, TracyMessage, TracySetThreadName, …), each written to use up its arguments so disabled builds get no unused-variable warnings. There are 99 call sites in src/, for example ZoneScopedN("SqlTransaction::TryCommit").
3. SqlStatistics is always compiled in and switched on at runtime with Enable()/Disable(). It uses relaxed atomics and records power-of-two latency histograms. Snapshot() returns a plain value type, which can be diffed, exported, or asserted on in a test. When both Tracy and statistics are on, every sample is also sent as a TracyPlot. Statistics never depends on Tracy, in either direction.
4. CI job tracy_capture. It builds tracy-capture and tracy-csvexport at the pinned tag and builds an example with Tracy on. It starts the capture, runs the example (which calls SqlStatistics::Enable()), then checks that the counters are non-zero and that at least one latency sample is non-zero, so the job fails if the instrumentation records nothing.

Proposal: three parts, each landable alone

Part 1: compile-time Tracy zones, behind MORPH_ENABLE_TRACY (default OFF)

• CMake: find_package(Tracy CONFIG QUIET), then CPM fallback. Use the same shape as core-cpp is getting in build: find core-cpp with find_package before CPMAddPackage, so packaged builds need no fetch #840, and the same tag Lightweight pins (v0.13.1). A process that links both libraries must end up with exactly one TracyClient, and two client copies of different versions in one process do not work. morph is an INTERFACE target, so the define and the Tracy::TracyClient link are INTERFACE too.
• Use prefixed macros, not Tracy's bare names. Lightweight defines stubs named ZoneScoped and so on. The examples include Lightweight and morph in the same translation units. If Lightweight had Tracy ON and morph OFF, morph's stubs would redefine Tracy's real macros. Proposed names: MORPH_ZONE("Bridge::executeVia"), MORPH_ZONE_TEXT(sv), MORPH_PLOT(name, v), MORPH_THREAD_NAME(name), MORPH_MESSAGE(sv), in include/morph/core/profiler.hpp. They forward to Tracy when it is enabled and are stubs otherwise.
• Header-only ODR hazard, stated plainly: these macros change inline function bodies. Every translation unit in a program must agree on MORPH_TRACY_ENABLED. An INTERFACE definition gives that by construction for CMake consumers. Document it for anyone else.
• Zones on the hot paths: Bridge::executeVia, BridgeSink settle, ActionDispatcher's runner (registry.hpp), the ModelStrands pump, RemoteServer::dispatchMessage/dispatchExecute, wire envelope encode/decode, ActionTraits toJson/fromJson at the codec boundary, the SocketBackend I/O thread's read and send, journal success/failure recording, and offline enqueue/replay.
• Thread names for pool workers, the socket I/O thread and the TimeoutScheduler thread.
• A dispatch crosses threads (caller → strand → callback executor), and a Tracy zone must begin and end on the same thread. So each phase is its own zone on its own thread, and the phases are linked by Context::requestId written as zone text. Do not try to build one zone spanning the whole dispatch.
• core-cpp's event loop and strands are core-cpp's to instrument. If they need it, that is a separate issue in core-cpp.

Part 2: an optional Tracy sink for morph::observe, plus an opt-in statistics collector

• morph::observe is already the runtime-toggled half of Lightweight's design. What it lacks is a Tracy connection. Add a helper, only when MORPH_ENABLE_TRACY is on, that installs a MetricSink sending each Metric to a fixed-name TracyPlot (morph.executeLatencyMs, morph.executeInFlight, …).
• The TraceSink maps to TracyMessage, not to zones, because beginSpan and endSpan can run on different threads.
• Design question: morph::observe holds one sink per kind, so installing a Tracy sink would replace a host's Prometheus sink. That needs either a tee/fan-out helper or a documented "profiling builds only" rule.
• Spec question (invariant 5): a Lightweight-style Snapshot() with latency histograms runs against the spec's "No sampling/aggregation/histograms in morph". It can be written as an optional sink implementation the host installs, so the emit path stays raw as the spec requires, and the benchmarks in Part 3 could read phase breakdowns from it. The spec owner should decide whether that sink ships in include/morph or stays in tests/bench.

Part 3: benchmarks that use both, and a CI capture job

• Add benchmarks for the uncovered paths listed above. Reuse morph_bench's method (best-of-N gating, distribution reporting), which is documented in its header together with the reasons for it. Catch2 is already a dependency, so its BENCHMARK is available if a micro-benchmark harness is wanted. That choice belongs in the PR, not in this issue.
• Upload bench_*.json / .jsonl as workflow artifacts on the job that runs them, so results can be compared across runs.
• Add a tracy-capture job modelled on Lightweight's. Build with MORPH_ENABLE_TRACY=ON, run morph_bench under tracy-capture, export with tracy-csvexport, and assert on specific zone names with non-zero counts (e.g. Bridge::executeVia, ActionDispatcher::dispatch). Also mutate the build so a zone is removed, and confirm the job fails, since a capture that "succeeds" with no zones would prove nothing (AGENTS.md, Verify rather than assert). Put it on the nightly workflow, not on pull_request, in line with the move of slow legs to nightly.

Out of scope

• Compile-time benchmarking (-ftime-trace over template instantiation of Completion<T>, BridgeHandler<Model> and ActionTraits<A>). That is where morph's real cost lies, but it is a different measurement with different tools and deserves its own issue.
• Instrumenting core-cpp.

What would change this

• Drop Part 1 if there is a reason Tracy cannot be an optional dependency of a header-only library. A consumer mixing Tracy-on and Tracy-off translation units in one binary is the case to check.
• Drop Part 2's collector if the spec owner reaffirms "no aggregation in morph". The Tracy sink alone still stands.
• Split this issue at triage into its three parts if they will not land in one lane. Each part is written to land without the others.