Skip to content

core, net, offline, ci: Tracy profiler zones behind MORPH_ENABLE_TRACY, with a nightly capture check - #861

Merged
Yaraslaut merged 4 commits into
masterfrom
lane/tracy-zones
Oct 3, 2026
Merged

Yaraslaut merged 4 commits into
masterfrom
lane/tracy-zones

Conversation

@Yaraslaut

@Yaraslaut Yaraslaut commented Oct 3, 2026 •

Copy link
Copy Markdown
Member

Closes #843 (Part 1 only, as rescoped at triage; Part 2 is #860).

Four commits, each droppable on its own:

  1. core, build: MORPH_ENABLE_TRACY and the MORPH_ZONE profiler macros. The option is OFF by default. It runs find_package(Tracy CONFIG QUIET) first and falls back to CPM wolfpld/tracy v0.13.1 under Lightweight's CPM name (tracy) and options (TRACY_ENABLE, TRACY_ON_DEMAND). MORPH_TRACY_ENABLED and Tracy::TracyClient are INTERFACE properties of morph::morph, and morphConfig.cmake finds Tracy for the consumer. A found client that lacks TRACY_ENABLE is a configure error. Tracy is added before core-cpp: see the core-cpp finding below. include/morph/core/profiler.hpp adds MORPH_ZONE, MORPH_ZONE_TEXT, MORPH_PLOT, MORPH_THREAD_NAME and MORPH_MESSAGE. None of Tracy's own names is defined. With Tracy off, each stub names its arguments only in decltype, so nothing is evaluated and no argument goes unused. The ODR rule is stated in the header and in the new docs/spec/core/profiler.md, and MSVC/clang-cl get a #pragma detect_mismatch. tests/test_profiler.cpp covers both modes.
  2. core, net, offline: profiler zones on the hot paths, and thread names. Each zone covers one phase on one thread. No zone spans a thread hand-off or a co_await. Phases are linked by Context::requestId as zone text. The zone list is below.
  3. ci: nightly Tracy capture asserts the named zones fire. This adds a tracy-capture job to nightly-slow-checks.yml (not to pull_request), wired into reconcile-issues. The job builds tracy-capture/tracy-csvexport at v0.13.1 and builds morph_bench and morph_bench_alloc with Tracy on. It captures each one and asserts that named zones have non-zero counts. scripts/check_tracy_capture.sh fails closed: no CSV, no rows, an unexpected header, a zone that is absent or counted zero, a binary built without Tracy (the capture never connects), or a hang all fail the job. scripts/test_check_tracy_capture.sh self-tests the assertion and runs first in the job. A new advanced option, MORPH_TRACY_ON_DEMAND (default ON), lets the job record from the first zone, and TRACY_NO_EXIT=1 makes the result independent of when the capture connects.
  4. core: profiler zones on RemoteServer's dispatch phases (added after core: stamp the verified principal on deregister, release only held references; spec + tests for the cancellation policy (#855, #858, #846) #859 merged). RemoteServer::dispatchMessage covers every decoded message and RemoteServer::dispatchExecute covers an execute's admission gates; both run on the server strand. The admitted run gets its own zones on the model's strand: startRemote, startTaskRemote and finishRemote. Every zone carries requestId as its text. The nightly morph_bench capture now also requires dispatchMessage, dispatchExecute, startRemote and finishRemote.

Zones

Zone Thread Zone text
Bridge::executeVia / executeAttachedVia / executeCreatingVia owner —
Bridge::dispatchNow owner (later, for a call that waited for its bind) bridge session requestId
ActionTraits::toJson, ActionTraits::resultFromJson remote backend's thread —
BridgeSink::settleValue, settleException, forward settling thread / owner —
LocalBackend::executeInto caller requestId
LocalBackend::startLocal, startTaskLocal, finishLocal model strand requestId
ModelStrands::task (around-task hook, every strand task) pool thread —
ActionDispatcher::dispatch, dispatchAsync model strand installed session requestId
ActionDispatcher::prepareAction (decode + pre-handler gates) model strand —
recordActionSuccess, recordActionFailure (journal) model strand —
wire::encode, wire::decode caller envelope requestId
SocketBackend::fileExecute I/O loop envelope requestId
SocketBackend::fileControl, dispatchIncomingEnvelope, drainFrames; ws::enqueueFrame (send side) I/O loop —
InMemoryOfflineQueue::enqueue, FileOfflineQueue::enqueue, SyncWorker::drain, SyncWorker::replay queue/worker owner —
RemoteServer::dispatchMessage, RemoteServer::dispatchExecute (admission) server strand envelope requestId
RemoteServer::startRemote, startTaskRemote, finishRemote model strand envelope requestId

Thread names: morph.pool for every ThreadPoolExecutor worker, and morph.io for the IoLoop thread. Since the one-owner change, that one thread carries every socket, every TimeoutScheduler deadline and the probe, so the ticket's "socket I/O thread" and "TimeoutScheduler thread" are now the same thread.

Finding: core-cpp pins a different Tracy (commented on #843)

core-cpp v0.5.0 has its own CORE_CPP_WITH_TRACY, which pins Tracy v0.14.1 under the same CPM name tracy. Under morph, core-cpp fetches nothing (CORE_CPP_FETCH_DEPS OFF).

  • Before this change: -DCORE_CPP_WITH_TRACY=ON failed configure: "Tracy is needed because CORE_CPP_WITH_TRACY holds, but neither the parent project nor find_package(Tracy 0.14.1) provides Tracy::TracyClient". This was measured with morph's Tracy block placed after core-cpp.
  • With this change: morph adds Tracy before core-cpp, so MORPH_ENABLE_TRACY=ON CORE_CPP_WITH_TRACY=ON MORPH_INSTALL=OFF configures. core-cpp's core-cpp-base and core-cpp-net build against the single v0.13.1 client. This was measured locally.
  • Still open: with MORPH_INSTALL=ON, core-cpp's install rules refuse to export its targets, because it links a fetched Tracy it did not install. That is core-cpp's install policy; the spec documents the workaround.

Verification

All of the following was measured locally on macOS 26 / arm64 with Homebrew clang 22.1.8. That is CI's CLANG_VERSION; clang 23 fails master on unrelated -Wlifetime-safety-*.

  • Tracy-OFF, Debug, -Weverything -Werror, MORPH_BUILD_NET=ON MORPH_BUILD_LOAD_TESTS=ON: builds with zero warnings, and build.ninja and CMakeCache.txt contain no Tracy target or include.
    • morph_tests: 1725 cases, 1724 passed, 1 failed as expected.
    • morph_net_tests: all 210 cases passed on the first two runs. Later runs fail in SocketServer: the accept flow keeps serving after pending connections are aborted before accept with TcpSocket::connect: could not connect to 127.0.0.1:<port>. The same Tracy-ON binary that passed the whole suite earlier now fails that case too, so I attribute it to machine state, not to this change. That attribution is inferred, not proven.
  • Mutation (OFF): making the MORPH_PLOT stub drop its arguments fails test_profiler.cpp with -Wunused-variable on depth and inFlight. GCC 15 -Wall -Wextra -Wunused-but-set-variable -Werror accepts the decltype stubs, including a local that is set and then used only by a macro.
  • Tracy-ON, Release, fetched v0.13.1: builds with zero warnings in morph's own TUs (Tracy's own TracyClient.cpp emits sprintf deprecation warnings). morph_tests passes (1723 cases, 1 failed as expected) and morph_net_tests passes (210 cases).
  • Capture, real run with tracy-capture/tracy-csvexport built locally from v0.13.1, via scripts/check_tracy_capture.sh on morph_bench:
    ok: zone wire::encode                             3250922
    ok: zone wire::decode                             1635462
    ok: zone ModelStrands::task                       1625460
    ok: zone ActionDispatcher::dispatch               1625460
    ok: zone ActionDispatcher::prepareAction          1596836
    
  • Capture with the RemoteServer zones (fourth commit, rebased on f7541f6), exit 0:
    ok: zone RemoteServer::dispatchMessage            1521299
    ok: zone RemoteServer::dispatchExecute            1521298
    ok: zone RemoteServer::startRemote                1521298
    ok: zone RemoteServer::finishRemote               1521298
    ok: zone wire::encode                             3042598
    ok: zone wire::decode                             1531300
    ok: zone ModelStrands::task                       1521298
    ok: zone ActionDispatcher::dispatch               1521298
    ok: zone ActionDispatcher::prepareAction          1494371
    
    After the rebase, Tracy-OFF morph_tests reports 1744 cases: 1743 passed, 1 failed as expected.
  • Capture goes red with a zone removed. I deleted MORPH_ZONE("ActionDispatcher::dispatch") (and its text line), rebuilt morph_bench, and ran the same command. Exit 1:
    ok: zone ModelStrands::task                       1651129
    error: zone ActionDispatcher::dispatch has no events in .../cap-mut/morph_bench.csv
    ok: zone ActionDispatcher::prepareAction          1576945
    error: 1 named zone(s) missing from .../cap-mut/morph_bench.csv
    
    After restoring the zone, the same command exits 0.
  • Fail-closed: capturing the Tracy-OFF morph_bench exits 1 with error: tracy-capture did not exit within 120s; killed it. test_check_tracy_capture.sh passes all 10 cases. With the checker's count test mutated (-gt 0 changed to -ge 0), the self-test fails 3 cases.
  • Doxygen (build-doc --target doc) passes, and the macros appear in globals_defs.html. Note: Doxygen does not fail on a missing @param for a function-like macro, so the macro docs were checked by hand.
  • clang-tidy-diff (clang-tidy 22, the repo's .clang-tidy) on the changed lines is clean apart from the mac-only poison-system-directories. The first run caught bugprone-sizeof-container on the original sizeof stubs, cppcoreguidelines-macro-usage, and readability-use-concise-preprocessor-directives; all three are fixed. That is why the stubs use decltype.
  • Lint: actionlint and shellcheck are clean, and clang-format 22 reports nothing on the changed files.

Not verified

  • The morph_bench_alloc capture (Bridge/LocalBackend zones) was not run successfully locally. On Apple, Tracy's client starts lazily and is never destroyed, so TRACY_NO_EXIT does not hold a short program open, and the capture never connected. On Linux the client is a static object and waits, so the nightly job should pass, but that is inferred from Tracy's source, not run. The Bridge zones were confirmed present in that binary (strings), not in a capture.
  • The nightly job has not run in CI at all. I did not dispatch it, because a workflow_dispatch run also runs reconcile-issues.
  • MSVC and clang-cl: not built locally. I assume decltype counts as a use for MSVC's C4189 but have not checked it. The detect_mismatch pragma is not exercised.
  • GCC with Tracy ON: not built. CI has no Tracy-ON GCC leg; the nightly job uses clang 22.
  • A Tracy found via find_package (installed or vcpkg): not tried.
  • Thread names: they compile and run, but nothing asserts them, because tracy-csvexport does not report thread names.

Review notes (done inline)

  • Zones are never open across a co_await. The send side is zoned at enqueueFrame, not in writerFlow, and the strand zone wraps one synchronous run().
  • A nested zone (for example SyncWorker::drain around replay) shadows morphProfilerZone on purpose, under a scoped -Wshadow suppression, so that MORPH_ZONE_TEXT binds to the innermost zone.
  • MORPH_ZONE uses ZoneNamedN plus static_assert(true) rather than ZoneScopedN, because whether Tracy's expansion includes the ; varies by compiler. With ZoneScopedN, the first ON build failed with -Wextra-semi-stmt.

🤖 Generated with Claude Code

MORPH_ENABLE_TRACY (default OFF) finds an installed Tracy, or fetches
wolfpld/tracy v0.13.1 through CPM under Lightweight's package name, tag and
options, so a process linking both libraries ends up with one TracyClient.
MORPH_TRACY_ENABLED and Tracy::TracyClient are INTERFACE properties of
morph::morph, and an installed morph finds Tracy for its consumer. A found
client without TRACY_ENABLE is a configure error, since every zone would
compile to nothing.

include/morph/core/profiler.hpp defines MORPH_ZONE, MORPH_ZONE_TEXT,
MORPH_PLOT, MORPH_THREAD_NAME and MORPH_MESSAGE, never Tracy's own names:
Lightweight stubs those, and the two share translation units. With Tracy off
each macro names its arguments only in decltype, so nothing is evaluated and no
argument goes unused; tests/test_profiler.cpp holds both under the full
warning set. The header and docs/spec/core/profiler.md state the ODR rule:
every translation unit must agree on MORPH_TRACY_ENABLED; MSVC and clang-cl
get a detect_mismatch link check.

Tracy is added before core-cpp: core-cpp's own CORE_CPP_WITH_TRACY takes the
client its parent provides, so a build with both on runs the one v0.13.1
client instead of failing to find one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Yaraslaut and others added 3 commits October 3, 2026 23:36
One zone per phase, on the thread that runs it: a dispatch crosses from the
caller to the model's strand to the settling thread, and a Tracy zone has to
begin and end on one thread, so no zone spans a hand-off or a co_await. The
phases of one call are linked by writing Context::requestId as zone text
where the phase can see the call's session.

Zoned: Bridge::executeVia (and its attached/creating forms), dispatchNow, the
ActionCall codec, BridgeSink's settles and forward; LocalBackend's
executeInto, start and finish; every model strand task (the around-task
hook); ActionDispatcher's dispatch, dispatchAsync and prepareAction; the
journal's outcome recording; wire::encode/decode; SocketBackend's file,
receive and frame drain, and the WebSocket send queue; the offline queues'
enqueue and SyncWorker's drain and per-item replay. The pool workers are
named morph.pool and the IoLoop thread, which carries every socket and
TimeoutScheduler deadline, morph.io.

RemoteServer is not zoned here; its dispatchMessage and dispatchExecute are
the next two.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A tracy-capture job on the nightly workflow builds tracy-capture and
tracy-csvexport at v0.13.1, builds morph_bench and morph_bench_alloc with
MORPH_ENABLE_TRACY=ON, runs each under a capture and fails unless every named
zone has a non-zero count: the codec, dispatcher and strand zones from
RemoteServer's dispatch, and Bridge/LocalBackend/BridgeSink from the local
path. Its result is reconciled into an issue like the other nightly legs.

scripts/check_tracy_capture.sh fails closed: no CSV, no rows, an unexpected
column layout, a named zone absent or counted zero, a program built without
Tracy, and a program or capture that does not exit are all failures.
scripts/test_check_tracy_capture.sh drives the assertion through each of
those from synthetic CSVs and runs first in the job.

MORPH_TRACY_ON_DEMAND (advanced, default ON) lets the job build the fetched
client to record from the first zone; with TRACY_NO_EXIT=1 the result then
does not depend on when the capture connects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RemoteServer::dispatchMessage covers every decoded message and
RemoteServer::dispatchExecute covers an execute's admission gates, both on the
server strand. The admitted run is zoned separately on the model's strand
(startRemote, startTaskRemote, finishRemote), so no zone spans the hand-off.
Each zone carries the envelope's requestId as its text. The nightly capture
now also requires the four zones morph_bench reaches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.82759% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
include/morph/core/registry.hpp 70.00% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

@Yaraslaut
Yaraslaut merged commit cc00977 into master Oct 3, 2026
39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci Subsystem: ci area: core Subsystem: core enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

observability: Tracy zones behind MORPH_ENABLE_TRACY (prefixed MORPH_ZONE macros) and a capture job that proves they fire

1 participant