debug(tracing): Instrument cross-thread span flake [DO NOT MERGE]#1238
Draft
lcian wants to merge 3 commits into
Draft
debug(tracing): Instrument cross-thread span flake [DO NOT MERGE]#1238lcian wants to merge 3 commits into
lcian wants to merge 3 commits into
Conversation
TEMPORARY instrumentation to diagnose the flaky `future_cross_thread_info_span` test that intermittently observes 0 envelopes. Records, via a global buffer, whether `on_new_span` builds a Transaction or child Span, and whether `on_close`/`finish` runs (and on which thread). The test dumps the buffer when the envelope count is wrong. Not for merge; used to capture a real CI reproduction. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MUAsFPraMdxyRXe8Qpswaq
Instructions and example for changelogPlease add an entry to Example: ## Unreleased
### Features
- Instrument cross-thread span flake [DO NOT MERGE] ([#1238](https://github.com/getsentry/sentry-rust/pull/1238))If none of the above apply, you can opt out of this check by adding |
Second instrumentation round for the cross-thread flake: - wrap the future's Span in a Drop-logging newtype to record the thread where the future's own handle is released; - tag thread1/thread2 ids (keyed by span name) and log on_enter/on_exit threads to build the cross-thread timeline; - add a "MAIN read transport" marker to order finish() vs the test read. Goal: determine whether on_close lands on a third thread (=> a second Span handle) or on the future's drop thread after a concurrent enter/exit (=> ordering). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MUAsFPraMdxyRXe8Qpswaq
Log every span in on_new_span *before* the span filter ignores it, including its contextual parent id. This catches library-emitted trace/debug spans and reveals whether any span is created with our transaction's id as its contextual parent (which would clone it and pin its ref_count until that child closes). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MUAsFPraMdxyRXe8Qpswaq
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Do not merge — temporary debugging
Adds throwaway instrumentation to diagnose the flaky
future_cross_thread_info_spantest insentry-tracing, which intermittently (~1/30 undercargo llvm-cov) observes 0 envelopes where it expects exactly one transaction.What it records
A global buffer (
sentry_tracing::repro_diag) captures:on_new_span: whether the sentry object built is a Transaction or a child Span (+ span id, thread id).on_close: whetherSentrySpanDatawas found andfinish()ran, on which thread — or which no-op branch was taken (no tracing span / noSentrySpanData).The test dumps the full buffer when the envelope count != 1, so a CI reproduction tells us definitively whether:
Spaninstead of aTransaction(finishing a child span emits no envelope), oron_close/finishsimply never ran on the joined thread before the assertion.Plan
Reproduce on CI via repeated reruns of the Code Coverage job, read the captured diagnostics, then revert this branch.