Skip to content

test(canopen): wait on observables instead of sleeping - #188

Merged
dborgards merged 7 commits into
mainfrom
claude/pensive-wozniak-ivbwba
Sep 27, 2026
Merged

dborgards merged 7 commits into
mainfrom
claude/pensive-wozniak-ivbwba

Conversation

@dborgards

@dborgards dborgards commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

What does this change?

This is the next step of #171 and covers the CANopen rows of the delay/sleep audit. The sleeps it replaces did not wait for a timer. They guessed how long something else would take to happen. The tests now wait for that event directly:

  • NMT Start applied: poll ICanOpenNode.State, which reads through the node's actor.
  • Boot-up arrived: a BootupWatch consumes the opening boot-up first, then waits for the one after the reset.
  • SDO session or init on the wire: wait for the server's init ack, or for the client's own init frame, on the bus.
  • Another writer blocked on the OD write gate: spin on the new internal ObjectDictionary.WriteGateWaiters counter.

The counter adds two interlocked operations around the gate in WriteUnsigned and Add. Both methods already take two locks, so the cost is small in comparison. Nothing outside the test project reads the counter.

Checks that something did not happen keep their wall-clock windows. Each commit message lists the affected tests. One of them is the RPDO drain in Tpdo_Emission_UnderConcurrentOdWrites_NeverTears, and ff50e60 has the full reasoning:

  • EmitTpdo sends each TPDO through its own Task.Run, so no frame can mark "everything before me has arrived".
  • The writer loop's change-of-state TPDOs make the number of frames still in flight unknown.

Mutation checks:

  • Replacing the write gate with a private lock in WriteUnsigned makes A_Typed_Write_Resolves_Its_Type_Under_The_Write_Gate and A_Direct_Write_Cannot_Land_Inside_A_ConfigureTpdo_Transaction fail.
  • Doing the same in Add makes A_Redeclaration_Waits_For_The_Write_In_Flight_On_The_Entry fail.
  • The commit messages record the NMT, boot-up and SDO mutations.

Type of change

  • feat — new behaviour (minor release)
  • fix / perf — bug or performance fix (patch release)
  • docs / test / refactor / chore / ci — no release
  • Breaking change (! in the title, plus a BREAKING CHANGE: footer explaining the migration)

Checklist

  • dotnet build CanKit.Pro.sln -c Release succeeds (with -p:CI=true)
  • dotnet test CanKit.Pro.sln -c Release passes (net10.0, locally)
  • Public API changes are documented with XML comments (none; the new member is internal)
  • New behaviour is covered by a test (no new behaviour)
  • The requirement or ADR this relates to is referenced (e.g. FR-RAW-031, ADR-7), if any (Replace wall-clock category-2 test sleeps (macOS flake risk) #171)

🤖 Generated with Claude Code

https://claude.ai/code/session_013WJ8h1ahw4Nj5dYuEWy34s

…nary

WriteUnsigned and Add already serialize on the private _writeGate mutex,
but nothing let a test observe that a concurrent writer had genuinely
reached it. Three CanOpenCommunicationProfileTests relied on a fixed
Thread.Sleep instead, assuming a concurrent "hammer" writer had reached
the gate within the sleep. Internal WriteGateWaiters (incremented before
lock (_writeGate), decremented once acquired) gives tests a real signal
to poll instead.

Refs #171

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013WJ8h1ahw4Nj5dYuEWy34s
…eeps with observables

Part of the #114/#171 delay-and-sleep audit ("CANopen NMT Start" and
"CANopen session, wire, and pump" tables). None of these sleeps waited
on a clock the node's actor had armed: they assumed a received frame
had been dequeued and ApplyNmtTransition had run, an SDO session had
been installed, an upload-init was on the wire, or the RPDO event pump
had drained.

- ICanOpenNode.State already round-trips through the node's actor, so
  a bounded poll on it (WaitUntilOperationalAsync) is a real barrier
  for "the NMT Start this test just sent has been applied" — no product
  change needed. Used in CanOpenDynamicMappingTests and the four
  Tpdo_*/Nmt_Broadcast tests in CanOpenNodeIntegrationTests that used to
  sleep 50-100 ms after sending Start.
- A local BootupWatch (the pattern CanOpenCommunicationProfileTests and
  CanOpenDeviceDescriptionTests already use) replaces the sleeps that
  waited to consume a node's opening boot-up or a reset's boot-up
  (Nmt_ResetNode_EmitsBootup, Nmt_ResetCommunication_Emits..., and the
  heartbeat-consumer arming sleep in Heartbeat_Consumer_FiresTimeout...).
- Sdo_ServerSupersede_EmitsWireAbort_ForPriorTransfer,
  Sdo_ClientResponseWithShortDlc_IsAcceptedAndCompletes, and
  Sdo_ClientSegmentedUploadResponse_OverMaxTransferBytes_AbortsOutOfMemory
  now wait for the server's session-install ack / the client's own
  init frame to be observed on the wire (FrameObserved) instead of
  guessing when a request "must" have been transmitted.
- Sdo_Segmented_Download_Wrong_Toggle_Aborts waits for the segmented
  download's init-ack (scs 0x60) instead of guessing 100 ms.
- Tpdo_Emission_UnderConcurrentOdWrites_NeverTears flushes the
  consumer's RPDO event pump with one more deterministic TPDO and
  waits for its own delivery (the pump is a single FIFO reader) instead
  of a fixed 200 ms, so a late torn payload can no longer slip past the
  assertions unsampled.

Left unconverted, with reasons:
- Every "nothing else was transmitted" wall-clock window whose effect
  the actor round trip cannot observe (Tpdo_ChangeOfState_DoesNotEcho_
  On_RpdoUnpack's echo-quiet window, Overlapping_Emcys_..., A_Guarding_
  Reply_..., the producer-tick windows) is left as-is per the audit's
  own guidance: shortening a green "did not happen" window only
  weakens it.
- Nmt_ResetCommunication_EmitsBootup_And_Settles_In_PreOperational's
  trailing state check needed no barrier at all: PerformNmtReset sets
  _state = PreOperational before it emits the boot-up frame the test
  already awaits, on the same actor, so the state is already correct
  by the time bootups.Second resolves.

Mutation-tested: ApplyNmtTransition (state never reaches Operational),
the two EmitHeartbeat(0x00) boot-up call sites (construction and
reset), AbortSupersededServerSession (stays silent), and the download-
segment toggle check each caught their guarded test(s) when broken and
were restored; src/ carries no leftover mutation. Ran the full CANopen
+ ApiApproval filter (424 tests, all green) and the three converted
files' tests 10x in a row (116 tests each run, all green).

Refs #171

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013WJ8h1ahw4Nj5dYuEWy34s
…aiters signal

Part of the #114/#171 delay-and-sleep audit's object-dictionary write-gate
sub-table. A_Redeclaration_Waits_For_The_Write_In_Flight_On_The_Entry,
A_Typed_Write_Resolves_Its_Type_Under_The_Write_Gate, and
A_Direct_Write_Cannot_Land_Inside_A_ConfigureTpdo_Transaction each held
the ObjectDictionary's write gate from inside a WriteValidator callback
and slept a fixed 100-200 ms, assuming a concurrent writer had reached
the same gate by then. They now spin (SpinUntilAtTheWriteGate) on the
ObjectDictionary.WriteGateWaiters seam added in the prior commit — real
evidence the other writer's own lock (_writeGate) attempt is blocked on
this one, not a guess at how long reaching it takes.

Left unconverted: A_Save_During_An_Nmt_Reset_Stores_All_Restored_Values_
Not_A_Mix's Thread.Sleep(300) is not the same shape. It runs inside an
NMT reset's Transaction(...), which already holds _writeGate for the
whole restore; the concurrent save's WriteRaw call is provably blocked
on that same lock the instant it is issued, regardless of the sleep's
length. The sleep is margin for the "if the save were wrongly not held
back it would have completed by now" direction (matching the audit's
"green"), not a guess about when the save reaches the gate, so it is
left as-is.

Mutation-tested: making Add's write-gate lock private (no longer the
shared gate) broke A_Redeclaration_Waits_For_The_Write_In_Flight_On_
The_Entry as expected, then reverted. Ran CanOpenCommunicationProfileTests
(67 tests) and the full CANopen + ApiApproval filter (424 tests) green,
and the three converted files' tests 10x in a row (116 tests/run) green.

Refs #171

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013WJ8h1ahw4Nj5dYuEWy34s
@cursor

cursor Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

PR Summary

Low Risk
Test and internal test-seam changes only; two interlocked ops on existing write-gate paths with no new public API.

Overview
Part of #171: CANopen integration tests stop using fixed Task.Delay / Thread.Sleep where they were only guessing timing, and instead wait on observable signals.

Production (test seam only): ObjectDictionary gains an internal WriteGateWaiters counter (Interlocked around lock (_writeGate) in WriteUnsigned and Add) so tests can tell when concurrent writers are actually queued on the write gate.

Tests: New helpers synchronize on real conditions—poll ICanOpenNode.State until Operational after NMT Start; BootupWatch for first vs post-reset boot-up heartbeats; bus observers / TCS for SDO segmented init acks and client init frames on the wire; SpinUntilAtTheWriteGate for write-gate race tests. Negative checks that need “nothing happened yet” (e.g. RPDO drain after concurrent TPDO) still use explicit wall-clock windows, with comments explaining why no flush barrier exists.

No public API or runtime behavior change beyond the internal waiter count used only from tests.

Reviewed by Cursor Bugbot for commit 366c488. Bugbot is set up for automated code reviews on this repo. Configure here.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-27T21:22:45.721330Z 366c488 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a0bcb53a30

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tests/CanKit.Pro.Tests/TestCases/CANopen/CanOpenNodeIntegrationTests.cs Outdated
The handler added to wait for the flush TPDO matched any RPDO from the
producer. An event still queued from the 2000 emissions before it is
raised to every handler subscribed when it runs, so it could complete
the flush early and let the counts be sampled while later deliveries,
a torn one included, were still pending (Codex on #188). The flush TPDO
now carries a payload nothing else emits, and the tear check ignores it.

Refs #171

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013WJ8h1ahw4Nj5dYuEWy34s
CodeQL on #188 flagged the foreach in both copies of
WaitUntilOperationalAsync as a missed All(...).

Refs #171

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013WJ8h1ahw4Nj5dYuEWy34s

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 00768e7948

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tests/CanKit.Pro.Tests/TestCases/CANopen/CanOpenNodeIntegrationTests.cs Outdated
@codecov

codecov Bot commented Sep 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

The RPDO flush added on this branch cannot order anything. EmitTpdo
sends each TPDO through its own Task.Run, so the flush TPDO can
overtake earlier ones on the wire (Codex on #188). Counting deliveries
does not work either: every WriteRaw in the writer loop also emits a
change-of-state TPDO, and a probe saw 16,000 to 35,000 of them arrive
for the loop's 2000 triggers. With no barrier available, the 200 ms
window from main is restored, with the reason recorded next to it.

Refs #171

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013WJ8h1ahw4Nj5dYuEWy34s

Copy link
Copy Markdown
Owner Author

macos-latest went red on 00768e7 in a UDS test, not in anything this branch changes.

  • What failed: UdsClientTests.P2_Ends_With_The_First_Frame_Of_A_MultiFrame_Response, with IsoTpTimeoutException: N_Cr timer expired waiting for next Consecutive Frame at UdsClientTests.cs:393. The other 1213 tests passed.
  • Why it isn't this branch's: the diff touches only CANopen tests and ObjectDictionary. That test, and the UDS and ISO-TP code it runs, are the same as on main.
  • It has happened before: the same test failed the same way on macOS for test: move ready-seam category-2 sleeps onto the virtual clock and observables #185 (run 36346544308). Both failures are recorded on Replace wall-clock category-2 test sleeps (macOS flake risk) #171 (comment).
  • The likely cause: the test's timing margin. The client allows N_Cr = 500 ms between Consecutive Frames, and the ECU paces them at STmin = 127 ms. That leaves roughly 370 ms per gap, over 12 gaps.
  • No fix exists yet. It belongs with the UDS P2 rows, which still need a time seam that actually holds.

00768e7 is no longer the head. ff50e60 has a fresh CI run of its own, so I'm not re-running the old one.


Generated by Claude Code

dborgards commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner Author

macos-latest failed on ff50e60 in a UDS test this branch does not touch. That head has since been superseded.

The failing test was UdsClientTests.A_Pending_Answer_Consumed_As_Another_Requests_Stray_Still_Extends_Its_Window (UdsClientTests.cs:988). The other 1213 tests passed.

Why it isn't this branch's:

  • The diff contains only CANopen tests and ObjectDictionary. The failing test and the UDS code it exercises are the same as on main.
  • The audit (docs/reviews/2026-09-25-delay-sleep-audit.md, row for this test) already records it red on macOS: run 36102259889, when that PR was docs-only.
  • Its margin is 200 ms. The ECU handler sleeps 400 ms inside a 600 ms P2.

No fix exists yet. It is one of the UDS P2 rows on #171 that still needs a time seam that holds.

Re-run: none was needed. The base merge 366c488 started fresh CI, and all legs are green on that head, macOS included (run 36351351035).


Generated by Claude Code

@dborgards
dborgards merged commit 6f8836b into main Sep 27, 2026
14 checks passed
@dborgards
dborgards deleted the claude/pensive-wozniak-ivbwba branch September 27, 2026 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants