fix(vfsevents): remove sleep-based races in vfsevents watcher tests - #341
Merged
Conversation
funkyshu
force-pushed
the
fix/340-gcsevents-race
branch
from
July 16, 2026 07:35
7323511 to
d1f993c
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the contrib/vfsevents GCS watcher test suite to remove sleep-based synchronization and eliminate race detector failures by waiting for receiveWithRetry completion, aligning with the stabilization approach used previously in s3events. It also records the fix in the contrib changelog.
Changes:
- Replaced
time.Sleep(50ms)waits in threegcseventswatcher tests with channel-based synchronization onreceiveWithRetrycompletion. - Added bounded timeout handling that cancels the context and attempts a short grace wait to reduce the chance of a lingering goroutine after subtest failure.
- Updated
contrib/vfsevents/CHANGELOG.mdunder[Unreleased]to document the fix (issue #340).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| contrib/vfsevents/watchers/gcsevents/gcsevents_test.go | Removes sleep-based races by synchronizing on receiveWithRetry completion and adding time-bounded failure handling. |
| contrib/vfsevents/CHANGELOG.md | Documents the test race fix under [Unreleased]. |
Comments suppressed due to low confidence (1)
contrib/vfsevents/watchers/gcsevents/gcsevents_test.go:550
time.After(2 * time.Second)in this select can’t be canceled when thedonecase wins, so repeated runs (e.g.,-count=200) will accumulate a large number of pending timers. Prefertime.NewTimer+defer timer.Stop()so the timeout timer is cleaned up immediately on success.
"eventTime": "2023-01-01T12:00:00Z",
"overwrittenByGeneration": "1111111111",
}
ctx := s.T().Context()
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Fixes #340. Same root cause as #339: TestEnhancedMetadata, TestOverwriteEventSuppression, and TestRetryBackoffTiming each start receiveWithRetry in a goroutine and then read test-local state (receivedEvent/receivedEvents) after a fixed time.Sleep(50ms), racing with the goroutine's write. Since RetryConfig.Enabled defaults to false, receiveWithRetry makes a single receive() call and returns, matching each test's Once() mock expectations. Fixed by waiting on receiveWithRetry's own completion via an error channel (rather than a sleep or a handler-only signal), and asserting it returns without error. On the timeout path, cancel the context immediately and give the goroutine a short bounded window to exit before failing, so it can't keep touching the mock after the subtest has already failed. Extracted the wait/cancel/grace-period logic (previously duplicated across all three tests) into a shared waitForReceive suite helper. Uses time.NewTimer + defer Stop() instead of time.After so the timers are released promptly instead of lingering until they fire, per review feedback. Also fixed the changelog entry to consistently qualify all three subtest names with TestGCSWatcherTestSuite/. Verified: go test -race -run TestGCSWatcherTestSuite -count=200 now passes cleanly (was previously failing on all three tests).
funkyshu
force-pushed
the
fix/340-gcsevents-race
branch
from
July 16, 2026 07:40
d1f993c to
b9bd43a
Compare
TestEnhancedMetadata only synchronized on the event handler firing (via an eventReceived channel), not on pollOnce's own completion. On CI (ubuntu, go1.26), the subtest could finish and its mock-expectation teardown could run before the background pollOnce goroutine made its DeleteMessage call, producing: FAIL: DeleteMessage(string,string) FAIL: 1 out of 2 expectation(s) were met. This is exactly the residual race risk flagged (but not fixed at the time, since it wasn't the primary target) during review of #339's fix for TestNonVersionedBucketMetadata. Fixed by switching TestEnhancedMetadata to the same completion-based synchronization (wait for pollOnce to fully return via an error channel) already used by TestNonVersionedBucketMetadata, extracting the shared wait/cancel/grace-period logic into a waitForPoll suite helper (mirroring the equivalent gcsevents helper added earlier in this PR). Uses time.NewTimer + defer Stop() to proactively avoid the un-stopped time.After timer issue raised in review of this PR. Verified: - go test -race -run TestS3WatcherTestSuite -count=300 ./watchers/s3events/... - go test -run TestS3WatcherTestSuite -count=1000 ./watchers/s3events/... (no -race, to stress test scheduling/timing rather than just data races) both pass cleanly with 0 failures. (cherry picked from commit b451efd)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #340. Also folds in the fix for a CI-observed flake in
s3events(was briefly its own PR, #342, now consolidated here since it's the same root-cause pattern).gcsevents (fixes #340)
Same root cause as #339:
TestEnhancedMetadata,TestOverwriteEventSuppression, andTestRetryBackoffTimingincontrib/vfsevents/watchers/gcseventseach startreceiveWithRetryin a goroutine and then read test-local state (receivedEvent/receivedEvents) after a fixedtime.Sleep(50ms), racing with the goroutine's write. Confirmed withgo test -race -run TestGCSWatcherTestSuite ./watchers/gcsevents/...onmain— all three fail reliably.Since
RetryConfig.Enableddefaults tofalse(zero value),receiveWithRetryshort-circuits to a singlereceive()call and returns, matching each test's.Once()mock expectations.s3events (CI flake found after merging #339)
CI on ubuntu + go1.26 hit:
in
TestEnhancedMetadata. That test only synchronized on the event handler firing (via aneventReceivedchannel), not onpollOnce's own completion — a residual race flagged (but not fixed at the time) during review of #339's fix for the siblingTestNonVersionedBucketMetadatatest. On CI's scheduler, the subtest could finish (and trigger its mock-expectation teardown) before the backgroundpollOncegoroutine'sDeleteMessagecall happened.Changes
gcseventstests now wait onreceiveWithRetry's own completion via an error channel (matching the hardened pattern from fix(vfsevents): remove sleep-based race in TestNonVersionedBucketMetadata #339, including its review feedback) instead of a fixed sleep, and assert the call returns without error. Extracted into a sharedwaitForReceivesuite helper.s3events/TestEnhancedMetadatanow uses the same completion-based synchronization asTestNonVersionedBucketMetadata, extracted into a sharedwaitForPollsuite helper.time.NewTimer+defer timer.Stop()(nottime.After), per review feedback on this PR, to avoid leaving timers pending when the happy path wins the select.contrib/vfsevents/CHANGELOG.mdunder[Unreleased].Test plan
go test -race -run TestGCSWatcherTestSuite -count=200 ./contrib/vfsevents/watchers/gcsevents/...— clean (previously failed on all three named subtests).go test -race -run TestS3WatcherTestSuite -count=300 ./contrib/vfsevents/watchers/s3events/...— clean.go test -run TestS3WatcherTestSuite -count=1000 ./contrib/vfsevents/watchers/s3events/...(no-race, to stress scheduling/timing rather than just data races) — clean.go test -race ./contrib/vfsevents/...— clean.go build ./... && go vet ./...forcontrib/vfsevents— clean.golangci-lint run ./watchers/gcsevents/... ./watchers/s3events/...— only pre-existing, unrelatednolintlintfindings (not touched by this PR).Made with Cursor