Skip to content

chore(release): give tests/migrate its own race shard so the 0.10.0 release workflow can pass - #1065

Merged
HuiJun merged 7 commits into
mainfrom
hotfix/0.10.0-migrate-shard
Oct 11, 2026
Merged

HuiJun merged 7 commits into
mainfrom
hotfix/0.10.0-migrate-shard

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Hotfix from main for the 0.10.0 release. Since #1033 every CircleCI develop build has failed in Go race tests: on the 4-vCPU rest node tests/migrate under -race went from 1370 s to past the 30-minute package timeout (panic: test timed out after 30m0s, FAIL github.com/Open-MBEE/OpenSysML/tests/migrate 1800.098s). main now carries #1033 through #1064, and every publish job in the release workflow requires Go race tests, so a rehearsal or a v0.10.0 tag fails the same way; the nightly, which only publishes green CircleCI commits, has been stalled as well.

  • Brings in ci(test): give tests/migrate its own race shard #1060: tests/migrate becomes a race shard of its own (scripts/race-shard.sh migrate, make test-shard SHARD=migrate, make coverage-shard SHARD=migrate); CircleCI go-race-test and go-coverage go from four to five executors, pr.yml from five to six race shards. Its changelog fragment is folded into the 0.10.0 ### Changed section.
  • Gives the CircleCI migrate shard the Makefile's 55-minute per-package timeout (make test-shard already uses 55 m). The package alone, with nothing beside it, takes 1423 s under -race on an idle 8-vCPU machine, so on the 4-vCPU large class a node of its own leaves no margin under 30 m.
  • Trims the pr.yml shard comment to two lines.

No test or gate changes: the same packages run under the same flags, one more job wide.

How it was verified

  • circleci config validate .circleci/config.yml passes; scripts/race-shard.sh migrate prints only tests/migrate and rest no longer lists it (119 packages).
  • go test -race -pgo=off -count=1 ./tests/migrate alone: ok 1422.985s (8 vCPU).
  • python3 scripts/changelog.py check, python3 scripts/check-doc-ids.py pass; exactly one ## 0.10.0 section.
  • The race run itself is CI's: this PR's Go race tests (migrate) / (rest) jobs, then the CircleCI release_rehearsal on main after merge, are the proof.

Checklist

  • make test and make lint pass locally (CI configuration change; the Go suite is unchanged)
  • Tests added or updated for the change (n/a: CI configuration)
  • Documentation extended where it already covers the surface (AGENTS.md shard list, Makefile help text, via ci(test): give tests/migrate its own race shard #1060)
  • Changelog: fragment folded into the existing 0.10.0 section (release preparation, per docs/project/releasing.md)
  • baselines regenerated and make docs-counts run if a gate count moved (n/a)
  • No internal work-item labels (waves, slices, F4, K5) in the body, docs, or changelog

Link to Devin session: https://nasa-jpl-demo.devinenterprise.com/sessions/3db6a090aa214667b2a66d372c6e36b1
Open in Devin Desktop: https://nasa-jpl-demo.devinenterprise.com/desktop/session/3db6a090aa214667b2a66d372c6e36b1?variant=devin
Requested by: @HuiJun

devin-ai-integration Bot and others added 5 commits October 10, 2026 18:36
Under -race the SysML v1 migrator's suite grew past 24 minutes on the pull-request
runners and past the 30-minute package timeout on CircleCI's develop builds, failing
every merge since the rest shard carried it. scripts/race-shard.sh migrate names it;
both CI configurations run six race shards and five coverage shards.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
Under -race the SysML v1 migrator suite outgrew the 30-minute package timeout on the CircleCI rest node, so the release workflow could not pass; it now runs as a shard of its own.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
The package already ran 1370-1800+ seconds under -race on the rest node, so the 30-minute default leaves no margin even on a node of its own.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration
devin-ai-integration Bot changed the base branch from develop to main October 10, 2026 23:47
HuiJun and others added 2 commits October 10, 2026 17:08
Under -race the SysML v1 migrator's suite grew past 24 minutes on the pull-request
runners and past the 30-minute package timeout on CircleCI's develop builds, failing
every merge since the rest shard carried it. scripts/race-shard.sh migrate names it;
both CI configurations run six race shards and five coverage shards.

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: jason.han <hanhuijun@gmail.com>
@devin-ai-integration
devin-ai-integration Bot marked this pull request as ready for review October 11, 2026 00:40
@HuiJun
HuiJun merged commit 0a4eb9a into main Oct 11, 2026
27 checks passed
@HuiJun
HuiJun deleted the hotfix/0.10.0-migrate-shard branch October 11, 2026 00:44

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

Devin Review

Comment thread .circleci/config.yml
Comment on lines +464 to +465
timeout=30m; [ "$shard" = migrate ] && timeout=55m
go test -v -race -pgo=off -timeout "$timeout" $pkgs

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Migration shard retains shorter silence limit

When tests/migrate exceeds 30 minutes, the 55-minute timeout cannot prevent CircleCI from terminating the silent command. go test buffers package output, so observed 32-minute runs still fail the release workflow.

Learn more

CircleCI stops a run step when it produces no output for no_output_timeout. The repository already documents that go test buffers a package's -v output until that package finishes in the solver-absent gate at .circleci/config.yml. This shard contains only tests/migrate, so a package run can remain silent for its entire 23–32 minute duration. Raising Go's test timeout to 55 minutes therefore leaves CircleCI's independent 30-minute silence limit as the effective cap.

Example: A loaded runner completes tests/migrate in 32 minutes. CircleCI sees no output after the initial shard message, kills the step at 30 minutes, and never reaches Go's 55-minute timeout.

Recommended fix: Raise this run step's no_output_timeout above the migration shard's 55-minute Go timeout, or stream periodic output without masking the test exit status.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a bug: go test -v streams a single package's output as it runs, so the silence limit is never reached. Evidence from the first release build with this change (pipeline 3995 on main): the migrate node ran 33.2 minutes under no_output_timeout: 30m and passed (ok github.com/Open-MBEE/OpenSysML/tests/migrate 1923.117s). The buffering noted on the REPL gate step applies to multi-package runs; the migrate shard is one package.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant