Skip to content

fix(stovepipe): Keep issues on task-wide failures - #958

Merged
prathshenoy merged 3 commits into
mainfrom
codex/stovepipe-taskwide-closure
Oct 27, 2026
Merged

fix(stovepipe): Keep issues on task-wide failures#958
prathshenoy merged 3 commits into
mainfrom
codex/stovepipe-taskwide-closure

Conversation

@prathshenoy

@prathshenoy prathshenoy commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

What:

  • Keep existing issues open when any task-wide failure is recorded for a build.
  • Apply all reported issues before reconciling absent ones, using the existing processing phases.

Why:

  • Avoid treating missing target results as recovery. A task-wide failure can mean validation never ran or could not identify failed units, so closing earlier issues would claim recovery without evidence.
  • Avoid a separate precheck and new storage operations for data we already need to read. Recording task-wide failures while applying reported issues lets the closing pass use a decision established over the whole build.

Test Plan

  • Add unit tests.
  • Verify terminal intent and task-wide failure information persist against a real MySQL database.

Revert Plan

  • Revert this PR before enabling issue-tracking consumption. The added columns can remain unused; once consumption is enabled, drain active passes before rolling back.

Issue Links

Stack

  1. feat(stovepipe): Retain versioned issue state #942
  2. feat(stovepipe): Store applied issue state #948
  3. feat(stovepipe): Apply issue cursor updates #949
  4. feat(stovepipe): Track issues in bounded passes #950
  5. feat(stovepipe): Publish issue-tracking updates #951
  6. fix(stovepipe): Keep issues on task-wide failures #958
  7. feat(stovepipe): Recover failed issue tracking #961
  8. feat(stovepipe): Wire issue tracking consumers #962

This was referenced Oct 27, 2026
Merged
Merged
@prathshenoy
prathshenoy added this pull request to stack #953 October 27, 2026 20:38
This was referenced Oct 27, 2026
Merged
Merged
Merged
@prathshenoy
prathshenoy force-pushed the codex/stovepipe-taskwide-closure branch from f876d43 to fa02eec Compare October 27, 2026 21:02
@prathshenoy
prathshenoy force-pushed the codex/stovepipe-taskwide-closure branch from fa02eec to 0668840 Compare October 27, 2026 21:12
@prathshenoy
prathshenoy marked this pull request as ready for review October 27, 2026 21:15
@prathshenoy prathshenoy mentioned this pull request Oct 27, 2026
Merged
2 tasks done
@prathshenoy
prathshenoy added this pull request to the merge queue Oct 27, 2026
Merged via the queue into main with commit 125bea0 Oct 27, 2026
38 checks passed
Base automatically changed from codex/stovepipe-row-handoff to main October 27, 2026 21:59
@prathshenoy
prathshenoy deleted the codex/stovepipe-taskwide-closure branch October 27, 2026 21:59
@prathshenoy prathshenoy mentioned this pull request Oct 27, 2026
Merged
2 tasks done
DivyanshGarg380 pushed a commit to DivyanshGarg380/submitqueue that referenced this pull request Oct 28, 2026
@prathshenoy
feat(stovepipe): Track issues in bounded passes (uber#950)
078c40d 
## Summary

**What**:
- Process build issues in bounded batches and request redelivery until
the pass finishes.
- Open reported issues during refreshes and close missing issues after
final analysis, except when the build was canceled.

**Why**:
- Avoid doing an entire queue’s issue tracking in one delivery. Saving a
cursor lets later deliveries continue the pass while completing any
interrupted issue and history writes first.
- Keep delayed notifications from undoing newer issue state. Finish an
accepted pass before admitting another build, reject older sources, and
ignore refreshes after finalization; leave consumer activation and
terminal recovery for follow-up work.

## Test Plan

- [x] Add unit tests.

## Revert Plan

- Revert dependent PRs first, then this PR. Issue-tracking consumption
remains disabled, so reverting does not interrupt an active
issue-tracking worker.

## Issue Links

- [CODEM-607](https://linear.app/uber/issue/CODEM-607/build-analysis)

## Stack

1. uber#942
1. uber#948
1. uber#949
1. uber#950
1. uber#951
1. uber#958
1. uber#961
DivyanshGarg380 pushed a commit to DivyanshGarg380/submitqueue that referenced this pull request Oct 28, 2026
@prathshenoy
feat(stovepipe): Publish issue-tracking updates (uber#951)
438ab74 
## Summary

**What**:
- Publish issue-tracking refresh and finalization notifications after
build analysis persists its results.
- Add the publication route without enabling issue-tracking consumption.

**Why**:
- Give issue tracking a reliable handoff from build analysis. Publishing
after persistence ensures it can read the build’s recorded issues, and a
failed publish can retry without losing the handoff. Require a complete
cumulative terminal result so finalization does not depend on an earlier
worker finishing its writes.
- Keep terminal recovery separate from this rollout. Registering only
the publication route allows the processing layers to be reviewed
without starting a consumer whose recovery policy is not yet wired.

## Test Plan

- [x] Add unit tests.

## Revert Plan

- Revert dependent PRs first, then this PR. Issue-tracking consumption
remains disabled, so reverting does not interrupt an active
issue-tracking worker.

## Issue Links

- [CODEM-607](https://linear.app/uber/issue/CODEM-607/build-analysis)

## Stack

1. uber#942
1. uber#948
1. uber#949
1. uber#950
1. uber#951
1. uber#958
1. uber#961
DivyanshGarg380 pushed a commit to DivyanshGarg380/submitqueue that referenced this pull request Oct 28, 2026
@prathshenoy
feat(stovepipe): Recover failed issue tracking (uber#961)
25bc437 
## Summary

**What**:
- Add dead-letter recovery that stops failed issue-tracking passes while
keeping completed issue changes.
- Record incomplete issue tracking without changing the build's
validation outcome.

**Why**:
- Unblock later builds when issue tracking exhausts normal retries.
Today the unfinished pass remains saved, so later notifications try to
resume the same failed work. Recovery needs to stop that pass without
throwing away issue changes that already succeeded.
- Keep delayed workers from changing issue state after recovery. Simply
clearing progress would let a paused worker write an old opening or
closing after a newer build has run. Recovery protects the current issue
and retains its history before allowing newer work through.

## Test Plan

- [x] Add unit tests.
- [x] Verify recovery progress persists against a real MySQL database.

## Revert Plan

- Revert this PR before enabling consumption. If already enabled, stop
consumers and reconcile recovery progress before rolling back so an
older consumer does not resume abandoned work.

## Issue Links

- [CODEM-607](https://linear.app/uber/issue/CODEM-607/build-analysis)

## Stack

1. uber#942
1. uber#948
1. uber#949
1. uber#950
1. uber#951
1. uber#958
1. uber#961
1. uber#962
DivyanshGarg380 pushed a commit to DivyanshGarg380/submitqueue that referenced this pull request Oct 28, 2026
@prathshenoy
feat(stovepipe): Wire issue tracking consumers (uber#962)
a158dfb 
## Summary

**What**:
- Enable normal issue tracking and dead-letter recovery in the reference
service.

**Why**:
- Consume the issue notifications that build analysis already publishes.
Until now they accumulated on the queue without updating current issues
or their history. Registering both consumers makes that path
operational, while deployments can still supply their own build-issue
detector.
- Keep failed tracking from blocking later builds indefinitely. Normal
processing uses bounded retries, and recovery stops incomplete work
safely instead of repeating the same failed pass forever.

## Test Plan

- [x] Add unit tests.
- [x] Verify analysis handoff and dead-letter recovery through the
running service and MySQL.

## Revert Plan

- Revert this PR to disable both consumers without deleting persisted
issue state, history, or progress.

## Issue Links

- [CODEM-607](https://linear.app/uber/issue/CODEM-607/build-analysis)

## Stack

1. uber#942
1. uber#948
1. uber#949
1. uber#950
1. uber#951
1. uber#958
1. uber#961
1. uber#962
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mnoah1 mnoah1 mnoah1 approved these changes
@behinddwalls behinddwalls Awaiting requested review from behinddwalls behinddwalls is a code owner
@sbalabanov sbalabanov Awaiting requested review from sbalabanov sbalabanov is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@prathshenoy @mnoah1