issue-project-automation: concurrent label events cause deadlocks, run cancellations, and label corruption #29

Checked for duplicates

Yes – this tracks two related PRs (#27, #28) and documents the remaining failures they introduced.

🐛 Describe the bug

The issue-project-automation.yml workflow (and the reusable workflows it calls, label-to-project.yml and add-issue-to-project.yml) have gone through two failed serialization attempts for a race condition. The system is currently broken in at least two distinct ways affecting all NASA-PDS repositories.

Root Problem: Read-Merge-Write Race in set_org_issue_field_values

When a user selects multiple labels simultaneously in the GitHub Issues UI, GitHub fires one labeled webhook event per label. Each event independently triggers issue-project-automation.yml, so N labels → N concurrent workflow runs for the same issue.

Each run eventually calls sync-fields in .github/scripts/project_automation.py, which sets the org-level Product and Work Area fields via the GitHub REST issue-field-values endpoint. That endpoint is a replace-all PUT — there is no per-field PATCH — so the script does a read-merge-write:

1. Read all current org field values for the issue
2. Overlay the fields being set
3. PUT the merged result back

With two concurrent runs both reading before either has written, the second writer's PUT replaces the entire field set with a version that does not include what the first writer just set. Any human-set fields (Priority, Effort, custom dates) set between the read and write are silently wiped.

Attempted Fix 1: PR #27 — Concurrency Groups on All Workflows

Added a concurrency group keyed by repository + issue_number with cancel-in-progress: false to three files:

• issue-project-automation.yml (top-level caller)
• add-issue-to-project.yml (reusable, called via workflow_call)
• label-to-project.yml (reusable, called via workflow_call)

Result: Deadlock on every single-label event

GitHub Actions detected a self-referential wait: the top-level run holds the concurrency group while waiting for the reusable workflow to complete, but the reusable workflow declares the same group and cannot acquire it because the caller is still holding it. Every labeled/unlabeled event was immediately cancelled:

Canceling since a deadlock was detected for concurrency group:
'issue-automation-NASA-PDS/registry-client-71' between a top level workflow
and 'add-new-issue-to-project'

Attempted Fix 2: PR #28 (open, not yet merged) — Remove Concurrency from Reusable Workflows

Removes the concurrency block from add-issue-to-project.yml and label-to-project.yml, leaving it only on issue-project-automation.yml where it belongs.

Result: Intermediate events silently cancelled when 3+ events fire simultaneously

GitHub Actions concurrency with cancel-in-progress: false allows at most one waiting run per group at a time. When 3 or more events fire simultaneously:

• Run A executes
• Run B queues
• Run C arrives → Run B is cancelled, C becomes the new waiter

Canceling since a higher priority waiting request for
issue-automation-NASA-PDS/pdc-cds-infra-154 exists

Any label-specific work in the cancelled run (add to build project, set sprint, cascade labels to sub-issues) never completes. There is no native GitHub Actions mechanism to queue more than one waiting run; this is a fundamental platform limitation.

Additional Symptom: Adding a Build Label Removes Other Labels on the Same Issue

Separately observed: adding a build label such as B19 to an issue that already has B18, sprint-backlog, or other labels results in those other labels disappearing from the issue. The exact mechanism is not yet confirmed — it may be a side effect of the cascade-label step, an interaction with the concurrency cancellation, or an unrelated bug in how gh api -X POST ... -f "labels[]={label}" behaves. This needs its own investigation.

🕵️ Expected behavior

• All label events (including 3+ fired simultaneously) should complete without cancellation, deadlock, or silent drop
• Adding one label should not remove or affect other labels already on the issue
• Org-level field values set by automation should not clobber human-set fields (Priority, Effort, etc.)
• Org-level field values set by a previous run should not be reverted by a concurrent run

📜 To Reproduce

1. Open any issue in a NASA-PDS repo that uses issue-project-automation.yml
2. Add 3+ labels simultaneously or in rapid succession via the GitHub Issues UI
3. Observe one or more of:
   • Deadlock cancellation in the Actions run log (if PR fix: remove duplicate concurrency group causing automation deadlock #28 not yet merged to your copy)
   • "Higher priority waiting request" cancellation in the Actions run log
   • Some labels not added to the correct build project or sprint
4. Separately: add one build label (e.g., B19) to an issue that already has other labels and observe if any existing labels disappear

Confirmed affected repos: NASA-PDS/registry-client (issue #71), NASA-PDS/pdc-cds-infra (issue #154)

🖥 Environment Info

• Affected files: .github/workflows/issue-project-automation.yml, .github/workflows/label-to-project.yml, .github/workflows/add-issue-to-project.yml, .github/scripts/project_automation.py
• Affected repos: All ~62 NASA-PDS repos with issue-project-automation.yml (both the org default and per-repo copies)
• Related PRs: Fix race condition: serialize per-issue automation to prevent field-write collisions #27 (merged, introduced deadlock), fix: remove duplicate concurrency group causing automation deadlock #28 (open, fixes deadlock but not queue overflow)

🩺 Proposed Solution

Step 1 — Remove all concurrency blocks from all issue-automation workflows

GitHub Actions concurrency is fundamentally unsuitable here:

The fix is to remove concurrency from:

• issue-project-automation.yml in NASA-PDS/.github (org default + template)
• All ~62 per-repo copies of issue-project-automation.yml

This reverts the scope of PR #27 for the top-level workflow.

Step 2 — Fix the race in set_org_issue_field_values at the script level

Since the fields being written (Product, Work Area) are deterministic per repo (same repo always maps to the same values), most concurrent writes write the same values and are harmless. The destructive case is when a human-set field (Priority, Effort) is read as null by one concurrent run and then PUT back as null by that run, overwriting a value set by a prior run.

Option A — Skip-if-already-set (simplest): Before writing, check whether the fields we intend to set already have the target values. If so, skip the PUT. After the first run sets them, all subsequent concurrent runs see the values are already correct and do nothing. This eliminates ~95% of concurrent writes.

Option B — Retry with re-read (robust): After writing, re-read the field values and verify our changes persisted. If another concurrent writer overwrote them, retry up to N times with exponential backoff and jitter. Convergence is guaranteed because all writers write the same auto-field values.

Option C — Conditional PUT (correct, not currently available): Use ETag/If-Match headers or per-field PATCH if GitHub adds support. Not available as of writing.

Recommendation: implement Option A immediately (low risk, one-line guard in _build_fields_to_write) and add Option B as a safety net.

Step 3 — Investigate the label removal symptom

Before closing this issue, confirm:

• Whether gh api -X POST repos/{repo}/issues/{n}/labels -f "labels[]={label}" is additive (does not touch existing labels)
• Which step in label-to-project.yml or project_automation.py is responsible for the label removal
• Whether this is the same root cause as the concurrency problem or a separate bug

For Internal Dev Team To Complete

⚙️ Engineering Details

🎉 Integration & Test

🤖 Generated with help from Claude Code