test: ~264 failing tests across 48 files, unmasked by fixing the babel 7/8 jest crash (PR #3306)

[eleshar]

Summary

Fixing PR #3306's Babel 7/8 dependency conflict (see that PR) let jest actually run for the first time in this repo — previously every test file crashed at load time with BABEL_VERSION_UNSUPPORTED before a single test executed, so npm test / CI reported no useful signal at all regardless of what was broken underneath.

With jest running, the full suite reports:

Test Suites: 48 failed, 204 passed, 252 total
Tests:       264 failed, 14 todo, 4306 passed, 4584 total

These 264 failures are pre-existing and unrelated to PR #3306 or the PRD-agent consolidation work — confirmed by spot-checking one representative case: agents/pr-creation-agent/skills/validate-branch-name.js never populates the scope/shortTitle fields its own test file (agents/pr-creation-agent/__tests__/validate-branch-name.test.js) expects, independent of Babel or any recent change. This has evidently been broken for a long time; it was simply invisible because jest could not run at all.

Scope

48 unique failing test files, spanning many unrelated subsystems:

• Agent-spec tooling: .github/scripts/__tests__/create-agent-spec.test.js, generate-agent-index.test.js, validate-agent-specs.test.js, workflow-integration.test.js
• Milestone assignment: __tests__/agents/milestone-assignment.test.js, scripts/agents/includes/__tests__/milestone-assignment.test.js, scripts/automation/__tests__/allocate-to-milestone.test.js, milestone-management.test.js, integration-workflow-milestone.test.js
• PR/branch/label agents: agents/pr-creation-agent/__tests__/validate-branch-name.test.js, orchestrate-pr-creation.test.js, integration/label-application-scenarios.test.js
• Metadata agent: agents/metadata-agent/__tests__/api/octokit-client.test.js, context/repo-context-detector.test.js, context/repo-templates.test.js
• Metrics: scripts/metrics/__tests__/integration.test.js, metrics-reporter.test.js, metrics-storage.test.js, trend-analyzer.test.js, scripts/metrics/integrations/__tests__/issue-templates.test.js, scripts/workflows/__tests__/metrics-reporting-orchestrator.test.js
• Linting agent integrations: scripts/agents/__tests__/linting-agent/integration/{block-plugin,control-plane,wordpress-plugin,wordpress-theme}.integration.test.js
• Validation scripts: scripts/validation/__tests__/{agent-frontmatter-pr-changes,issue-template-contract,openspec-labels,project-meta-sync,validate-frontmatter,validate-frontmatter-changed-files}.test.js
• Automation/performance: scripts/automation/__tests__/{staging-validation,performance/phase-2b-validation,performance/phase-2c-validation,performance/performance-benchmarking-phase-2c,performance/generate-phase-2c-reports}.test.mjs
• Release: agents/release/gates/__tests__/release-gates.test.js, scripts/agents/__tests__/release.agent.mcp.test.js
• Misc: agents/meta-agent/__tests__/frontmatter-validation.test.js, __tests__/agents/remediation-checklist-generator.test.js, scripts/__tests__/collect-link-targets.test.js, scripts/agents/__tests__/project-meta-sync.agent.test.js, scripts/agents/includes/__tests__/{changelogUtils,check-template-labels,field-parity,header-footer,label-heuristics}.test.js, scripts/workflows/branch-policy/__tests__/validate-main-branch-pr.test.js

Full list (deduplicated, alphabetical):

.github/scripts/__tests__/create-agent-spec.test.js
.github/scripts/__tests__/generate-agent-index.test.js
.github/scripts/__tests__/validate-agent-specs.test.js
.github/scripts/__tests__/workflow-integration.test.js
__tests__/agents/milestone-assignment.test.js
__tests__/agents/remediation-checklist-generator.test.js
agents/meta-agent/__tests__/frontmatter-validation.test.js
agents/metadata-agent/__tests__/api/octokit-client.test.js
agents/metadata-agent/__tests__/context/repo-context-detector.test.js
agents/metadata-agent/__tests__/context/repo-templates.test.js
agents/pr-creation-agent/__tests__/integration/label-application-scenarios.test.js
agents/pr-creation-agent/__tests__/orchestrate-pr-creation.test.js
agents/pr-creation-agent/__tests__/validate-branch-name.test.js
agents/release/gates/__tests__/release-gates.test.js
scripts/__tests__/collect-link-targets.test.js
scripts/agents/__tests__/linting-agent/integration/block-plugin.integration.test.js
scripts/agents/__tests__/linting-agent/integration/control-plane.integration.test.js
scripts/agents/__tests__/linting-agent/integration/wordpress-plugin.integration.test.js
scripts/agents/__tests__/linting-agent/integration/wordpress-theme.integration.test.js
scripts/agents/__tests__/project-meta-sync.agent.test.js
scripts/agents/__tests__/release.agent.mcp.test.js
scripts/agents/includes/__tests__/changelogUtils.test.js
scripts/agents/includes/__tests__/check-template-labels.test.js
scripts/agents/includes/__tests__/field-parity.test.js
scripts/agents/includes/__tests__/header-footer.test.js
scripts/agents/includes/__tests__/label-heuristics.test.js
scripts/agents/includes/__tests__/milestone-assignment.test.js
scripts/automation/__tests__/allocate-to-milestone.test.js
scripts/automation/__tests__/integration-workflow-milestone.test.js
scripts/automation/__tests__/milestone-management.test.js
scripts/automation/__tests__/performance/generate-phase-2c-reports.test.mjs
scripts/automation/__tests__/performance/performance-benchmarking-phase-2c.test.mjs
scripts/automation/__tests__/performance/phase-2b-validation.test.mjs
scripts/automation/__tests__/performance/phase-2c-validation.test.mjs
scripts/automation/__tests__/staging-validation.test.js
scripts/metrics/__tests__/integration.test.js
scripts/metrics/__tests__/metrics-reporter.test.js
scripts/metrics/__tests__/metrics-storage.test.js
scripts/metrics/__tests__/trend-analyzer.test.js
scripts/metrics/integrations/__tests__/issue-templates.test.js
scripts/validation/__tests__/agent-frontmatter-pr-changes.test.js
scripts/validation/__tests__/issue-template-contract.test.js
scripts/validation/__tests__/openspec-labels.test.js
scripts/validation/__tests__/project-meta-sync.test.js
scripts/validation/__tests__/validate-frontmatter-changed-files.test.js
scripts/validation/__tests__/validate-frontmatter.test.js
scripts/workflows/__tests__/metrics-reporting-orchestrator.test.js
scripts/workflows/branch-policy/__tests__/validate-main-branch-pr.test.js

Confirmed example (not a guess)

agents/pr-creation-agent/skills/validate-branch-name.js's validateBranchName() return object never includes scope or shortTitle — only valid, errors, type. Its own test file (agents/pr-creation-agent/__tests__/validate-branch-name.test.js) asserts result.scope / result.shortTitle on ~20 of its 35 cases, all failing with Received: undefined. This is a genuine implementation gap in the skill itself, unrelated to Babel, PR #3306, or PRD-agent work.

Suggested next step

Triage in batches by subsystem (milestone-assignment tests share one bug fix, most likely; linting-agent integration tests probably share another). Each failure needs the same discipline as any fix: verify against current code before changing anything, since (as with the PRD-agent doc tests) some failures may turn out to be tests that drifted from legitimately-evolved implementation rather than real regressions.

🤖 Generated with Claude Code

[linear-code]

LS-4125

[eleshar]

Root-caused `scripts/validation/tests/validate-frontmatter-changed-files.test.js` (listed in this issue's scope) precisely, in case it helps whoever picks this cluster up:

`FileDiscovery.findChangedFiles()` (`scripts/validation/validate-frontmatter.js:399`) does `.map((rel) => path.resolve(rootDir, rel))`. The test hardcodes `rootDir = '/repo'` and expects the result to stay `/repo/...` (POSIX-style). On Windows, `path.resolve('/repo', 'x')` resolves `/repo` as "root of the current drive" (Node's documented `path.win32` behaviour), producing `C:\repo\x` instead — not a bug in the source's path handling (native paths are fine for the actual `fs` calls that consume this output), just a test that hardcoded a platform-specific expected value.

Confirmed via a clean `develop` checkout (no local changes) — reproduces identically, so it's genuinely pre-existing, not introduced by anything recent.

Suggested fix: compute the expected values via the same `path.resolve(rootDir, rel)` calls at test-time instead of hardcoding `/repo/...` strings, so the assertion verifies the real behavior (filtering/dedup/sort) without being tied to a separator style.

[eleshar]

Found the same root cause hits a second file too: `scripts/validation/tests/validate-links.test.js` — 9 failures, all `toHaveLength` mismatches where the actual array has one extra entry with a Windows-style `resolved: "C:\root..."` path. Same underlying pattern as the comment above: tests hardcode POSIX-style expected paths/behaviour against a `path.resolve`-based implementation that's correctly platform-aware. Confirmed via a clean `develop` checkout, not introduced by anything recent.

[eleshar]

Status update 2026-09-23: full suite on develop @ 5fb6cc0 now reports 7 failed suites / 5 failed tests (5201 passed). The 264-failure inventory above is pre-restructure and obsolete for most entries. Current failures re-baselined in #3472; reference-detection subset tracked in #3460. Leaving open as the canonical failing-tests tracker.