Migrate Claude e2e failure analysis to shared claude-ai-helpers step-registry pattern

[kaovilai]

Summary

Migrate OADP's e2e failure analysis from the current in-repo Claude CLI setup to the shared claude-ai-helpers step-registry pattern now used by other CI teams (e.g. medik8s, hypershift).

Current approach (this repo)

Added in #2038, present on oadp-dev and oadp-1.6:

• build/ci-Dockerfile installs Node.js + @anthropic-ai/claude-code (or the claude.ai/install.sh installer on oadp-dev) directly into the test-oadp-operator image.
• Makefile's test-e2e target invokes tests/e2e/scripts/analyze_failures.sh after Ginkgo exits, wiring GOOGLE_APPLICATION_CREDENTIALS / CLAUDE_CODE_USE_VERTEX / ANTHROPIC_VERTEX_PROJECT_ID from files expected under /var/run/oadp-credentials/gcp-claude-code-*.
• Those gcp-claude-code-* files were never actually added to the oadp-credentials vault collection / to the openshift/release config, so in practice analyze_failures.sh always hits the "Vertex AI not configured — skipping" branch and no-ops today.

New approach (already used by medik8s, hypershift, security/adversary-scan, etc.)

openshift/release publishes a shared claude-ai-helpers base image (ci namespace, built from openshift-eng/ai-helpers) with Claude Code and CI-analysis skills preinstalled, plus a shared, already-provisioned credential test-credentials/sa-claude-openshift-ci. Consuming repos just add a step-registry post-step to their ci-operator config test — no changes needed inside the repo's own container image or Makefile. See medik8s-analyze-e2e-failure / hypershift-analyze-e2e-failure refs in openshift/release for the pattern.

A companion openshift/release PR adds oadp-analyze-e2e-failure and wires it into the oadp-1.6 and oadp-dev ci-operator configs (the only branches currently carrying the Makefile hook — oadp-1.3/oadp-1.4/oadp-1.5 were checked and don't have it).

Requirements / follow-up changes needed in this repo

• Remove the Node.js/Claude CLI install steps from build/ci-Dockerfile. (Migrate Claude e2e failure analysis to claude-ai-helpers step-registry #2408, [oadp-1.6] Migrate Claude e2e failure analysis to claude-ai-helpers step-registry #2409)
• Remove the test-e2e Makefile hook that shells out to tests/e2e/scripts/analyze_failures.sh (and the GOOGLE_APPLICATION_CREDENTIALS/CLAUDE_CODE_USE_VERTEX/ANTHROPIC_VERTEX_PROJECT_ID wiring in Makefile). (Migrate Claude e2e failure analysis to claude-ai-helpers step-registry #2408, [oadp-1.6] Migrate Claude e2e failure analysis to claude-ai-helpers step-registry #2409)
• Retire (or repurpose for local/manual use only) tests/e2e/scripts/analyze_failures.sh — analysis now runs as a post-step outside the test container, driven from openshift/release. (Migrate Claude e2e failure analysis to claude-ai-helpers step-registry #2408, [oadp-1.6] Migrate Claude e2e failure analysis to claude-ai-helpers step-registry #2409)
• No vault/credential changes needed here — the new post-step uses the already-shared test-credentials/sa-claude-openshift-ci secret, so the planned gcp-claude-code-credentials / gcp-claude-code-project-id additions to the oadp-credentials collection are no longer necessary.
• Update CLAUDE.md's "Automated Failure Analysis with Claude" section and docs/design/claude-prow-failure-analysis_design.md to describe the new architecture. (Migrate Claude e2e failure analysis to claude-ai-helpers step-registry #2408, [oadp-1.6] Migrate Claude e2e failure analysis to claude-ai-helpers step-registry #2409)

Pull Requests

• Migrate Claude e2e failure analysis to claude-ai-helpers step-registry #2408 — oadp-dev
• [oadp-1.6] Migrate Claude e2e failure analysis to claude-ai-helpers step-registry #2409 — oadp-1.6 (backport, same diff)

Follow-up (separate scope)

While verifying post-step compatibility across branches, found oadp-1.4's test-e2e target never generates junit_report.xml (missing --ginkgo.junit-report flag, unrelated to the Claude hook — tracked separately in #2410/#2411).

References

• PR to analyze failures based on the Tigers PR. #2038 (original in-repo implementation)
• openshift/release inspiration PRs: Add AI analysis to medik8s jobs release#82802, ROSAENG-64876: Add adversary security scan for rosa-hyperfleet release#83297
• Companion openshift/release PR: oadp-operator: migrate e2e Claude failure analysis to claude-ai-helpers post-step release#83957

Note

Responses generated with Claude

[kaovilai]

Companion openshift/release PR: openshift/release#83957

Note

Responses generated with Claude

[kaovilai]

Suggested merge order

1. oadp-operator: migrate e2e Claude failure analysis to claude-ai-helpers post-step release#83957 merges first — adds the oadp-analyze-e2e-failure post-step + config wiring for oadp-1.6 and oadp-dev. This is additive only (new post-step, new env), doesn't touch this repo, safe to land independently.
2. Verify on both branches: open a throwaway PR (or watch the next periodic run) on oadp-dev and oadp-1.6, confirm oadp-analyze-e2e-failure shows up in the Prow job's steps and produces claude-failure-analysis.md in artifacts on a real e2e failure (or exits early with "passed — skipping analysis" on green runs).
3. Only after step 2 is confirmed working on both branches, open the in-repo cleanup PR(s) here (per the checklist above) — separately for oadp-dev and oadp-1.6 since they're independent branches:
   • Remove Claude CLI install from build/ci-Dockerfile
   • Remove the test-e2e Makefile hook + Vertex env wiring
   • Retire/repurpose tests/e2e/scripts/analyze_failures.sh
   • Update CLAUDE.md / docs/design/claude-prow-failure-analysis_design.md
4. Land oadp-dev cleanup first, then cherry-pick/backport the same cleanup to oadp-1.6.

Don't remove the old in-repo hook before step 2 is confirmed — if the new post-step turns out to be broken/misconfigured, the old (currently silently no-op'd) path is otherwise the only fallback, and pulling both at once would leave zero failure-analysis coverage while debugging.

Note

Responses generated with Claude