Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 8 additions & 14 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,28 +213,22 @@ E2E tests in presubmit CI are automatically triggered via OpenShift's Prow infra

### Automated Failure Analysis with Claude

When E2E tests fail in Prow CI, Claude Code automatically analyzes the failures and generates a comprehensive report.
When E2E tests fail in Prow CI, an `openshift/release` step-registry post-step analyzes the failures and generates a comprehensive report — following the shared `claude-ai-helpers` pattern also used by `medik8s`, `hypershift`, and other CI teams.

**How it works**:

1. After test execution completes with failures, the analysis script (`tests/e2e/scripts/analyze_failures.sh`) is invoked
2. Claude runs in headless mode (`--print` flag) for non-interactive CI automation via Vertex AI
3. Claude analyzes artifacts written by the E2E test code: JUnit reports, must-gather diagnostics, and per-test pod logs
4. A detailed markdown report is generated at `${ARTIFACT_DIR}/claude-failure-analysis.md`
5. The report includes root cause analysis, known flake detection, and actionable recommendations
1. The `oadp-analyze-e2e-failure` post-step (defined in `openshift/release`, not this repo) runs after the E2E test step, using the shared `claude-ai-helpers` base image (built from `openshift-eng/ai-helpers`) with Claude Code preinstalled.
2. It authenticates via the shared, already-provisioned `test-credentials/sa-claude-openshift-ci` credential — no per-repo vault/credential setup needed.
3. Claude analyzes artifacts written by the E2E test code: JUnit reports, must-gather diagnostics, and per-test pod logs.
4. A detailed markdown report is generated in the job's artifact directory.
5. The report includes root cause analysis, known flake detection, and actionable recommendations.

**Important**: Claude analyzes only artifacts generated during test execution (JUnit, must-gather, per-test logs). Prow's build-log.txt is written by CI infrastructure after tests complete and is not available during analysis.

**Accessing the analysis**:

- Find `claude-failure-analysis.md` in the Prow artifacts directory alongside other test outputs
- URL pattern: `https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_oadp-operator/<PR>/<job-name>/<build-id>/artifacts/claude-failure-analysis.md`
- Find the generated report in the Prow artifacts directory alongside other test outputs.

**Configuration**:

- Analysis requires Vertex AI credentials configured in the CI environment
- Gracefully skips if credentials are not available (no impact on test execution)
- Can be disabled by setting `SKIP_CLAUDE_ANALYSIS=true`
- **Automatic secret redaction**: API keys, tokens, passwords, and credentials are automatically redacted from output
**Local/manual use**: `tests/e2e/scripts/analyze_failures.sh` is kept in this repo for running the same analysis by hand against a local `make test-e2e` run; it is no longer invoked by CI or the `test-e2e` Makefile target.

For more details, see the [design document](docs/design/claude-prow-failure-analysis_design.md).
11 changes: 0 additions & 11 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -1073,17 +1073,6 @@ test-e2e: test-e2e-setup install-ginkgo $(if $(MUST_GATHER_REPO),build-must-gath
-skipMustGather=$(SKIP_MUST_GATHER) \
$(HCP_EXTERNAL_ARGS) \
|| EXIT_CODE=$$?; \
if [ "$(OPENSHIFT_CI)" = "true" ]; then \
if [ -f /var/run/oadp-credentials/gcp-claude-code-credentials ]; then \
export GOOGLE_APPLICATION_CREDENTIALS=/var/run/oadp-credentials/gcp-claude-code-credentials; \
export CLAUDE_CODE_USE_VERTEX=1; \
export CLOUD_ML_REGION=global; \
if [ -f /var/run/oadp-credentials/gcp-claude-code-project-id ]; then \
export ANTHROPIC_VERTEX_PROJECT_ID=$$(cat /var/run/oadp-credentials/gcp-claude-code-project-id); \
fi; \
fi; \
./tests/e2e/scripts/analyze_failures.sh $${EXIT_CODE:-0}; \
fi; \
exit $${EXIT_CODE:-0}

.PHONY: test-e2e-cleanup
Expand Down
27 changes: 0 additions & 27 deletions build/ci-Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,6 @@ WORKDIR /go/src/github.com/openshift/oadp-operator

COPY ./ .

# Make analysis script executable for CI execution
RUN chmod +x tests/e2e/scripts/analyze_failures.sh

# Install kubectl (multi-arch)
ARG TARGETARCH
# --retry: curl's default retry set already covers exit 6 (could not resolve
Expand All @@ -28,30 +25,6 @@ RUN export KV_VERSION=$(curl --retry 5 --retry-delay 5 -s https://storage.google
chmod +x virtctl && \
mv virtctl /usr/local/bin/

# Install Claude CLI (native binary, no Node.js dependency)
RUN curl --retry 5 --retry-delay 5 -fsSL https://claude.ai/install.sh | bash && \
ln -sf ~/.local/bin/claude /usr/local/bin/claude && \
claude --version

# Clone openshift/velero source code for failure analysis
# Uses oadp-dev branch to match OADP operator development
# Retried like the curl fetches above -- same category of transient
# network/DNS blip, just via git instead of curl (no built-in --retry flag).
# rm -rf before each attempt: a clone that fails partway through can leave a
# non-empty target directory behind, which would make the next attempt fail
# with "destination path already exists" instead of actually retrying.
# The trailing `test -d .../.git` is load-bearing, not decoration: without it,
# `&& break || sleep 5` makes the loop's own exit status the last `sleep`'s
# (always 0), so RUN succeeds even after all 3 clone attempts fail -- silently
# shipping an image with no velero checkout instead of failing the build here.
RUN for i in 1 2 3; do \
rm -rf /go/src/github.com/openshift/velero && \
git clone --depth 1 --branch oadp-dev \
https://github.com/openshift/velero.git \
/go/src/github.com/openshift/velero && break || sleep 5; \
done; \
test -d /go/src/github.com/openshift/velero/.git

RUN go mod download && \
mkdir -p $(go env GOCACHE) && \
chmod -R 777 ./ $(go env GOCACHE) $(go env GOPATH)
Loading