Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 105 additions & 0 deletions .github/workflows/cc-parity.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# Opt-in bundle-scale parity (#9346). Non-required until maintainers promote it.
name: cc-parity

on:
pull_request:
types: [opened, synchronize, reopened, labeled]
workflow_dispatch:

permissions:
contents: read

concurrency:
group: cc-parity-${{ github.event_name }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
changes:
if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'run-cc-parity')
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
pull-requests: read
outputs:
relevant: ${{ steps.filter.outputs.relevant }}
steps:
- id: filter
env:
GH_TOKEN: ${{ github.token }}
EVENT_NAME: ${{ github.event_name }}
PR_NUMBER: ${{ github.event.pull_request.number }}
run: |
set -euo pipefail
if [ "$EVENT_NAME" != pull_request ]; then
echo 'relevant=true' >> "$GITHUB_OUTPUT"
exit 0
fi
cc_files=$(gh api "repos/$GITHUB_REPOSITORY/pulls/$PR_NUMBER/files" --paginate --jq '.[].filename')
# An empty listing must not quietly turn a requested gate green.
if [ -z "$cc_files" ] || grep -E '^(crates/|Cargo\.(toml|lock)$|rust-toolchain|\.cargo/|\.github/workflows/cc-parity\.yml$|scripts/cc_parity_gate\.py$|tests/(test_cc_parity_gate\.py$|cc-parity/))' <<< "$cc_files" > /dev/null; then
echo 'relevant=true' >> "$GITHUB_OUTPUT"
else
echo 'relevant=false' >> "$GITHUB_OUTPUT"
fi

cc-parity:
needs: changes
if: needs.changes.outputs.relevant == 'true'
# The bundle's IR construction exceeds the ARM runner's 7 GB RAM.
runs-on: macos-15-intel
timeout-minutes: 90
env:
CARGO_BUILD_JOBS: '4'
CARGO_INCREMENTAL: '0'
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false

- name: Test the gate's failure paths
run: python3 -m unittest discover -s tests -p test_cc_parity_gate.py -v

- name: Set scratch work directory
run: echo "CC_PARITY_WORK=$RUNNER_TEMP/cc-parity" >> "$GITHUB_ENV"

# This job uses the macOS SDK, not the preinstalled simulator images.
# simctl unmounts runtime images before deleting their backing storage.
- name: Free simulator runtime disk space
run: |
sudo xcrun simctl runtime delete all
df -h /

- name: Fetch and verify the pinned bundle
run: python3 scripts/cc_parity_gate.py prepare --work-dir "$CC_PARITY_WORK"

- name: Install LLVM 22
run: |
set -euo pipefail
brew install llvm@22 2>/dev/null || brew install llvm
cc_llvm_prefix="$(brew --prefix llvm@22 2>/dev/null || brew --prefix llvm)"
"$cc_llvm_prefix/bin/llvm-config" --version | grep -q '^22\.'
echo "LLVM_SYS_221_PREFIX=$cc_llvm_prefix" >> "$GITHUB_ENV"

- uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2
with:
shared-key: cc-parity-wasm-host
save-if: ${{ github.event_name == 'workflow_dispatch' && github.ref == 'refs/heads/main' }}

- name: Build compiler, then all runtime archives together
run: python3 scripts/cc_parity_gate.py build --work-dir "$CC_PARITY_WORK"

- name: Compile the pinned bundle natively
run: python3 scripts/cc_parity_gate.py compile --timeout 4500 --work-dir "$CC_PARITY_WORK" --perry "$GITHUB_WORKSPACE/target/perry-dev/perry"

- name: Check help and version offline against golden bytes
run: python3 scripts/cc_parity_gate.py check --work-dir "$CC_PARITY_WORK"

- name: Upload compiler logs and parity results
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: cc-parity-results
path: ${{ runner.temp }}/cc-parity/logs/
if-no-files-found: warn
retention-days: 7
1 change: 1 addition & 0 deletions changelog.d/9793-cc-parity-gate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add an opt-in `run-cc-parity` CI gate that compiles pinned Claude Code 2.1.112 and checks native help/version output against offline Node goldens.
1 change: 1 addition & 0 deletions docs/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,7 @@
- [Geisterhand (UI Fuzzer)](testing/geisterhand.md)
- [Node Compatibility Matrix](testing/node-compat-matrix.md)
- [CI Tiers (PR gate / sweep / full)](testing/ci-tiers.md)
- [Claude Code Bundle Parity](testing/cc-parity.md)
- [CI Gate Scheduling](testing/ci-gate-scheduling.md)

# CLI Reference
Expand Down
101 changes: 101 additions & 0 deletions docs/src/testing/cc-parity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# Claude Code bundle parity

The `cc-parity` workflow compiles the standalone Claude Code **2.1.112** npm
bundle with Perry and compares native `--help` and `--version` stdout with
checked-in Node output. It covers the bundle-scale regressions described in
[#9346](https://github.com/PerryTS/perry/issues/9346).

## Opt in

Apply **`run-cc-parity`** to a PR changing compiler/runtime crates, build inputs,
or the gate itself. The workflow also supports manual dispatch. Unlabelled PRs
skip every job; labelled documentation-only PRs skip the expensive job. A new
commit supersedes the previous run on the same PR.

This starts as a **non-required** check. Adding it to branch protection is a
separate maintainer decision after successful hosted runs. It has no push,
schedule, or release-tag trigger and is independent of `run-extended-tests`.

The expensive job uses one `macos-15-intel` runner. Its
[14 GB RAM allocation](https://docs.github.com/en/actions/reference/runners/github-hosted-runners)
provides more headroom for bundle IR construction than the 7 GB ARM runner.
The job removes unused simulator images and disables Cargo incremental artifacts
to leave disk space for LLVM and the native archives. The issue estimated 25–40
minutes for bundle compilation; local validation took **57 minutes 25 seconds**
on macOS arm64 with five LLVM workers. The hosted Intel run with four workers
remains to be measured. Allow additional time for toolchain setup, especially on
a cold cache. The job has a 90-minute cap, compilation a 75-minute cap, and each
CLI invocation a 60-second cap. Timings are recorded for diagnosis, not compared
with a performance threshold.

## What the check proves

`tests/cc-parity/manifest.json` pins the npm tarball and extracted `package/cli.js`
by both size and SHA-256. Only that regular file is extracted; no package install
hooks run. The compiler is built first, then the runtime, stdlib, Wasm host, and
all native extension archives are built together with `perry-runtime/wasm-host`.
This avoids stale runtime copies in extension archives (#6303). Compilation uses
`--no-auto-optimize --no-cache --enable-wasm-runtime`, with four LLVM workers
(`PERRY_CODEGEN_UNIT_JOBS=4`) to use the Intel runner's four cores within its
memory budget.

The runtime arm requires a native Mach-O executable. Each invocation gets its own
temporary HOME, XDG directories, working directory, and TMPDIR, with a small
environment allowlist and no inherited credentials or compiler tuning knobs.
macOS `sandbox-exec` denies network access; the gate fails if that sandbox is
unavailable. The harness tests include an attempted connection to prove the
network restriction is active.

Both commands must exit zero before their deadlines and produce exactly the
golden bytes: **9,175 bytes** for help and **22 bytes** for version. A crash,
timeout, empty output, or one-byte difference fails. The manifest also pins the
goldens themselves, so changing a golden without updating its identity fails.

Downloading the bundle, LLVM, and Rust dependencies requires network access
during setup. The two CLI executions are offline and use no Node installation
or API key. The artifact contains source identity, build/compile logs, actual
stdout/stderr, and JSON results; it excludes the downloaded bundle and executable.

## Run locally on macOS

From the repository root, with LLVM 22 and the pinned Rust toolchain available:

```bash
export LLVM_SYS_221_PREFIX="$(brew --prefix llvm@22)"
export CARGO_BUILD_JOBS=4
cc_work="$(mktemp -d)"
python3 -m unittest discover -s tests -p test_cc_parity_gate.py -v
python3 scripts/cc_parity_gate.py prepare --work-dir "$cc_work"
python3 scripts/cc_parity_gate.py build --work-dir "$cc_work"
python3 scripts/cc_parity_gate.py compile --timeout 4500 --work-dir "$cc_work" --perry "$PWD/target/perry-dev/perry"
python3 scripts/cc_parity_gate.py check --work-dir "$cc_work"
```

If using `CARGO_TARGET_DIR`, pass the compiler in that directory instead. A local
tarball can be supplied to `prepare --archive <path>`; the same hashes are still
required. Inspect `$cc_work/logs/` for output differences and failure details.
Never run the bundle using your regular HOME: Claude can write its configuration
even on startup paths.

## Refresh the pin and oracle deliberately

2.1.112 is a standalone `cli.js` release. A newer package may have a different
distribution shape; confirm it still supplies the full bundle before changing
the pin. Update the manifest's version, URL, archive identity, and bundle identity
from the exact public npm tarball, then run `prepare` again.

The recorded reference used Node **v26.5.1** on macOS arm64. To verify that oracle
with the same scratch environment and network sandbox:

```bash
python3 scripts/cc_parity_gate.py check --work-dir "$cc_work" --node "$(command -v node)"
```

This writes `logs/node-help.stdout`, `logs/node-version.stdout`, and
`logs/node-parity.json`. A deliberate version refresh may fail the old golden
comparison; inspect both command results, require zero exit codes and no timeout,
and review the output changes before copying those two stdout files into
`tests/cc-parity/`. Update their byte counts and SHA-256 values and the oracle
provenance in the manifest. Rerun the Node check, harness tests, native compilation,
and native check. Commit the manifest and goldens together; never accept output
from a failing native executable as the new oracle.
4 changes: 4 additions & 0 deletions docs/src/testing/ci-tiers.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,10 @@ window (`previous sweep SHA .. this sweep SHA`), exactly as for the six-hourly g

## Opting a PR into more

- **`run-cc-parity` label** — runs the [Claude Code bundle parity gate](cc-parity.md)
on changes to crates, build inputs, or the gate. This independent, initially
non-required check compiles pinned Claude Code 2.1.112 and compares offline
native help/version output with checked-in Node goldens on one macOS runner.
- **`run-extended-tests` label** — promotes the PR's `test.yml` run to the `full`
tier AND enables the PR arm of every satellite gate. Use it for GC / codegen
changes that should be measured before merge, and for anything touching a
Expand Down
Loading
Loading