Support exact OpenClaw candidate packages in native evals

[vincentkoc]

Summary

Allow native OpenClaw evaluation campaigns to install an exact local package
tarball instead of requiring a registry-published harness version.

Motivation

ShellBench can reproduce released OpenClaw versions, but it cannot measure a
candidate OpenClaw fix before publication. That blocks clean baseline/candidate
ablations for benchmark-driven product work and encourages ad hoc remote
machine edits that are not auditable or resumable.

The campaign needs to preserve the candidate package name, version, SHA-256,
and resume identity so a result can be tied to the exact bits under test.

Proposed approach

Add an OpenClaw-only --openclaw-package-tarball input that:

• validates the package name and version before leasing,
• stages the archive with restrictive permissions,
• uploads and installs the exact artifact during bootstrap,
• records its SHA-256 in fleet, run, and audit metadata,
• rejects resume when candidate identity changes or is missing.

Registry-based runs remain unchanged when the option is absent.

Alternatives considered

• Publishing temporary npm versions: pollutes the registry and couples
  evaluation to release operations.
• Editing a warmed machine: not reproducible, not resumable, and not auditable.
• Building OpenClaw inside every lease: slower and introduces build-environment
  variance into the harness comparison.

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed August 2, 2026, 3:14 PM ET / 19:14 UTC.

Summary
This member-authored feature request remains necessary on current main: native evaluation accepts and verifies only the runner and task archives, then bootstraps OpenClaw from the registry. The linked open implementation PR already owns the requested work, so this issue should remain open while maintainers decide the acceptable artifact-execution boundary.

Reproducibility: not applicable. this is a missing capability rather than a report of broken existing behavior. Current-main source confirms the registry-only bootstrap and absence of a candidate-package input.

Ways to help us reproduce this

• Add a screenshot or short recording showing the behavior.
• Add expected vs actual behavior.

Maintainer decision needed

Question	Recommendation
Should native evaluation accept operator-supplied local OpenClaw package tarballs on leased hosts that receive provider credentials, and what minimum admission, retention, and resume-identity constraints must apply?	Approve a constrained opt-in path: Accept only an explicit OpenClaw-only tarball input with local validation, restrictive staging, SHA-256 provenance, strict resume matching, and documented operator responsibility.

Why: Current main has no equivalent capability, while the requested feature deliberately expands the executable-artifact boundary of the native fleet; source inspection cannot determine the repository’s intended trust policy.

Next step
The linked open pull request already owns implementation; the remaining action is a maintainer decision on the artifact security boundary, not a second repair PR.

Security
Needs attention: The issue proposes a new executable-artifact path onto leased hosts that receive provider credentials, so the trust boundary requires maintainer approval before implementation lands.

Review details

Best possible solution:

Land the paired implementation only with an explicit opt-in, OpenClaw-only artifact policy that verifies and records package identity, protects the staged archive and credentials, and rejects resumes whose candidate package identity differs.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this is a missing capability rather than a report of broken existing behavior. Current-main source confirms the registry-only bootstrap and absence of a candidate-package input.

Is this the best way to solve the issue?

Unclear: the paired pull request matches the requested architecture, but maintainers must first choose the acceptable trust boundary for installing a local artifact on credentialed leased hosts.

AGENTS.md: not found in the target repository.

Remaining risk / open question:

• A local package tarball is executable code installed on leased hosts that also receive provider environment secrets; maintainers need an explicit artifact-admission and retention policy before accepting this capability.
• Changing candidate identity during a resumed campaign could invalidate benchmark provenance unless the package name, version, and digest are bound to the run index and audit outputs.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Label changes

Label justifications:

• P2: This is a bounded benchmark-workflow capability with meaningful provenance and security implications, but no demonstrated production outage or user data loss.
• impact:security: The proposal adds installation of operator-supplied executable packages on leased machines supplied with provider environment credentials.
• impact:other: Exact artifact identity materially affects auditability and comparability of native benchmark results outside the more specific impact taxonomy.

Evidence reviewed

Security concerns:

• [medium] Define the candidate-artifact trust boundary — scripts/native_eval/bootstrap_beast.sh:82
  A local npm package tarball becomes executable during bootstrap on remote leased machines that receive provider environment data; admission, archive handling, provenance, and cleanup requirements need explicit policy.
  Confidence: 0.95

What I checked:

• Current main lacks candidate-package input: The native fleet configuration and CLI accept task, runner, and environment inputs, but no OpenClaw package tarball option or corresponding configuration field. (scripts/native_eval/fleet.py:199, 884dd1bb5511)
• Current main only stages runner and task archives: Input preparation validates and hashes the runner and task archives, and fleet metadata records only those archive digests; no candidate OpenClaw artifact identity is retained. (scripts/native_eval/fleet.py:445, 884dd1bb5511)
• Bootstrap remains registry-based: The remote bootstrap installs openclaw@$OPENCLAW_VERSION through npm, confirming that current main cannot install an exact locally supplied candidate package. (scripts/native_eval/bootstrap_beast.sh:82, 884dd1bb5511)
• Linked implementation candidate: The open linked pull request explicitly closes this issue and proposes the tarball input, artifact digest provenance, remote installation, and resume binding. It is not merged, so it is not evidence that current main implements the request. (3da8ec53b796)
• Feature provenance and likely ownership: Available local history shows the native matrix runner and its subsequent stabilization/runbook work were authored by Vincent Koc; the feature trail appears to begin with the auditable native matrix runner. (scripts/native_eval/fleet.py:445, 69f75c6629c4)
• Maturity classification unavailable: The required shortlist helper could not classify this issue because the checkout does not contain qa/maturity-scores.yaml; no maturity label is selected.

Likely related people:

• Vincent Koc: Available history attributes the native matrix runner, subsequent result-stability work, and the research campaign runbook to Vincent Koc; the linked implementation follows that established subsystem. (role: native-evaluation feature owner and recent area contributor; confidence: high; commits: 69f75c6629c4, b9acd9f7a010, 569b5c39c783; files: scripts/native_eval/fleet.py, scripts/native_eval/bootstrap_beast.sh, tests/test_native_eval_fleet.py)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.