Repository navigation
test: verify live judge artifact inspection on a compatible runner - #70
Merged
Merged
Conversation
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
Patrick-Erichsen
changed the title
clawsweeper
Bot
added
clawsweeper:intake:feature
|
Codex review: needs maintainer review before merge. What this changesAdds an opt-in Ubuntu 22.04 workflow that runs the real ClawHub judge against a harmless package and verifies its inspection receipt. Example: A maintainer manually enables judge-inspection for inspection-smoke@1.0.0.
Review scores
ProductKind: Test only · Worth it: Yes · Fix scope: Complete Merge readiness✅ Ready for maintainer review Keep this PR open for landing: it adds useful live judge coverage absent from main, follows the existing inspection contract, and introduces no actionable defect. Priority: P3 Before mergeNone. FindingsNone. Agent review detailsHow this fits togetherGitHub Actions builds the CLI, supplies local scanner fixtures and a restricted environment, invokes the existing ClawHub judge profile, and checks the resulting JSON report. flowchart LR
A[Manual dispatch] --> B[Ubuntu 22.04 runner]
B --> C[Harmless package and scanner fixtures]
C --> D[ClawScan CLI]
D --> E[Read only Codex judge]
E --> F[Receipt validation]
F --> G[Retained report]
Technical reviewBest possible solution: Land the bounded manual smoke workflow while preserving the existing judge sandbox and receipt validator. Do we have a high-confidence way to reproduce the issue? This verification-only PR supplies paired live Ubuntu runs and a successful final-head smoke run; no target code was executed during this review. Is this the best way to solve the issue? The focused workflow exercises the existing CLI and production judge profile without duplicating validation logic or weakening sandbox policy. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against 5be7f8e97fa4. Provenance checked
TestingProof path: shipped entry point. SecurityNone. EvidenceWhat I checked:
Likely related people:
LabelsLabel changes:
Label justifications:
Rating scale6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior. WorkflowClawSweeper edits this one comment on every review. Comment Reviewed October 20, 2026, 4:28 PM ET / 20:28 UTC. |
fix: render the challenged file in judge receipt examples
Select Go 1.27.3 for source, CI, and release builds while retaining the Go 1.27.1 language minimum. All Go workflows select the toolchain from go.mod. Update the README and changelog to match. On native macOS, the Go 1.27.2 baseline reaches GO-2026-6809 through `HuggingFaceBenchmarkClient.FetchSkillTrustBenchRows` at `internal/runner/benchmark.go:835`, then `bufio.Scanner.Scan` and `x509.Certificate.Verify`. The same symbol-level `govulncheck ./...` scan is clean on Go 1.27.3. Validation: `go build ./...`, `go test -count=1 ./...`, `go vet ./...`, npm wrapper and package-builder tests, and `node scripts/build-npm-package.mjs --version v0.0.0 --pack --smoke` pass. Independent P0–P3 autoreview found no actionable issues. A patch release will follow after exact-head CI and release preflight pass. This PR also prepares the dated v0.2.1 changelog so the same reviewed source can pass the tag-driven release preflight. The empty Unreleased section is retained for subsequent work.
Explicitly select Go 1.27.3 in every setup-go invocation, preserving the Go 1.27.1 source minimum and existing source toolchain declaration. Follow-up to #72: go-version-file makes setup-go select the language minimum before consulting the toolchain directive. The explicit version prevents this older setup selection in CI, release, benchmark, and results workflows. All six published v0.2.1 binaries already independently verify as Go 1.27.3 through source toolchain switching; no replacement release is needed. Validation: actionlint passes; independent P0–P3 autoreview found no actionable issues. Exact-head CI verifies both Linux and macOS packaging.
Apply the reviewed eMudhra incident remediation. Validation and remaining platform/deployment qualifications are recorded in the pull request.
A successful judge process can omit the required artifact path in its inspection receipt. Preserve that failed result and re-evaluate once with a fresh workspace/challenge under the original shared judge deadline. Never retry valid security verdicts or command failures; exhausted invalid receipts still fail closed.
A killed scanner run previously retained neither its final artifact nor the active stage. Emit allowlisted stderr phase and direct-child process events when CLAWSCAN_PROGRESS=1 so the worker can attribute an unfinished phase. Keep deadlines, raw scanner evidence and final verdict validation unchanged; this is diagnostic enablement, not a timeout recovery claim.
Add a manual Ubuntu 22.04 check for the ClawHub judge's real package-file inspection and retain its redacted receipt. Keep normal CI opt-out, preserve read-only execution, and pin the patched Go toolchain. Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com>
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in live check that the ClawHub judge can inspect package files in the worker's restricted environment. Ubuntu 24.04 rejects the nested bubblewrap setup even when a simple sandbox command succeeds; the retained check uses Ubuntu 22.04 and preserves the read-only judge sandbox.
The workflow builds ClawScan, installs the pinned Codex CLI, scans a harmless local package using scanner fixtures, and verifies the completed inspection receipt, manifest SHA-256, and inspected path. Normal push and pull-request CI leave this paid check disabled. It neither accesses registry state nor publishes packages.
Updated against current main with the explicit Go 1.27.3 pin, operator documentation, and a changelog entry crediting @Patrick-Erichsen.
Validation
8ba57605078bf24ed3016ec531a59740f7597a17.