Revise agent skills based on cross-harness transcript audit - #5
Open
Kabilan108 wants to merge 6 commits into
Open
Revise agent skills based on cross-harness transcript audit#5Kabilan108 wants to merge 6 commits into
Kabilan108 wants to merge 6 commits into
Conversation
📝 WalkthroughWalkthroughThis PR updates Claude and Codex model configuration, rewrites guidance across multiple agent skill documents, and changes the niri compositor config for a Helium Agents DevTools window rule. ChangesModel and runtime configuration
Agent skills documentation updates
Niri compositor config
Estimated code review effort: 2 (Simple) | ~15 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@agents/claude/CLAUDE.md`:
- Around line 77-79: The review example under CLAUDE.md is inconsistent with the
new override rule because `codex exec review` still uses the default full-access
settings from `~/.codex/config.toml`. Update the example so the review command
explicitly includes the same sandbox/effort overrides used elsewhere, or clearly
label review runs as an exception. Use the `codex exec review` example in this
document as the place to make the change.
In `@agents/skills/btca-local/SKILL.md`:
- Around line 14-43: The BTCA repo-selection guidance is too permissive when no
repo reference is provided, which can lead to using the wrong checkout. Update
the instructions in SKILL.md so the repo resolution flow in the
guidelines/workflow explicitly requires an exact user-provided repo reference
(or an explicit default repo) before cloning/searching; otherwise, ask for
clarification instead of guessing from context. Keep the existing symbols and
sections like the repo-resolution guideline, work dir setup, load, and search
steps aligned with this stricter behavior.
In `@agents/skills/frontend-design/SKILL.md`:
- Line 32: The Typography guidance in the frontend-design skill has conflicting
defaults around system fonts, so tighten it into one consistent rule. Update the
Typography section to keep the “existing products should use established type
scale/font tokens and local/system fonts when appropriate” guidance, and remove
any blanket prohibition that contradicts that exception. Make sure the final
wording clearly distinguishes standalone work from existing products without
implying system fonts are always disallowed.
In `@agents/skills/niri-computer-use/SKILL.md`:
- Around line 17-34: The Quick Start in the skill documentation still hardcodes
the dotfiles location as the default ACU_DIR, so Claude-installed copies can
resolve to a missing path. Update the ACU_DIR guidance in SKILL.md to derive
from the actual skill root or clearly fall back to the
~/.claude/skills/niri-computer-use location when the dotfiles path is absent,
and make sure the helper command examples consistently use that resolved
directory.
In `@agents/skills/review-swarm/SKILL.md`:
- Line 181: The review-swarm skill contract is contradictory because it mentions
a separate implementation phase after the read-only review, which conflicts with
the “read-only end-to-end” requirement. Update the wording in SKILL.md around
the review workflow to keep the skill strictly read-only and move any
fix/validation path out of this document, using the surrounding review-swarm
instructions as the reference point.
In `@agents/skills/tmux/SKILL.md`:
- Around line 118-123: The tmux SKILL preflight example only validates
send-to-pane, but the surrounding wording refers to all helper wrappers and
their flag shapes. Update the example near the send-to-pane, launch-agent, and
poll-agents references to run each helper’s --help check, or narrow the prose so
it only claims validation for send-to-pane.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 69539e0f-2be2-404c-afb3-770922766b97
📒 Files selected for processing (19)
agents/claude/CLAUDE.mdagents/claude/settings.jsonagents/codex/config.tomlagents/skills/agent-browser/SKILL.mdagents/skills/btca-local/SKILL.mdagents/skills/commit/SKILL.mdagents/skills/deslop/SKILL.mdagents/skills/frontend-design/SKILL.mdagents/skills/handoff/SKILL.mdagents/skills/helium-browser-use/SKILL.mdagents/skills/helium-browser-use/references/remote.mdagents/skills/learn-skill/SKILL.mdagents/skills/learn/SKILL.mdagents/skills/niri-computer-use/SKILL.mdagents/skills/pair-programmer/SKILL.mdagents/skills/playground/SKILL.mdagents/skills/review-swarm/SKILL.mdagents/skills/tmux/SKILL.mdhome/desktop/wayland/compositors/niri/config.kdl
Audited all 14 shared skills against ~620 transcript hits from the local and sietch tracer archives (claude + codex), then applied the evidence-backed fixes: - commit: nested-repo staging, pre-staged-change hygiene, concrete binary/large-file check (gif/h5 fixtures slipped through the old rule), generated-churn exclusions, dedupe allowed-tools - deslop: trigger on natural "clean up staged changes" asks, scope rules replacing bare $1, keep-list, audit-only mode, drop commit/reset perms - handoff: allow user-specified durable destinations (temp-only rule conflicted with real usage), restore gather-state step and doc structure that produced the best historical handoffs, hollow-handoff guard - pair-programmer: hard stop at the plan (sessions drifted into implementing after "outline what we need"), clarified clarify-vs-ask, output shape - frontend-design: context-fit direction for utilitarian surfaces, design-system discovery first, conditional typography, verification checklist, accessibility baseline, do-not-use triggers - tmux: helper preflight, existing-pane recipe with verification, remote SSH tmux examples, tee-to-log for long jobs, scrollback limits, codex subcommand --help checks - agent-browser: doctor/env preflight (stale AGENT_BROWSER_EXECUTABLE_PATH), auth-profile validation, --full warning, shared-browser discipline, screenshot/errors fallback, non-trigger boundaries - helium-browser-use: Helium-mention trigger, jq-filtered /json/list (raw listing leaked extension targets), read-only existing-tab rules, profiling guardrails, stale-tunnel recovery in remote.md - niri-computer-use: $ACU_DIR pattern (relative paths broke from repo cwds), acu-state JSON shape + null focus handling, focus guard, by-window screenshots, interruption-risk tiers, troubleshooting section - learn: propose-only mode for audit sessions, CLAUDE.md conflict handling, duplicate-check across levels, tighter exclusions - learn-skill: context-visibility guard (forked runs lost the session), shared agents/skills placement, harness-neutral skill-creator handoff, Evidence field - playground: mandatory template read, xdg-open on Linux, smoke-test checklist, innerHTML escaping warning - review-swarm: sandbox-level read-only enforcement + post-review dirty check (a "read-only" reviewer committed once), reviewer-shape adaptations, untracked-file scoping, REVIEW.md discovery, Codex spawn fallback, fix/re-review handoff - btca-local: fix unclosed workflow tag, repo-reference normalization, vendored-checkout rule, commit-pinned citations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- frontend-design: reword system-font ban so it no longer contradicts the local/system-font exception in the Typography bullet - tmux: preflight all three helper wrappers, not just send-to-pane Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
4.5 months of cross-harness transcript data (2,317 main sessions): playground had 2 real uses, deslop 1, learn-skill 3. deslop overlaps the harness-native /simplify and /code-review --fix; learn-skill is superseded by the tracer-digest pipeline and the audit playbook. Rewrote the skills README to match the actual skill set (it described a long-removed one). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
48-session Spark review batch + deterministic metrics over the refreshed dataset (through 2026-07-12), findings segmented by model + harness: - handoff: proactive phase-boundary trigger — the single most repeated recommendation across both harnesses for long sessions - commit: confirm ambiguous commit outcomes via git log -1 before retrying (duplicate/empty-commit retries observed on codex) - agent-browser: snapshot at milestones, one verification pass per action batch (snapshot-polling loops were the top browser churn) - tmux: never resend a command to a pane without capture-pane confirming the prior attempt's outcome - codex AGENTS.md: skill routing table — codex had no pointer to the shared skills, explaining systematic review-swarm/pair-programmer misses in review-shaped codex sessions Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tag every digest item with model + harness (behavior differs materially across them; never generalize from one), and add a skill-friction section plus explicit high-signal anti-pattern shapes (repeated commands, writes without checks, repeated corrections) from the usage-analysis findings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codifies the audit procedure (source refresh, deterministic rebuild, invocation indexing, spark review batch, model+harness-segmented synthesis, user-approved cuts) so future rounds are a defined process instead of ad-hoc handoffs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
An audit of all 14 skills in
agents/skills/against real usage: every session transcript in the local and sietch tracer archives (Claude Code + Codex, ~620 indexed invocation hits) was mined per skill — did it trigger when it should, did the agent follow it, which commands failed, and where users had to correct course. The full per-skill reports (with transcript paths and verbatim evidence) are saved locally at~/.local/share/skill-audit/2026-07-02/— kept out of this public repo intentionally.Headline findings
find/rgfile listings mentioning a SKILL.md path, not actual skill reads. The per-skill reports flag which sessions were real.scripts/...from repo roots, playground's macOS-onlyopen, stale helper flags in tmux, a staleAGENT_BROWSER_EXECUTABLE_PATHbreaking launches.docs/handoffs/artifacts), helium's create-your-own-tab rule (read-only inspection of user tabs is often the point), review-swarm's fixed four-reviewer shape.git statuscheck), and binary fixtures (.gif,.h5) were committed despite the commit skill's binary ban (now backed by a concrete--numstatcheck)./learn-skillfork saw only its own invocation and produced nothing; it now verifies context visibility before proposing.Per-skill changes
git -C), pre-staged changes treated as user-owned, concrete binary/size check, generated-churn exclusions, allowed-tools dedup + read-only additions$1; keep-list; audit-only mode; dropped commit/reset from allowed-tools--helpchecks--fullsize warning; shared-browser discipline; screenshot/errors fallback when snapshots lie; non-trigger boundaries/json/list(raw listing printed extension internals); read-only existing-tab rules; profiling guardrails; stale-tunnel recovery in remote.md$ACU_DIRabsolute-path pattern;acu-state --jsonshape + nullfocused-windowhandling; focus guard with restore; by-window screenshots; interruption-risk tiers; troubleshooting; survey.md marked as stale-by-defaultagents/skills/placement; harness-neutral skill-creator handoff; Evidence field per proposalxdg-openon Linux; smoke-test checklist;innerHTMLescaping warning<workflow>tag; repo-reference normalization; vendored/local-checkout rule; commit-pinned citationsNot changed
🤖 Generated with Claude Code
Summary by CodeRabbit
Round 2 (2026-07-12)
Rebased onto current
nixos(round-1 niri-computer-use edits dropped — the upstreamacurework absorbed them) and re-ran the audit against a rebuilt dataset: 4,697 sessions through 2026-07-12, with every finding segmented by model + harness (behavior differs materially between gpt-5.x versions and between Opus/Fable), plus a 48-session LLM review batch.Cuts (usage over 4.5 months, approved):
playground(2 sessions),deslop(1 — overlaps harness-native /simplify),learn-skill(3 — superseded by tracer-digest + the new audit playbook). Skills README rewritten to match reality.Revisions (each tied to an observed failure mode):
handoff— proactive phase-boundary trigger; the most repeated recommendation across both harnesses for long sessionscommit— confirm ambiguous commit outcomes (git log -1) before retrying; duplicate-retry ambiguity observedagent-browser— snapshot at milestones, one verification pass per action batch; snapshot-polling loops were the top browser churntmux— never resend a pane command withoutcapture-paneconfirming the prior attemptAGENTS.md— skill routing table; codex had no pointer to shared skills, explaining systematic review-swarm/pair-programmer misses in review-shaped codex sessionsNew:
skill-auditplaybook skill codifying the audit procedure;tracer-digestprompt now tags findings by model/harness and extracts skill friction.Deferred: stop hooks → #9. Segmented evidence shows repeated-command loops are codex-only (28% of gpt-5.5 sessions, 0% on gpt-5.6) and missing-verification is claude-only and fading (18% opus → 8% fable) — neither clears the bar on current models.