Skill eval harness + eval-validated skill/CLI improvements: Common Tasks, inline-image discoverability (issue #207)#251
Merged
Merged
Conversation
Issue #207 claims agents reading the skill skip dedicated CLI subcommands and reach for raw GraphQL via `linear api`. Before changing the skill text, this adds an eval that can actually measure that: codex exec runs each task prompt in a fully isolated environment (fresh CODEX_HOME + fake HOME so the globally installed skill can't leak in, recording shims for linear/curl/ npx/npm, workspace-write sandbox) and a deterministic grader classifies route choice and flag correctness from the recorded invocations. Cases: five recipe families with development + holdout prompts, plus two controls where GraphQL is genuinely the right route. Outcome rules were declared before the baseline ran (see evals/linear-cli-skill/README.md). Baseline result, 36 trials at low effort on gpt-5.6-sol: 30/30 supported tasks full success, 6/6 controls on linear api. The premise of #207 did not reproduce in this configuration — with the skill actually read, routing is already perfect. Two earlier baseline runs were voided during harness development because stateless canned outputs (issue view contradicting the subject's own update; ENG-prefixed identifiers for OPS-team requests) baited subjects into GraphQL investigation and contaminated the signal; the shim is now consistency-aware. Related to #207
…ated Adds ~7 copy-pasteable recipes near the top of SKILL.template.md (and the generated SKILL.md): filtered queries via issue query (with the issue list/mine alias gotcha spelled out), my-issues, create with --description-file and --no-interactive, update state/assignee/labels (noting label replacement semantics), comment from file, view/URL. The boundary note stays generic so it doesn't teach the eval's control answers. Post-change eval, same frozen 36-trial protocol as the baseline: 29/30 supported tasks full success, holdout 15/15, controls 6/6 still correctly choosing linear api — no overcorrection from the new recipes. The single failure was a subject first trying the skill's documented npx alternative (npx @schpet/linear-cli issue create ...), which the frozen grader counts as a bypass; it was not a GraphQL fallback and the task then completed correctly via the CLI. With the baseline already at 30/30, the eval finds no measurable routing headroom at this configuration; the change is validated as non-regressing rather than as an improvement. See evals/linear-cli-skill/results/comparison.md. Related to #207
Agents asked to put a visible screenshot on an issue reach for
`issue attach`, which uploads the file but creates a sidebar link
attachment that never renders inline — while the success output
("Attachment created") convinces them the image is visible. The working
path, `issue comment add --attach`, has existed since v2.0.0 but nothing
pointed at it: the skill had no image guidance and the flag was buried in
a reference table.
Three coordinated changes, validated as experiment 2 of the skill eval:
- Skill: a Common Tasks recipe for visible images via
`issue comment add --attach`, with an explicit warning about
`issue attach`'s sidebar-only behavior.
- CLI: `issue attach` now says it created a sidebar link attachment,
and for images prints a copy-pasteable hint suggesting
`issue comment add --attach` (shell-quoted, --public preserved).
Help descriptions updated on both commands.
- Eval: new frozen image family (trap-phrased development prompt,
comment-phrased holdout) plus a sidebar-control case graded on
positionals, with pre-declared outcome rules, CLI/API control split,
binary-safe fixture checks, and version-matched shim output.
Result (rules frozen before baseline): image-development went 0/3 to 3/3
— every baseline trial fell into the attach trap and wrongly reported
success; every post-change trial routed straight to the recipe. Image
family 3/6 to 6/6 lands in the pre-declared partial-baseline band, so it
is reported as consistent with improvement (exploratory Fisher p = 0.09)
rather than confirmed. Controls held except one known npx-version-check
grader artifact, adjudicated by an Opus gold-label pass (17/18 agreement
with the deterministic grader).
schpetbot
force-pushed
the
skill-eval-207
branch
from
July 23, 2026 17:41
40e2658 to
df96f02
Compare
schpetbot
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to #207.
Two eval-validated changes to the skill (and, in the second case, the CLI), sharing one codex-based eval harness.
1. Skill eval harness + Common Tasks section (experiment 1)
Builds an eval that empirically measures the issue's claim — that an agent reading the skill skips dedicated CLI subcommands and reaches for raw GraphQL via
linear api— then uses it to baseline the current skill and validate a "Common Tasks" recipes section.gpt-5.6-sol, low effort)linear apiThe premise of #207 did not reproduce in this setup. With the skill actually read (verified per-trial via the codex event stream), routing was already perfect before any change. Per the outcome rules declared before the baseline ran (
evals/linear-cli-skill/README.md), that is reported as the finding rather than manufacturing harder prompts after the fact. The Common Tasks section is therefore shipped as validated non-regressing — in particular, the new recipes did not overcorrect the controls away from legitimatelinear apiuse — not as a measured improvement.The one post-change failure was a subject first trying the skill's documented npx alternative (
npx --yes @schpet/linear-cli issue create …); the frozen grader counts any npx use as a bypass. It was not a GraphQL fallback, and the trial then completed correctly vialinear issue create.2. Inline-image discoverability fix (experiment 2)
Agents asked to put a visible screenshot on an issue reach for
issue attach, which creates a sidebar link attachment that never renders inline — while its success output convinces them the image is visible. The working path,issue comment add --attach, has existed since v2.0.0 but nothing pointed at it. (Observed in real-world agent use; reproduced by the eval below.)Three coordinated changes:
issue comment add --attach, with an explicit warning aboutissue attach's sidebar-only behavior.issue attachnow reports "Sidebar link attachment created" and, for images, prints a copy-pasteable hint suggestingissue comment add --attach(shell-quoted,--publicpreserved). Help descriptions updated on both commands.imagefamily (trap-phrased development prompt, comment-phrased holdout) plus a sidebar control graded on positional args, with pre-declared outcome rules, run under separate condition names (image-baseline/image-post-change) so experiment 1's committed results stay frozen. Both conditions run all 15 cases, so the old families double as a regression guard.image-development("attach the screenshot … so it is visible")image-holdout("add a comment … visible inline")issue attachEvery baseline
image-developmenttrial fell into the trap — ranissue attach, "verified" viaissue view --json, and wrongly reported success. Every post-change trial routed directly to the recipe (no runtime-hint recovery needed). With baseline image-family success at 3/6, the pre-declared rules place this in the partial-baseline band: reported as consistent with improvement (exploratory one-sided Fisher p = 0.09), not statistically confirmed.* The sidebar-control miss is the same known npx-version-check grader artifact as experiment 1's single failure — the trial routed correctly. All 18 image/control trials were additionally gold-labeled by a Claude Opus subagent blind to the deterministic grades: 17/18 agreement, with the sole disagreement being that npx trial (labeler: goal achieved). A post-experiment grading hardening (verify the target issue id) is disclosed in the findings; regrading the committed records under it changed no counts.
Full findings:
evals/linear-cli-skill/results/image-comparison.md.