Skip to content

Report issues and read the fleet from the CLI (0.6.0) - #14

Merged
JOsacky merged 7 commits into
mainfrom
claude/cloud-cli-report
Sep 29, 2026
Merged

JOsacky merged 7 commits into
mainfrom
claude/cloud-cli-report

Conversation

@JOsacky

@JOsacky JOsacky commented Sep 29, 2026

Copy link
Copy Markdown
Member

Two things a person on a machine could not do from the terminal: tell the Skillhook team about a problem, and see the rest of the fleet. Released as 0.6.0.

skillhook cloud report: problem reports from a paired machine

skillhook cloud report "GitHub deliveries fail since the update" --body-file notes.txt --job 20260929T101500Z-a1b2c3 --email ada@example.com
skillhook cloud report "Where do hosted URLs come from?" --kind question --no-diagnostics
skillhook cloud report "Replays hang" --body - --dry-run < notes.txt    # the exact JSON, nothing sent
  • One request, POST {cloud.url}/api/agent/issues with the machine token, only when a person runs it. It prints Reported as #N: <url> and whether a confirmation email went out; --json prints the cloud's answer.
  • It carries the person's title and body (--body TEXT, --body -, --body-file), --kind, --severity, --email, and references (--job, --delivery, --skill). Unless --no-diagnostics, it adds what the machine already knows: skillhook, Node, OS and architecture, cloud.mode, the link's state, {runner, ready} per runner, and the health summary with the failing and warning checks. These come from the running server's cached answers, or from a quick local check (no network probes) when no server runs. A server answer of another shape falls back to the basics rather than failing the report.
  • Every .env value is scrubbed from the title, body, references and diagnostics before anything leaves, as for the link. A contact address that is itself a .env value is refused. Payloads, logs, prompts and job output never go. docs/cloud.md lists exactly what is sent.
  • Each report carries a report_id (a UUID), the same on every attempt. Network errors, timeouts and 5xx are retried (three attempts); a 429 is retried only after a retry_after_ms of at most 15 s; any other 4xx is never retried. The cloud files a retried report once. The client reads the answer leniently, so a field a newer cloud adds cannot turn a filed report into a failure and a second report.
  • Needs a paired machine (cloud.enabled and SKILLHOOK_CLOUD_TOKEN); otherwise it points to skillhook cloud connect or the dashboard / hosted MCP. Refused under SKILLHOOK_NO_CLOUD=1 and to an http URL. A refused token, a disabled machine, a rate limit and a cloud without the route (an HTML 404) each get a plain message.
  • The operation is shared (src/cloud/report.ts). The MCP tool cloud_report_issue sends the same report, with dry_run to show the person first.
  • --body stays a switch for deliveries show <id> --body and takes its text only after cloud report. skillhook cloud <subcommand> --help now prints the usage instead of running the subcommand (a report must not go out on --help).

Reading the fleet with an organisation API key

pbpaste | skillhook cloud login --key -    # checked with GET /api/v1/me, kept in .env, never printed
skillhook cloud machines
skillhook cloud jobs --waiting --machine mac-mini
skillhook cloud job 20260929T101500Z-a1b2c3
skillhook cloud logout
  • login --key shc_…|- stores SKILLHOOK_CLOUD_API_KEY in .env (mode 600). The environment variable works too (CI) and wins. logout removes the key from .env.
  • machines (name, status, mode, version, last seen), jobs [--machine M] [--skill S] [--status ST] [--outcome O] [--waiting] [--limit N] [--before C] and job <id> (status, outcome, the pending question, the result excerpt) read /api/v1. Tables by default; --json prints the API's JSON. Answers are parsed with loose schemas, since the cloud may add fields.
  • The machine token is never used for these reads, so a paired machine cannot read the rest of its organisation.
  • A key must look like one (shc_ then letters, digits, -, _), so a pasted second line or the machine token is refused before anything is sent.
  • A key never goes to the built-in placeholder URL: until pairing, cloud.url or SKILLHOOK_CLOUD_URL names a cloud, the reads refuse. Requests are HTTPS only and are refused under SKILLHOOK_NO_CLOUD=1.
  • src/cloud/http.ts now reads RFC 9457 problems (code, detail, request_id) as well as the agent API's error / message. A 401 says to run skillhook cloud login; a 403 names the missing fleet:read scope. A token that is not one line of printable characters is refused before it becomes a header (the runtime quotes a refused header in its error), and a token never appears in a transport error message.

The protocol contract (additive, PROTOCOL_VERSION stays 1)

export const ISSUE_KINDS = ["bug", "question", "feature", "other"] as const;
export const ISSUE_SEVERITIES = ["low", "normal", "high", "urgent"] as const;
// LIMITS.max_issue_report_bytes: 64 * 1024 (the request body)

export const IssueDiagnosticsSchema = z.object({
  skillhook_version: z.string().max(64).optional(),
  node_version: z.string().max(64).optional(),
  os: z.string().max(64).optional(),
  arch: z.string().max(32).optional(),
  mode: z.enum(MACHINE_MODES).optional(),
  link: z.object({ state: z.enum(LINK_STATES), reason: z.enum(LINK_REASONS).optional(), last_error: z.string().max(500).optional() }).loose().optional(),
  runners: z.array(z.object({ runner, ready: z.boolean() }).loose()).max(3).optional(),
  health: z.object({ ok: z.boolean(), summary, failing: z.array(z.object({ id: z.string().max(200), status: z.enum(CHECK_STATUSES), message: z.string().max(500).optional() }).loose()).max(50).optional() }).loose().optional(),
}).loose();

export const IssueReportRequestSchema = z.object({
  title: z.string().min(1).max(200),
  body: z.string().max(20_000).optional(),
  kind: z.enum(ISSUE_KINDS).optional(),           // the cloud defaults to "bug"
  severity: z.enum(ISSUE_SEVERITIES).optional(),  // the cloud defaults to "normal"
  contact_email: z.string().email().max(320).optional(),
  job_id: id.optional(),                          // the machine's own job id
  delivery_id: id.optional(),
  skill: name.optional(),
  diagnostics: IssueDiagnosticsSchema.optional(),
  /** A client-generated idempotency key; a retry with the same report_id returns the original report instead of filing a second one. */
  report_id: z.string().min(8).max(100).regex(/^[A-Za-z0-9_-]+$/).optional(),
}).strict();

export const IssueReportResponseSchema = z.object({ ok: z.literal(true), issue_id: id, number: z.number().int().min(1), url: z.string().url().max(2048), acknowledged: z.boolean() }).strict();

POST /api/agent/issues, Authorization: Bearer <SKILLHOOK_CLOUD_TOKEN>, JSON body ≤ 64 KiB. A 200 answer is an IssueReportResponse, the same for a new report and for a replay of the same report_id; acknowledged says whether a confirmation email went out. Errors use the agent API's shape {ok: false, error, message, retry_after_ms?}: 401 invalid_token, 403 machine_disabled, 400 invalid_request, 413 payload_too_large, 429 rate_limited, 500 server_error. Documented in docs/cloud-protocol.md#issue-reports.

Rules and docs

  • AGENTS.md's "Outbound requests are opt-in and enumerated" rule now lists the person-invoked cloud commands: one request each, only when run, only to the machine's cloud URL over HTTPS. cloud report and the API-key commands refuse under SKILLHOOK_NO_CLOUD=1.
  • Updated: docs/cloud.md (what cloud report sends, what the API-key commands read), docs/cloud-protocol.md, docs/mcp.md, docs/security.md, docs/operations.md, the README command table, llms.txt, the setup skill and CHANGELOG.md.
  • CodeQL: no new regexes with ambiguous repetition on untrusted text, flag lookups use a Map, and nothing writes user-supplied keys.

Landing this

Merging bumps package.json to 0.6.0, so the Release workflow tags v0.6.0 and Publish ships @meterapp/skillhook@0.6.0. The cloud (MeterApp/skillhook-cloud, which vendors 0.5.0 today) needs 0.6.0 for IssueReportRequestSchema. Until its /api/agent/issues route is deployed, cloud report says the cloud does not take reports yet.

Tests

  • npm run check: 43 files, 279 tests, build, schemas and release metadata (0.6.0).
  • New tests cover:
    • the schemas (valid, invalid, limits, report_id);
    • the report with and without diagnostics, both from a running server (src/test-support/fake-server.ts) and from a local check with the fake CLIs;
    • scrubbing of .env values in the title, body, references and diagnostics;
    • the not-paired, kill-switch, https and size refusals;
    • every error the cloud can answer, and which of them are retried;
    • the retry and idempotency path, including an answer lost after the report was filed;
    • the HTTP client's problem parsing and credential redaction;
    • login keeping the key without printing it (flag and stdin), a two-line key, and no key sent without a cloud URL;
    • 401 and 403 messages, table output and --json, the machine token never reaching /api/v1, and --help;
    • the MCP tool, including dry_run.
  • The fake cloud serves /api/agent/issues (once per report_id, with scripted failures) and the /api/v1 reads for one key.
  • An independent review pass over the diff found issues (the placeholder URL, credentials in fetch errors, --help, --diagnostics false, odd server answers, unscrubbed references, strict answer parsing); all are fixed here, with tests.

🤖 Generated with Claude Code

JOsacky and others added 7 commits September 29, 2026 11:11
The contract for POST /api/agent/issues, which the cloud side implements and
`skillhook cloud report` will use: IssueReportRequestSchema (title, body,
kind, severity, contact email, the job, delivery or skill it is about, and
diagnostics), IssueDiagnosticsSchema (versions, platform, cloud mode, link
state, runner readiness, the health summary with the failing checks; loose,
bounded like MachineInfo) and IssueReportResponseSchema (issue id, number,
URL, whether a confirmation email went out), with ISSUE_KINDS,
ISSUE_SEVERITIES and LIMITS.max_issue_report_bytes (64 KiB). Additive:
PROTOCOL_VERSION stays 1. Errors keep the agent API's shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`skillhook cloud report "<title>"` (and the MCP tool cloud_report_issue) sends
the Skillhook team a report through POST /api/agent/issues with the machine
token: the person's title and body (--body TEXT, --body - or --body-file),
kind, severity, a contact address, the job, delivery or skill it is about,
and unless --no-diagnostics what the machine already knows (versions,
platform, cloud mode, the link's state, runner readiness, the health summary
with the failing and warning checks), from the running server's cached
answers or a quick local check. Every .env value is scrubbed from all of it
before it leaves, as for the link; payloads, logs, prompts and job output
never go. --dry-run prints the exact JSON. It needs a paired machine and
refuses under SKILLHOOK_NO_CLOUD=1 or to an http URL; what the cloud answers
(a refused token, a disabled machine, a rate limit, a cloud without the
route) is said plainly. The shared operation lives in src/cloud/report.ts.

`--body` is a switch for `deliveries show` and takes its text only after
`cloud report`. The fake cloud takes reports; src/test-support/fake-server.ts
stands in for a running server. AGENTS.md's outbound rule lists the command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`skillhook cloud login --key shc_…|-` checks an organisation API key with
GET /api/v1/me and keeps it in .env as SKILLHOOK_CLOUD_API_KEY (mode 600,
never printed; the environment variable wins, for CI); `cloud logout`
forgets it. `cloud machines`, `cloud jobs [--machine M] [--skill S]
[--status ST] [--outcome O] [--waiting] [--limit N] [--before C]` and
`cloud job <id>` read /api/v1 with it and print tables, or the API's JSON
with --json, parsed with loose schemas since the cloud may say more. The
machine token is never used for them, so pairing gives a machine no view of
the rest of its organisation; keys that do not start with shc_ are refused.
Requests go only to the machine's cloud URL over HTTPS, only when run, and
not under SKILLHOOK_NO_CLOUD=1.

The HTTP client reads the public API's RFC 9457 problems (code, detail,
request_id) as well as the agent API's error and message; a refused key says
to log in again and a missing scope names it. The fake cloud serves the
/api/v1 reads for one key. Docs list exactly what each command reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cloud_report_issue takes dry_run, like `cloud report --dry-run`: it returns
exactly what would be sent (scrubbed, with the diagnostics) and sends
nothing, so an agent can show the person the report before it goes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The machine list is read only for the tables, and the filters are the only
thing that goes with a read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The report contract gains `report_id`, a client-generated idempotency key
(8 to 100 of A-Z a-z 0-9 _ -): a retry with the same id returns the original
report instead of filing a second one, and the answer is the same either
way. `cloud report` generates one per report and sends it with every
attempt; network errors, timeouts and 5xx are retried (three attempts), a
429 only after a retry_after_ms of at most 15 seconds, another 4xx never.
The fake cloud files a report once per id and can drop a connection or lose
an answer after filing, which the tests use.

From review:
- A token or key that is not one line of printable characters is refused
  before it becomes a header (the runtime quotes a refused header in its
  error), and a token never appears in a transport error message.
- API keys must look like one (shc_ and then letters, digits, - and _), and
  never go to the built-in placeholder cloud URL: until pairing,
  `cloud.url` or SKILLHOOK_CLOUD_URL names a cloud, the reads refuse.
- `skillhook cloud <subcommand> --help` prints the usage and does nothing
  else; `--diagnostics false` (0, no, off) turns diagnostics off.
- The references and the contact address are scrubbed like the rest (a
  contact address that is a value from .env is refused); diagnostics from a
  server answer of another shape fall back to the basics instead of
  failing the report; the client reads the cloud's answer leniently, so a
  field a newer cloud adds does not turn a filed report into a failure; a
  404 is "the cloud does not take reports yet" only when it is not the
  cloud's own answer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`skillhook cloud report` and the MCP tool cloud_report_issue: problem reports
to the Skillhook team from a paired machine, with scrubbed diagnostics and an
idempotent report_id; the issue-report schemas in
@meterapp/skillhook/protocol; and `cloud login|logout|machines|jobs|job`,
the organisation's fleet read with an organisation API key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JOsacky
JOsacky merged commit 9159e85 into main Sep 29, 2026
6 checks passed
@JOsacky
JOsacky deleted the claude/cloud-cli-report branch September 29, 2026 15:40
JOsacky added a commit that referenced this pull request Sep 29, 2026
A safety fix: most subcommands ignored `--help` / `-h` and did their
work anyway.

```bash
skillhook jobs prune --help         # pruned jobs
skillhook service install --help    # installed (and started) the launchd/systemd service
skillhook config set port 1 --help  # wrote skillhook.json
skillhook link --help               # linked the current directory
skillhook serve --help              # started the server
```

0.6.0 (#14) fixed this for `skillhook cloud <subcommand> --help` only,
inside `cloud.ts`.

## The fix: one check, before any command runs

- `COMMANDS` in `src/commands/main.ts` now pairs every command with its
usage (`{ run, usage }`), and `main()` checks `--help` / `-h` right
after resolving the command, before any command code runs. It prints the
usage and exits 0. A new command cannot be registered without a usage,
and cannot forget the check.
- `skillhook help <command> [subcommand]` does the same thing
(`skillhook help` alone still prints the overview).
- `--json` prints `{ "ok": true, "command": "jobs", "usage": "…" }`.
- The per-command checks in `cloud.ts` and `update.ts` are gone, because
the central one covers them.
- `serve`, `doctor`, `health`, `runners`, `mcp` and `url` had no usage
text and now have one. `init`'s usage now lists `--runner shell`, which
the command already accepted.
- `skillhook job <subcommand> --help` follows `jobCommand`'s dispatch.
The agent subcommands (`progress|ask|outcome|note|context`, or no
subcommand) print the job API usage. Everything else (`job list`, `job
prune`, …) is really `jobs …`, so it prints the `jobs` usage.
- Each command's private `USAGE` became an exported `<NAME>_USAGE`, as
`INIT_USAGE` and `UPDATE_USAGE` already were. Most of the diff in
`src/commands/*.ts` is this mechanical rename.

## The test (`src/cli.test.ts`)

It runs every entry of `COMMANDS` and every subcommand in three forms:
`… --help`, `… -h --json` and `help …`. That is about 400 invocations,
against a `tempHome()` set up so that each line has something to act on:
a job to prune or answer, a delivery to replay, secrets and config to
change, a linked project, a scheduled skill and a stored API key. Every
invocation must:

- exit 0 and print exactly the command's usage (or the JSON above), with
nothing on stderr;
- leave the home unchanged (a snapshot of every entry with its mtime and
content, compared after each invocation);
- never read stdin.

The table must also cover every subcommand the usage texts document
(`skillhook jobs prune …`, `skillhook projects add|remove …`), and every
`COMMANDS` key (aliases reuse their command's lines). As a control, the
same `jobs prune` line without `--help` does prune, and the snapshot
shows it.

It is hermetic even if the fix regresses. The env sets
`SKILLHOOK_NO_UPDATE_CHECK=1`, unreachable registry and cloud URLs,
`SKILLHOOK_NO_CLOUD=1` and fake claude/codex. `installService`,
`uninstallService`, `restartService`, `enableExposure` and
`disableExposure` are stubbed with `vi.mock` for this file, so a broken
check can never touch the real launchd/systemd service or Tailscale
Funnel. The test asserts none of them was called.

I checked that the test catches the bug. With the check skipped for
`jobs`, it fails on `expected 'Removed 1 old job(s)' to contain
'skillhook jobs prune [--keep N]'`. With a stray file write added to the
help path, the snapshot names the line and shows the new file. With a
subcommand missing from the table, it fails on `skillhook jobs prune
--help: expected [...] to include 'prune'`. The whole test takes about
0.3 s.

`npm run check` passes: typecheck, 280 tests, build, schema and release
metadata. CHANGELOG has an `## Unreleased` entry. README (global
options) and AGENTS.md (registering a command) mention the new rule.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant