npm bin/copilot is a loader, not a version pin: same path served 1.0.77 then 1.0.78 101s apart; --prefer-version works but is undocumented

[ErikPlachta]

Describe the bug

The globally-installed npm shim — $(npm prefix -g)/bin/copilot — is a loader, not a version pin. Two invocations of the same path, 101 seconds apart with nothing touched in between, ran two different CLI versions (1.0.77, then 1.0.78), while the installed npm packages continued to declare 1.0.77. Nothing an offline check can inspect changed.

This is fine for interactive use, and I'm not asking for auto-update to stop being the default. The problem is for reproducible evals and benchmarks: pointing a harness at that path is the natural way to "pin the CLI," and it silently isn't one. A run can straddle two CLI versions, and every result stamped "pinned 1.0.77" becomes unverifiable after the fact.

There is a mechanism that does pin — --prefer-version <version> — but it is not in --help and I only found it by reading the loader. Meanwhile the two knobs that are documented (--no-auto-update, COPILOT_AUTO_UPDATE=false) do not freeze the version to the installed one (details below).

The good part, stated up front: the CLI already emits copilotVersion on session.start. That is the only reason we caught this — our harness compares observed-vs-pinned per trial and quarantined the run after one request instead of twelve. A harness that checks copilot --version once at startup would not have noticed. Please keep that field.

Affected version

1.0.77 → 1.0.78 (the swap itself). macOS 15 / arm64, node v24.18.1, npm global-prefix install.

Steps to reproduce the behavior

Observed live, then re-confirmed today from the same machine.

1. The original observation (2026-08-05, macOS/arm64). With COPILOT_CLI_PATH="$(npm prefix -g)/bin/copilot", copilot --version read 1.0.77, and a session started immediately after recorded:

{"type":"session.start","data":{"sessionId":"06959cca-…","copilotVersion":"1.0.77",
 "startTime":"2026-08-05T01:39:53.221Z", …}}

101 seconds later, the same path, same shell, nothing installed or changed in between:

2026-08-05T01:41:34.180Z [INFO] Starting Copilot CLI: 1.0.78
2026-08-05T01:41:34.784Z [INFO] No update needed, current version is 1.0.78, fetched latest release is v1.0.78
2026-08-05T01:41:43.170Z [DEBUG] … User-agent: copilot/1.0.78 (client/github/cli darwin v24.18.1) …

A 1.0.78/ directory appeared in ~/Library/Caches/copilot/pkg/darwin-arm64/ at 01:39:56Z — three seconds after that session.start, i.e. during the first session. Throughout, both @github/copilot/package.json and @github/copilot-darwin-arm64/package.json still read 1.0.77.

2. Current state on the same machine — the installed npm packages still declare 1.0.77, and the pkg cache holds three builds side by side:

$ node -p "require('.../node_modules/@github/copilot/package.json').version"
1.0.77
$ node -p "require('.../@github/copilot/node_modules/@github/copilot-darwin-arm64/package.json').version"
1.0.77
$ ls ~/Library/Caches/copilot/pkg/darwin-arm64/
1.0.64-0  1.0.77  1.0.78

3. What each knob actually does (all four run just now, back to back, same shell; the pkg cache was unchanged before and after):

$ copilot --version
GitHub Copilot CLI 1.0.78.          # installed package says 1.0.77

$ copilot --no-auto-update --version
GitHub Copilot CLI 1.0.78.          # documented flag — does NOT pin to the installed build

$ COPILOT_AUTO_UPDATE=false copilot --version
GitHub Copilot CLI 1.0.78.          # documented env var — same

$ CI=true copilot --version
GitHub Copilot CLI 1.0.78.          # CI default (auto-update off) — same

$ copilot --prefer-version 1.0.77 --version
GitHub Copilot CLI 1.0.77.          # this one works — and is not in --help

--help lists --no-auto-update ("Disable downloading CLI update automatically…"). It does not mention --prefer-version.

Expected behavior

Three things, in priority order. None of them is "stop auto-updating."

1. Document that the npm bin/copilot path does not pin a version. It resolves to the newest build in the shared pkg cache, which can change between two consecutive invocations. Anyone wiring a harness, benchmark, or eval to that path is reasonably assuming otherwise.

2. Document --prefer-version <version> as the supported pin (or provide a supported env-var equivalent for tools that build an argv they don't fully control). It already does exactly the right thing — it's just unfindable. This is the smallest possible fix and it closes most of the gap.

3. Clarify or fix --no-auto-update / COPILOT_AUTO_UPDATE=false. Reading the loader source, --no-auto-update, --prefer-version, and COPILOT_AUTO_UPDATE=false all clear the same "auto-update enabled" predicate that guards the redirect-to-newest-cached-build branch, so I expected --no-auto-update to run the installed 1.0.77. It ran 1.0.78. I can't tell from outside whether that's a bug or whether the flag is scoped strictly to "don't download" while the cache redirect is separate — the help text is consistent with the narrower reading. Either way, a user who reaches for --no-auto-update to stabilize a version will be misled, so it's worth stating explicitly which it is.

Keep emitting copilotVersion on session.start — per-session observability is what makes this detectable after the fact, and it's the difference between a quarantined run and twelve silently-invalid results.

Additional context

• OS / arch: macOS 15, arm64 (Apple Silicon). Node: v24.18.1 (harness) / v22.22.3 (shell). Shell: zsh. Install: npm install -g @github/copilot into a non-default global prefix.
• Everything above is what was observed on this one machine with these two versions. I haven't tried to reproduce on Linux or Windows, and I'm not claiming anything about the update trigger's timing beyond the timestamps shown.
• Scope note for fairness: this only bites non-interactive/automated use. For an interactive terminal session, auto-updating to the newest cached build is a sensible default.
• Detection cost us 1 premium request instead of 12, entirely because of the session.start.copilotVersion field.