test: layered test strategy — unit, command, container

[joshiste]

Establishes the three-level test structure discussed after #35, and fills the two gaps that had no coverage at all: the interactive prompts, and the process contract.

Level	Tool	Covers
Unit	vitest	A single function or class, no I/O
Command	vitest + msw + @inquirer/testing	A command end to end in process, including its prompts
Container	e2e/run.sh + expect	Only what needs a real process

A test goes at the lowest level that can hold it. CONTRIBUTING.md documents this.

Why the prompts had no coverage

Under vitest process.stdout.isTTY is undefined, so confirm() returned its non-interactive default and never even imported the prompt. config profile add and select were untested in any form, and cancellation was checked by throwing a hand-made ExitPromptError rather than by cancelling anything.

@inquirer/testing drives the real prompts through a shared screen, so these tests go through the same code a user does — the question sequence, the offered default, the validators as wired, and the profile the flow writes.

The existing non-interactive tests are kept, not converted. The three confirm() call sites pass three deliberately different values for defaultWhenNonInteractive, and those decide what happens in CI, where most runs of this CLI happen. Converting them would have swapped that coverage rather than added to it.

Reaching a prompt requires faking isTTY, since confirm() checks it before deciding to prompt at all. Whether that check reads a real terminal correctly cannot be answered by a test that disables it, so it is left to the container level.

Also now covered: the "an experiment is already running, run it in parallel?" recovery, driven through the prompt in both directions. That path was dead code until #35 stopped the response body being consumed before it got there, and has been reachable but unexercised since.

Why a container level as well

Four things no in-process test can reach: the exit status a pipeline reads, a real terminal, the spawn of a subcommand, and the artifact users install. abortExecution deliberately returns instead of exiting under NODE_ENV=test — that is what makes command-level tests possible, and equally why they can never assert an exit code.

Eleven checks, run against the built image so the npm tarball, bin entry, shebang and runtime package.json lookup are exercised on the way:

  ok    --version succeeds
  ok    --help succeeds
  ok    a subcommand is spawned and runs
  ok    an unknown command fails
  ok    a missing access token fails
  ok    an unreachable platform fails
  ok    output is clean when piped
  ok    output is coloured on a terminal
  ok    profile add stores what was typed
  ok    ctrl-c during a prompt exits 130
  ok    ctrl-c leaves no stack trace and no profile

They stay at "did it exit correctly" on purpose — assertions about content belong at the command level, where a failure points at a line rather than a terminal transcript.

Checked that the suite can fail: deleting the binary turns all eleven red. It did not start that way. Two of them asserted only absences — no escape codes, no stack trace, no profile — and every one of those also holds for a CLI that was never invoked. The first fix was still too weak, because 2>&1 catches the shell's own "not found" and made the output look present; they now assert on text the CLI itself prints and on a marker proving the prompt was reached.

The harness caught a real problem on its first CI run

The first push failed two checks that passed locally. They were genuine — for 4.3.6, which is what CI was testing.

docker/setup-buildx-action creates a docker-container builder, which cannot read the local image store. So FROM steadybit/cli:latest in the derived test image resolved from Docker Hub rather than from the image built seconds earlier in the same job. Building the harness against the published image reproduces CI's output exactly, same two checks in the same order — and both are new here, so the released CLI legitimately fails them.

The red build was not the dangerous part. After the next release the tag would have matched again and the suite would have gone green while still exercising a stale artifact.

Fixed with two independent guards:

• No derived image. e2e/ is mounted into the image that was built, so nothing resolves a tag and there is no opportunity to resolve the wrong one. Local and CI run an identical command, so they cannot drift.
• CI tags its build by commit, not latest. A tag that cannot exist in a registry fails loudly instead of quietly finding something else.

Known trade-off: expect is installed at test time

With no derived image, the pty allocator the interactive checks need is installed when the suite runs:

(1/3) Installing tzdata (2026c-r0)
(2/3) Installing tcl (8.6.17-r1)
(3/3) Installing expect (5.45.4-r5)

Three packages from https://dl-cdn.alpinelinux.org — signature-verified by apk, but an unpinned third party fetched on every run. Cost is ~1.1s of a ~7.4s suite, so this is about the dependency, not the time.

What it means in practice:

• An Alpine CDN outage reds an unrelated PR. It fails loudly (cannot install expect, which the interactive checks need) rather than silently skipping the interactive checks, which is the right failure mode but still a failure.
• A network-restricted runner cannot run the suite at all. Verified: --network none fails at this step.
• The installed version drifts over time. Immaterial for spawn/expect/send, but it is an uncontrolled input.

The alternative, if this becomes annoying: restore the derived image and build it with the default builder rather than buildx's container driver, which does read the local image store. Verified both halves:

FROM steadybit/cli:ci-<sha>        → builds, resolves the LOCAL image
FROM steadybit/cli:ci-<unknown>    → "failed to resolve"

So the commit-tag guard alone is enough to make the original bug loud, and expect would move back to build time — where the job already requires network to pull node:24-alpine and run npm ci, so it would add no new class of dependency. I went with mounting because one mechanism with no tag resolution at all is harder to get wrong, but that reasoning is worth revisiting rather than treating as settled.

Prerequisite included

service.ts resolved the config directory at module load, so nothing could point the CLI at a different home once it was imported; the existing test worked around it with a top-level-await import. That dance is gone, which is what made command-level tests of the profile flows possible.

Writing the test for it surfaced a second problem: the directory-creation memo was a single flag, so a changed HOME left the new directory unmade while every write into it failed with ENOENT. Reviewing that fix surfaced a third: the directory memo followed HOME but the read memo did not, so a changed home produced correct directories holding the previous home's contents. All three — directory, profiles, active profile — now go through one helper keyed on the config directory, which is less code than the two mechanisms it replaces.

Two error messages also named a function instead of the path they reported, and the active-profile read named the profiles file rather than its own.

Verification

106 tests, up from 88. npm run ci green at each of the five commits so the history bisects cleanly, and the container suite passes both piped and with -t, on arm64 and amd64.