[matthewgrossman]

Summary

A worked example of a test that needs a specific deployment feature, Keycloak. tmachine declares an environment that runs Keycloak and publishes how to reach it as OPENSHELL_TEST_* variables. The runner passes those variables to the tests, and a suite-local fixture reads them. The Keycloak provider-refresh test is ported to testkit on top of this.

Second in a linear stack: on top of #4773 (the runner), and #4774 (the conformance suite port) goes on top of this. Draft for discussion.

Related Issue

Part of #3954

How it fits together

1. tmachine declares the environment. ubuntu-docker-rootful-keycloak in tests/tmachine/config.nix runs the usual Docker setup plus ansible/playbooks/keycloak.yaml. That playbook starts Keycloak with --restart unless-stopped and writes /etc/openshell-test/env.d/keycloak.env. Setup is cached as a disk image, so the Keycloak image is cached too, and the container comes back on every boot.
2. The runner passes the variables through. After boot, test_env.rs reads every /etc/openshell-test/env.d/*.env file and rejects names outside the OPENSHELL_TEST_ prefix (errors never print values). guest_env adds the variables to the testkit-ready and guest nextest commands. The runner has no idea what Keycloak is.
3. The test helper reads them. KeycloakContext::new(cx).await reads OPENSHELL_TEST_KEYCLOAK_{ISSUER,CLIENT_ID,USERNAME,PASSWORD} and waits for the realm to answer. Set the same variables to run the suite against your own deployment.

Run it locally:

nix run .#openshell-test-runner -- \
  --environment ubuntu-docker-rootful-keycloak --installer deb \
  --family features --package keycloak

Combinations such as Podman + Keycloak are more tmachine environments. They can be generated as a cross product later without changing the runner or the tests.

Changes

• testkit: Cli::run_with_env and Context::cli_with_env, so secrets reach the CLI through env vars rather than argv.
• tmachine: the ubuntu-docker-rootful-keycloak environment. Environment setup can now take inputs, used for the realm file; environments without inputs keep their cache keys. The keycloak role adds a restart policy.
• Runner: test_env.rs, and guest_env takes the published variables.
• Keycloak fixture (tests/suites/features/keycloak): KeycloakContext derefs to testkit::Context and adds password_grant(), revoke() and token_endpoint(). A missing variable fails the test.
• Ported test: provider_refresh.rs, with the same CLI calls and checks as the old one and run-unique provider and profile names.
• Lockfile: tests/suites/features/Cargo.lock is aligned with the root lock and added to the alignment check.
• CI: an E2E Suites matrix row for features/keycloak on the new environment; the tmachine disk cache key includes the realm file.
• Docs: an "Environments with services" paragraph in TESTING.md.

The old tests/suites/features/provider-refresh/keycloak crate and its tmachine testsuite are unchanged.

Open questions

• Env vars versus a config file the fixture reads directly, and the exact variable names. For now tests read env vars only; letting testkit read env.d itself is deferred.
• The variables, including the throwaway realm password, reach the guest as env arguments over SSH.
• Provider and profile cleanup runs whether the test passes or fails, but a panic or a killed process skips it.

Testing

• Runner, tmachine, testkit and features workspace unit tests and clippy pass
• Lockfile alignment check and its tests pass
• cargo nextest list -p keycloak lists keycloak::provider_refresh (host target)
• The Keycloak environment ran green in a tmachine VM in Branch E2E on df2bcbd (run), before the rebase onto the current feat(tests): add the tmachine Session API and run a conformance smoke test on tmachine #4773, which changed only lockfiles. That run's one failure was the existing Kubernetes Rust smoke job, which this PR doesn't touch.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

Label test:e2e applied, but pull-request/4891 does not exist yet. A maintainer needs to comment /ok to test df2bcbd332a27e575dba7413b368b82abbd0fdc3 to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

[matthewgrossman]

/ok to test df2bcbd

[drew]

The provider-refresh port preserves the existing test's assertions. Two concerns are noted inline: service credentials can appear in SSH error context, and the runner's two-minute per-test limit is shorter than the fixture's three-minute readiness allowance.

This branch also needs restacking onto #4773: it currently lacks the latest log-retention and Core E2E gating changes, and GitHub reports merge conflicts.

Validation: runner unit tests, eight tmachine tests, two lockfile-check tests, and alignment checks for all three enrolled workspaces passed. VM E2E suites were not rerun.

[drew]

Service credentials can appear in CI errors. Previously the runner passed only gateway and CLI settings over SSH; these additional variables include service passwords. Session::exec_timeout adds format!("run {argv:?} in tmachine VM") as error context, and the runner prints the full error chain. An SSH timeout or launch failure therefore prints the credential values, despite the parser avoiding them in its own errors. Please redact environment values in that context or pass the service variables through a guest file.

[matthewgrossman]

Good catch: the argv context in exec_timeout bypasses the parser's redaction. Fixed in f644909 by keeping the values in the guest. The runner still checks env.d, but it no longer puts any value in a command. The nextest command now runs through a small sh wrapper that exports each env.d/*.env line literally (IFS= read -r + export "$line", not source) before execing nextest, so only OPENSHELL_GATEWAY and OPENSHELL_BIN appear in argv. The parser no longer trims, to match the wrapper, and a unit test runs the wrapper against files with spaces, $, backticks and = in values.

[drew]

The runner terminates each test before this readiness allowance expires. Previously Keycloak readiness was checked by Ansible before starting the test process; this fixture now waits inside the test with a three-minute allowance, while tests/runner/src/bundle.rs sets slow-timeout = { period = "30s", terminate-after = 4 }. A slow Keycloak restart can therefore cause the test to be killed after two minutes instead of allowing readiness to finish, and skips provider/profile cleanup. Please move service readiness outside the test budget or increase the applicable nextest timeout.

[matthewgrossman]

Agreed, the budgets conflicted. Fixed in 0f9ad51: READY_TIMEOUT is now 90s, with a comment tying it to the runner's 2-minute per-test budget, so a slow Keycloak fails with a readiness error instead of being killed. The wait runs before the test creates its provider or profile, so a kill there wouldn't leave anything to clean up anyway. Keycloak restarts while the runner waits for the gateway, so in practice it's ready before the first test. If 90s turns out to be flaky, I'll move readiness out of the test.