Skip to content

Commit 50a3237

Browse files
matthewgrossmanmatthewgrossman
authored
test(features): run feature tests against environment services, starting with Keycloak (#4891)
* feat(testkit): pass extra environment variables to the CLI Add Cli::run_with_env and Context::cli_with_env so tests can hand credentials to the CLI by name instead of on its command line. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * feat(tmachine): add an Ubuntu Docker environment that runs Keycloak Environment setup can now take inputs, like installers do, and hashes them only when present so existing cached disks stay valid. ubuntu-docker-rootful-keycloak runs the Docker setup plus a Keycloak playbook that starts the container with a restart policy, since setup is cached as a disk, and writes how to reach it as OPENSHELL_TEST_* lines in /etc/openshell-test/env.d/keycloak.env. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * feat(test-runner): pass the variables an environment publishes to tests After boot, read every /etc/openshell-test/env.d/*.env file, accept only OPENSHELL_TEST_* names, and add the variables to the testkit-ready and nextest commands through guest_env. Errors name the key, never the value. The runner doesn't know which services an environment runs; suites read the variables, so they also run against a deployment that sets them. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(features): port Keycloak provider refresh to testkit Add a keycloak features crate with a suite-local KeycloakContext fixture that reads OPENSHELL_TEST_KEYCLOAK_*, waits for the realm to answer, and provides password grants, revocation and the token endpoint. Port the provider refresh test onto it. Align the features lockfile with the root and add it to the alignment check. The old provider-refresh crate stays until the required lanes switch. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * ci(tests): run the Keycloak features suite in E2E Suites Add a matrix row for features/keycloak on ubuntu-docker-rootful-keycloak, include the Keycloak realm in the tmachine disk cache key, and document environments that provide services in TESTING.md. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(testing): describe how to add a feature suite Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(tmachine): publish test variables through a shared task file Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(test-runner): keep published test variables out of SSH commands The runner passed the variables in env.d on the nextest command line, so an SSH timeout or launch failure printed their values, which can be credentials, in the error context. The runner now only checks the files; the guest shell exports them before starting nextest, so only the gateway name and CLI path appear in the command. Lines are read literally on both sides: no trimming, and only empty lines and lines starting with # are skipped. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(features): fit Keycloak readiness in the per-test timeout The fixture waited up to three minutes for Keycloak inside the test, but the runner's nextest profile kills a test after two. Wait up to 90 seconds so a slow Keycloak fails with a readiness error instead. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(test-runner): run the export script test with shell builtins only The Nix build runs the runner's tests in a sandbox without env on PATH, so the test failed there and the runner didn't build. Print the exported variables with printf instead. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(features): name the Keycloak suite after what it tests The suite was named after the service it needs rather than what it tests. Rename it to keycloak-provider-refresh and move the Keycloak fixture into a support crate at tests/support/keycloak, so other suites that need an OIDC issuer can depend on it. TESTING.md now asks for a descriptive suite name instead of prescribing one. Lint and format the new crates in CI and mise. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(features): bound each Keycloak request in the fixture poll_until checks its deadline only between attempts, and curl had no time limit, so a Keycloak that accepts connections but stalls could hold one attempt until nextest killed the test. Give each request 10 seconds. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * ci(tests): key the tmachine disk cache by environment and installer Both E2E suites rows computed the same cache key. Only the first job to save it got its disks in, and actions/cache never saves after an exact hit, so the other row rebuilt its setup and install layers on every run. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(testkit): keep the gateway selection over caller CLI env run_with_env applied the caller's variables last, so one named like the gateway, endpoint or workspace variable could silently point the CLI at another gateway. Apply them first so testkit's selection wins. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(features): leave the test body room after Keycloak readiness poll_until checks its deadline only between attempts, so a 90 second wait could run for about 102 seconds with the request time limit and the poll interval, leaving the test body under 20 seconds of the 2 minute budget. Wait 75 seconds and spell out the margin. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * ci(tests): include config.nix in the tmachine disk cache key tmachine's setup hash covers each environment's playbook list and setup inputs, which config.nix declares. Changing only config.nix left the cache key unchanged, so the rebuilt layers were never saved again. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(testing): add the lint step for new feature suites CI formats and lints feature suites one package at a time, so a suite added by following TESTING.md got no fmt or clippy coverage. Also say what test_env actually guarantees about published values. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(features): report Keycloak suite cleanup failures The provider and profile cleanup discarded both errors and failed exit codes, so leftovers went unnoticed. Print them without failing the test. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
1 parent 73ae222 commit 50a3237

24 files changed

Lines changed: 5758 additions & 446 deletions

File tree

‎.github/workflows/branch-checks.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -234,6 +234,8 @@ jobs:
234234
cargo fmt --manifest-path tests/tmachine/Cargo.toml --all -- --check
235235
cargo fmt --manifest-path tests/runner/Cargo.toml --all -- --check
236236
cargo fmt --manifest-path tests/suites/conformance/Cargo.toml -p sandbox -- --check
237+
cargo fmt --manifest-path tests/suites/features/Cargo.toml -p keycloak-provider-refresh -- --check
238+
cargo fmt --manifest-path tests/support/Cargo.toml --all -- --check
237239
238240
cargo-lockfiles:
239241
name: Cargo lockfiles
@@ -296,6 +298,8 @@ jobs:
296298
cargo clippy --locked --manifest-path tests/tmachine/Cargo.toml --all-targets -- -D warnings
297299
cargo clippy --locked --manifest-path tests/runner/Cargo.toml --all-targets -- -D warnings
298300
cargo clippy --locked --manifest-path tests/suites/conformance/Cargo.toml -p sandbox --all-targets -- -D warnings
301+
cargo clippy --locked --manifest-path tests/suites/features/Cargo.toml -p keycloak-provider-refresh --all-targets -- -D warnings
302+
cargo clippy --locked --manifest-path tests/support/Cargo.toml --all-targets -- -D warnings
299303
300304
rust-test:
301305
name: Rust test (${{ matrix.system }})

‎.github/workflows/e2e-suites.yml‎

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,11 @@ jobs:
5050
family: conformance
5151
package: sandbox
5252
runner: ubuntu-24.04
53+
- environment: ubuntu-docker-rootful-keycloak
54+
installer: deb
55+
family: features
56+
package: keycloak-provider-refresh
57+
runner: ubuntu-24.04
5358
runs-on: ${{ matrix.runner }}
5459
timeout-minutes: 90
5560
steps:
@@ -77,13 +82,17 @@ jobs:
7782
uses: actions/cache@caa296126883cff596d87d8935842f9db880ef25 # v5.1.0
7883
with:
7984
path: ~/.cache/tmachine
80-
# tmachine keys its disk layers on a hash of the Ansible tree and
81-
# the cache versions in setup.rs and install.rs. actions/cache
82-
# doesn't save after an exact key hit, so a key that misses one of
83-
# those would rebuild the changed layers on every run. No
84-
# restore-keys fallback: restoring an older entry would carry its
85-
# stale disks into the new one, which then only grows.
86-
key: tmachine-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('tests/tmachine/ansible/**', 'tests/tmachine/images.nix', 'tests/tmachine/src/qemu/setup.rs', 'tests/tmachine/src/qemu/install.rs') }}
85+
# tmachine keys its disk layers on a hash of the Ansible tree, the
86+
# environments' playbook lists and setup inputs (config.nix, and
87+
# files such as the Keycloak realm file) and the cache versions in
88+
# setup.rs and install.rs. actions/cache doesn't save after an
89+
# exact key hit, so a key that misses one of those would rebuild the
90+
# changed layers on every run. Environments share no layers, and
91+
# only the first job to save a key gets its disks in, so each
92+
# environment and installer needs its own key. No restore-keys
93+
# fallback: restoring an older entry would carry its stale disks
94+
# into the new one, which then only grows.
95+
key: tmachine-${{ runner.os }}-${{ runner.arch }}-${{ matrix.environment }}-${{ matrix.installer }}-${{ hashFiles('tests/tmachine/ansible/**', 'tests/tmachine/config.nix', 'tests/tmachine/images.nix', 'tests/tmachine/src/qemu/setup.rs', 'tests/tmachine/src/qemu/install.rs', 'scripts/keycloak-realm.json') }}
8796

8897
- name: Download integration inputs
8998
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1

‎TESTING.md‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -230,6 +230,63 @@ an explicit matrix and runs the runner once per row. Branch E2E calls it when
230230
the `test:e2e` label is set, and a failure fails the Core E2E result. To run
231231
a suite in another environment on CI, add a row.
232232

233+
### Adding a feature suite
234+
235+
Feature suites live in the `tests/suites/features` workspace, one crate per
236+
suite, with a name that says what the suite tests.
237+
`tests/suites/features/keycloak-provider-refresh` is the example to copy.
238+
239+
1. Add a crate under `tests/suites/features/<name>` that depends on
240+
`openshell-testkit`, and add it to `members` in
241+
`tests/suites/features/Cargo.toml`.
242+
2. Write the tests in `tests/*.rs` as `#[testkit::test]` functions.
243+
3. Re-lock the workspace: `cargo metadata --manifest-path
244+
tests/suites/features/Cargo.toml`, then
245+
`uv run tasks/scripts/check_cargo_lockfile_alignment.py --fix`.
246+
4. Add `cargo fmt` and `cargo clippy` lines for the package (`-p <name>`) to
247+
`tasks/rust.toml` and to the Format and Lint steps in
248+
`.github/workflows/branch-checks.yml`.
249+
5. Add a row for the suite to the matrix in `.github/workflows/e2e-suites.yml`.
250+
251+
If the feature only needs the gateway, run the suite in an existing
252+
environment such as `ubuntu-docker-rootful`, and skip the rest of this
253+
section.
254+
255+
**Suites that need a service.** A feature that needs something besides the
256+
gateway, such as an OIDC issuer, runs in a tmachine environment that provides
257+
it. The environment and the suite share a contract: a set of
258+
`OPENSHELL_TEST_<SERVICE>_*` environment variables that say how to reach the
259+
service.
260+
261+
- **Environment.** Add an environment to `tests/tmachine/config.nix` whose
262+
setup playbook starts the service (with a restart policy, since setup is
263+
cached as a disk) and writes the variables as `KEY=VALUE` lines to
264+
`/etc/openshell-test/env.d/<service>.env`. Pass files the playbook needs
265+
through the environment's setup `inputs`, and add them to the tmachine disk
266+
cache key in `e2e-suites.yml`. The runner rejects names without the
267+
`OPENSHELL_TEST_` prefix, and the tests get the variables in the guest, so
268+
values never appear in a runner command. Values are taken literally, without
269+
quoting.
270+
- **Fixture.** Put the service's fixture in a support crate under
271+
`tests/support/<service>`, so any suite can depend on it, or reuse the
272+
existing one. It defines the variable names as constants and a
273+
`<Service>Context::new(cx)` that reads them, waits for the service, and
274+
wraps the testkit `Context`. Each test calls it on its first line.
275+
Nextest runs each test in its own process, so the fixture builds its
276+
clients there from the variables. A missing variable fails the test; it
277+
never skips it.
278+
- **Documentation.** Document the variables in the fixture's module docs, so
279+
anyone can run the suite against their own deployment by setting them.
280+
281+
The Keycloak fixture is `tests/support/keycloak`. Run the suite that uses it
282+
with:
283+
284+
```shell
285+
nix run .#openshell-test-runner -- \
286+
--environment ubuntu-docker-rootful-keycloak --installer deb \
287+
--family features --package keycloak-provider-refresh
288+
```
289+
233290
### Python E2E (`e2e/python/`)
234291

235292
`mise run e2e:python` builds `openshell/e2e-python:dev` from

‎tasks/rust.toml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,8 @@ run = [
4040
"cargo clippy --manifest-path tests/tmachine/Cargo.toml --all-targets -- -D warnings",
4141
"cargo clippy --manifest-path tests/runner/Cargo.toml --all-targets -- -D warnings",
4242
"cargo clippy --manifest-path tests/suites/conformance/Cargo.toml -p sandbox --all-targets -- -D warnings",
43+
"cargo clippy --manifest-path tests/suites/features/Cargo.toml -p keycloak-provider-refresh --all-targets -- -D warnings",
44+
"cargo clippy --manifest-path tests/support/Cargo.toml --all-targets -- -D warnings",
4345
]
4446
run_windows = "powershell -NoProfile -ExecutionPolicy Bypass -File tasks/scripts/windows-msvc.ps1 lint native"
4547
hide = true
@@ -56,6 +58,8 @@ run = [
5658
"cargo fmt --manifest-path tests/tmachine/Cargo.toml --all",
5759
"cargo fmt --manifest-path tests/runner/Cargo.toml --all",
5860
"cargo fmt --manifest-path tests/suites/conformance/Cargo.toml -p sandbox",
61+
"cargo fmt --manifest-path tests/suites/features/Cargo.toml -p keycloak-provider-refresh",
62+
"cargo fmt --manifest-path tests/support/Cargo.toml --all",
5963
]
6064
hide = true
6165

@@ -71,6 +75,8 @@ run = [
7175
"cargo fmt --manifest-path tests/tmachine/Cargo.toml --all -- --check",
7276
"cargo fmt --manifest-path tests/runner/Cargo.toml --all -- --check",
7377
"cargo fmt --manifest-path tests/suites/conformance/Cargo.toml -p sandbox -- --check",
78+
"cargo fmt --manifest-path tests/suites/features/Cargo.toml -p keycloak-provider-refresh -- --check",
79+
"cargo fmt --manifest-path tests/support/Cargo.toml --all -- --check",
7480
]
7581
hide = true
7682

‎tasks/scripts/check_cargo_lockfile_alignment.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,10 @@
3535
# Workspaces whose shared crates must match the root lockfile. Add a
3636
# workspace here once its lockfile has been aligned with `--fix`.
3737
ALIGNED_WORKSPACES = (
38+
"tests/support",
3839
"tests/testkit",
3940
"tests/suites/conformance",
41+
"tests/suites/features",
4042
)
4143

4244

‎tests/runner/src/execute.rs‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,9 @@
77
//!
88
//! Tests find the gateway the way the CLI does, so the runner points them at
99
//! the one the installer registered with `OPENSHELL_GATEWAY` and at the
10-
//! installed CLI with `OPENSHELL_BIN`.
10+
//! installed CLI with `OPENSHELL_BIN`. Anything else the environment provides,
11+
//! such as Keycloak, the tests learn from the variables it publishes (see
12+
//! [`crate::test_env`]).
1113
1214
use std::fmt;
1315
use std::path::Path;
@@ -19,6 +21,7 @@ use tmachine::Session;
1921
use tmachine::config::Config;
2022

2123
use crate::bundle::{Bundle, PROFILE, READY_BINARY};
24+
use crate::test_env;
2225

2326
const GUEST_DIR: &str = "/var/tmp/openshell-tests";
2427
/// The nextest log's name, in the guest directory and in the results.
@@ -225,6 +228,9 @@ async fn run_tests(session: &Session, run: &Run<'_>) -> Result<Status> {
225228
format!("OPENSHELL_GATEWAY={}", run.gateway),
226229
format!("OPENSHELL_BIN={openshell_bin}"),
227230
];
231+
test_env::check(session)
232+
.await
233+
.context("check the variables the environment publishes for tests")?;
228234

229235
let guest = GUEST_DIR;
230236
checked(
@@ -289,9 +295,15 @@ async fn run_tests(session: &Session, run: &Run<'_>) -> Result<Status> {
289295
nextest.extend(["-E", filter]);
290296
}
291297
// nextest writes its output to a file in the guest, so a run that times
292-
// out or loses SSH still leaves a log for `collect_results`.
298+
// out or loses SSH still leaves a log for `collect_results`. The guest
299+
// shell exports the published variables itself, so their values, which
300+
// can be credentials, never appear in the command (or its errors).
293301
let log = format!("{guest}/{NEXTEST_LOG}");
294-
let mut command = vec!["sh", "-c", r#"exec "$@" >"$0" 2>&1"#, &log];
302+
let script = format!(
303+
r#"exec >"$0" 2>&1; {}; exec "$@""#,
304+
test_env::export_script(test_env::ENV_DIR)
305+
);
306+
let mut command = vec!["sh", "-c", &script, &log];
295307
command.extend(nextest);
296308
let output = session.exec_timeout(&command, RUN_TIMEOUT).await?;
297309
Ok(Status::of_nextest(

‎tests/runner/src/main.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ use crate::execute::{Run, Status};
1717

1818
mod bundle;
1919
mod execute;
20+
mod test_env;
2021

2122
/// Build a test family for the guest, boot a tmachine environment with
2223
/// OpenShell installed, run the tests in it, and check the results.

0 commit comments

Comments
 (0)