infra: Full migration of emulated baremetal e2e tests to storm-based runner - #724
Draft
frhuelsz wants to merge 41 commits into
Draft
infra: Full migration of emulated baremetal e2e tests to storm-based runner#724frhuelsz wants to merge 41 commits into
frhuelsz wants to merge 41 commits into
Conversation
frhuelsz
changed the title
Add the foundations for porting the pytest E2E host-state validation suite into Go storm test cases, plus the first ported marker (base): - storm/utils/trident/hoststatus.go: hybrid Host Status parser (typed core accessors + gabs escape hatch), GetHostStatus over SSH. - storm/utils/sysinspect: base-tier SSH parsers (blkid, lsblk, mount, /dev/md RAID resolution, passwd/group, efibootmgr). - storm/e2e/validate: SoftAsserter (interim soft-assert accumulator) and base validation (partitions, users, uefi-fallback) ported from base_test.py, including the non-verity A/B active-volume path check. - scenario wiring: validate-install / validate-ab-update-* cases and expectedActiveVolume tracking (flipped after each A/B update). All unit-tested (stdlib testing); storm-trident builds and discovers base_vm-host. Not yet integration-tested on a real VM. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Port extensions_test.py: for each configured sysext/confext, verify the extension path exists on the host and that the extension is active per 'systemd-<type> status --json'. Self-selects when the Host Config declares sysexts/confexts. - storm/utils/sysinspect/systemd_ext.go: systemd-sysext/confext status JSON parser. - storm/e2e/validate/extensions.go: ValidateExtensions + HasExtensions. - scenario/validate.go: wire extensions into validate-* cases. Unit-tested; storm-trident builds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- rollback (rollback_test.py): validate rolled-back servicing state, health-check lastError, active volume (absent when not-provisioned), and the health-check failure log messages. Self-selects via a top-level 'health' section in the Host Config (HasRollbackIntent), replacing base validation and expecting not-provisioned. - ab_update_staged (ab_update_staged_test.py): validate the staged servicing state and unchanged active volume. Wired inline into the split A/B flow between stage and finalize. Unit-tested; storm-trident builds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Port verity_test.py::test_verity_root: confirm /dev/mapper/root exists and veritysetup reports it active/verified/read-only, then validate the data and hash devices correspond to the expected block devices, handling both A/B (active volume of data/hash pairs) and non-A/B configs, and both partition and RAID backing devices. Self-selects when the Host Config declares a verity device. - storm/utils/sysinspect/veritysetup.go: veritysetup status parser. - storm/utils/sysinspect/blkid.go: add full device Path to BlkidEntry. - storm/e2e/validate/verity.go: ValidateVerity + HasVerity. - scenario/validate.go: wire verity into validate-* cases. Unit-tested; storm-trident builds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add SSH parsers needed for the encryption validation port: cryptsetup status + luksDump (JSON), dmsetup info, findmnt, active swaps, readlink -f, getenforce/setenforce, and blkid --output export. All unit-tested. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Port encryption_test.py::test_encryption (the largest marker). For each configured encryption volume, validate: - the backing device (partition or RAID array) is crypto_LUKS; - LUKS2 metadata via cryptsetup luksDump (digest/keyslot/token, TPM2 policy differing for UKI vs grub images) with the SELinux permissive workaround; - device-mapper state via dmsetup info (LUKS2 vs PLAIN for swap); - cryptsetup status (cipher/keysize, in-use vs active); - mount/swap/active status via findmnt/swapon, accounting for A/B pair membership and the active volume. Self-selects when the Host Config declares encryption volumes; uses the scenario's isUki param for the expected TPM2 policy. Unit-tested (parsers + HC helpers); storm-trident builds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Code review found HasVerity self-selected on any storage.verity entry, so usr-verity configs (verity on /usr, plain root) would run root-verity validation and false-fail with 'no verity configuration found for root device'. The ported verity_test.py::test_verity_root only validates root verity and only ran on root-verity configs. Gate HasVerity on whether the root filesystem is itself a verity device, matching that scope. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
checkTridentViaSshAfterInstall hardcoded expectSuccessfulCommit=true, so health-check rollback scenarios (health-checks-install), whose install deliberately fails and rolls back, would false-fail before rollback validation runs. Gate expectSuccessfulCommit on a new hasRollbackIntent() (top-level 'health' in the Host Config), matching the pipeline's --expect-failed-commit handling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Widen invert.py ALLOWED_CONFIGS from [base] to the set validated end-to-end
on a real VM: base, simple, misc, raid-{mirrored,resync-small,small},
encrypted-{partition,raid,swap}, root-verity. These exercise the base,
encryption, and root-verity validation paths (confirmed via local VM runs:
cryptsetup/luksDump/dmsetup/findmnt for encryption, veritysetup for verity,
plus a full A/B update cycle for root-verity).
Held back with documented prerequisites:
- UKI/usr-verity family (combined, memory-constraint-combined, rerun,
usr-verity, usr-verity-raid): Secure Boot rejects the UKI kernel unless
the image signing cert is injected via --signing-cert (not passed by
local runs or the storm CI template yet).
- health-checks-install: install-os must tolerate the expected rollback
phonehome failure (ignorePhonehomeFailures not wired).
- extensions: needs sysext/confext injection (edit_host_config.py port).
- split: needs the trident-split-installer ISO.
Two of these gaps (UKI signing, rollback-install) were found by validating
before enabling, and would have failed in CI.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Wire the container-runtime prep the pytest suite did at connection time and enable the VM/CONTAINER e2e stage: - scenario: on every fresh SSH connection for the container runtime, disable SELinux enforcement and load the Trident container image into Docker (prepareContainerRuntime, called from populateSshClient so it re-runs after post-reboot reconnects). LoadTridentContainer existed but was never called. - invert.py: add 'container' to ALLOWED_RUNTIMES. - storm_e2e.yml: un-comment the VM CONTAINER test_execution_template stage (BM stays gated on bare-metal host setup). Validated end-to-end on a real VM: base_vm-container passed clean (OK: skipped 5; passed 11; total 16), including a full A/B update cycle with Trident run via 'docker run trident/trident:latest' and the SELinux/docker prep re-running after the A/B reboot. The CI already renames container images via 'storm-trident helper prepare-images' and sets the container ISO/tarball params, so the stage has the plumbing it needs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove accidentally-committed junit.xml/junit-hci.xml (storm-trident -j output from local validation runs) and gitignore junit*.xml, logs*/, out*/ so local run artifacts are not tracked. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove accidentally-committed metrics-ab-update-1-ab-update.jsonl and trident-clean-install-metrics.jsonl (storm-trident run outputs written to the repo root) and gitignore logstream*.log and *metrics*.jsonl so these local run artifacts are not tracked. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
health-checks-install deliberately fails its health checks and rolls back, so Trident phones home a failure and netlaunch returns a PhoneHomeFailureError. install-os treated this as fatal, so the config could not run. Mirror the legacy pipeline's --only-print-exit-code handling: for rollback-intent scenarios (hasRollbackIntent), log the phonehome failure as expected and continue to post-install validation instead of failing. Re-enable health-checks-install in the e2e allowlist. Validated end-to-end on a real VM: health-checks-install_vm-host passed (OK: skipped 1; passed 5; total 6) with install-os PASS, check-trident-ssh PASS (expected failed commit), and validate-install PASS running ValidateRollback (trident get + health-check-failure log inspection). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
UKI/usr-verity images boot their kernel directly through firmware Secure
Boot, which rejected them ('Access denied') without the image signing cert.
Wire the cert and fix a related isUki bug:
- test_execution_template.yml: pass --signing-cert with the usrverity test
image's ca_cert.pem (at System.ArtifactsDirectory/usrverity-testimage)
when present. Enrolling the test CA is harmless for grub images, so it is
passed unconditionally when available.
- invert.py: set isUki=true for configs whose test-selection.yaml declares
the 'uki' marker (mirrors the pytest isUki fixture). Previously isUki was
never set, so the encryption validation used the wrong (non-UKI) expected
TPM2 policy for UKI images.
- invert.py: enable usr-verity and usr-verity-raid.
Validated end-to-end on a real VM: usr-verity passed (OK 11/16) with the UKI
booting cleanly (no Access denied), usr-verity-raid passed (OK 5/6), and base
still passed WITH the cert injected (OK 11/16 — no grub regression).
Encryption+UKI configs (combined, memory-constraint-combined, rerun) remain
held: their encrypted root on an A/B pair under usr-verity trips the base A/B
path check (tracked separately).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
base_test.py::test_partitions asserts the active A/B volume's device path only when the volume ID appears in Host Status partitionPaths; when it does not, it is a no-op. The Go port instead failed in that case, so configs whose root sits on the A/B pair but whose mounted device Trident reports under a different key (combined, memory-constraint-combined, rerun) failed with 'active volume "root-a" missing from partitionPaths'. Match the Python tolerance: assert on match, log-and-skip when absent. Combined with the earlier isUki fix, this unblocks the encryption+UKI configs. Enable combined, memory-constraint-combined, rerun. Validated end-to-end on a real VM: combined passed (OK 11/16) with encryption luksDump running under the correct UKI TPM2 policy and the ab-path match skipped for root-a; rerun passed (OK 11/16). memory-constraint-combined shares the same validated path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The split config tests the split clean-install flow (stage and finalize as separate steps) using the trident-split-installer ISO. Enable it in ALLOWED_CONFIGS. The storm CI template already special-cases split to use the trident-split-installer ISO. Validated end-to-end on a real VM: split_vm-host passed (OK 5/6) doing a clean install via the split installer ISO and base validation. (The split config has no abUpdate, so no A/B cases run; the separate split-A/B-update path is exercised by abUpdate configs at ring pre+.) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the OCI Host Configuration edits that the pipeline's trident-prep step did via edit_host_config.py, so the storm flow can run configs that pull extension images (and, at ACR rings, COSI images) from an OCI registry: - new scenario args --sysext-oci-url/--sysext-sha384, --confext-oci-url/ --confext-sha384, and --oci-image-url. - prepareHostConfig injects os.sysexts / os.confexts entries and overrides image.url when the corresponding args are provided (applyOciOverrides). This is the Go foundation for enabling the extensions config; unit-tested. Pipeline wiring (build+push ext images to ACR and pass the args) and enabling extensions in the allowlist follow separately. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Enable the extensions config in the storm e2e flow by building sample
sysext images, pushing them to ACR, and injecting the resulting OCI URL +
sha384 into the Host Configuration (consumed by the Go applyOciOverrides
added earlier).
- e2e-template.yml / storm_e2e.yml: thread acrServiceConnectionName down to
test_execution_template.yml.
- test_execution_template.yml:
- add the trident_e2e_params variable group (provides ACR_NAME).
- resolve the configuration name + runtime from the matrix SCENARIO at
runtime (fixes the split handling, which referenced an undefined
compile-time tridentConfigurationName).
- for the extensions config, build sysext images (build-extension-images),
push them to ACR (acr-push via the ACR service connection), and pass
--sysext-oci-url/--sysext-sha384 to storm-trident run.
- invert.py: enable the extensions config.
NOTE: pipeline validation pending — verified locally via YAML parse, go
build/test, and template preview, but the ACR push + OCI pull path can only
be exercised by a real pipeline run.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Allow the testing-trident.yml shortcut pipeline (which runs selected suites against pre-built artifacts) to run the storm E2E tests: - storm_e2e.yml: add testingRun (consume DownloadTestingElements instead of the in-pipeline build stages) and runVMHost/runVMContainer gating params. Defaults preserve the full-pipeline behavior. - test-template.yml + testing-trident.yml: add runStormE2EVMHost / runStormE2EVMContainer selection params and a stormTestRing selector (pr-e2e/ci/pre/full-validation), include storm_e2e with testingRun=true, and extend the artifact-download gating to cover the storm selections. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…path In a testingRun (testing-trident.yml shortcut), the storm E2E matrix and validation logic must reflect the branch under test, not a possibly-stale go-tools artifact downloaded from another build. Previously the shortcut downloaded storm-trident from the source build, so e.g. running the shortcut against a main artifact only produced the base scenario even though the branch enables many more. - new build-storm-trident.yml steps template (update-protoc, protoc-gen-go, make bin/storm-trident) mirroring building-tools.yml. - storm_e2e.yml DefineTests: when testingRun, build storm-trident from source (on the build-capable named pool) instead of downloading go-tools, so the generated matrix reflects the branch. - test_execution_template.yml: when testingRun, build storm-trident from source after downloading test images (overriding the downloaded binary). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The standalone 'make bin/storm-trident' target only regenerates the embedded configurations; the protobuf (pkg/tridentgrpc) and TLS cert (pkg/rcp/tlscerts) sources are generated by other targets (bin/netlaunch) in a full build. The shortcut build-storm-trident step failed with 'pattern client.crt: no matching files' and missing tridentpbv1 packages. Generate both before building, so the standalone build succeeds. Verified locally from a clean state (removed generated sources, ran the sequence, forced a full rebuild). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously the testing-trident.yml shortcut rebuilt storm-trident inside both the DefineTests_E2E job and every test-execution job. The test jobs first download the go-tools artifact (a stale main storm-trident) into bin/, and make's timestamp check could skip the rebuild, silently using the stale binary (all scenarios reported "not found"). Build storm-trident once in a dedicated BuildStormTridentE2E stage and publish it as the storm-trident-e2e artifact. Both DefineTests_E2E and the test jobs now download that artifact (overriding the go-tools binary) instead of building in-job. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…arse
The validate-* case collapses many sub-checks into one storm test case.
Record every sub-check outcome (pass and fail) in SoftAsserter and log an
ordered PASS/FAIL Summary at the end of the case, so it is visible exactly
which checks ran even when the case passes.
Also fix ParseSystemdExtStatus: systemd emits the 'extensions' field as a
bare string ('none' for an empty hierarchy, or a single extension name),
not only as an array. Decode string|array|null and drop the 'none'
sentinel. This unblocks the extensions scenario validation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Trident forbids sealing encryption to PCR 7 (secure-boot-policy) for a UKI
target OS when Trident runs inside a container: Secure Boot measures the
host boot chain, not the container's, so the sealed policy can never
reproduce. combined, memory-constraint-combined, and rerun all seal to
PCR 7 under a UKI layout, so they failed clean install on vm-container
('Since Trident is running in a container, PCR 7 cannot be used for
encryption in the target UKI OS').
Self-select this constraint from the Host Configuration: drop the container
runtime for any config that carries the uki marker and encrypts volumes
sealed to PCR 7. Non-UKI encryption (encrypted-partition/raid/swap) and
UKI-without-encryption (usr-verity) keep both runtimes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ion configs dea6f0f excluded combined/rerun from the container runtime because they failed clean install with 'PCR 7 cannot be used for encryption in the target UKI OS'. That diverged from the legacy suite, which DOES run both in container: the VM glue added in #221 (.pipelines/templates/stages/testing_vm/netlaunch-testing.yml) rewrites the Host Configuration to set storage.encryption.pcrs = [boot-loader-code, kernel-boot] (dropping PCR 7) for usr-verity UKI images on the container runtime. Port that transform into storm HC prep (applyContainerPcrExclusion) and remove the invert.py container exclusion, so combined/rerun run in both runtimes exactly like legacy. memory-constraint-combined stays host-only because target-configurations.yaml does not list it under container. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The storm E2E extensions path pushed sysext images to ACR but never removed them, leaking a new tagged image per build. The legacy suite cleans up via remove-from-acr.yml (storm-trident script acr-delete). Add the equivalent cleanup step to the storm test-execution job: an always()-conditioned AzureCLI task that runs acr-delete for the extensions config (no-op otherwise), reconstructing the tag base from build id + config + deployment environment and using the SYSEXT_REPO set by the push step. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
storm-trident supports -j/--j-unit to emit JUnit XML, but the E2E job neither requested it nor published anything, so storm runs produced no Tests-tab results or JUnit artifact. Pass -j to write JUnit XML to the output dir and publish it via the shared handle-junit-test-results.yml (PublishTestResults@2 + build artifact, succeededOrFailed so failures upload too), one JUnit file per matrix scenario. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…/container merge storm emits JUnit testcases with an empty classname and identical case names (install-os, validate-install, ...) across runtimes. ADO keys test identity on classname + name, so the host and container variants of a scenario collided/merged in the Tests tab. Set each <testcase> classname to the scenario id (e.g. base_vm-host) before publishing so the two variants are distinct. Uses a targeted sed rather than an XML parser because storm embeds raw serial-console control characters in the CDATA that break strict XML parsing (ADO's parser tolerates them). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pin github.com/microsoft/storm to 6b7ba8c (branch user/frhuelsz/live-ado-groups) to validate live Azure DevOps collapsible groups around each storm test case's streamed output in a real pipeline run. This is a temporary branch pin (pseudo-version), to be replaced with a released storm tag once the feature is validated and merged upstream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…blish PublishTestResults@2 rejected the storm JUnit with 'hexadecimal value 0x00 is an invalid character' because storm copies raw serial-console output (including NUL and other C0 control bytes) verbatim into the system-out CDATA, which XML 1.0 forbids. Strip those bytes (keeping tab/LF/CR) before publishing, alongside the existing classname qualification. Byte filtering is used rather than an XML parser because the document is not well-formed until the bytes are removed. The durable fix lives in storm's reporter (a CDATA scrubber, tracked separately); drop this once that lands. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Update the storm pin from 6b7ba8c to d021951 (branch user/frhuelsz/ live-ado-groups). The first prototype wrapped each case's streamed output in ADO groups but emitted the ##[group] markers on stdout while the (started)/PASS status lines went to stderr; the ADO agent orders the two streams by arrival, so boundary lines fell inside the wrong group. d021951 routes all of a case's live output (started line, group markers, streamed detail incl. in-case logrus, status line) onto a single stream in program order, so (started)/STATUS render outside the fold and bracket it correctly. Still a temporary branch pin, to be replaced with a tagged storm release. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…pped The four ab-update-split-* cases skip on rings below 'prerelease', but validate-ab-update-split was registered without that gate, so it ran (and would validate against a host that never underwent the split update) even when the split update itself was skipped. Extract the ring gate into a shared skipIfSplitTestsDisabled helper and apply it to the split validation case too, so they skip together. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ced-rollback)
Legacy runs a forced-rollback A/B update for every A/B config
(e2e-test-abupdate-scenario.yml --forced-rollback): it injects failing
health checks so Trident rolls back to the current volume, then validates
the rolled-back state. Port this into TridentE2EScenario:
- Refactor abUpdateOs to take abUpdateOptions{split, expectRollback}. When
expectRollback is set, the post-reboot service check expects a FAILED
commit and the expected active volume is not flipped (the host returns to
its current volume).
- Add auto-rollback cases (sync-hc, inject-hc, upload-hc, update,
validate-auto-rollback) inside the HasABUpdate() gate, so they self-select
for the 10 A/B configs exactly like the legacy abActiveVolume!=null gate.
- injectRollbackHealthChecks appends the same two failing checks the legacy
ab-update helper adds (script 'exit 1' + systemd non-existent-service*),
both gated to the ab-update phase, keeping the image unchanged.
- validateAutoRollback asserts the provisioned/unchanged-volume rollback
outcome via the existing ValidateRollback (ports rollback_test.py).
Unit tests cover registration order and the health-check injection.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…uble reboot Local VM validation surfaced two issues in the auto-rollback path: 1. Reusing the current image for the forced-rollback update failed Trident's duplicate-filesystem-UUID validation before the health checks could even run. Add auto-rollback-update-hc (updateHostConfig) so the rollback update points at a fresh image, exactly like a normal A/B update; the injected failing health checks then actually fire. 2. A forced rollback reboots TWICE (into the staged volume to run the failing health checks, then back to the current volume once rolled back). The old single-reconnect + single service-check landed mid-second-reboot and failed with 'SSH connection needs to be re-established'. Add waitForFailedCommitAfterRollback, which re-dials a fresh SSH client on every attempt until the service settles in its failed-commit state (mirroring the legacy check-trident-service helper). Also drop the expectRollback special-case from runTridentUpdate: the rollback update reboots normally (hits the REBOOTING message), and swallowing non-zero statuses there would have masked genuine pre-reboot failures like the duplicate-fs-uuid error above. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Legacy runs a return A/B update into OS A after the auto-rollback ("Stage
and finalize A/B update into target OS A"). Add it to TridentE2EScenario
inside the HasABUpdate() gate:
- ab-update-2 reuses the auto-rollback's image version rather than bumping.
Test images alias only two real COSIs (odd version -> image.cosi/v1, even
-> image_v2.cosi/v2), so bumping to the next even version would collide
with the active volume's UUID; reusing the odd version keeps a distinct
UUID. This mirrors legacy running the auto-rollback with
incrementUpdateVersion=false so the return update reuses the same version.
- ab-update-2-clear-hc strips the failing health checks the auto-rollback
injected so this update commits (flips active to A) instead of rolling
back again, mirroring the legacy ab-update helper run without
--forced-rollback.
- Refactor updateHostConfig into updateHostConfigToVersion(bump) with a
reuse-version variant; validate-ab-update-2 reuses validateHostState.
Registration-order unit tests updated. VM validation pending.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Port the legacy VM-only manual-rollback step (storm-trident helper manual-rollback) plus its follow-up validation into TridentE2EScenario, self-selecting on HasABUpdate() && IsVM(): - manual-rollback publishes the rollback chain + pre-rollback datastore as artifacts, then stages and finalizes 'trident rollback', which reboots the host back into the previously-committed volume. A manual rollback reboots exactly once (no health checks), so the standard single reconnect + successful-commit check applies. It flips expectedActiveVolume so the follow-up validation checks the rolled-back volume. - validate-manual-rollback reuses validateHostState, which self-selects base + encryption/verity checks from the Host Config (superset of the legacy 'base [or encryption]' pytest run). - Registered last, after the split A/B update, matching legacy ordering. The rolled-back volume is derived from storm's tracked expectedActiveVolume, so it is correct regardless of ring (split flips the volume only at pre+). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Port the legacy VM-only rebuild-raid step into TridentE2EScenario, self-selecting independently of A/B on HasRebuildableRaid() && IsVM() (storage.raid present and not usr-verity; TODO(12277) UKI/verity rebuild). This covers both A/B RAID configs (raid-small, raid-resync-small) and non-A/B ones (raid-mirrored, encrypted-raid). - Add testVmHostInfo.FailAndReplaceDataDisk: via the libvirt API, force the VM off, delete a RAID member data disk volume, recreate it blank at the same path/size, and power the VM back on (the legacy helper shelled out to virsh + qemu-img). - rebuild-raid-fail-disk drives that, reconnects to the degraded host, and confirms the servicing commit is intact; rebuild-raid runs 'trident rebuild-raid'; validate-rebuild-raid reuses validateHostState. - Add hostconfig.HasRebuildableRaid + hasUsrVerity getters with unit tests (RAID config registers rebuild-raid; usr-verity-raid does not). Registered as an independent block after the A/B/manual-rollback block, so non-A/B RAID configs run install -> validate -> rebuild-raid. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…elinux/check-tracing) Port the legacy host-only check-selinux and check-tracing steps as a validate-host-diagnostics case registered right after validate-install, self-skipping on the container runtime: - ValidateSelinuxDenials runs audit2allow against the audit log and surfaces any SELinux denials, failing only if the command cannot run (matching the helper, which does not hard-fail on denials). - ValidateJournaldTracing asserts Trident's commit tracing metric (trident_start) reached journald under the trident-tracing identifier. - ValidateTraceFileMetric asserts the servicing feature-usage metric (host_config_feature_usage) was captured in the clean-install trace-stream file (netlisten's trident-clean-install-metrics.jsonl); an unconfigured trace file is skipped. Adds SoftAsserter.Passf, unit tests for the trace-file metric parsing, and a registration-order test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fold the versioning half of the legacy prepare-images helper into the scenario: a prepare-test-images setup step (after prepare-hc) ensures the versioned COSIs the A/B sequence requests exist in the test image dir. v1 (<type>.cosi) and v2 (<type>_v2.cosi) are distinct real images that must be provided; v3+ are created as hard-links following prepare-images' scheme — odd versions alias v1, even versions alias v2 — so each version's filesystem UUID differs from the active volume. No-op for non-A/B configs and OCI-hosted images (staged in ACR by the pipeline). This removes the local requirement to hand-create image_vN hard-links: a scenario now runs from just the two real images. It is idempotent, so it is a no-op in the pipeline where download-test-images already materializes v3/v4 via 'prepare-images -v 4' (that shared step still serves the legacy path, so it stays until legacy retirement). Adds unit tests for the hard-link scheme (odd->v1, even->v2, existing files left as-is, missing source errors) and a registration-order test. Locally validated base_vm-host end-to-end from only v1+v2: prepare-test-images created v3, and auto-rollback + ab-update-2 (which consume v3) passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The rollback test logic already lived in storm, but its orchestration was still spread across two YAML templates, so the scenario could not run on a dev box without replicating pipeline steps by hand. Derive the test profile from the image flavor. The skips, UKI and secure boot settings were a pure function of the flavor, computed by a 39-line chain of bash conditionals. A single --flavor argument now determines them in Go, removing that bash along with the skipExtensionTesting, skipRuntimeUpdateTesting, skipNetplanRuntimeTesting and isUki parameters. The mapping is behavior preserving and locked by tests, including the quirk that the "qemu" flavor keeps secure boot despite setting UKI (the old bash gated only on the literal name "uki"). An unknown flavor is now rejected at parse time instead of silently ignored. Self-provision the test sysext images. prepare-qcow2 and the extension update both require test-sysext-N.raw in the artifacts directory, but they were only ever produced by a pipeline step that ran a script and moved the output into place, so a local run failed later with a confusing "failed to find extension file". A prepare-extensions case now builds them directly into the artifacts directory, and is idempotent so reruns are cheap. It is skipped when extension testing is skipped, which the pipeline step could not do: it built all three images even for flavors that never use them. build_extension_images grows an output directory argument and stages its intermediate tree in a temp dir rather than the working directory, which is what forced the pipeline to move the files in the first place. Publish JUnit results. Rollback reported nothing, so a failure was an opaque non-zero exit; per-case results now reach the Tests tab. The serial console output storm embeds can carry control bytes that XML 1.0 forbids, so the file is scrubbed and its empty classname qualified before upload. Remove TestCaseToRun. Storm is a sequential runner whose cases share state, so selecting a single case is not meaningful, and nothing invoked it for rollback. It also marked every other case as skipped rather than filtering, polluting results with meaningless skips. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
While the legacy and storm E2E suites run in parallel, both push their test sysexts to sysext-<runtime> and derive the same tag from the shared build id. Each suite builds its own images, and mksquashfs embeds timestamps, so the bytes differ: the last pusher wins and the other job fails validation with a SHA384 mismatch. This took down the extensions config in both suites in build 1174767. The ACR cleanup was a second collision vector, since either suite could delete images the other was still using. Push to sysext-storm-<runtime> instead. The cleanup step follows automatically, as acr-push exports the repo it used via SYSEXT_REPO. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
frhuelsz
force-pushed
the
user/frhuelsz/storm-port
branch
from
August 4, 2026 18:26
3c9a81e to
ec782a5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🔍 Description