test(system): chain tests on a rolled-out fleet, with a per-chain go/no-go report (abilityai/trinity-enterprise#794)

[dolho]

Summary

These are the 1.0 system test's chains (ent#783). Each chain checks that a change in one part of a rolled-out system shows up in another part. A chain passes only on records the system wrote: platform rows, files the platform wrote into a container, and the platform's own responses. It never passes on an agent's report that it is done.

Chain	On dev
J19 An objective reaches the person	Runs on any stack with Docker. All six platform steps passed on a live stack. The brief step is recorded as not run, so the chain reports partial (see below).
J18 A library change reaches every holder	Runs when given a test skills repo (CHAIN_SKILLS_REPO, CHAIN_SKILLS_GITHUB_TOKEN); otherwise not run
J15 Roll-out · J16 Canon fact · J17 Fact routing	Skeletons that report not run, blocked by ent#793 (the reference company) and ent#804 (change propagation / fact routing)

How it works

• tests/system_chains/

  • A report plugin plus the five chains, which are J15–J19 in tests/journeys/catalog.yaml.
  • They are collected only when TRINITY_CHAIN_TESTS=1, so they never run inside run-full.sh or the per-PR journey lane.
  • The journey tier's create, poll and key helpers are reused rather than duplicated.

• The report (chain-report.json and chain-report.md) gives each chain one verdict:

  • passed;
  • failed, at a named step;
  • partial;
  • not run, with a reason.

  Only passed counts as evidence. A skip, a step that couldn't run, or a test that recorded no step is never reported as passed. The summary is a go only when every chain passed.

• scripts/system/run_chains.sh runs the chains against a target and exits 0 only when every chain passed.

• .github/workflows/system-chains.yml is manual-dispatch: it boots a ref, runs the chains and uploads the report. GitHub registers manual dispatch only from the default branch, so this can't be dispatched until the next release cut carries it to main. The exception is recorded in the bug(ci): issue promotion has not run since #2769 — the pull_request_target trigger is inert until it reaches main #2814 parity guard; until then, use run_chains.sh.

• Docs: docs/testing/SYSTEM_CHAINS.md, and a regenerated docs/testing/JOURNEYS.md.

Findings from building it

1. J19 brief step. The platform puts no objective number into a brief, so any number in a brief is written by the model. The chain doesn't accept that, so it reports this step as not run (verdict partial) until the platform carries the number. Follow-up to file.
2. J18 and fleet re-inject. Re-injecting agents after a library sync is off by default (skills_library_auto_reinject_enabled). With it off, a library change reaches an agent that holds the skill only when that agent restarts. The chain turns it on for the run, says so in the report, and restores the old value afterwards.
3. Soft-deleted agents keep their seat. J19's first real runs showed this: an agent that holds a seat (ent#811, abilityai/trinity-enterprise#818) still holds it after it's deleted, because the seat-holder row is only cleaned up at the hard purge. A new agent then can't take that seat (409 naming the deleted agent). I'm reporting it on fix(security): sandbox voice panel HTML in iframe (CRITICAL XSS) #818. The chain now uses a fresh seat id per run, so it doesn't depend on earlier runs cleaning up.
4. Readiness race. An agent reports running a moment before its own server accepts file writes (503). The chains wait for the server the way any client must.

Test Plan

• tests/unit/test_ent794_chain_report.py has 9 tests on the verdict rules. As a mutation check, I made partial report as passed; 2 of them go red.
• Guards pass: catalog (test_2338, with the count bumped to 19), Python-version parity, workflow trigger parity, and root test placement (100 passed).
• Live run: scripts/system/run_chains.sh -k j19 against a dev stack.
  • J19 is partial: 6 of 6 platform steps passed and the brief step was not run. Exit code 3, as designed.
  • The full run reports J15/J16/J17/J18 as not run, each with its reason.
• J18 needs a test skills repo before it can run (see above).

Refs abilityai/trinity-enterprise#794. This is partial: the issue stays open for J15–J17, which depend on ent#793 and ent#804, and for J18's repo.

🤖 Generated with Claude Code

[dolho]

Live test matrix against a local dev instance (2026-10-07)

I ran scripts/system/run_chains.sh against a running dev stack. The stack was v1.0.0-aws.2, with the #3271 assignments and seat code loaded.

#	Case	Expected	Result
A	Collection gate off (TRINITY_CHAIN_TESTS unset)	nothing collected	✅ 0 collected
B	Runner without a password	exit 2 with a message	✅
C	Full run	J19 partial at the brief step; J15–J18 not run with reasons; exit 3; nothing left behind; the re-inject setting unchanged	✅
D	J19 with an impossible stale deadline	failed at "when recording stops, the number reads stale on the card"; exit 1; agent cleaned up	✅ after fix
E	J18 with an unreachable repo and a bad token	not run, with the reason	✅ after fix
F	Wrong admin password	all 5 chains reported failed with "setup failed before the chain started: … 401"; exit 1	✅ after fix
G	A second run straight after (fresh seat and objective ids per run)	partial again; nothing left behind	✅
H	CHAIN_KEEP_AGENTS=1	agent kept and listed in the report as kept agents	✅

Defects the matrix found, now fixed in 4fbfd5125

• F: when the suite-wide login failed, every chain errored before its fixture ran. The chains were missing from the report, and the empty report exited 3 ("not yet"). A chain that dies that early is now reported failed, built from its marker, and the reason is pytest's crash line. A report with no chains exits 1.
• E: a network error during J18's repo precondition check read as a failure. An unreachable repo now means the chain is not run; the check retries three times first.
• D: a real failure and a partial or not-run chain both exited 3. The exit codes are now 0 (all passed), 1 (a chain failed, or no chain was reported) and 3 (only partial or not run).

What J19 left on the kept agent, read back from the platform: the agent holds chain-pipeline-owner-* (seat_source: holds). The objective "Reach 100 qualified pipeline this quarter" is owned, and chain_pipeline reads 42 / 100, stale: true.

🤖 Generated with Claude Code

[github-actions]

⚠️ Nightly unit-suite found regressions when this PR is merged into dev (3 of 3 seeds).

Regression details (head_sha: `a8bae72ee49c1b6db48c57ded4adc05d34b9862e`)

Seed 12345

Backend unit-suite regression diff

Per-XML totals

Side	Path	Total	Pass	Fail	Error	Skip
base	junit-base-pr3331-12345.xml	24211	24093	0	0	118
head	junit-head-pr3331-12345.xml	24223	24101	4	0	118

❌ New failures introduced by HEAD (4)

Tests failing under HEAD that did not fail under BASE in any seed:

• [F] test_2080_harness_contract::test_every_test_directory_is_covered_by_a_tier
• [F] test_2080_harness_contract::test_the_directory_guard_passes_on_the_real_tree
• [F] test_2339_testing_docs_consolidated::test_docs_testing_top_level_is_exactly_the_live_set
• [F] test_lint_sys_modules::test_committed_baseline_matches_current_repo_state

Legend: [F] = assertion failure, [E] = collection or fixture error.
Identity = (classname, name, kind); union taken across all input XMLs.

---

Seed 67890

Backend unit-suite regression diff

Per-XML totals

Side	Path	Total	Pass	Fail	Error	Skip
base	junit-base-pr3331-67890.xml	24211	24093	0	0	118
head	junit-head-pr3331-67890.xml	24223	24101	4	0	118

❌ New failures introduced by HEAD (4)

Tests failing under HEAD that did not fail under BASE in any seed:

• [F] test_2080_harness_contract::test_every_test_directory_is_covered_by_a_tier
• [F] test_2080_harness_contract::test_the_directory_guard_passes_on_the_real_tree
• [F] test_2339_testing_docs_consolidated::test_docs_testing_top_level_is_exactly_the_live_set
• [F] test_lint_sys_modules::test_committed_baseline_matches_current_repo_state

Legend: [F] = assertion failure, [E] = collection or fixture error.
Identity = (classname, name, kind); union taken across all input XMLs.

---

Seed 99999

Backend unit-suite regression diff

Per-XML totals

Side	Path	Total	Pass	Fail	Error	Skip
base	junit-base-pr3331-99999.xml	24211	24093	0	0	118
head	junit-head-pr3331-99999.xml	24223	24101	4	0	118

❌ New failures introduced by HEAD (4)

Tests failing under HEAD that did not fail under BASE in any seed:

• [F] test_2080_harness_contract::test_every_test_directory_is_covered_by_a_tier
• [F] test_2080_harness_contract::test_the_directory_guard_passes_on_the_real_tree
• [F] test_2339_testing_docs_consolidated::test_docs_testing_top_level_is_exactly_the_live_set
• [F] test_lint_sys_modules::test_committed_baseline_matches_current_repo_state

Legend: [F] = assertion failure, [E] = collection or fixture error.
Identity = (classname, name, kind); union taken across all input XMLs.

Reproduce locally: git merge dev && ( cd tests && python -m pytest unit/ -m "not slow" -p randomly --randomly-seed=12345 )

[vybe]

merge-train: ejected from this batch — rides the next train once fixed. The Tier 1 failures are this PR's own:

1. lint (sys.modules pollution check): tests/unit/test_ent794_chain_report.py:22 — bare sys.modules[...] assignment; use monkeypatch.setitem.
2. regression diff — 4 new failures: test_2080_harness_contract::test_every_test_directory_is_covered_by_a_tier and ::test_the_directory_guard_passes_on_the_real_tree (the new tests/system_chains/ needs a tier), test_2339_testing_docs_consolidated::test_docs_testing_top_level_is_exactly_the_live_set (new docs/testing/SYSTEM_CHAINS.md), test_lint_sys_modules::test_committed_baseline_matches_current_repo_state.

The body also has no closing keyword — add Fixes abilityai/trinity-enterprise#<N> so the issue promotes on merge.

[vybe]

/validate-pr (Lane C, because of the new workflow): rebased onto dev, with the registry.json conflict resolved by keeping both sides. Fixed the merge-train ejection causes in 549f222: system_chains/ is now an opt-in NON_TIER_DIRS entry; the chain doc moved to tests/system_chains/README.md, with a pointer in STRATEGY.md, so docs/testing/ stays the #2339 live set; and the sys.modules registration is scoped through MonkeyPatch.context. All checks green, including regression diff and the sys.modules lint. The workflow is dispatch-only with contents: read, and inputs.ref is used only in with:. Security greps came back clean.