[dolho]

Summary

pytest-randomly is kept out of tests/requirements-test.txt on purpose, because pytest auto-discovers installed plugins. Nothing enforced that: once the plugin landed in a local virtualenv, every local run was shuffled with no signal, and latent order dependencies got triaged as product regressions.

• No signal → refuse or announce. The new tests/testing_utils/randomly_guard.py is imported as testkit.randomly_guard, through the tracked testkit → testing_utils symlink.
  • If the plugin is active and the run did not opt in, pytest stops before collection with a UsageError that names both ways out.
  • If the run opted in, it prints a banner with the seed and how to reproduce or disable the shuffle. Opt-in means any of: --randomly-seed=N, -p randomly, TRINITY_RANDOM_ORDER=1, or PYTEST_ADDOPTS.
  • Opt-in is read from the invocation itself, because pytest-randomly resolves an implicit seed inside its own pytest_configure.
  • Both conftests call the guard with tryfirst, because the sealed unit/ tests never load the root conftest.
• Scripts pin deterministic order. run-core.sh, run-full.sh, run-integration.sh and run-smoke.sh pass -p no:randomly.

The acceptance criteria, item by item

• Banner or fail-fast: done, as above. Checked live in a venv with pytest-randomly 4.1.0:

  • a plain run refuses with the message;
  • -p randomly --randomly-seed=4242 prints the banner;
  • TRINITY_RANDOM_ORDER=1 and PYTEST_ADDOPTS both opt in.

• Scripts pin order: done for all four run-*.sh scripts.

• The three unit-tier order dependencies: no longer reproduce on dev. test_1081_physical_meter, test_2638_subscription_switch_on_turn and test_subscription_auto_switch_pingpong were run together under eight seeds: 1, 2, 3, 12345, 67890, 99999, 4242 and 777 (97/97 each). test_1081 is unchanged since the report, so this matches the sys.modules baseline restore in the unit conftest. I could not reproduce the reporter's own seed, so nothing is quarantined; if one comes back, the guard now tells you it was a shuffle.

• The four API-tier order dependencies: already fixed on dev by test: fix order-dependent and stale tests surfaced by the full suite run #3681 (2026-10-14).

  • test_folders_default_values, test_info_no_config and test_get_config_no_config use isolated_agent.
  • test_round_robin_distribution skips on a foreign-subscriptions precondition.

• Does CI's three-seed run hit these? Every CI job that installs pytest-randomly already passes -p randomly --randomly-seed=N:

  • backend-unit-test.yml: the head seed, the inline base and push-absolute runs;
  • backend-unit-nightly.yml and dev-nightly.yml: the unit matrix.

  The jobs that run pytest without a seed (dev-nightly's live suite, integration-nightly, journey-smoke, schema-parity, backend-image-smoke) don't install the plugin, so the guard does nothing there and no workflow changes are needed. The nightly three-seed runs on dev and on open PRs are what exercise order independence. Those seeds do not hit the reported orderings today, and this run reproduced none of them either.

Changes

• tests/testing_utils/randomly_guard.py (new), tests/unit/test_2891_randomly_guard.py (new, 24 tests).
• tests/conftest.py, tests/unit/conftest.py: wire the guard into pytest_configure and pytest_sessionstart.
• tests/run-{core,full,integration,smoke}.sh: -p no:randomly.
• tests/registry.json entry.

Behaviour change for local runs: in a venv with pytest-randomly installed, any pytest call that neither passes -p no:randomly nor opts in now stops with an explanation, instead of silently shuffling. That's the point of the issue.

Test Plan

• cd tests && pytest -p no:randomly unit/test_2891_randomly_guard.py: 24 pass. They also pass under -p randomly --randomly-seed=…, TRINITY_RANDOM_ORDER=1 and PYTEST_ADDOPTS; the refusal test now clears an inherited opt-in.
• Live checks of the refusal and the banner, as above.

Fixes #2891

🤖 Generated with Claude Code

[obasilakis]

Ran it in a scratch venv with pytest-randomly and xdist 3.6.1: a plain run refuses; -p randomly --randomly-seed=4242 passes with one banner; -n 2 passes (workers get the full args); -p no:randomly passes. The three unit files from the issue pass 97/97 together under 8 seeds.

Minor:

• tests/testing_utils/randomly_guard.py:96 — banner() docstring says call from a trylast pytest_configure; it is called from pytest_sessionstart.
• tests/testing_utils/randomly_guard.py:71 — the PYTEST_ADDOPTS check misses the one-token -prandomly form the argv check handles.
• The evidence for the three unit-tier deps is isolated runs; the failures came from a full unit run. A full unit/ run under one seed (or a green nightly) would close it.