Build/Test Tools: Add plugin compatibility testing workflow.

[adamsilverstein]

Claude Code built this workflow, requirements and steering from me:

Adds a workflow that installs the top plugins from the WordPress.org directory one at a time against a given version of WordPress and checks that nothing fatals. The idea is to catch the WP Rocket class of breakage - a popular plugin fataling on every request against a new core version, taking real sites offline - while there is still time to fix core or reach the plugin author. See adamsilverstein#64 and https://x.com/austinginder/status/2090199834787541074

A Trac ticket is being opened for this and will be linked here once it exists. Core needs a ticket before anything can be committed, so this stays a draft until then.

How it works

plugin-compatibility.yml queries https://api.wordpress.org/plugins/info/1.2/ for the most popular plugins at run time, so there is no list to go stale, and splits the slugs into 5 shards. Each shard calls reusable-plugin-compatibility.yml, which installs WordPress with WP_DEBUG and WP_DEBUG_LOG on, starts php -S, and then for each plugin installs it, activates it, runs wp eval, requests / and /wp-login.php, checks wp-content/debug.log, and removes the plugin before moving to the next one.

WP_DEBUG_DISPLAY is left off on purpose so the site behaves the way a production site does - a fatal is an empty page and an HTTP 500 rather than a printed stack trace. The fatal error handler is disabled too, otherwise recovery mode swallows the fatal and deactivates the plugin mid-test.

Runs are workflow_dispatch (so a release lead can point it at a beta or RC) and weekly against nightly. It is signal-only, not a check on every commit - a third party plugin breaking shouldn't turn core CI red on unrelated work.

How has this been tested

The workflow has not run on GitHub Actions yet. Fork runs are blocked by the github.repository == 'WordPress/wordpress-develop' guards that the other workflows use, so the first real run will have to happen after this lands or via a temporary guard removal on a branch. Everything below was verified locally instead.

actionlint 1.7.12 with shellcheck 0.10.0 on PATH reports 0 errors for both new files, and 0 errors across the whole .github/workflows directory. zizmor 1.24.1 with --persona=regular --strict-collection reports one unpinned-images finding on the database service image, which is the same finding it already reports for install-testing.yml and reusable-upgrade-testing.yml, and it exits 0 in the --format=sarif mode the lint workflow uses.

The matrix builder was run locally against the live API. With plugin-count=100 it returned 100 slugs split into 5 shards of 20, each valid JSON. With plugin-count=10 it returned 5 shards of 2. With a count smaller than the shard count (3) it returned 3 shards of 1 rather than empty shards.

The per-plugin loop was extracted from the YAML and run unchanged in a container with WP-CLI, PHP 8.3 and MySQL 8.4, which is as close to the runner as could be managed locally:

• The top 10 popular plugins against nightly - all 10 passed, and the whole thing including the WordPress download took about 85 seconds, so a shard of 20 should sit well under the 20 minute target.
• A nonexistent slug was reported as SKIPPED and did not fail the job.
• Four deliberately fataling test plugins, one per detection path, all reported FAIL with the right reason: fatal on plugin load caught at activation, fatal on template_redirect caught as HTTP 500 on /, fatal on shutdown caught at activation, and a fatal in wp_footer (which still returns HTTP 200 because output already started) caught in debug.log.
• After a fataling plugin, wp-content/plugins and active_plugins were both back to a clean state and the next plugin in the list still ran and passed.

Not tested: multisite, PHP versions other than 8.3, MariaDB, and the Slack notification and failed-workflow jobs, which are copied from install-testing.yml unchanged.

Types of changes

• Add .github/workflows/plugin-compatibility.yml, the caller, which builds the plugin matrix and fans out to 5 shards.
• Add .github/workflows/reusable-plugin-compatibility.yml, which installs WordPress and tests one shard of plugins in isolation.
• Write a per-plugin results table to the workflow summary and fail the job only when a plugin fatals.

Open questions

• Is 5 shards of 20 the right shape? The count is an input, so a release lead could run 250 against an RC, but that would make each shard 50 plugins.
• Should this hook into .version-support-*.json for the PHP version instead of pinning 8.3, or is one current PHP version the right scope for a smoke test?
• Premium plugins like WP Rocket itself can't be fetched from the .org API, so the exact incident that prompted this wouldn't have been caught. Is there an appetite for a vendor-supplied zip input down the road, or does that raise too many licensing questions?
• Does any of this overlap with what Plugin Check or Tide already do? Nothing there appears to run plugins against unreleased core, but confirmation from someone closer to that infrastructure would help.

AI Use

Code and description written with 🤖 Claude Code, working from acceptance criteria in the linked issue. I will review and test.