[v3] feat: reference plugin packages, and the runner surface they need by monikon22 · Pull Request #48 · Drownek/plugwright

monikon22 · 2026-08-23T13:30:59Z

Second PR in the series (#46), on top of #47. Thanks for taking that one as a merge commit — the SHAs
survived, so this branch shows only its own 16 commits instead of replaying the whole of PR 1.

Rebased onto v3-dev at e89bde6, your chore: bump to 3.0.0-dev.0. No conflicts. Kotlin
compiles, and tsc --noEmit is clean in runner-package, auth-authme-package and
console-rcon-package.

PR 1 added the plugin host. This one ships two plugins written against it, and fills in the runner
API they turned out to need.

Opening this as a draft on purpose. Two questions below need your answer before it's ready to
merge: what npm scope the new packages publish under, and whether they share the runner's version
number or carry their own. Both are cheap to change now and expensive to change in PR 6. The code
is finished and reviewable in the meantime — I'll mark it ready as soon as you've picked.

The packages

@plugwright/console-rcon gives an environment a console over RCON, which is the only way to
reach a server plugwright did not start.

@plugwright/auth-authme answers a login wall. onPlayerCreate fires on every connection — the
first bot of a test, a second bot from createPlayer(), every player.rejoin(), the admin-bot
channel — so authentication happens once per connection rather than once per run:

export default definePlugin({
    name: 'authme',
    async onPlayerCreate(player, { account }) {
        if (account.auth === 'microsoft') return;
        // wait for the prompt, answer it, wait for the confirmation
    },
});

An environment declares them by name and the build installs them:

plugins {
    npm("@plugwright/auth-authme") { options["password"] = botPassword }
    local("stand-reset")   // src/test/e2e/plugins/stand-reset.ts
}

One consequence of taking your per-player message buffers: the AuthMe plugin matches prompts
against player.messageBuffer rather than a session-wide one. Two bots authenticating at the same
time used to be able to answer each other's prompt; now each only sees its own.

The scope name is your call

I named them @plugwright/*, and I should flag that, because your published package is
@drownek/plugwright. Merging this as-is means the repo publishes under two scopes.

My reasoning was that these are named after the project rather than after you, and that an org
scope is something you can grant other people access to later, if a third-party plugin ever wants
to live next to the reference ones. A personal scope can't do that. But it's a preference, not a
requirement, and it's your namespace, so I'd rather ask than assume.

Nothing under @plugwright is published today, so for these two packages there's no migration
either way — only a decision about what the first publish is called.

There's a second half to it, though, and I'd rather put it in front of you than let it arrive as a
surprise. If you do go with @plugwright, the runner should follow, or the split you were trying
to avoid just moves somewhere worse: two reference plugins under the project's scope and the
runtime they load into under a personal one. I'm happy to do that migration and I think 3.0 is the
moment for it, since it's already a breaking release and users are editing their build scripts
anyway. It's 47 references across 35 files, plus npm deprecate on @drownek/plugwright pointing
at the new name, and it should be its own PR rather than something smuggled into this one. The
Gradle plugin id and the me.drownek.plugwright Kotlin package would stay exactly as they are —
this is only about npm.

So the real choice is between one small rename now and one larger one later, or no renames at all.
I don't think either is wrong.

If you want to keep @plugwright, what it needs from you before PR 6 wires up publishing:

An npm org called plugwright, created under your account. The free tier covers unlimited
public packages.
Nothing in the manifests. Both already carry "publishConfig": { "access": "public" }, which
is what stops a scoped package defaulting to restricted on its first publish.
Publishing access for the release workflow. release.yml already asks for id-token: write
and passes --provenance, so I assume you're on trusted publishing rather than an
NPM_TOKEN. Worth checking how that behaves for a package that doesn't exist yet — as far as I
can tell a trusted publisher is configured per package, which would mean the first publish of
each has to go out from your machine (2FA, or a granular token) before
npm trust github @plugwright/console-rcon --file release.yml can take over. If you'd rather
not deal with that ordering, a granular token scoped to the org, wired as NODE_AUTH_TOKEN in
the publish step, covers the first release and you can move to OIDC afterwards.
Provenance is unaffected. It attests the GitHub repo, and both manifests already point
repository.url at Drownek/plugwright, so the npm scope not matching your GitHub name
doesn't matter.

If you'd rather stay on one scope, say so and I'll fold the rename into this PR:
@drownek/plugwright-console-rcon and @drownek/plugwright-auth-authme. That's 27 references
across 17 files — two package manifests, the example's package.json, four docs pages, three
Kotlin files and external.ts — mechanical, and it costs you no setup at all, since you already
own that scope.

Either answer is fine by me. I'd just like it settled here rather than in PR 6, where it would
mean rewriting the publishing commits.

And their version numbers

Same kind of question, so I'll ask it in the same place. Both new packages carry 3.0.0-dev.0
right now, the number in version.txt. The alternative is to start each at 1.0.0 and let them
move on their own schedule.

I tried it the other way first, and it went badly enough to be worth reporting: the two packages
sat at 1.0.0 for several runner releases, because the bump script only knew about
runner-package. Nothing in @plugwright/auth-authme@1.0.0 said which runner it was built
against, and by then there were three candidates. A shared number answers that by existing.

The cost of lockstep is real and I'd rather name it than sell around it. A plugin that hasn't
changed still gets a new version every time the runner is released, so its major number stops
meaning "this plugin's API broke". My read is that it doesn't cost much here, because the API
these two are written against is the runner's plugin host — when the runner takes a major, the
thing they implement usually did change. That's a judgement, though, not a fact.

Independent versions are a defensible position, but it's more than a different starting number.
The bump script has to carry three versions instead of one. The release has to work out which
packages actually changed since the last tag, because npm rejects a republish at the same version.
And the peerDependencies range stops being a formality and becomes a compatibility matrix you
maintain by hand.

Worth deciding now rather than in PR 6, because PR 6 assumes lockstep:
feat(bump): move the plugin packages with the runner teaches bump-version.js all three package
directories and refreshes the file: lockfiles that copy the runner's version. Changing course
later means rewriting that commit.

Runner changes that came out of writing the plugins

requires can demand a specific capability value, not just a capability name.
Op and command sync go through a console that answers, so they work on a server the runner does
not own.
The CLI is a bin entry.
Optional packages resolve from the test project rather than the runner's own tree.
Authentication happens before the spawn wait, not after — a login wall holds the spawn.

Keeping the password out of the log

/register and /login take the password as an argument, so the AuthMe plugin has to put it on
the wire in the clear. What was avoidable is the log: every run printed
Chatting: /register hunter2 hunter2 verbatim, and the reports and the CI output kept it.

chat now takes a list of values to redact from the line it logs:

player.chat(`/login ${password}`, { secrets: [password] });

The message sent to the server is untouched; only the logged copy changes, and it reads
/login [REDACTED]. The list comes from the caller on purpose. chat would otherwise have to
recognise every plugin's command shapes to find the secret argument, and a guess that misses fails
open — it prints the password. An empty list is the default, so existing calls are unaffected.

Worth being clear about the limit of this. It does not make the password a secret in any wider
sense: a bot has to send it as plain text over the protocol, and a plugin option still travels
into the generated config.json as a plain value. Carrying a secret reference there would mean
widening PluginRef.options from Map<String, String> to Map<String, Any>, which breaks the
published Kotlin API and gives up compile-time typing on plugin options, so I left it alone.
Passwords worth protecting belong in the environment's account pool, where they already travel as
a reference and are resolved only when a bot leases the account.

Two small corrections

Both new packages now carry 3.0.0-dev.0 rather than the stale 2.0.4-dev.0 they were written
at, and their peerDependencies on @drownek/plugwright is >=3.0.0-dev.0. They call
definePlugin, which doesn't exist before 3.0, so the >=2.0.0 I originally had there would have
allowed a runner that can't load them. scripts/bump-version.js still only bumps
runner-package; teaching it about the other two is PR 6's job, so I left it alone rather than
half-wiring it here.

The second one is already on v3-dev, from PR 1. ExternalMode asks npm for
@plugwright/console-rcon@^1.0.0, a range I wrote before the package existed. The package ships
at the project version, so that range resolves to nothing: the npm install --no-save fails,
installMissingRunnerPackages logs a warning and carries on, and the environment then reports the
package as missing at run time. Dropping the version leaves it as "whatever the test project
already has", which is what the two @drownek/plugwright refs beside it already do. The example
never hit this, because it links all three packages with file:.

Docs

docs/environments.mdx, docs/external-servers.mdx, docs/plugins.mdx, docs/custom-modes.mdx
and docs/reports.mdx are new. configuration.mdx, examples.mdx and test-filtering.mdx are
updated.

Testing

example_plugin now runs its suite twice: against the Paper server plugwright starts, and against
a second server started by hand from start.sh, reached over RCON with an admin bot. The second
one is what keeps ExternalMode honest.

Merging

Same request as last time, for the same reason: merge commit or rebase and merge, not
squash and merge. PRs 3 through 6 are stacked on this branch, and squashing would collapse
these 16 commits into one, so the next PR would show all of this again under new SHAs. Squashing
is fine on the last PR of the series, or when v3-dev goes to master.

I'll rebase PR 3 onto v3-dev and open it once this merges.

Implements ServerConsole over the Source RCON protocol directly on top of net.Socket - no third-party dependency, and kept out of @drownek/plugwright's own dependency list since it's only needed when a build script declares console { rcon { ... } }. executeAndWait resolves from the server's own response packet, so it doesn't need the minecraft:say <syncId> round-trip the stdio and admin-bot consoles rely on.

@plugwright/auth-authme: on every bot connection - initial join, rejoin, and the external mode's admin-bot console, all of which go through the same onPlayerCreate hook - waits for the login or register prompt and answers it: /register <pass> <pass> for a freshly generated account (account.justCreated), /login <pass> otherwise. Commands and prompt/success patterns are configurable options; Microsoft accounts are skipped since AuthMe never prompts them. Ships a preflight spec (auth.spec.js) so a broken login flow surfaces as a named failure at the top of the report instead of buried in the first user test that happens to create a bot.

runnerPackages() was declared by every mode and read by nobody, so an optional runner package such as the RCON console could never actually reach the test project. plugwrightCompileTests now installs the ones missing from node_modules, merged across environments, and only warns when an install fails: the runner already reports the missing package with the context to fix it.

A dynamic import of a package this one doesn't depend on — an optional console, a third-party mode, a plugin — resolved relative to the runner's own location, which finds nothing when the runner is a linked checkout instead of an entry in the test project's node_modules. Fall back to resolving from the test project, and include the underlying error in the missing-package message.

PluginsSpec was in plugwright-external, so `plugins { npm(...) }` only existed on external environments. Nothing about a runner plugin is mode-specific: a local server running an authentication plugin needs the login hook exactly as much as a remote one does. Move the spec into plugwright-api and give LocalEnvironmentSpec the same block. An npm-named plugin now also joins the environment's runner packages, so plugwrightCompileTests installs it instead of leaving the runner to fail on a package nobody fetched. A plugin given as a path is left alone.

The runner resolves `local` and `external` by mode id, since both are compiled into it. Anything else has to be imported from a package, and the config file had no field saying which one — so a custom mode always died with "mode X, which this runner cannot run yet". The first RunnerPackageRef naming an export is that package, so write it into environment.runtime for every mode but the two built-in ones. Also fixes the misspelled "Environment sumarries" header.

join() waited for the spawn event and only then fired onPlayerCreate. A server with a login wall never spawns an unauthenticated player, so the hook that would have logged the bot in never ran and every test died on a 30s spawn timeout. Wait for the play state instead, run the hook there, then wait for the spawn. Message listeners now go up before the first await as well: the login prompt arrives immediately, and a prompt that lands before the buffer exists is one no authentication plugin can answer.

makeOp waited for "Made X a server operator" in the player's chat, and deOp waited for a `say` marker to come back. Both assume the whole server log reaches the bot, which is true for the stdio console and false for RCON: the answer goes back over the RCON socket, so every op-dependent test timed out against an external server. When the console returns responses, run the command through it and read the answer. Also expose executeAndWait on ServerWrapper, and report op: true for an external environment once a console channel answers — having a console is what being able to op means.

requires: ['console'] is satisfied by an RCON console, which answers its own commands and nothing else. Reading the server log needs more than that, so a log matcher on such an environment neither skipped nor worked — it timed out after the full assertion timeout with no explanation. requires now accepts 'key:value' ('consoleOutput:full'), and the server log matcher fails immediately with the level it found and the requires clause that would have skipped it.

The runner takes a --config file and needs no Gradle, but there was no bin entry, so `npx plugwright --config …` did not resolve to anything.

account.justCreated said whether to register or log in. It is a hint from the account pool and it is wrong the moment a pool account outlives the run that created it, which is the second run against any stand: the plugin sent /register for an account the server already knew. Wait for either prompt and answer the one that arrived, register first since AuthMe's register prompt mentions the password too. Match only messages newer than the step they belong to — a greeting with "welcome" in it was passing for a login confirmation, and tests started before the player could run a command. After registering, wait for the login AuthMe performs itself, or send it when it doesn't. Adds a `password` option for accounts an environment invents rather than leases, which is how the local mode names its throwaway bots.

Five pages for what the last phases added: how environments and modes relate and what tasks each produces, what changes when the server isn't yours, the runner plugin contract, the report formats, and a guide for writing a mode of your own. Configuration keeps its flat-property reference and gains the environments block that supersedes it; test filtering gains the capability and environment filters. READMEs updated to match.

The example described one implicit local server. It now declares two environments explicitly: `local`, which downloads Paper and installs AuthMe next to the plugin under test, and `stand`, which connects to a server started by hand from the same run directory and leaves it running. `local` writes an AuthMe config a bot can get through — the stock one asks for the password in a dialog, allows one registration per IP, and treats a test suite as a bot attack — and logs every bot in through @plugwright/auth-authme. `stand` leases four accounts from a pool, reaches the console over RCON, and resets op and inventory between tests with a local plugin, since a leased account carries over whatever the last test left on it. What it cannot reset is excluded by name; the one test that reads the server log now says requires: ['consoleOutput:full'] and skips there instead. 47 pass on local, 33 pass and 14 skip on the stand.

`chat` logs every message it sends, which is what makes a run readable — until a plugin sends a credential. An authentication plugin has no choice but to put the password in a chat command, and the log then carries it in the clear. `chat(message, { secrets: [...] })` redacts the listed values from the logged copy only; the server still receives the message unchanged. The list comes from the caller because the caller is the only one who knows: `chat` would otherwise have to recognise every plugin's command shapes, and a guess that misses fails open — it prints the secret. An empty list, which is the default, redacts nothing and leaves existing calls as they were.

Both chat calls put the password on the wire in the clear, because AuthMe's `/register` and `/login` take it as an argument and there is no other way to answer them. What was avoidable is the log: every run printed `Chatting: /register hunter2 hunter2` verbatim, and the reports and CI output kept it. The calls now declare the password as a secret, so the logged copy reads `/register [REDACTED] [REDACTED]` while the server still receives the real command. This does not make the password a secret in any wider sense — a bot has to send it as plain text over the protocol, and a plugin option still travels as a plain value into the generated config. Accounts whose password is worth protecting belong in the environment's pool, where it travels as a reference and is resolved only when a bot leases it.

…nnot match ExternalMode asked npm for "@plugwright/console-rcon@^1.0.0", written before the package existed. The package ships at the project version, so that range never resolves: the install fails, the task logs a warning, and the environment reports a missing package at run time. Leaving the version off means "whatever the test project already has", which is what the two @drownek/plugwright refs beside it already do.

Drownek · 2026-08-23T13:56:20Z

Decisions:

Scope: I grabbed the @plugwright org on npm, so let's go with Option A and publish everything under it. And yeah, 3.0 is the right time to move the main runner package over too - include that migration wherever it fits (PR 6 works).

Versioning: lockstep, agreed. Same version number across plugins and runner saves us compatibility headaches later. Keep it at 3.0.0-dev.0.

On publishing (Trusted Publishing vs Token): to sidestep OIDC bootstrapping issues on brand-new scoped packages, I'll generate a granular NPM token scoped to the org and wire it as NODE_AUTH_TOKEN in GitHub Secrets for the first release. Once they've published once, we can move to OIDC.

Mark it ready for review and I'll merge.

Drownek · 2026-08-23T14:08:11Z

Since I've got maintainer perms, I just marked it ready and merged it myself - figured that beats waiting on a round trip.

v3-dev's prepped. Go ahead and open next PR whenever.

monikon22 added 16 commits August 23, 2026 15:59

chore(runner): expose the CLI as a bin entry

71d68ef

The runner takes a --config file and needs no Gradle, but there was no bin entry, so `npx plugwright --config …` did not resolve to anything.

Drownek marked this pull request as ready for review August 23, 2026 14:07

Drownek merged commit 67e78ce into Drownek:v3-dev Aug 23, 2026

monikon22 mentioned this pull request Aug 23, 2026

[v3] feat: a fixed directory layout for the test workspace #49

Merged

Navigation Menu

[v3] feat: reference plugin packages, and the runner surface they need - #48

[v3] feat: reference plugin packages, and the runner surface they need#48
Drownek merged 16 commits into
Drownek:v3-devfrom
monikon22:pr/2-plugin-packages

monikon22 commented Aug 23, 2026 •

edited

Loading

Uh oh!

Drownek commented Aug 23, 2026

Uh oh!

Drownek commented Aug 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

monikon22 commented Aug 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The packages

The scope name is your call

And their version numbers

Runner changes that came out of writing the plugins

Keeping the password out of the log

Two small corrections

Docs

Testing

Merging

Uh oh!

Drownek commented Aug 23, 2026

Uh oh!

Drownek commented Aug 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

monikon22 commented Aug 23, 2026 •

edited

Loading