dba-runner

dbaai_bench runs one model against one task on servers you already have. This runs many models against many tasks on servers it creates itself, marks the results with its own commands, and destroys the servers afterwards.

uv sync                                     # installs ../dbaai_bench as an editable checkout
uv run python dbrun.py suites               # what tests exist
uv run python dbrun.py run -m sonnet,gpt-oss-120b -s mysql-install

One run leaves a directory like this:

runs/20260822-143512/
  run.json          what was asked for, and the exact command that asked
  results.jsonl     one line per finished cell, appended as it happens
  leaderboard.md    the table, the tables that explain it, and the tasks it scored
  results.csv       one row per stage, for a spreadsheet
  events.jsonl      droplets created and destroyed, with their ids
  known_hosts       this run's host keys, not yours
  cells/<model>/<suite>/01-<stage>/     <suite>-<os>-<db> for whichever axis is wide
      task.md         the task exactly as the model received it
      grade.json      every check, what it returned, and the score
      bench.stdout.txt
      bench/          the do-dba run directory: transcript.jsonl, report.md, secrets.json

What it is doing

• A cell is one model against one suite, on one operating system, for one database. It owns its own droplets, its own subprocess and its own directory, so cells run four at a time and a model that wedges costs one cell.
• A suite is one TOML file in suites/. It says how many servers the work needs and lists the stages in order.
• A stage is one task handed to the model, plus the checks that mark it. One stage means a fresh-droplet single task. Several stages mean the same droplets, in order - which is how you test work that builds on itself: set up replication, then upgrade both servers underneath it.
• A lease is the droplets a cell holds from its first stage to its last.

Each stage ends with two verdicts, and they are allowed to disagree:

status	what do-dba concluded from the model's own VERIFY commands - done, unverified, exhausted, ...
score	the fraction of check weight the runner's own commands found satisfied

A stage that reports done and scores 40% is the interesting case: the model convinced the harness and not the suite. The leaderboard lists every one of them under Said done, was not. The checks were written before any model saw the task, and they run over the runner's own SSH connection after the model's process has exited.

Getting set up

Two things are needed, and the second one is optional.

A model gateway. The same keys dbaai_bench uses - OPENROUTER_API_KEY, or DIGITALOCEAN_INFERENCE_KEY with --provider digitalocean. The runner reads its own .env first and the bench's second, so a key that already works there works here - this project's .env only needs the names it is actually overriding. A name left blank in it is dropped rather than read as an empty answer, so it falls through to the bench's file instead of shadowing the real key there.

doctl, to create droplets.

winget install DigitalOcean.doctl        # or: scoop install doctl
doctl auth init                          # paste a personal access token
doctl account get                        # check it

A token in $DIGITALOCEAN_ACCESS_TOKEN is used in preference to whatever doctl auth init stored, so a run can be pointed at a different account without touching your own login. It is passed to doctl through the environment and never as an argument, because arguments are visible in the process list.

Skip all of that to run against servers you already have:

uv run python dbrun.py run -m sonnet -s mysql-install --host root@203.0.113.10

Those are never destroyed, and every cell gets the same machines one after another - fine for trying the runner out, not a matrix.

Running a matrix

# every suite in suites/, three models, four cells at a time
uv run python dbrun.py run -m sonnet -m gpt-oss-120b -m qwen3-max

# one suite, models from a file, a hard cost ceiling
uv run python dbrun.py run --models-file models.txt -s mysql-replication --max-cost 20

# see what it would do and spend nothing
uv run python dbrun.py run -m sonnet --dry-run

# mix gateways in one run: provider:model overrides the default
uv run python dbrun.py run -m sonnet -m do:openai-gpt-oss-120b

# the same models and suites on three operating systems, compared
uv run python dbrun.py run -m sonnet -s mysql-replication \
    --os ubuntu-26 --os debian-13 --os rocky-9

# and the same again across three databases
uv run python dbrun.py run -m sonnet -s mysql-replication \
    --db 9.7 --db percona-8.4 --db mariadb-11.4

# one task, three engines: the install suites are written check for check
uv run python dbrun.py run -m sonnet \
    -s mysql-install -s postgres-install -s mongodb-install

Model names are resolved against the gateway's catalogue before anything is created, the same partial-name matching dba.py -m does. A typo fails in two seconds instead of after forty droplets.

Suites are matched by name, unique prefix or path, so -s postgres-t finds postgres-tune. A prefix that matches several suites is refused and says which, which is also the quickest way to remember the names: -s mongodb lists the three MongoDB suites rather than guessing between them. With no -s, every suite in suites/ runs.

The matrix is built suite-major: every model gets tried on the first suite before any model sees the second. A run stopped early is then a comparison rather than one model's complete row.

Ctrl+C sets a flag every cell polls. Running stages are killed, pending cells are cancelled, the droplets are destroyed, and the cells that already finished are tabulated - then --resume picks up the rest:

uv run python dbrun.py run -m sonnet -m opus --resume 20260822-143512

--resume skips the (model, suite, OS, database) cells that finished - the same pair on the run's other image, or its other database, is still owed its droplets. A cell the runner broke - a timeout, a droplet that never came up - is not counted as finished, because there is nothing to learn from it.

Operating systems

--os (spelled --image if you prefer) is repeatable, and each one it is given is another axis of the matrix: cells become models x suites x OSes, one OS at a time within each suite, and one leaderboard compares them. Given one --os, or none (the default is ubuntu-26-04-x64), nothing changes - the OS is recorded but stays out of the names and the tables, because one OS is not a comparison.

• Friendly names. rocky-9 resolves against your own account's image list, so it finds rockylinux-9-x64 without you looking it up. A name that matches several images, or none, fails before a droplet exists and says which images it was choosing between.
• A suite may decline. os_family says which package manager a suite's shell was written for, and a suite that cannot run on one of the run's images is left out of that column with a not run: line on the plan, rather than being graded on a dnf it never had. If that empties the matrix, the run stops instead of starting.
• A suite may pin. A suite with its own image runs once, on that image, whatever --os says - it is testing something about that OS specifically.
• The OS is part of a cell's identity: droplet names, directory names, the row labels, results.jsonl, the CSV and --resume all carry it.

--os and --host are mutually exclusive: an operating system is something the runner chooses when it creates a server, and your own servers already have one.

Databases

Six products across three engines, and none of them is a drop-in for any other. --db (spelled --mysql if you prefer - the axis was called that when MySQL was all it had) is repeatable the same way --os is, and each value is another axis of the matrix:

uv run python dbrun.py run -m sonnet -s mysql-install \
    --db 9.7 --db percona-8.4 --db mariadb-11.4

# the same question, asked of the other two engines
uv run python dbrun.py run -m sonnet -s postgres-tune --db pg-18 --db pg-16
uv run python dbrun.py run -m sonnet -s mongodb-install --db mongodb-8.0 --db psmdb-8.0

Whatever you write is what the models are asked for. There is no list of versions to be on and no list of names to be on, because a benchmark whose vocabulary lags its subject cannot ask the only question worth asking the week MariaDB 12.3 ships.

written	means
mysql-9.7, percona-8.4, mariadb-11.4	that flavour at that version
pg-18, postgres-17, mongodb-8.0, psmdb-8.0	the other two engines, in the words their own documentation uses
mariadb-12.3, mysql-8.0.36, pg-19	the same, at any version you like - nothing is checked against a list
percona, mariadb, postgres, psmdb	that flavour at its newest version here, which is the one thing the built-in versions are still for
9.7, 8.4	that version of MySQL Community - a bare number is a MySQL, because that is where this axis started
"Percona Server 8.4", "MariaDB 11.4 rc"	your words, reaching the task exactly as written
"FerretDB 2.0"	a product this runner has never heard of

One token is the shorthand: mysql (Community), percona (Percona Server for MySQL), mariadb, postgres, mongodb (Community), percona-mongodb, the vendors' own words (percona-server-for-mysql, maria-db, postgresql, pgsql, psmdb, mongodb-org) and a version if you want one. It resolves to the vendor's own name for the product, so --db psmdb-8.0 asks the model for "Percona Server for MongoDB 8.0".

Anything with a space in it is prose, kept word for word: --db "Percona Server 8.4" asks for exactly that and is recorded as percona-server-8.4. The runner reads three things out of it and edits nothing - a trailing version if the last word looks like one, which of the six flavours it is, and which engine that flavour belongs to, because grading has to know both what a server should look like and which suites could honestly mark it. The words are read most-specific-first and the MySQL ones last, since "Percona Server for MongoDB" contains percona and is not a MySQL. A name with none of them in it is a flavour of its own, identified by grepping whatever the server says about itself - VERSION(), version(), db.version() - for the name itself, and offered to every suite on the axis, because which engine somebody meant by "FerretDB 2.0" is not something a runner should be guessing.

Two things are still refused, before anything is created: a value with no name in it, and two values that would be recorded as one row (percona-8.4 and "Percona 8.4" are both percona-8.4 on a leaderboard). Everything else the plan says rather than refuses, because a typo and a product released last Tuesday are the same string:

  db       myqsl-8.4
  myqsl-8.4: 'myqsl' is not a flavour anything here knows, so every suite on the
  --db axis - whichever database it is written for - will be asked for 'myqsl 8.4'
  and will identify what turns up by grepping the version it reports for 'myqsl'
  - did you mean mysql?

A version outside the built-in list gets the quieter version of that, naming the ones this runner has been kept up to date with. Both land above the confirmation prompt, which is the point: a database nobody can install grades every model at zero and looks exactly like every model failing.

One --db, or none, changes nothing about the matrix - the database is recorded on every row but stays out of the cell names and the table headings, which have nothing to distinguish. The default is one per engine (mysql-9.7, postgres-18, mongodb-8.0), so a bare dbrun run asks each suite for its own engine's. The plan still prints the db line whenever it is not the default, because one database is a decision even when it is not an axis.

• The axis is really one axis per engine. --db percona-8.4 --db psmdb-8.0 does not cross every suite with both: nothing can grade both, and a MySQL suite asked for MongoDB would score zero for having been asked the wrong question. So a suite takes the values on its own engine's line, and an engine the run never named falls back to that engine's default rather than dropping its suites - --db mariadb-11.4 says which MySQL to compare and says nothing at all about MongoDB, and reading it as "no MongoDB today" would shorten the matrix by six suites without a word.
• A suite may decline. db_flavour lists the flavours a suite's checks can honestly mark, and one the suite has not claimed is left out with a not run: line on the plan. mysql-tune's tuning stage rests on performance_schema.variables_info, which MariaDB has not got, so grading it there would score a MariaDB tuner zero for MariaDB not being MySQL. A flavour from outside the six is nobody's to decline - no suite could have named a product that did not exist when it was written - so every suite on the axis is asked for it, and the warning above is where the doubt goes.
• A suite may sit the axis out. No db_flavour at all means the suite is not on this axis: it runs once however many databases the run compares, and its rows record no database. That is mysql-restore, whose cloud-init installs whatever the distribution calls mysql-server. Note the asymmetry with os_family, where saying nothing means runs on anything - a suite that mentions the product but declares no flavour, or declares flavours and never mentions the product, is refused at load time rather than run.
• The database is part of a cell's identity: droplet names, directory names, the row labels, results.jsonl, the CSV and --resume all carry it.

--db and --host are not mutually exclusive, unlike --os: the product is what the task asks the model to install, not what the server already boots.

Money

Two meters run at once, and the runner shows both.

• Models. --max-cost USD is the run's ceiling: once spend reaches it no new cell starts, and the report says which were skipped. A cell already running is left alone - it is bounded by its own --max-cost-per-stage, and killing a cell halfway leaves a half-built server and a result worth nothing. Suites can set their own per-stage cap, and most of the ones here do.
• Droplets. Billed by the hour whether or not anybody remembers them. Every droplet is tagged dbaai-runner, dbrun-<runid> and suite-<name>, and its id is written to events.jsonl before the model is handed anything. So whatever happens to the process:

uv run python dbrun.py reap                        # anything this runner ever left behind
uv run python dbrun.py reap --run 20260822-143512  # just that run's
uv run python dbrun.py reap --dry-run              # list them and stop

--keep leaves the droplets up for a post-mortem. It is the one way this runner spends money after it exits, so it says so on the way in and on the way out.

--effort {low,medium,high} is the one flag that changes what a run costs without changing anything the plan counts. It reaches every stage as dba.py --effort, asking the model to think before each step, and the gateway turns that one word into whatever the model upstream wants - so it is worth comparing models on rather than a knob only one of them has. Thinking is billed as output tokens: expect several times the model spend of the same matrix without it, and expect the suites' own max_cost ($1.50 to $5 a stage across the ones here) to be reached sooner, which shows up as stages stopping on cost rather than on a verdict - so the plan warns about it above the confirmation prompt, beside the cost caps. Unset is the absence of the flag rather than a word meaning none: a reasoning model goes on reasoning at the gateway's default, exactly as in every run recorded so far. The effort is part of each stage's recorded command line in grade.json, so a cell that thought is distinguishable from one that did not long after the run.

Writing a suite

A suite is a TOML file in suites/. The short form is one task:

name = "mysql-install"
description = "Install MySQL 9.7, a database, and a login user for it"
nodes = 1
max_steps = 25
max_cost = 1.5

task = '''
Install MySQL 9.7 from the vendor's own apt repository on this server.
Create a database called `app` and a login user called `app_rw`.
'''

  [[check]]
  name = "mysql is running and starts at boot"
  weight = 2
  command = "systemctl is-active --quiet mysql && systemctl is-enabled --quiet mysql"

  [[check]]
  name = "the app database exists"
  command = "mysql -N -B -e \"SHOW DATABASES\" | grep -qx app"

The long form is [[stage]] tables with [[stage.check]] under each, which is what a suite that reuses its droplets looks like - see suites/mysql-repl-upgrade.toml.

Suite keys

key	meaning
name	the suite's name, and part of its droplets' hostnames
description	one line, shown in listings
nodes = N	N servers, unnamed. The harness calls them node1, node2, ... and the model works out the roles
names = [...]	N servers whose roles the suite has decided; the model is told these names and follows them
size / image / region	override the run's droplet settings for this suite. An image here takes the suite off the --os axis: it runs once, on that image
os_family	which OS families the suite's shell is written for - "debian" (apt, so Ubuntu and Debian both) or "rhel" (dnf), one or a list. --os leaves the suite out of an image it has not claimed
db_flavour	which databases the suite's checks can honestly mark - "mysql", "percona", "mariadb", "postgres", "mongodb", "percona-mongodb", one or a list. All of them must be the same engine, since one suite's checks cannot mark two. Saying nothing takes the suite off the --db axis rather than putting it on every column of it. Spelled mysql_flavour in a suite written before there was anything but MySQL, and still read
cloud_init / cloud_init_file	user-data for the droplets, for a fixture that should be in place before the model logs in
stop_on_fail	end the cell if a stage does not pass (off by default)

Stage keys

key	meaning
id	short, used as the directory name and in the tables
task / task_file	the prose the model is given. This is the specification: a vague task grades vaguely
title	one line for the progress display
setup	commands the runner runs before the model is told anything - the fixture the task starts from. A failure here is filed as setup-failed and never blamed on the model
setup_host	which servers the setup runs on (default: all)
weight	this stage's share of the cell's score
max_steps	steps the model gets (default 30)
max_cost	dollars this one task may spend
timeout	wall clock for the whole stage, in seconds (default 3600)
command_timeout	per command on the server, in seconds (default 300)

Check keys

key	meaning
name	what the tables call it
command	shell, run by the runner over its own connection. Exit 0 is a pass
host	"*" every server must pass (the default), "one" exactly one must, "any" at least one must, or a node's name
weight	its share of the stage score (default 1)
expect_exit	when success is not exit 0
contains / not_contains	when the exit code is not the answer - SHOW REPLICA STATUS exits 0 either way
timeout	seconds (default 60)

host = "one" and host = "any" are what make a role-agnostic suite gradeable: exactly one of these servers is a replica that is caught up, without the suite dictating which one the model should have chosen. The nodes that satisfied it are recorded, so the report can say which way round the model built it.

Use "one" for a role only one server can hold and "any" for one that several can. The difference is worth getting right: two servers that both take writes and both have binary logging on are two standalone databases, and under "any" they would collect the marks for a replication pair. Both replication suites use "one" for every check that names a role.

Every check and setup command is told about the lease it is running in:

DBRUN_SELF              this node's label
DBRUN_NODES             every label, space separated
DBRUN_PEERS             the other labels
DBRUN_HOST_<LABEL>      that node's public address
DBRUN_PRIVATE_<LABEL>   its private address (the public one if it has none)

which is how a check can require that replication reads from the peer's private address rather than merely from somewhere.

A suite on the --db axis is told which database this cell asked for as well:

DBRUN_DB                mysql-9.7, percona-8.4, pg-18, psmdb-8.0, ferretdb-2.0
DBRUN_DB_ENGINE         mysql | postgres | mongodb | empty for a product from outside
DBRUN_DB_FLAVOUR        the flavour: percona | mariadb | postgres | percona-mongodb | ...
DBRUN_DB_VERSION        9.7          empty if none was asked for
DBRUN_DB_NAME           Percona Server for MySQL      (the vendor's words, or yours)
DBRUN_DB_TITLE          Percona Server for MySQL 8.4  (exactly what the task asks for)
DBRUN_DB_VERSION_RE     ^9\.7        what to grep the reported version for, empty for "any"
DBRUN_DB_MATCH          percona      what the version and what it says about itself must match
DBRUN_DB_REJECT         percona|mariadb   what it must not, or empty

Every one of them except _ENGINE is exported under DBRUN_MYSQL_* as well, because the axis had that name when MySQL was all it had and a suite written then still greps $DBRUN_MYSQL_MATCH. Nine variables of duplication is a cheap price for an unset $DBRUN_MYSQL_VERSION_RE being impossible: an empty pattern is a version check that passes for every version.

A check written against those needs nothing else to grade a product this runner has never heard of: an empty _REJECT and an empty _VERSION_RE both mean "no constraint", which is what grep -q "" does anyway, so --db "MariaDB Enterprise" grades as MariaDB at any version without a line of the suite changing.

Patterns and not only names, because every suite that cares asks the same two questions - is this the version that was asked for, and is it the right vendor - and neither is a string comparison. VERSION() is not enough on its own: Percona's is a bare 8.4.3-3 and names the vendor only in @@version_comment, and Community is the one product that has to be identified by what it is not. So a check greps both fields together:

version=$(q "SELECT VERSION()"); comment=$(q "SELECT @@version_comment")
echo "version=${version:-none} comment=${comment:-none}"
[ -n "$version" ] || exit 1
printf '%s\n' "$version" | grep -q "$DBRUN_DB_VERSION_RE" || exit 1
printf '%s %s\n' "$version" "$comment" | grep -Eqi "$DBRUN_DB_MATCH" || exit 1
[ -z "$DBRUN_DB_REJECT" ] || ! printf '%s %s\n' "$version" "$comment" | grep -Eqi "$DBRUN_DB_REJECT"

The same three lines carry to the other engines, and only the two fields change. PostgreSQL is one project, so version() and server_version are all there is to read. MongoDB is the awkward one: Percona tracks upstream's numbering exactly, so db.version() cannot tell a community server from a Percona one and the mongodb suites grep the version together with the installed package names - which is the only place either vendor writes itself down plainly.

In the task - and only there - {{db}} writes the product in as it was asked for, with {{db_name}}, {{db_version}}, {{db_flavour}} and {{db_engine}} for the parts. {{mysql}}, {{mysql_name}}, {{mysql_version}} and {{mysql_flavour}} still render the same things. Commands are given the variables above instead, deliberately: check commands are full of docker ps --format '{{.Names}}' and other Go templates that nothing here may touch. A {{db_ver}} that nothing replaces is refused at load time rather than handed to a model as literal braces.

Give a check a product-neutral name. Where the points went groups by check name, and it answers as the database and version asked for is one comparable row where it answers as MySQL 9.7 would be three that are not.

Checks are read-only by convention, and the starter suites break that convention deliberately twice: one check writes a marker row on whichever server accepts writes, and the next looks for that row on a server that refuses them. Checks run in the order they are written, which is what makes that pair work - and a write that arrives is the only real evidence that replication replicates.

A stage with no checks is refused at load time. Grading a stage on the model's own account of itself is the one thing the harness under test already refuses to do.

The starter suites

suite	servers	OS	db	what it is
mysql-install	1	debian	all three MySQLs	install the database, a schema, a scoped user, not open to the world
mysql-restore	1	debian	-	a dropped database and last night's dump - cloud-init and setup build the situation
mysql-replication	2	debian, rhel	all three MySQLs	replication over the private network, roles left to the model
mysql-repl-upgrade	2	debian, rhel	mysql	two stages on the same pair: replicate, then move to Percona without breaking it
mysql-tune	1	debian	mysql, percona	two stages: install it, then size it to the machine - graded as ratios of the server's own memory, and on whether the settings would survive a restart
mysql-group-replication	1	debian, rhel	mysql, percona	three containers, three ONLINE members, writes crossing all of them
postgres-install	1	debian	postgres	the same task as mysql-install, in PostgreSQL's vocabulary: a cluster, a database, a login role that is not a superuser
postgres-tune	1	debian	postgres	install it, then size it - shared_buffers as a ratio of the machine, and pending_restart to catch an ALTER SYSTEM that was never restarted into
postgres-replication	2	debian, rhel	postgres	streaming replication over the private network, roles left to the model
mongodb-install	1	debian	mongodb, psmdb	install it, turn access control on, a database with a collection in it, a user scoped to it
mongodb-tune	1	debian	mongodb, psmdb	install it, then tune the server and its host: the WiredTiger cache, transparent huge pages, swappiness - and each of those in a way that outlives a reboot
mongodb-replication	2	debian, rhel	mongodb, psmdb	a two-member replica set with a keyfile, over the private network, roles left to the model

They are the tasks this benchmark was built for, and they are not cheap: the runs they were written from took 40-70 steps and half an hour each. Start with -s mysql-install and one model.

The OS column is each suite's os_family, and it is what --os rocky-9 picks up: the install and tune suites are written around a vendor apt repository, so they are left out of an rhel column rather than failed on it. The replication suites grade a thing the database does rather than a way of installing it, and run on either family.

The db column is each suite's db_flavour, and what --db mariadb-11.4 or --db psmdb-8.0 picks up. Three suites per engine, deliberately the same three tasks: install, tune and replication are written check for check against each other, so a model's PostgreSQL score is readable beside its MySQL one and the difference is the database rather than the marking. On the MySQL line, mysql-install and mysql-replication mark all three flavours; mysql-repl-upgrade is Community-only because becoming Percona is the task; mysql-tune and mysql-group-replication need something MariaDB has not got. On the MongoDB line every suite marks both builds, because what tells them apart is the installed package name and not a check - which makes --db mongodb-8.0 --db psmdb-8.0 the cleanest comparison in the whole set. PostgreSQL has one flavour: what Percona distributes is PostgreSQL, with the same version(), so a second column would be noise - the axis worth comparing there is the major version, and --db pg-18 --db pg-16 is two repositories, two data directories and one set of checks. mysql-restore has no db_flavour at all and runs once whatever the run compares.

Reading the results

uv run python dbrun.py report                     # rebuild the last run's tables
uv run python dbrun.py report 20260822-143512

results.jsonl is authoritative and append-only; the leaderboard and the CSV are derived from it and can be rebuilt at any time, including after a scoring change.

The leaderboard has five sections: the score matrix, Said done, was not, Where the points went (every check and how many attempts satisfied it - the fastest way to find a check that is wrong rather than hard), Cells that did not get a fair run, and The tasks - the prose the models were given, quoted once per distinct wording, because a suite's paragraphs get edited between runs and a percentage is not a result without them.

A run that compared operating systems gets a sixth, By operating system, and one that compared databases gets By database. Either way every table that names a cell grows that axis in its row labels: rows read `sonnet @ rockylinux-9` or `sonnet / mariadb-11.4`, or both at once, and Where the points went counts each check per OS and per database rather than pooling them - because a check that passes on one image and not the other, or on MySQL and not on MariaDB, is the whole reason to run both. The tasks splits the same way, naming the database whenever a stage's {{db}} made two wordings out of one paragraph.

Layout

dbrun.py              entry point
dba_runner/
  cli.py              subcommands, screens, preflight
  matrix.py           cells, the thread pool, the budget, cleanup
  suites.py           the TOML format and its loader
  products.py         the --db axis: engines, flavours, versions, what a check is told
  provision.py        doctl, leases, droplets, reap
  harness.py          running dba.py as a subprocess and reading it back
  grade.py            the checks, over the runner's own SSH connection
  results.py          results.jsonl, the leaderboard, the CSV
suites/               the tests
tests/                offline suites - no network, no token, no money
run_tests.py          runs them all, one line per suite

do_dba is imported for its SSH transport and terminal handling, and dba.py is run as a subprocess: the thing being measured is allowed to fail badly, and in a subprocess a wedged model costs one cell and a timeout.

Tests

uv run python run_tests.py
uv run python run_tests.py suites grade

Nothing there touches the network, a DigitalOcean account or a model. doctl is a fake that answers from a script, the bench is a fake dba.py that writes a transcript, and the servers are fakes that answer the checks - so the whole matrix, including a cell that times out and a droplet that leaks, runs on a laptop with no token and no money.

Three of them grade shell rather than Python. suites loads all twelve shipped suites and puts every check and setup script through sh -n, renders every task once per flavour the suite claims and fails on a {{ that survived, and asserts that each suite accepts its own engine's default database - a suite that refused it would be silently skipped by an ordinary run. tune runs mysql-tune's tuning checks against nine fake servers - one tuned well, one untouched, one whose settings would not survive a restart, one with performance_schema switched off - and asserts which checks should fail on each. repl grades both MySQL replication suites against twelve pairs of servers that are really directories with a stub mysql on their PATH: a correct pair, the same pair the other way round, an older server that spells every field Slave_, a pair replicating over the public internet, a replica that still takes writes, a faultless pair of MariaDB, two standalone databases, a dead peer. Three of the twelve are the --db axis: a Percona pair graded as a Community run and a Community pair graded as a Percona one both fail on the version check alone, and the Percona pair graded as the Percona run it is comes out clean. The suite file is byte-identical across all three - only the exported environment differs, which is the whole claim the axis makes.

That stub is an executable and not a shell function on purpose, because the client is where a suite goes wrong: this one refuses \G when the statement arrives through -e, exactly as the real one does, and a helper that asked that way scored a working replica zero on every model of two whole runs. One scenario grades a correct pair with that old helper put back and asserts it still comes out at the 0.625 the run recorded - so the day the stub stops being awkward, the test says so instead of passing everything.

Both stub-server graders are MySQL, and that is the gap in this battery. The four PostgreSQL and MongoDB tune and replication suites are loaded, rendered and parsed by suites, but nothing offline runs their checks against a fake server yet, so their marking has been argued rather than tested. Closing it means a stub psql and a stub mongosh of the same kind as the mysql one - awkward in the same places the real clients are awkward, which for mongosh means an adminCommand that answers {ok: 0} instead of failing.

A suite that mis-marks does not look like a broken suite. It looks like a model that failed.

Notes

• runs/ holds real credentials. The bench's secrets.json and the servers' generated passwords end up under it. It is in .gitignore and should stay there.
• Its own known_hosts. Per run, inside the run directory. DigitalOcean reissues addresses, and neither your ~/.ssh/known_hosts filling with dead droplets nor a recycled address looking like an attack is wanted.
• cloud-init is waited for. A freshly booted image is still installing things for a minute or two, holding the apt or dnf lock, and a model whose first step is apt-get would meet "Could not get lock" through no fault of its own. That is the difference between grading a model and grading a race.
• Multi-stage suites rely on do-dba's server-side credential store. Stage one's generated passwords stay on the server in /etc/profile.d/dba-secrets.sh, which is how stage two can log in at all. --no-server-secrets in the bench would break that, so the runner does not pass it.