feat(db): move every query into one database layer and support MySQL and MariaDB by Pfannkuchensack · Pull Request #9702 · invoke-ai/InvokeAI

Pfannkuchensack · 2026-10-08T01:06:57Z

Summary

Every SQL statement of the app now lives in one place, invokeai/app/services/shared/database/, built on SQLAlchemy Core, and InvokeAI can keep its data in MySQL 8.4 or MariaDB 10.11 instead of the SQLite file. SQLite stays the default and keeps working as before.

The problem. SQL was spread over ~23 storage services (364 execute calls) on one sqlite3 connection behind a process-wide lock. Every database request of every user waits for every other one, and no server database could be used. With multi-user mode, that lock is the ceiling on how many people one install can serve.

What changes

shared/database/: Database (engines, transactions, retries of lost races, error translation), schema/ (table metadata that matches the migrated SQLite schema exactly, checked by a parity test), queries/ (one module per domain), dialect.py (the few constructs that differ per backend), copy.py.
Every storage service is ported (users, boards, projects, workflows, library, models, images/videos, intermediates, gallery, image index, session queue, image moves, fonts, scripts). Services hold no SQL; an AST guard test keeps it that way (the two SQLite scratch files of gallery maintenance and of the output-folder walk are not the app's database, and are listed there with the reason). The transitional shared/sqlite/ facade and all 20 SQLite triggers are gone (their logic is in the app, so it runs on every backend).
Invariants that relied on the process lock (queue claims and transitions, enqueue admission, last admin, media protection) are atomic in SQL, with named row locks (db_locks) on servers.
MySQL/MariaDB via db_url / INVOKEAI_DB_URL (pip install "invokeai[mysql]"; included in the Docker image). Startup checks the server version and max_allowed_packet, and holds an instance lock: one InvokeAI process per database; a second one refuses to start.
invoke-db-copy moves an install's SQLite database into a new server database (preflight, then row-by-row verification), and with --to-sqlite FILE back into a new SQLite file.
New migrations are portable (Alembic operations); the 64 existing ones stay as the SQLite upgrade path, and a server database starts at the newest schema.

Before / after (one InvokeAI process, 20,000 images, request mix of reads and writes, median of 3 runs):

Simultaneous users	SQLite	MySQL 8.4	MariaDB 10.11
1	125 req/s	72 req/s	116 req/s
8	121 req/s	409 req/s	437 req/s
32	131 req/s	396 req/s	398 req/s
Median response at 32 users	218 ms	2.5 ms	2.4 ms
4 users while a 5th selects all 20k images, repeatedly	38 req/s	239 req/s	263 req/s

A single request is slower on a server (a network round trip per query, pure-Python driver); from two users on, the servers answer more requests, and a slow request no longer holds up everyone else.

Related Issues / Discussions

Replaces the closed v6 attempt #9071 (SQLModel, one class, no migrations for external databases), whose design is not reused.

QA Instructions

Test servers (any free ports):

docker run -d --name invokeai-test-mysql84 -p 127.0.0.1:33084:3306 --tmpfs /var/lib/mysql -e MYSQL_ROOT_PASSWORD=invokeai-test mysql:8.4 --disable-log-bin
docker run -d --name invokeai-test-mariadb1011 -p 127.0.0.1:33011:3306 --tmpfs /var/lib/mysql -e MARIADB_ROOT_PASSWORD=invokeai-test mariadb:10.11
uv sync --extra test --extra mysql
uv run pytest -n 4                                                    # SQLite
INVOKEAI_TEST_DB_URL=mysql+pymysql://root:invokeai-test@127.0.0.1:33084/invokeai_test \
  uv run pytest -n 4 -m "uses_database and not sqlite_only and not slow"
INVOKEAI_TEST_DB_URL=mariadb+pymysql://root:invokeai-test@127.0.0.1:33011/invokeai_test \
  uv run pytest -n 4 -m "uses_database and not sqlite_only and not slow"

Results on the final commit (Windows, Python 3.12):

SQLite: 13,204 passed, 212 skipped. 2 failed: test_16_channel_vae_loader and minimax_h3/test_text_encoder_checkpoint, whose fixtures are Git LFS pointers in my checkout (unrelated to this PR).
MySQL 8.4: 795 passed. MariaDB 10.11: 795 passed. CI runs both as new lanes in python-tests.yml.
My local Docker MariaDB ends a lock wait early with error 1205 now and then (its VM clock jumps); a race test can then fail and passes on rerun. MySQL never does.
ruff check / ruff format --check clean; openapi.json and schema.ts regenerated; docs build passes.

End to end:

A copy of a real development database (1,072 rows in 32 tables) went through invoke-db-copy --check and the copy into both servers, with every table verified. InvokeAI on each server answered boards, images, workflows, queue, models, presets, prompts and wildcards exactly as on SQLite; a second process was refused; the runtime config shows no password.
Server → SQLite with --to-sqlite gives a file the app opens without migrating; queue ids continue after the server's counter.
After merging main: the app starts on SQLite and on MySQL, serves its main endpoints, and the new vacuum endpoint answers 204.

SQLite cost of the layer, each port measured against the code right before it (SQLite file, synchronous=full, 4 interleaved runs):

Kind of call	Example	Before	After
Point read	image record by name	20.6 µs	40.9 µs
Large read	gallery page of 100	19.5 ms	19.8 ms (+2.8 %)
Large read	all image names, 20k	6.13 ms	6.61 ms (+8.9 %)
Write	save an image record	467 µs	562 µs
Write	enqueue a batch of 10	62.2 ms	61.7 ms
Faster	sync bundled style presets	8.61 ms	0.94 ms (−89 %)
Faster	intermediates cleanup pass, one owner	3.35 ms	0.67 ms (−80 %)
Faster	workflow counts by tag	5 statements	1 (−79 %)

Point reads pay a fixed ~10–30 µs for SQLAlchemy and an explicit transaction; writes of several statements +20–100 µs; queue writes are 20–25 % faster without the triggers. scripts/benchmark_database.py reproduces the SQLite numbers.

Review

Material findings were resolved, among them: queue claims and cancels racing without the process lock; deadlocks from lock order between projects and boards; a lost instance lock after a server-side idle timeout; transient database errors hidden behind routes' own 404/500 answers (now 503 with Retry-After); oversized values only failing in the middle of a copy; MySQL serving a stale AUTO_INCREMENT from its statistics cache.

Remaining limitations, not addressed here:

Postgres is not supported yet; the layer is written for it (no MySQL-only syntax outside dialect.py), but there is no CI lane.
One InvokeAI process per database; Galera and other multi-writer setups are not supported.
move_all_images advances one image per window without moves (20k images in place take 20k queries); ImageService.copy does not take image_mutation_lock.
Font listing by hash is a table scan on servers.
Shutdown does not join the session processor's worker threads before closing the database (noisy tracebacks only).

Compatibility / Rollout

SQLite installs: five portable migrations run at the first start (lock rows, intermediates state, two lock rows, dropping the 20 triggers), with the usual automatic backup. To go back to an older version, restore that backup. A cover image's deletion no longer updates boards.updated_at (that was a trigger).
API: client-chosen keys are bounded where a server stores them (max_length 255, emails 254, paging ge/le): over-long values now get 422 instead of being stored. A busy, unreachable or exhausted database answers 503. invokeai-web exits with 3 when the app refuses to start (it exited 0).
Python: new dependencies sqlalchemy and alembic; extra mysql (PyMySQL). The import path invokeai.app.services.shared.sqlite is gone, as are the re-exports of invokeai.app.services.fonts, and scripts/remove_orphaned_models.py (the orphaned models dialog and API cover it, but not offline use or file-level deletion).
Merged since this PR was opened, and ported onto the layer here: Store image-index embeddings as fp16 #9696 (fp16 embeddings: encoding column), Add gallery maintenance UI #9700 (gallery maintenance: image locations, Database.backup, which on a server database it skips, as the operator backs that up), feat(ui): reliability pass plus canvas, gallery and frontend layout features #9682 (queue listing index, now a portable migration with the same id that creates it on SQLite only, and its planner guards as dialect.Unindexed), Fix model manager cache and recovery failures #9669 (tests), Gallery: Browse large collections with a sparse page cache (infinite scrolling) #9683 (gallery item location: one ranked statement over both halves, in the name listing's order).
Open PRs that touch the same code and need a port onto the layer, whichever merges second: Feat: Remote Workers Panel #9642 (raw SQL through session_queue._db).

Checklist

The PR has a short but descriptive title, suitable for a changelog
Meaningful regression coverage added / updated where needed; obsolete tests/code removed
Persisted-state and API changes include required migrations / compatibility validation
Relevant performance/efficiency opportunities considered; material claims have evidence
Material review findings resolved and relevant checks rerun
Documentation added / updated (if applicable)
Updated What's New copy (if doing a release after this PR)

Adds shared/database: SQLAlchemy Core engines (SQLite behind the process lock, pooled MySQL/MariaDB), query modules behind Database.queries, backend-neutral errors and a guard ratcheting SQL outside it. SqliteDatabase becomes a cursor facade over it; a nested cursor transaction now joins the outer one. CI runs the layer's tests against MySQL 8.4 and MariaDB 10.11.

Adds shared/database/schema, one module per domain, with types that keep SQLite's declarations and bound keys on MySQL/MariaDB. A parity test holds the metadata to the migrated SQLite schema; server tests check that the tables behave alike. Translates SQLite's ON DELETE RESTRICT error and refuses mysql+ URLs that reach a MariaDB server.

Migrations after PORTABLE_CUTOVER use Alembic operations and run on every backend; the 63 before it stay SQLite-only. A new MySQL or MariaDB database is created from the metadata and seeded from the migration chain, under a migration lock. Adds the row copy core and frozen cutover schemas that hold server migrations to the metadata.

A point read through SQLAlchemy Core and an explicit transaction costs about 11 µs more on SQLite than the raw cursor did; the 10 µs floor failed every ported point read. 50 µs is the planned budget.

Users, app settings, client state and media references run on Database.queries; units retry deadlocks (queries.run) and build DTOs outside the transaction (@mapped). A db_locks table, the first portable migration, keeps the last-admin and first-admin checks atomic on MySQL and MariaDB. Service tests use a new `database` fixture and run on SQLite, MySQL and MariaDB.

Boards and their image and video memberships run on Database.queries; board listings test visibility with EXISTS instead of DISTINCT joins (get_all of 200 boards 1546 -> 626 µs). dialect.OrderedJoin keeps SQLite's membership-first plan, and a board's counts come from one statement per media kind. Tests reach the SQLite facade through a mock_sqlite_database fixture instead of service internals.

@Locking

Project records run on Database.queries; saves, board claims and deletes lock the project's row before its board's, so concurrent writes on MySQL and MariaDB decide on the rows they write. Writes return the document they stored instead of decoding it again (a save 1589 -> 1517 µs); listings break created_at ties by project id. Adds a @Locking decorator for row locks, and race tests that pause a real write while another one waits for it.

The workflow library runs on Database.queries; saves, deletes and visibility changes lock the workflow's row before their checks, and a creation sent twice at once yields one workflow. Searches match %, _ and \ literally and ignore case on every backend, listings break ties by workflow id, and tags are counted in one statement (counts_by_tag of 5 tags -79 %). Client-chosen list sizes no longer multiply cached statements, and the library page reads each workflow once (-13 %).

Style presets, system prompts and wildcards run on Database.queries; creates return what they stored instead of reading it back, and owner-scoped prompt updates and deletes decide in one statement. The bundled style presets are replaced in one transaction (start-up sync 8.6 -> 0.9 ms), listings break ties of equal names by id, and an unknown wildcard owner raises ForeignKeyViolation. Their common modules no longer load FastAPI or dynamicprompts with the database layer, and the benchmark script times the library's reads and writes.

Model records and relationships run on Database.queries; an update locks the model's row, so concurrent edits on MySQL and MariaDB both apply, and the orphan scan reads every registered path from the record store, also of records whose config no longer validates. Listings break ties by key and keep SQLite's NOCASE ordering through dialect.CaseInsensitiveOrder (now also for boards, style presets and system prompts); hash lookups return the oldest record first, and dialect.insert_ignore skips a pair already related. Removes the unused list_models, ModelSummary, get_model_by_hash and get_related_model_keys_batch.

Image and video records run on the database layer on every backend; the storages lose their SQL and the Sqlite prefix. Listings break ties by name, search matches % and _ literally, day filters are timestamp ranges, and storage errors are no longer reported as missing rows; the unused get_image_dates is removed. Adds the JsonString and fixed_limit dialect helpers and the capture_statements and explain_query_plan test helpers.

Record services validate metadata, so the database layer no longer loads torch; column types adapt once per backend. Removes record delete exceptions nothing raises and stale column lists; the last representable day and an empty day match nothing. Adds tests for video listings, intermediate deletes and chunking, field round trips, paging, cross-account shares and membership plans.

…tables The cleanup's classification, previews, windows and delete guard run on the database layer; session holds and unmeasurable marks move from TEMP tables into tables the service empties at start. A media_protection lock serialises a cleanup's check and delete with every write that protects media, and windows page by creation time and name instead of rowid. Adds the bound-set and sql_true dialect helpers, shared lock acquisition, and the frozen migration copy of the media-reference writers.

The polymorphic gallery listing, name lists, date boards and board summaries run on the database layer on every backend. Date boards and board counts follow the gallery's views, so canvas-owned images neither count as assets nor make or cover a day. Search matches % and _ literally, day filters are ranges, and a listing's total counts without the board join.

Move the image index's SQL into queries/image_index.py; guarded writes lock their parent row shared, and a vocabulary replace takes a new image_index_vocabulary lock, so concurrent writes stay clean on servers. Read embeddings in one bound-set statement, share one board-access rule, and fetch query results whole.

Move the queue's reads (items, listings, ids, summaries, status/batch/destination counts, workflow-call chains, enqueue receipts) into queries/session_queue.py; the writes stay on the legacy cursor until the next step. Pages keep every filter after the keyset, origin prefixes match case-insensitively with wildcards escaped, and a chain is read in one transaction.

Port the queue's writes into queries/session_queue.py and replace SqliteSessionQueue with SessionQueue: claims and status changes are conditional UPDATEs, bulk changes lock their rows first, and enqueue, retry and child enqueue take a new session_queue_admission lock while sharing media protection with the intermediates cleanup.

Move the image move journal's SQL into queries/image_moves.py and run ImageMoveService on Database: the commit is one unit that repoints the job's moved images in a single joined UPDATE, guarded by their old subfolder, and rolls back when any moved image is not where the move put it. Move tests run on every backend.

Move the font catalog's SQL into queries/fonts.py and run FontService on Database: listings skip the count when the page ends them, rescans run one at a time and write only changed rows, and uploads write their file outside any transaction. The fonts package no longer re-exports the service, which closed an import cycle.

Run gallery_maintenance and import_images through the app's config and records, with consistent online backups (Database.backup, now also the migrator's), and drop remove_orphaned_models.py in favour of the Model Manager's orphan cleanup, whose user-guide section is corrected.

A portable migration drops the 20 triggers that stamped updated_at and the queue's started_at, completed_at and session_revision: the application sets these on every backend, as servers always required.

Open the app database through Database and init_database, drop SqliteDatabase, init_db and Database.legacy_cursor, move SQLiteDirection to shared/pagination.py unchanged, and empty the access guard's allowlist; tests use sqlite_cursor for their own SQL.

A db_url setting selects a server database: startup checks its version and max_allowed_packet, holds an instance lock for the process, and warns beside an unused SQLite database; busy-database errors answer 503, the CLIs and maintenance scripts open the configured database and refuse one the app has not migrated.

…copy invoke-db-copy checks a snapshot for orphans and oversized values, copies it into an empty server database and verifies every table by checksum. Adds the MySQL/MariaDB configuration page, links it from the YAML, Docker and admin guides, and installs the mysql extra in the Docker image.

Batch, queue, client-state, project, cover-image and relationship keys take at most 255 characters, a reserved workflow id 36, and board and workflow paging is bounded. Addresses at special-use domains are held to 254 characters, and the model store refuses keys and paths longer than its columns on every backend.

…copy The instance lock is checked every minute and taken again when the server ends its session; one SessionLock serves it and the migrator. Busy, unreachable or exhausted databases answer 503 also behind a route's own error, and URLs that cannot be read never echo their password. invoke-db-copy checks key and computed column lengths and row sizes, clears SET NULL orphans, copies into db_url by default and writes the migrator's records only after verifying every table.

A connection lost during a statement stays an error: a server also drops the connection of an oversized statement, which no retry fixes. invoke-db-copy reports snapshot and check failures without a traceback, and its vocabulary check ignores format characters as the server collations do.

uvicorn returns normally when the lifespan fails, so a refused start (e.g. a database another process serves) exited 0 and looked like a clean stop to supervisors.

invoke-db-copy --to-sqlite FILE copies the db_url database into a new SQLite file under the instance lock, verifies every table and moves it into place without overwriting a file. Ids continue after the server's AUTO_INCREMENT counters, read past MySQL's statistics cache, so no queue item id is issued twice.

joshistoast · 2026-10-08T06:11:59Z

#9700 is now merged

Ports invoke-ai#9700 (gallery maintenance: image locations, Database.backup with upstream's reserve-and-clean-up, no file backup of a server database), invoke-ai#9682 (queue listing index as a portable migration with the same id, SQLite only; planner guards as dialect.Unindexed; limited and one-pass status reads) and invoke-ai#9669's tests onto the database layer. The access guard lists the two SQLite scratch files of gallery maintenance and the output-folder walk, which are not the app's database.

Ports invoke-ai#9683's gallery item location onto the gallery queries (ROW_NUMBER over both halves, in the name listing's order) and its router test onto the layer. Re-locks uv.lock from main's with the layer's sqlalchemy, alembic and pymysql (uv 0.11.28, no user index).

…db-access-layer

Re-locks uv.lock from main's (diffusers 0.41.0, invoke-ai#9774) with the layer's sqlalchemy, alembic and pymysql.

## Summary `test_workflow_last_run_at.py::test_update_last_run_at_sets_timestamp` fails intermittently, mostly on Windows CI. - The fixture stores the 43 bundled workflows, then the test creates one workflow and expects it in the first 10 results of a listing sorted newest first by `created_at`. - `created_at` has millisecond resolution. On Windows' ~15 ms clock the new workflow usually gets the same timestamp as the bundled ones. - Ties sort by `workflow_id` descending. A random UUID sorts above the bundled `default_...` ids only about 1 time in 7 or 8, so the new workflow usually falls off the page and the test's `next(...)` raises `StopIteration`. The listing now includes only user workflows, which is what the check is about. Nothing else changes. ## Related Issues / Discussions Seen as the `windows-cpu` failure on #9694. The test came in with #9702. ## QA Instructions - Forced tie (a throwaway test setting every row's `created_at` to the same value), 20 runs each: the old query lost the workflow in 19, the new one in none. - `uv run --no-sync pytest tests/app/services/workflow_records`: 137 passed. - `ruff check` and `ruff format --check` are clean. ## Review No material findings. This is a one-line test change. ## Checklist - [x] _The PR has a short but descriptive title, suitable for a changelog_ - [x] _Meaningful regression coverage added / updated where needed; obsolete tests/code removed_ - [ ] _Persisted-state and API changes include required migrations / compatibility validation_ - [ ] _Relevant performance/efficiency opportunities considered; material claims have evidence_ - [x] _Material review findings resolved and relevant checks rerun_ - [ ] _Documentation added / updated (if applicable)_ - [ ] _Updated `What's New` copy (if doing a release after this PR)_

Pfannkuchensack added 30 commits October 4, 2026 09:14

chore(db): allow 50 µs per call in the database benchmark

26a9f06

A point read through SQLAlchemy Core and an explicit transaction costs about 11 µs more on SQLite than the raw cursor did; the 10 µs floor failed every ported point read. 50 µs is the planned budget.

refactor(db): drop the SQLite triggers

0216f01

A portable migration drops the 20 triggers that stamped updated_at and the queue's started_at, completed_at and session_revision: the application sets these on every backend, as servers always required.

fix(db): count merged rows inside invoke-db-copy's failure handling

22c6500

fix(app): exit with a failure code when the app refuses to start

1ecdbc0

uvicorn returns normally when the lifespan fails, so a refused start (e.g. a database another process serves) exited 0 and looked like a clean stop to supervisors.

Pfannkuchensack requested review from blessedcoolant and joshistoast as code owners October 8, 2026 01:06

Pfannkuchensack added the DO NOT MERGE label Oct 8, 2026

Pfannkuchensack removed the DO NOT MERGE label Oct 9, 2026

lstein assigned JPPhoto Oct 9, 2026

lstein added the 7.0.0 label Oct 9, 2026

lstein added this to Invoke - Community Roadmap Oct 9, 2026

Merge branch 'main' into feat/db-access-layer

ca72217

joshistoast self-assigned this Oct 10, 2026

Pfannkuchensack added 3 commits October 10, 2026 03:20

Merge remote-tracking branch 'origin/feat/db-access-layer' into feat/…

b7e6b13

…db-access-layer

Merge upstream main into feat/db-access-layer

4206805

Re-locks uv.lock from main's (diffusers 0.41.0, invoke-ai#9774) with the layer's sqlalchemy, alembic and pymysql.

joshistoast approved these changes Oct 10, 2026

View reviewed changes

joshistoast merged commit 71aeb79 into invoke-ai:main Oct 10, 2026
17 checks passed

lstein mentioned this pull request Oct 10, 2026

fix(model-manager): prevent stale in-flight model loads from reentering the cache #9694

Merged

5 of 7 tasks

Pfannkuchensack deleted the feat/db-access-layer branch October 10, 2026 18:08

lstein mentioned this pull request Oct 10, 2026

test(workflows): fix flaky last_run_at listing test on Windows #9788

Merged

3 of 7 tasks

Navigation Menu

Repository navigation

feat(db): move every query into one database layer and support MySQL and MariaDB - #9702

feat(db): move every query into one database layer and support MySQL and MariaDB#9702
joshistoast merged 38 commits into
invoke-ai:mainfrom
Pfannkuchensack:feat/db-access-layer

Pfannkuchensack commented Oct 8, 2026 •

edited

Loading

Uh oh!

joshistoast commented Oct 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Pfannkuchensack commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issues / Discussions

QA Instructions

Review

Compatibility / Rollout

Checklist

Uh oh!

joshistoast commented Oct 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Pfannkuchensack commented Oct 8, 2026 •

edited

Loading