fix(sql): schema-changing statements require tinycloud.sql/schema (TC-1581, TC-1585, TC-1586)

CHANGELOG.md

@@ -2,6 +2,12 @@
 
 ## [Unreleased]
 
+### Changes
+
+- Schema-changing SQL statements now require `tinycloud.sql/schema` (TC-1581). `tinycloud.sql/write` covers data statements only (`INSERT`, `UPDATE`, `DELETE`, `SELECT`). `CREATE`, `ALTER` and `DROP` of tables, views and indexes need a schema-conferring ability: `sql/schema`, `sql/admin` or `sql/*`. This is judged for every statement of a request against all the SQL abilities the invocation holds, in `query`, `execute`, the `schema` list of `execute`, `batch` and named statements, and again by the SQLite authorizer, which now also refuses `CREATE TRIGGER`. A client that holds both `write` and `schema` (the SDK sends both for `execute({ schema })` and for migrations) is unaffected; the SDK has requested `sql/schema` for DDL since 2.5.0. `PRAGMA` still needs `sql/admin`. The storage-quota gate classifies a statement as a write whoever may run it. Configurable with `[global.storage.sql] ddl_requires_schema = "enforce" | "log" | "off"` (default `enforce`): `log` runs the statement and records it, with one constant warning per request and a counter by service and statement kind, and no SQL text, space ids, DIDs or table names; `off` restores the earlier rule that `write` also suffices for the statements `log` counts; the other changes in this release (the `CREATE TRIGGER` refusal, the nested-block-comment refusal, the CTE write classification, the schema-table and PRAGMA restrictions) apply in every mode. `log` counts only statements the parser classifies as schema-changing. A request that mixes `schema` entries and `sql` is not transactional: the schema entries run first and can stay applied if a later statement is refused. No migration; the default needs no compose change. See `docs/sql-schema-authority.md`.
+- Fixed `WITH … INSERT` and `WITH … UPDATE` SQL statements being classified as reads (TC-1585). They skipped the storage-quota gate, emitted no write-hook events and reported no touched table, so a write that changed the database was not made durable until a later write did. Any statement tree that contains an `INSERT` or `UPDATE` is now a write: it is quota-gated, publishes write events for its table, is persisted, and is refused to a read-tier grant and to a `readOnly` caveat. `WITH … DELETE` is still not accepted by the SQL parser, for any ability.
+- Connection-level SQLite state no longer outlives a request on a database's shared connection (TC-1586). An `sql/admin` holder may now run only an allow-list of PRAGMAs: the plain schema reports, the database header values `user_version` and `application_id` (set or read), the check pragmas `integrity_check`, `quick_check` and `foreign_key_check`, and read-only reports such as `journal_mode`, `foreign_keys` and `page_count`. Any other PRAGMA, including every form that sets connection-level or process-wide settings, is refused. After any request in which the authorizer allowed a PRAGMA outside the plain schema reports, the node also resets the connection to its opening state, for in-memory and file-backed databases alike; if that reset fails, the database's actor is retired and the next request rebuilds it from the durable copy. The SQLite authorizer also no longer lets a data statement write `sqlite_master`, `sqlite_schema` or their temp counterparts: only a schema change (which writes them internally), a pure query, and `admin` do. `sql/write` keeps writing application tables.
+
 ## [1.21.5] - 2026-10-20
 
 ### Release notes

docs/sql-schema-authority.md

@@ -0,0 +1,54 @@
+# SQL schema authority (TC-1581)
+
+`tinycloud.sql/write` covers data statements. Statements that change the schema need `tinycloud.sql/schema`.
+
+| Statement | Needs |
+|---|---|
+| `SELECT` | `read`, `write`, `admin` or `*` |
+| `INSERT`, `UPDATE`, `DELETE`, `REPLACE`, and `WITH … INSERT/UPDATE` | `write`, `admin` or `*` |
+| `CREATE`/`ALTER`/`DROP` of tables, views and indexes (including `CREATE TABLE … AS SELECT` and temp objects) | `schema`, `admin` or `*` |
+| `PRAGMA` | `admin` or `*` (unchanged) |
+| `ATTACH`, `DETACH`, `VACUUM`, `ANALYZE`, `REINDEX`, `CREATE VIRTUAL TABLE`, `CREATE TRIGGER`, transaction control | never |
+
+`admin` implies `schema` in the capability registry (`capabilities.json`); `*` implies every SQL ability. `write` does not imply `schema`, and `schema` does not imply `write` or `read`: a holder of `schema` alone can run schema statements only.
+
+## How a request is judged
+
+An invocation can carry several SQL abilities. The node judges each statement against all of them, so a request that holds both `write` and `schema` can run data and schema statements together. That is what the SDK sends for `execute({ schema: [...] })` and for `migrations.apply`. A request that mixes the two statement kinds and holds only one of the abilities is refused.
+
+This applies to every request path that runs SQL: `query`, `execute`, each entry of the `schema` list of `execute`, `batch`, and named statements (`executeStatement`). A named statement that holds DDL needs `schema` on the invocation like the same SQL sent directly.
+
+The SQLite authorizer enforces the same rule on the engine's own callbacks, so a statement the parser misclassifies is still refused.
+
+## Configuration
+
+`[global.storage.sql] ddl_requires_schema` in `tinycloud.toml` (env `TINYCLOUD_STORAGE__SQL__DDL_REQUIRES_SCHEMA`; a new CVM env name needs `allowed_envs`):
+
+| Value | Behaviour |
+|---|---|
+| `"enforce"` (default) | A schema statement without a schema-conferring ability is refused with `403`. |
+| `"log"` | The statement runs as before. Each such statement increments `tinycloud_ddl_without_schema_total{service="sql",kind=…}` and the request logs one constant warning (`sql ddl allowed without schema ability (ddl_requires_schema = log)`). Neither carries SQL text, space ids, DIDs or table names. `kind` is one of `create_table`, `create_view`, `create_index`, `alter_table`, `drop`. |
+| `"off"` | The earlier rule: `write` also suffices for the schema statements `log` counts. The other changes in this document (the `CREATE TRIGGER` refusal, nested block comments, CTE write classification, schema-table and PRAGMA restrictions) apply in every mode. |
+
+`log` counts only statements the parser classifies as schema-changing. A statement the parser does not recognise as such is still refused by the SQLite authorizer unless the policy allows it, and is not counted.
+
+A request that carries both `schema` entries and `sql` is not transactional. The `schema` entries run first and can stay applied if a later statement is refused (only the storage-quota guard rolls a refused request back).
+
+A statement that nests block comments (`/* … /* … */ … */`) is refused for every ability: SQLite ends a comment at the first `*/`, the SQL parser nests them, and the two would read different statements.
+
+The warning is not mapped in the log shipper (`log-shipper/diagnostics.json`), so it ships as an unclassified `node_warning` until it is.
+
+## Clients
+
+The SDK has requested `tinycloud.sql/schema` for DDL since js-sdk 2.5.0 and sends `[write, schema]` for requests that mix both. A client or manifest that runs DDL under `sql/write` only needs `tinycloud.sql/schema` added to its grant.
+
+## Connection state and SQLite's schema tables (TC-1586)
+
+A database's SQLite connection is shared by every request on it, so connection-level state must not outlive the request that set it.
+
+- The SQLite authorizer allows anyone only the plain schema reports (`table_info`, `table_list`, `table_xinfo`, `database_list`, `index_list`, `index_info`, `foreign_key_list`).
+- An `admin` (or `*`) holder may also set and read `user_version` and `application_id` (database header values), run the check pragmas `integrity_check`, `quick_check` and `foreign_key_check`, and read (not set) `schema_version`, `page_count`, `page_size`, `freelist_count`, `data_version`, `encoding`, `journal_mode`, `foreign_keys`, `writable_schema`, `query_only`, `collation_list`, `compile_options` and `pragma_list`.
+- Every other PRAGMA is refused for everyone, including all forms that set connection-level or process-wide settings and maintenance forms such as `wal_checkpoint`, `optimize` and `incremental_vacuum`.
+- After any request in which the authorizer allowed a PRAGMA outside the plain schema reports, the node resets the connection to the state it was opened in, for an in-memory database (journal mode `memory`) and a file-backed one (WAL). If the reset fails, the database's actor is retired and the next request rebuilds it from the durable copy.
+
+The SQLite authorizer also refuses a data statement that writes `sqlite_master`, `sqlite_schema` or their temp counterparts. Only a schema change (which writes them internally), a pure query (a `pragma_*` table-valued function reads the schema through them) and `admin` may.

test/m1-realdata-e2e/tests/cross_layer_contract.rs

@@ -1180,7 +1180,7 @@ async fn seed_listen_sql(sql: &SqlService, space: &SpaceId) -> Result<()> {
             ],
         },
         None,
-        "tinycloud.sql/write".to_string(),
+        "tinycloud.sql/admin".to_string(),
     )
     .await?;
     sql.execute(
@@ -1861,7 +1861,7 @@ async fn seed_grant_vector_sql(sql: &SqlService, space: &SpaceId) -> Result<()>
             ],
         },
         None,
-        "tinycloud.sql/write".to_string(),
+        "tinycloud.sql/admin".to_string(),
     )
     .await?;
     Ok(())

test/n4-mounted-e2e/src/main.rs

@@ -551,7 +551,7 @@ async fn seed_sql(rocket: &Rocket<Build>) -> Result<()> {
         .state::<tinycloud_core::sql::SqlService>()
         .context("SQL service missing")?;
     use tinycloud_core::sql::{SqlRequest, SqlValue};
-    service.execute(&space, "documents", SqlRequest::Execute { sql: "CREATE TABLE IF NOT EXISTS shared_documents (document_id INTEGER PRIMARY KEY, markdown TEXT NOT NULL)".into(), params: vec![], schema: None }, None, "tinycloud.sql/write".into()).await.map_err(|error| anyhow::anyhow!("SQL schema seed: {error}"))?;
+    service.execute(&space, "documents", SqlRequest::Execute { sql: "CREATE TABLE IF NOT EXISTS shared_documents (document_id INTEGER PRIMARY KEY, markdown TEXT NOT NULL)".into(), params: vec![], schema: None }, None, "tinycloud.sql/schema".into()).await.map_err(|error| anyhow::anyhow!("SQL schema seed: {error}"))?;
     service
         .execute(
             &space,

test/w5-policy-runtime-node-e2e/tests/challenge_to_native_read.rs

@@ -168,7 +168,7 @@ secret = "{}"
                 params: vec![SqlValue::Text("alpha".to_string()), SqlValue::Integer(111)],
             },
             None,
-            "tinycloud.sql/write".to_string(),
+            "tinycloud.sql/admin".to_string(),
         )
         .await?;
     sql_service

test/w5-policy-runtime-node-e2e/tests/sql_isolation.rs

@@ -150,7 +150,7 @@ async fn legacy_web_artifacts_load_via_invoke_after_alias_cutover_and_restart()
             physical,
             serde_json::from_value(sql_write_body(row))?,
             None,
-            "tinycloud.sql/write".into(),
+            "tinycloud.sql/admin".into(),
         )
         .await?;
     }
@@ -322,7 +322,9 @@ async fn legacy_web_artifacts_load_via_invoke_after_alias_cutover_and_restart()
         &owner_vm,
         vec![],
         &fresh_resource,
-        "tinycloud.sql/write",
+        // `sql_write_body` creates its table and inserts a row: `admin`
+        // carries both (schema changes need `schema`, TC-1581).
+        "tinycloud.sql/admin",
         "urn:uuid:00000000-0000-4000-8000-000000000024",
     )?;
     let (status, response) = invoke_json(&client, fresh_write, &sql_write_body("new-row")).await;
@@ -528,7 +530,7 @@ async fn legacy_alias_collision_refuses_the_whole_transaction() -> Result<()> {
             name,
             serde_json::from_value(sql_write_body("row"))?,
             None,
-            "tinycloud.sql/write".into(),
+            "tinycloud.sql/admin".into(),
         )
         .await?;
     }
@@ -575,7 +577,7 @@ async fn legacy_inventory_attributes_from_invoked_abilities_without_a_grant() ->
         &owner_vm,
         vec![],
         &resource,
-        "tinycloud.sql/write",
+        "tinycloud.sql/admin",
         "urn:uuid:00000000-0000-4000-8000-000000000025",
     )?;
     let (status, response) = invoke_json(&client, header, &sql_write_body("digest-row")).await;
@@ -588,7 +590,7 @@ async fn legacy_inventory_attributes_from_invoked_abilities_without_a_grant() ->
         "only",
         serde_json::from_value(sql_write_body("legacy-row"))?,
         None,
-        "tinycloud.sql/write".into(),
+        "tinycloud.sql/admin".into(),
     )
     .await?;
     let items = database_migration::inventory(&conn, &datadir).await?;
@@ -608,7 +610,7 @@ async fn legacy_inventory_attributes_from_invoked_abilities_without_a_grant() ->
         &owner_vm,
         vec![],
         &resource,
-        "tinycloud.sql/write",
+        "tinycloud.sql/admin",
         "urn:uuid:00000000-0000-4000-8000-000000000026",
     )?;
     let (status, response) = invoke_json(&client, unfenced_write, &sql_write_body("blocked")).await;
@@ -994,7 +996,7 @@ async fn sql_paths_do_not_share_databases_and_grants_are_exact() -> Result<()> {
     let tempdir = TempDir::new()?;
     let engine = Engine {
         service: "sql",
-        write_ability: "tinycloud.sql/write",
+        write_ability: "tinycloud.sql/admin",
         read_ability: "tinycloud.sql/read",
         grant_seed: b"tc780-n1-sql-read-appA-connectors",
         nonce_prefix: "000000000a",
@@ -1046,7 +1048,7 @@ fn regression_engine(service: &'static str) -> Result<Engine> {
     Ok(match service {
         "sql" => Engine {
             service,
-            write_ability: "tinycloud.sql/write",
+            write_ability: "tinycloud.sql/admin",
             read_ability: "tinycloud.sql/read",
             grant_seed: b"tc780-n2-ancestor-sql",
             nonce_prefix: "000000000c",

test/w5-policy-runtime-node-e2e/tests/tc119_registry_wire_paths.rs

@@ -200,7 +200,7 @@ secret = "{}"
                 params: vec![SqlValue::Integer(1)],
             },
             None,
-            "tinycloud.sql/write".to_string(),
+            "tinycloud.sql/admin".to_string(),
         )
         .await?;
 

tinycloud-core/src/data_owners.rs

@@ -898,7 +898,7 @@ mod tests {
             &other_sql,
             "default",
             "main",
-            "tinycloud.sql/write",
+            "tinycloud.sql/schema",
             crate::sql::SqlRequest::Execute {
                 sql: "CREATE TABLE notes (id INTEGER)".to_string(),
                 params: vec![],