refactor(proxy): route each statement through one refuse-by-default decision

CHANGELOG.md

@@ -37,6 +37,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ### Fixed
 
+- **Unknown column error message**: the type-checking error for a column a table does not have now names the column and the table the right way round, for example `Column: email not found for table: users`. It previously swapped them.
+
 - **Schema-qualified table names**: Proxy now resolves a schema-qualified name such as `public.users` to the table in that schema, so statements that use one are encrypted instead of failing type checking. This includes a table whose name is also used in another schema on the search path. Proxy keys encryption configuration by the unqualified table name, so when same-named tables in different schemas declare a column with different EQL domains, statements on that column are refused rather than encrypted with the other table's configuration. Same-named tables that declare the same domain share one configuration and key descriptor, so their ciphertexts and search terms are interchangeable.
 
 - **Prepared statement recovery and settings isolation**: failed or discarded protocol messages no longer replace encryption metadata for statements PostgreSQL retained. Pipelined close-and-reprepare recovers immediately, and refused operations use names that remain unpredictable within PostgreSQL's lookup limit. Unsupported keyset values fail explicitly, and stale prepared statements signal driver cache invalidation.
@@ -75,6 +77,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
   Compatibility note: while encrypted columns are configured, Proxy now refuses writes to tables in schemas outside the search path it loads, to views and foreign tables (Proxy loads only base tables), and writes that use a bare table name found in more than one schema on that search path. Before upgrading, check that applications do not write to such tables through Proxy, or qualify the table name with its schema.
 
+- **Writes that fail type checking**: when encrypted columns are configured, including legacy EQL v2 columns, Proxy now refuses a statement that writes data and cannot be type checked, such as `MERGE` or SQL `PREPARE` of an `INSERT`, whether or not `CS_DEVELOPMENT__ENABLE_MAPPING_ERRORS` is set. Previously such a statement was forwarded as written, so its values could reach PostgreSQL in plaintext. This applies to writes to any table while any column is encrypted, including bulk `UPDATE ... FROM (VALUES ...)` and `INSERT ... SELECT ... FROM unnest(...)`, which the type checker does not support yet, and to a write inside `COPY (...) TO`. Read-only statements that cannot be type checked are still forwarded, except one that uses an operation an encrypted column's EQL type does not support, which is now refused. That covers a comparison such as `<` on a column with no ordering index, whose operand would otherwise reach PostgreSQL as plaintext, and `ORDER BY`, `GROUP BY`, `DISTINCT` or `UNION` over an encrypted column without the ordering or equality index they need, which PostgreSQL would otherwise evaluate over the ciphertext and return wrong results without an error. See [Unsupported encrypted operation](docs/errors.md#mapping-unsupported-encrypted-operation). Proxy now tracks each connection's native temporary tables, and checks each statement in a simple query against the schema as the encryption-neutral DDL earlier in that query leaves it, so writes to those tables type check like any other. A write to a bare name that names both a temporary and a permanent table is refused as ambiguous; qualify it, as in `pg_temp.name`. See [Ambiguous table](docs/errors.md#mapping-ambiguous-table). Every statement in a simple query is now checked, including those after one that cannot be type checked. See [Statement could not be type checked](docs/errors.md#mapping-statement-could-not-be-type-checked).
+
+  Compatibility note: before upgrading, check that applications do not send these writes through Proxy while encrypted columns are configured: `MERGE`; SQL `PREPARE` of a write; a write inside `COPY (...) TO`; bulk `UPDATE ... FROM (VALUES ...)`; `INSERT ... SELECT ... FROM unnest(...)`; and other writes Proxy cannot type check, including writes to tables with no encrypted columns. Reads that use an operation an encrypted column does not support, including `ORDER BY`, `GROUP BY`, `DISTINCT` or `UNION` over a column without the index they need, are now refused too.
+
 - **Short encryption results**: Proxy now refuses a statement when the encryption service returns fewer values than it was given, instead of sending the unmatched parameters unencrypted. See [Encryption result count mismatch](docs/errors.md#encrypt-result-count-mismatch).
 
 - **Updated PostgreSQL client library**: Proxy is now built against `tokio-postgres` 0.7.18, which resolves [GHSA-3gjw-f78c-vvpw](https://github.com/advisories/GHSA-3gjw-f78c-vvpw). No configuration change is required.

docs/errors.md

@@ -18,6 +18,8 @@
   - [Unmodelled DDL](#mapping-unmodelled-ddl)
   - [Unmappable encrypted column](#mapping-unmappable-encrypted-column)
   - [Unresolvable table](#mapping-unresolvable-table)
+  - [Ambiguous table](#mapping-ambiguous-table)
+  - [Unsupported encrypted operation](#mapping-unsupported-encrypted-operation)
   - [Unknown prepared statement](#mapping-unknown-prepared-statement)
   - [Internal Error](#mapping-internal-error)
 
@@ -245,7 +247,18 @@ CipherStash Proxy checks SQL statements against the database schema to transpare
 
 The behaviour of Proxy depends on the `mapping_errors_enabled` configuration.
 
-When `mapping_errors_enabled` is `false` (the default), then type check errors are logged, and the statement is passed through to the database.
+When `mapping_errors_enabled` is `false` (the default), then type check errors are logged, and a read-only statement is passed through to the database.
+A statement that writes data (`INSERT`, `UPDATE`, `DELETE` or `MERGE`, including one inside `PREPARE`, `EXPLAIN`, `COPY (...) TO` or a `WITH` clause) is refused instead whenever any column is encrypted, because Proxy cannot tell whether it writes plaintext to an encrypted column.
+This applies to every table, including tables with no encrypted columns, because Proxy cannot tell which columns a write it cannot type check reaches.
+For example, these statements are refused while any column is encrypted:
+
+- `UPDATE users SET name = v.name FROM (VALUES (1, 'Ada')) AS v(id, name) WHERE users.id = v.id`
+- `INSERT INTO users (id) SELECT x FROM unnest(ARRAY[1, 2]) AS x`
+- `INSERT INTO names SELECT relname FROM pg_catalog.pg_class`
+
+Proxy type checks writes to the connection's native temporary tables, and to tables that encryption-neutral DDL earlier in the same simple query creates or alters, so those are not refused.
+`CREATE TABLE IF NOT EXISTS` is the exception: the table may already exist with encrypted columns that Proxy has not loaded, so a write after it in the same simple query is refused unless an earlier statement created the table.
+In a simple query, a statement is also refused when another statement in the same query needs encrypting.
 
 When `mapping_errors_enabled` is `true`, then type check errors are raised, and statement execution halts.
 
@@ -254,7 +267,7 @@ Configure this setting with the environment variable `CS_DEVELOPMENT__ENABLE_MAP
 In our experience, most production systems have a relatively small number of columns that require protection.
 As SQL is large and complex, instead of blocking statements with type check errors that are false negatives, the default behaviour of Proxy is to allow the statement.
 
-However, this does mean it is possible that a statement that references encrypted columns cannot be type-checked, and it will be passed through to the database.
+However, this does mean it is possible that a read-only statement that references encrypted columns cannot be type-checked, and it will be passed through to the database.
 When a statement is passed through to the database, the database's column constraints (provided by EQL) will catch the statement, and return a PostgreSQL error.
 
 Example constraint error:
@@ -421,7 +434,7 @@ Statement refers to table 'users', which is not in the database schema Proxy has
 
 ### Notes
 
-The refusal applies to `INSERT`, `UPDATE`, `DELETE` and `MERGE`, including one inside `PREPARE`, `EXPLAIN` or a `WITH` clause, when any column is encrypted.
+The refusal applies to `INSERT`, `UPDATE`, `DELETE` and `MERGE`, including one inside `PREPARE`, `EXPLAIN`, `COPY (...) TO` or a `WITH` clause, when any column is encrypted.
 It applies wherever the statement names the table, so `INSERT INTO users ... SELECT ... FROM other` is refused when `other` cannot be resolved.
 Read-only statements are not refused.
 When no columns are encrypted, Proxy forwards these statements.
@@ -443,6 +456,57 @@ If the table is in a schema outside the search path, add that schema to the sear
 If the table was created on another connection, retry once Proxy has reloaded the database schema.
 
 
+<!-- ---------------------------------------------------------------------------------------------------- -->
+
+
+## Ambiguous table <a id='mapping-ambiguous-table'></a>
+
+A statement that writes data refers to a table by a bare name that the connection uses for both a temporary table and a permanent table.
+PostgreSQL resolves the name to the temporary table, but if that table has been dropped somewhere CipherStash Proxy cannot see, such as inside a function or a `DO` block, the name means the permanent table.
+Proxy cannot tell which table the statement writes to, so it refuses the statement rather than forward its values unencrypted.
+
+### Error message
+
+```
+Statement refers to table 'notes', which names both a temporary table and a permanent table, so Proxy cannot tell which one it writes to. Qualify the name, as in 'pg_temp.notes'.
+```
+
+### Notes
+
+The refusal applies to the same statements as [Unresolvable table](#mapping-unresolvable-table), and only when any column is encrypted.
+Read-only statements are not refused.
+
+### How to fix
+
+Qualify the table name: `pg_temp.notes` for the temporary table, or the permanent table's schema, as in `public.notes`.
+Alternatively, give the temporary table a name no permanent table uses.
+
+Proxy assumes PostgreSQL's default search order, in which `pg_temp` comes first.
+It does not track `SET search_path`, so a client that puts `pg_temp` later in its search path should qualify table names.
+
+
+<!-- ---------------------------------------------------------------------------------------------------- -->
+
+
+## Unsupported encrypted operation <a id='mapping-unsupported-encrypted-operation'></a>
+
+A statement uses an operation on an encrypted column that the column's EQL type does not support, such as `<` or `ORDER BY` on a column with no ordering index, or `LIKE` on a column with no match index.
+CipherStash Proxy cannot encrypt the operand for that operation, so it refuses the statement rather than send the operand to the database as plaintext.
+
+### Error message
+
+```
+Statement uses an operation that an encrypted column's EQL type does not support: Type `...` does not satisfy bounds `Ord`.
+```
+
+### Notes
+
+The refusal applies to reads as well as writes, and whether or not `CS_DEVELOPMENT__ENABLE_MAPPING_ERRORS` is set.
+
+### How to fix
+
+Change the statement to use only operations the column supports, or change the column's EQL domain type to one that supports the operation.
+
 
 <!-- ---------------------------------------------------------------------------------------------------- -->
 

packages/cipherstash-proxy-integration/src/extended_protocol_error_messages.rs

@@ -48,10 +48,12 @@ mod tests {
         let sql = "INSERT INTO encrypted (id, encrypted_unconfigured) VALUES ($1, $2)";
         let result = client.query(sql, &[&id, &encrypted_text]).await;
 
+        // EQL Mapper cannot type check a write to a column it does not know, so
+        // Proxy refuses it rather than forward the values as written.
         assert_db_error(
             result,
             "ERROR",
-            "column \"encrypted_unconfigured\" of relation \"encrypted\" does not exist",
+            "Statement could not be type checked: Column: encrypted_unconfigured not found for table: encrypted. For help visit https://github.com/cipherstash/proxy/blob/main/docs/errors.md#mapping-statement-could-not-be-type-checked",
         );
     }
 

packages/cipherstash-proxy/CONTEXT.md

@@ -54,14 +54,28 @@ _Avoid_: session (`start_session`, `SessionId` and the
 `..._statements_session_duration_seconds` metric all use this sense and are misnamed).
 
 **Passthrough**:
-Traffic forwarded without encryption or rewriting. Four independent conditions produce
-it and they are not interchangeable: no encrypt config is loaded, mapping is disabled by
-configuration, `SET CIPHERSTASH.UNSAFE_DISABLE_MAPPING` was issued on this connection, or
-the **type-check fallback** forwarded a statement EQL Mapper could not type check while
-mapping errors are off. The fallback never applies to a refusal, and never to a simple
-Query with another statement that needs mapping.
+Traffic forwarded without encryption or rewriting. Mapping disabled by configuration
+forwards the whole connection. Otherwise each statement's **route** decides, and a
+statement is forwarded only for a named reason: `SET CIPHERSTASH.UNSAFE_DISABLE_MAPPING`
+was issued on this connection, EQL Mapper does not type check its kind, it touches no
+encrypted column, or the **type-check fallback**. These are not interchangeable.
 Name the condition rather than saying "passthrough" alone.
 
+**Route**:
+What Proxy does with one statement from a Query or Parse: map it, forward it for a named
+reason, or refuse it. Refusing is the default. A failed type check falls back to
+forwarding only for a coverage gap (the statement could not be type checked, or an
+internal mapper error), only while mapping errors are off, and never in a simple Query with another statement to map.
+When encrypted columns are configured, a write never falls back.
+Each statement in a simple Query is type checked against a **batch view**: the effective schema plus the encryption-neutral DDL of the statements before it.
+
+**Connection temporary tables**:
+The native temporary tables one connection has created, which the type checker resolves before permanent tables, as `pg_temp` is searched first.
+A table is added when PostgreSQL confirms its `CREATE`, and removed by `DROP`, `DISCARD`, `ON COMMIT DROP` at commit, or rollback of the transaction or savepoint that created it.
+A bare name that also names a permanent table is ambiguous and does not resolve, because a stale record would otherwise hide the permanent table.
+A write that uses it is refused with its own error.
+Proxy does not track `SET search_path`, so it assumes PostgreSQL's default of searching `pg_temp` first; a client that moves `pg_temp` later in its search path can make a bare name mean a permanent table Proxy has not loaded.
+
 ### Encryption
 
 **Column**:

packages/cipherstash-proxy/src/error.rs

@@ -59,30 +59,17 @@ pub enum Error {
 }
 
 impl Error {
-    /// Whether this error must be returned to the client even when the proxy is
-    /// configured to fall back to forwarding statements it could not map.
-    ///
-    /// That fallback exists for mapper *coverage gaps*: a statement Proxy does
-    /// not understand is passed through unchanged, on the reasoning that it
-    /// probably touches nothing encrypted. Some errors are not coverage gaps but
-    /// deliberate refusals, where forwarding the statement is the very harm the
-    /// error was raised to prevent. Those are listed here and are never
-    /// downgraded to a passthrough.
-    pub fn must_fail_closed(&self) -> bool {
+    /// Whether this error is a mapper *coverage gap*: EQL Mapper could not type
+    /// check the statement, which is not evidence that it touches anything
+    /// encrypted. Only a coverage gap may fall back to forwarding the
+    /// statement as written. Every other error is a refusal, so an error
+    /// added later fails closed without being listed anywhere.
+    pub fn is_coverage_gap(&self) -> bool {
         matches!(
             self,
-            // Forwarding a statement that references a legacy EQL v2 column
-            // stores plaintext in a column its operator believes is encrypted
-            // (CIP-3688). No configuration may turn that back on. Nor may a
-            // write to a table Proxy cannot resolve be forwarded.
             Error::Mapping(
-                MappingError::UnmappableEncryptedColumn { .. }
-                    | MappingError::UnresolvableTable { .. }
-                    | MappingError::DependentStatementAfterDdl
-                    | MappingError::DependentStatementAfterSettings
-                    | MappingError::PreparedSettingsChanged
-                    | MappingError::UnmodelledDdl
-            ) | Error::Encrypt(EncryptError::InvalidInboundEqlPayload)
+                MappingError::StatementCouldNotBeTypeChecked(_) | MappingError::Internal(_)
+            )
         )
     }
 }
@@ -158,6 +145,12 @@ pub enum MappingError {
     #[error("Statement refers to table '{table}', which is not in the database schema Proxy has loaded, so Proxy cannot tell whether it writes to an encrypted column. For help visit {}#mapping-unresolvable-table", ERROR_DOC_BASE_URL)]
     UnresolvableTable { table: String },
 
+    #[error("Statement refers to table '{table}', which names both a temporary table and a permanent table, so Proxy cannot tell which one it writes to. Qualify the name, as in 'pg_temp.{table}'. For help visit {}#mapping-ambiguous-table", ERROR_DOC_BASE_URL)]
+    AmbiguousTable { table: String },
+
+    #[error("Statement uses an operation that an encrypted column's EQL type does not support: {}. For help visit {}#mapping-unsupported-encrypted-operation", _0, ERROR_DOC_BASE_URL)]
+    UnsupportedEncryptedOperation(String),
+
     #[error("Column '{table}.{column}' is declared as '{column_type}', a legacy EQL v2 type that this version of CipherStash Proxy cannot encrypt or decrypt. Statements referencing '{table}' are refused so that plaintext is never written to the column. Migrate '{table}.{column}' to an EQL v3 domain type. For help visit {}#mapping-unmappable-encrypted-column", ERROR_DOC_BASE_URL)]
     UnmappableEncryptedColumn {
         table: String,

packages/cipherstash-proxy/src/postgresql/context/mod.rs

@@ -1346,9 +1346,10 @@ where
             .map_err(|_| crate::error::ContextError::ProtocolStateUnavailable.into())
     }
 
-    /// Returns the resolver for this connection's effective schema snapshot.
+    /// Returns the resolver for this connection's effective schema snapshot,
+    /// with its native temporary tables.
     pub fn get_table_resolver(&self) -> Arc<TableResolver> {
-        self.schema_middleware.resolver()
+        self.schema_middleware.effective_resolver()
     }
 
     /// Records schema intent for a parsed prepared statement.
@@ -1732,21 +1733,14 @@ where
         Ok(())
     }
 
-    /// Records the native tables a simple-query batch creates, so the batch's
-    /// later statements can write to them.
-    pub fn note_native_tables(&self, statements: &[sqltk::parser::ast::Statement]) {
-        self.schema_middleware.note_native_tables(statements);
-    }
-
-    /// Whether every table `statement` writes to is a native table this
-    /// connection created that the schema may not model yet.
-    pub fn writes_only_native_tables(&self, statement: &sqltk::parser::ast::Statement) -> bool {
-        eql_mapper::write_targets(statement).is_some_and(|targets| {
-            !targets.is_empty()
-                && targets
-                    .iter()
-                    .all(|target| self.schema_middleware.is_native_table(target))
-        })
+    /// The resolver to type check each statement of a simple-query batch with,
+    /// which sees the encryption-neutral schema changes of the statements
+    /// before it.
+    pub fn batch_table_resolvers(
+        &self,
+        statements: &[sqltk::parser::ast::Statement],
+    ) -> Vec<Arc<TableResolver>> {
+        self.schema_middleware.batch_resolvers(statements)
     }
 
     /// Whether any column is encrypted. A column can be encrypted without