fix(proxy): refuse COPY FROM into encrypted columns, check statements after COPY FROM STDIN

CHANGELOG.md

@@ -81,6 +81,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
   Compatibility note: before upgrading, check that applications do not send these writes through Proxy while encrypted columns are configured: `MERGE`; SQL `PREPARE` of a write; a write inside `COPY (...) TO`; bulk `UPDATE ... FROM (VALUES ...)`; `INSERT ... SELECT ... FROM unnest(...)`; and other writes Proxy cannot type check, including writes to tables with no encrypted columns. Reads that use an operation an encrypted column does not support, including `ORDER BY`, `GROUP BY`, `DISTINCT` or `UNION` over a column without the index they need, are now refused too.
 
+- **`COPY ... FROM` into encrypted columns**: when encrypted columns are configured, Proxy now refuses a `COPY ... FROM` (from the client, a file or a program) that would fill an encrypted column, with an error that names the column, as Proxy does not encrypt `COPY` data. Previously such a `COPY` was forwarded, and stored plaintext in the encrypted column. A `COPY` with no column list fills every column, so it is refused when the table has any encrypted column. A `COPY ... FROM` into a table Proxy cannot resolve, or into a column it has not loaded, is refused too, whatever `CS_DEVELOPMENT__ENABLE_MAPPING_ERRORS` is set to. A simple query that has a `COPY` and a statement Proxy encrypts is refused; send the `COPY` in a query of its own. See [COPY into encrypted column](docs/errors.md#mapping-copy-into-encrypted-column).
+
+- **Statements after `COPY ... FROM STDIN`**: Proxy now forwards a `COPY ... FROM STDIN` into columns that are not encrypted, so `psql`'s `\copy` and driver `COPY` APIs can bulk-load plaintext tables through Proxy. Previously Proxy refused the statement as invalid SQL unless it ended with `;`. With a `;`, Proxy read every later statement in the same simple query as copy data and forwarded it unchecked, so a plaintext `INSERT` into an encrypted column could reach PostgreSQL. Those statements are now checked and encrypted like any other.
+
+  Compatibility note: before upgrading, check that applications do not `COPY` into encrypted columns through Proxy; load those columns with `INSERT` instead.
+
 - **Short encryption results**: Proxy now refuses a statement when the encryption service returns fewer values than it was given, instead of sending the unmatched parameters unencrypted. See [Encryption result count mismatch](docs/errors.md#encrypt-result-count-mismatch).
 
 - **Updated PostgreSQL client library**: Proxy is now built against `tokio-postgres` 0.7.18, which resolves [GHSA-3gjw-f78c-vvpw](https://github.com/advisories/GHSA-3gjw-f78c-vvpw). No configuration change is required.

docs/errors.md

@@ -19,6 +19,9 @@
   - [Unmappable encrypted column](#mapping-unmappable-encrypted-column)
   - [Unresolvable table](#mapping-unresolvable-table)
   - [Ambiguous table](#mapping-ambiguous-table)
+  - [COPY into encrypted column](#mapping-copy-into-encrypted-column)
+  - [COPY into unknown column](#mapping-copy-into-unknown-column)
+  - [COPY with a statement that uses an encrypted column](#mapping-copy-with-encrypted-statement)
   - [Unsupported encrypted operation](#mapping-unsupported-encrypted-operation)
   - [Unknown prepared statement](#mapping-unknown-prepared-statement)
   - [Internal Error](#mapping-internal-error)
@@ -488,6 +491,80 @@ It does not track `SET search_path`, so a client that puts `pg_temp` later in it
 <!-- ---------------------------------------------------------------------------------------------------- -->
 
 
+## COPY into encrypted column <a id='mapping-copy-into-encrypted-column'></a>
+
+A `COPY ... FROM` statement would load data into an encrypted column.
+CipherStash Proxy does not encrypt `COPY` data, so it refuses the statement rather than store plaintext in the encrypted column.
+
+### Error message
+
+```
+COPY into encrypted column 'users.encrypted_email' is not supported, as Proxy does not encrypt COPY data.
+```
+
+### Notes
+
+The refusal applies to a `COPY ... FROM STDIN`, and to a `COPY ... FROM` a file or a program.
+A `COPY` with no column list loads every column of the table, so it is refused when the table has any encrypted column.
+A `COPY ... FROM` into a table with no encrypted columns, or into only the plaintext columns of a table, is forwarded to PostgreSQL with its data.
+A `COPY ... TO` of a table or of a query that reads is forwarded, and returns encrypted columns as ciphertext, as Proxy does not decrypt `COPY` data.
+A `COPY ... FROM` into a table Proxy cannot resolve is refused as an [Unresolvable table](#mapping-unresolvable-table) or an [Ambiguous table](#mapping-ambiguous-table).
+That includes a table created with `CREATE TABLE ... (LIKE ...)` or `CREATE TABLE ... AS`, whose columns Proxy does not know.
+When no columns are encrypted, Proxy does not check the table of a `COPY`.
+A `COPY` after DDL that may change which columns are encrypted, in the same simple query, is still refused, as an `INSERT` would be.
+
+### How to fix
+
+Load the encrypted columns with `INSERT` statements through CipherStash Proxy, which encrypts the values.
+Use `COPY` only for columns that are not encrypted.
+
+
+<!-- ---------------------------------------------------------------------------------------------------- -->
+
+
+## COPY into unknown column <a id='mapping-copy-into-unknown-column'></a>
+
+A `COPY ... FROM` statement names a column that is not in the database schema CipherStash Proxy has loaded for its table.
+Proxy cannot tell whether the column is encrypted, so it refuses the statement.
+
+### Error message
+
+```
+COPY into column 'users.nickname', which is not in the database schema Proxy has loaded, is not supported, as Proxy cannot tell whether it is encrypted.
+```
+
+### Notes
+
+The refusal applies only when any column is encrypted, and whatever `CS_DEVELOPMENT__ENABLE_MAPPING_ERRORS` is set to.
+
+### How to fix
+
+Check that the column exists and that its name is spelled as PostgreSQL resolves it.
+If the column was added on another connection, retry once Proxy has reloaded the database schema.
+
+
+<!-- ---------------------------------------------------------------------------------------------------- -->
+
+
+## COPY with a statement that uses an encrypted column <a id='mapping-copy-with-encrypted-statement'></a>
+
+A simple query contains a `COPY` together with a statement that uses an encrypted column.
+CipherStash Proxy rewrites such a query to encrypt it, and cannot keep the `COPY` in the rewritten query.
+
+### Error message
+
+```
+COPY must be sent in a query of its own when the query has a statement that uses an encrypted column.
+```
+
+### How to fix
+
+Send the `COPY` in a query of its own.
+
+
+<!-- ---------------------------------------------------------------------------------------------------- -->
+
+
 ## Unsupported encrypted operation <a id='mapping-unsupported-encrypted-operation'></a>
 
 A statement uses an operation on an encrypted column that the column's EQL type does not support, such as `<` or `ORDER BY` on a column with no ordering index, or `LIKE` on a column with no match index.

packages/cipherstash-proxy-integration/Cargo.toml

@@ -16,6 +16,7 @@ cipherstash-config = { workspace = true }
 cipherstash-proxy = { path = "../cipherstash-proxy/" }
 chrono = { version = "0.4.39", features = ["clock"] }
 fake = { version = "4", features = ["chrono", "derive"] }
+futures-util = { version = "0.3", features = ["sink"] }
 hex = "0.4.3"
 postgres-types = { version = "0.2.9", features = ["derive"] }
 rand = "0.9"

packages/cipherstash-proxy-integration/src/copy_from_stdin.rs

@@ -0,0 +1,68 @@
+//! `COPY ... FROM STDIN` through Proxy, which tokio-postgres sends with the extended protocol.
+//! Proxy forwards a copy into a table with no encrypted columns, and refuses one into an
+//! encrypted column, as it does not encrypt copy data.
+
+#[cfg(test)]
+mod tests {
+    use crate::common::{clear, connect_with_tls, random_id, trace, PROXY};
+    use bytes::Bytes;
+    use futures_util::SinkExt;
+    use std::pin::pin;
+
+    #[tokio::test]
+    async fn copy_into_a_plaintext_table() {
+        trace();
+        clear().await;
+        let client = connect_with_tls(*PROXY).await;
+
+        let (first, second) = (random_id(), random_id());
+        let mut sink = pin!(client
+            .copy_in("COPY plaintext (id, plaintext) FROM STDIN")
+            .await
+            .unwrap());
+        sink.send(Bytes::from(format!("{first}\talice\n{second}\tbob\n")))
+            .await
+            .unwrap();
+        assert_eq!(sink.as_mut().finish().await.unwrap(), 2);
+
+        let rows = client
+            .query(
+                "SELECT plaintext FROM plaintext WHERE id = ANY($1) ORDER BY plaintext",
+                &[&vec![first, second]],
+            )
+            .await
+            .unwrap();
+        let copied: Vec<String> = rows.iter().map(|row| row.get(0)).collect();
+        assert_eq!(copied, vec!["alice", "bob"]);
+    }
+
+    #[tokio::test]
+    async fn copy_into_an_encrypted_column_is_refused() {
+        trace();
+        clear().await;
+        let client = connect_with_tls(*PROXY).await;
+
+        // tokio-postgres' `Display` no longer includes the server's message
+        let refusal = |err: tokio_postgres::Error| err.as_db_error().unwrap().message().to_owned();
+        let expected = "COPY into encrypted column 'encrypted.encrypted_text' is not supported";
+
+        let err = client
+            .copy_in::<_, Bytes>("COPY encrypted (id, encrypted_text) FROM STDIN")
+            .await
+            .err()
+            .unwrap();
+        assert!(refusal(err).starts_with(expected));
+
+        // A COPY with no column list fills every column, including the encrypted ones
+        let err = client
+            .copy_in::<_, Bytes>("COPY encrypted FROM STDIN")
+            .await
+            .err()
+            .unwrap();
+        assert!(refusal(err).starts_with(expected));
+
+        // The connection is still usable
+        let rows = client.simple_query("SELECT 1").await.unwrap();
+        assert!(!rows.is_empty());
+    }
+}

packages/cipherstash-proxy-integration/src/lib.rs

@@ -1,5 +1,6 @@
 mod common;
 mod connection_resilience;
+mod copy_from_stdin;
 mod decrypt;
 mod diagnostics;
 mod disable_mapping;

packages/cipherstash-proxy/CONTEXT.md

@@ -67,6 +67,7 @@ reason, or refuse it. Refusing is the default. A failed type check falls back to
 forwarding only for a coverage gap (the statement could not be type checked, or an
 internal mapper error), only while mapping errors are off, and never in a simple Query with another statement to map.
 When encrypted columns are configured, a write never falls back.
+A `COPY ... FROM` is not type checked: when encrypted columns are configured, it is forwarded only if every column it fills resolves to a native column.
 Each statement in a simple Query is type checked against a **batch view**: the effective schema plus the encryption-neutral DDL of the statements before it.
 
 **Connection temporary tables**:

packages/cipherstash-proxy/src/error.rs

@@ -148,6 +148,15 @@ pub enum MappingError {
     #[error("Statement refers to table '{table}', which names both a temporary table and a permanent table, so Proxy cannot tell which one it writes to. Qualify the name, as in 'pg_temp.{table}'. For help visit {}#mapping-ambiguous-table", ERROR_DOC_BASE_URL)]
     AmbiguousTable { table: String },
 
+    #[error("COPY into encrypted column '{table}.{column}' is not supported, as Proxy does not encrypt COPY data. For help visit {}#mapping-copy-into-encrypted-column", ERROR_DOC_BASE_URL)]
+    CopyIntoEncryptedColumn { table: String, column: String },
+
+    #[error("COPY into column '{table}.{column}', which is not in the database schema Proxy has loaded, is not supported, as Proxy cannot tell whether it is encrypted. For help visit {}#mapping-copy-into-unknown-column", ERROR_DOC_BASE_URL)]
+    CopyIntoUnknownColumn { table: String, column: String },
+
+    #[error("COPY must be sent in a query of its own when the query has a statement that uses an encrypted column. For help visit {}#mapping-copy-with-encrypted-statement", ERROR_DOC_BASE_URL)]
+    CopyWithEncryptedStatement,
+
     #[error("Statement uses an operation that an encrypted column's EQL type does not support: {}. For help visit {}#mapping-unsupported-encrypted-operation", _0, ERROR_DOC_BASE_URL)]
     UnsupportedEncryptedOperation(String),
 

packages/cipherstash-proxy/src/postgresql/context/lifecycle.rs

@@ -146,7 +146,7 @@ impl<S: EncryptionService> Context<S> {
             }
         };
         self.prepare_schema_for_statement().await?;
-        if eql_mapper::requires_type_check(statement) {
+        if eql_mapper::requires_schema(statement) {
             self.wait_for_schema_execution().await;
             self.ensure_schema_modelled()?;
         }

packages/cipherstash-proxy/src/postgresql/context/mod.rs

@@ -1539,7 +1539,7 @@ where
         for (statement, _, changes) in
             settings_protocol::project_settings_effects(&state.settings, statements)?
         {
-            if changed && eql_mapper::requires_type_check(statement) {
+            if changed && eql_mapper::requires_schema(statement) {
                 return Ok(true);
             }
             changed |= changes;

packages/cipherstash-proxy/src/postgresql/context/settings_protocol.rs

@@ -170,7 +170,7 @@ impl SettingsProtocol {
             Prepared {
                 operation,
                 command,
-                needs_mapping: eql_mapper::requires_type_check(statement),
+                needs_mapping: eql_mapper::requires_schema(statement),
                 settings,
                 literal_sensitive: false,
             },