Push down JSONB existence predicates

[k-bx]

Summary

• Push PostgreSQL jsonb ? text predicates down for native ClickHouse JSON columns.
• Preserve PostgreSQL top-level object-key and array-string semantics.
• Support compatibility views that parse JSON documents stored in ClickHouse String columns, while leaving arbitrary JSON expressions local.
• Handle nullable String-backed documents without producing unsupported nullable nested array types in ClickHouse.

Validation

• make -j2
• Focused PostgreSQL 18 regression test for native JSON, nullable String-backed JSON, combined predicates, aggregate pushdown, and local fallback
• Tested on a real-world production canary system with PostgreSQL 18 and ClickHouse 26.3: EXPLAIN (FORMAT JSON, VERBOSE) confirmed that JSONHas, nullable-document handling, and count were all executed remotely; a month-range aggregate completed successfully, and a pushed single-row predicate matched a deliberately local PostgreSQL evaluation on the same replicated row

[theory]

Thank you, looks handy, but the tests are insufficient. Can you improve them?

[theory]

This test does not properly validate execution on the ClickHouse side. Please create the corresponding tables and data in ClickHouse and execute the queries with and without EXPLAIN (VERBOSE COSTS OFF to show both the CH SQL generated and its successful execution.

[k-bx]

Updated in 9fa88ff.

The regression now creates the native JSON and Nullable(String)-backed tables in ClickHouse, inserts representative object, array, scalar, and NULL data, and runs each predicate both with EXPLAIN (VERBOSE, COSTS OFF) and as an executing query. The expected output includes the full generated ClickHouse SQL and returned rows for native JSON, String-backed compatibility views, combined predicates, and the intentionally local fallback.

Validation:

• make -j2
• Full focused regression against PostgreSQL 18 / ClickHouse 26.3 on an isolated Kharkiv test database
• The same test update was applied to agent/v0.3.2-analytics-backports at 1869dcf

The PR branch was also rebased onto current main. GitHub Actions are currently awaiting maintainer approval.

[k-bx]

@theory I want to add that this has been used in (relatively-heavy) production instances at my org for quite some time, ever since PR was opened

[theory]

Almost there. I'm unable to push to your branch for some reason, so I pushed to the k-bx-agent/jsonb-exists-pushdown in our repo. Please have a look. In addition to the comments here in the PR, the results for ClickHouse 23.8-24.3 are unexpected. Here's the diff:

--- test/expected/jsonb_exists.out	2026-08-05 16:45:10
+++ test/expected/jsonb_exists_1.out	2026-08-05 16:52:46
@@ -84,7 +84,9 @@
  id 
 ----
   1
-(1 row)
+  2
+  3
+(3 rows)

Those results are quite wrong.

[theory]

Wow, okay, I think this makes sense now that I format it to make it more legible:

WHERE (
    (
        if(
            isNull(toJSONString(document)) OR isNull('key'),
            NULL,
            multiIf(
                JSONType(ifNull(toJSONString(document), 'null')) = 'Object',
                    JSONHas(ifNull(toJSONString(document), 'null'), 'key'),
                JSONType(ifNull(toJSONString(document), 'null')) = 'Array',
                arrayExists(
                    x -> JSONType(x) = 'String' AND JSONExtractString(x) = 'key', 
                    JSONExtractArrayRaw(ifNull(toJSONString(document), 'null'))
                ),
                0
            )
        )
    )
)

I get it for String, but is all that really necessary for a ClickHouse JSON column?

[theory]

Woof, okay, this makes sense:

SELECT id
FROM jsonb_exists.string_documents
WHERE (
    (
        if(
            isNull(document) OR isNull('key'),
            NULL,
            if(
                NOT isValidJSON(ifNull(document, 'null')),
                throwIf(1, 'invalid input syntax for type json'),
                multiIf(
                    JSONType(ifNull(document, 'null')) = 'Object',
                        JSONHas(ifNull(document, 'null'), 'key'),
                    JSONType(ifNull(document, 'null')) = 'Array',
                        arrayExists(
                            x -> JSONType(x) = 'String' AND JSONExtractString(x) = 'key',
                            JSONExtractArrayRaw(ifNull(document, 'null'))
                        ),
                    0
                )
            )
        )
    )
)

I don't suppose we can include any contextual information for the error? Maybe it could include the text it tires to pars into JSON.

[theory]

Okay, trying to understand this. I gather the point is to allow ? to work on a text column that Postgres implicitly uses jsonb_in to convert before evaluating. Is that correct?

I wonder if we'd be better off adding explicit pushdown for jsonb_in() (and perhaps other functions) as its own thing. That would mean adding case F_JSONB_IN: to lookup_builtin_func and creating the appropriate departed for it. Probably fine to do as a follow-up PR, but I suspect this won't be the last time we need to push down _in functions.

[k-bx]

Pushed 8f1948b, rebased onto current main, addressing the August 5 review and the related edge cases surfaced in #344:

• Native ClickHouse JSON now deparses directly to JSONHas(document, key) and is shipped only on ClickHouse 24.8+, avoiding the incorrect legacy Object('json') key-existence behavior.
• String-backed compatibility views ship on ClickHouse 23.8+ and preserve PostgreSQL object-key, array-string, and top-level scalar-string semantics. The scalar "key" case is now covered explicitly.
• Invalid String-backed JSON now raises an error that includes the offending document text.
• The explicit jsonb_in(text::cstring) recognition is documented in code as a narrowly scoped compatibility-view special case; general jsonb_in() pushdown remains suitable for a follow-up.
• Test setup now uses clickhouse_perform.
• The array predicate no longer places the RHS expression inside a fixed-name lambda scope, so a query such as document ? jsonb_exists_element cannot be captured by the lambda parameter. A column-RHS regression verifies the pushed query and returned rows.
• Added version-specific expected output for ClickHouse 24.8+, 23.8–24.3, and 23.3; older unsupported forms stay local.

Validation completed successfully:

• make -j2
• PostgreSQL 18 + ClickHouse 26.3.12.3
• PostgreSQL 18 + ClickHouse 23.8.16.40
• PostgreSQL 18 + ClickHouse 23.3.22.3

The 26.3 and 23.8 runs execute the supported predicates remotely; the 23.3 run confirms safe local evaluation.

[theory]

I don't know what "compatibility views" means, it's not a term of art in this project. I presume you mean a text foreign table column that maps to a JSON column in ClickHouse, yes?

[k-bx]

Sorry, “compatibility view” was imprecise. I mean a PostgreSQL view over a foreign table whose underlying ClickHouse column is String containing serialized JSON. The foreign column is mapped as PostgreSQL text, and the view exposes it as jsonb using jsonb_in(text_column::cstring).

So it is not a text foreign column mapping to a ClickHouse JSON column. I’ll replace the term with that explicit description in the code and documentation.

[k-bx]

Addressed in ce56e48, rebased onto the latest main:

• Removed the JSONB-specific ClickHouse version gates. Older native JSON implementations now return a remote error without crashing the PostgreSQL backend, following the existing project convention.
• Restored the original subplan_gate_user_mapping name; the broader rename is gone.
• Avoided column capture altogether. The document and RHS column expressions are now outside the lambda body. The RHS is supplied through a parallel array, and the lambda references only its local parameters. The regression deliberately names the RHS foreign column candidate, the same as the lambda parameter, and verifies the correct rows (1, 2, 4, 6).

Validated with PostgreSQL 18 against ClickHouse 26.3, 23.8, and 23.3. The older versions produce the expected native-JSON remote error while the String-backed cases continue to execute correctly. make -j2 also passes.