From 81db83e92190f1548b94477f0fc0bedc5968c58f Mon Sep 17 00:00:00 2001 From: bRong Njam Date: Mon, 12 Oct 2026 18:16:19 +0800 Subject: [PATCH 01/20] fix(sirius): pin merged bounded MO input coalescing --- docs/design/sirius-public-campaign.md | 25 +++++++++++++++++++++++++ third_party/sirius | 2 +- 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/docs/design/sirius-public-campaign.md b/docs/design/sirius-public-campaign.md index 2dcd3b283f60e..bd7229adcb54b 100644 --- a/docs/design/sirius-public-campaign.md +++ b/docs/design/sirius-public-campaign.md @@ -318,6 +318,31 @@ not restore admission or publish complete evidence. Small live-writer tests exercise quota growth during finalization/export and preserve the original reduced-quota counterexamples. +## Bounded input batching integration + +Sirius #31 merged as `b953bb5a87788e17557929987208b3a0068d933b`. +This follow-up pins that merged revision and rebuilds the guarded release SDK. +Native input now collects toward the existing 32 MiB expanded target, flushing +on EOS, source bounds or producer credit pressure. Both readiness and direct +claims enforce the policy; immutable size/string/NULL summaries remain charged +to the 64 MiB input window. Public ABI and exact numeric semantics are preserved. + +The user approved the batching-plus-MO-integration plan with EOS/pressure-only +flushing on 2026-10-12. Premerge development validation uses MO base `64b7a43852`, +Sirius `955b6881` versus `585536bf`, the same frozen provider and immutable +datasets, two streams, one excluded warmup and five measured repetitions. Q1 GPU +medians improve from 0.775129 to 0.185689 seconds at SF1 and from 9.296148 to +2.217461 seconds at SF10; median input units fall from 600 to 12 and 7,264 to 117. +All 22 SF1 public comparisons pass exactly. These targeted development results +retain their identities and do not replace the original D release records. + +Merged-source release validation requires the complete 744-execution timing and +correctness schedule plus the separate 264-execution resource schedule, existing +numeric/lifecycle/sanitizer gates and a fresh matched full-suite regression limit +of 5%. Generate captures outside Git and preserve their actual source/binary/SDK +identities. E and numeric execution changes remain separate; no sidecar repository +retirement is required. + ## Release validation record The generated development and release captures are preserved in contributor-fork diff --git a/third_party/sirius b/third_party/sirius index 955b6881afa6e..b953bb5a87788 160000 --- a/third_party/sirius +++ b/third_party/sirius @@ -1 +1 @@ -Subproject commit 955b6881afa6e92d0995f43e3fe9c383d86981c0 +Subproject commit b953bb5a87788e17557929987208b3a0068d933b From e84abbdedc002f19118879676f81685602cc6b70 Mon Sep 17 00:00:00 2001 From: bRong Njam Date: Mon, 12 Oct 2026 19:37:50 +0800 Subject: [PATCH 02/20] docs(sirius): record merged batching release validation --- docs/design/sirius-public-campaign.md | 37 +++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/docs/design/sirius-public-campaign.md b/docs/design/sirius-public-campaign.md index bd7229adcb54b..b1f9ce7222380 100644 --- a/docs/design/sirius-public-campaign.md +++ b/docs/design/sirius-public-campaign.md @@ -343,6 +343,43 @@ of 5%. Generate captures outside Git and preserve their actual source/binary/SDK identities. E and numeric execution changes remain separate; no sidecar repository retirement is required. +The merged-source release capture uses MO `81db83e921` on base `080365f013`, +Sirius `b953bb5a` and guarded SDK +`4f2235b64347a2642dbad1328de2f031cfbfc401ce114ed55bafcb0e68802ee6`. +All 744 timing/correctness executions and the separate 264 resource executions +pass, with six zero-growth SF1/SF10 stream 1/2/4 allocation profiles. A fresh +744-execution old-native campaign uses the same MO base and semantic inputs. + +| GPU median full suite | Old pin `955b6881` | Merged pin `b953bb5a` | +| --- | ---: | ---: | +| SF1 | 4.216747 s | 2.537705 s | +| SF10 | 49.884309 s | 28.475628 s | + +Median suites and sums of query medians pass the 5% regression limit. Q1 medians +fall from 0.876390 to 0.230886 seconds at SF1 and from 9.446339 to 2.359481 seconds +at SF10; median input units fall from 670 to 12 and 7,258 to 117. GPU full-suite +execution remains slower than native MO on this RTX 3070. + +Final validation closes all ten lifecycle cases, fresh production numeric ABI +and 96 MiB result ownership checks, public runner/compiler normal and race +cleanup checks, and actual GPU data/cancellation plus cuVS/Sirius coexistence. +Native control normal/ASan+UBSan+LSan/TSan runs pass 91 cases/750 assertions; +their actual `585536bf` execution revision is retained and qualified by complete +native-tree equality with merged `b953bb5a`. Fresh GPU memory checking passes +30 cases/34,509 assertions. Four raw sanitizer statuses remain 99, reconciled +only against the exact Native A CUB and no-GPU-work cuDF pinned-pool controls +and all six current steady profiles. Fresh matched numeric cases each retain +21 samples; the maximum ratio is 1.021216 against the 1.1 limit. + +Linked native test builds refreshed a receipt-covered static archive. The real +SDK target and guarded consumer build regenerate validation SDK `f65b21f4`, +with identical native source, compilers and explicit link arguments. Reports +retain this actual consumer execution context separately from the preserved +measured service/runtime bundle and SDK `4f2235b6`; no capture is relabeled. +The artifact acceptance command reports local completion, with required CI and +reviewer approval remaining. Generated captures and raw repetitions stay outside +Git; the original D archive below retains its original measurements. + ## Release validation record The generated development and release captures are preserved in contributor-fork From a9758045e334ee156c4bc0cb91bd454042531ac1 Mon Sep 17 00:00:00 2001 From: bRong Njam Date: Mon, 12 Oct 2026 19:55:02 +0800 Subject: [PATCH 03/20] test(compile): select centroid transport in IVF protocol fixture --- docs/design/sirius-public-campaign.md | 12 ++++++++++++ pkg/sql/compile/vector_scan_protocol_test.go | 8 ++++++++ 2 files changed, 20 insertions(+) diff --git a/docs/design/sirius-public-campaign.md b/docs/design/sirius-public-campaign.md index b1f9ce7222380..9286c3d1decbb 100644 --- a/docs/design/sirius-public-campaign.md +++ b/docs/design/sirius-public-campaign.md @@ -380,6 +380,18 @@ The artifact acceptance command reports local completion, with required CI and reviewer approval remaining. Generated captures and raw repetitions stay outside Git; the original D archive below retains its original measurements. +The delivery sync includes authoritative main `11914526ba` (MEDIUMINT, +prepared-parameter, SHOW GRANTS, window and Parquet fixes). Canonical SQL, +native bridge/lowering, reader ownership, public campaign evidence and +collectors are unchanged. The new type/protocol branches concern MEDIUMINT or +prepared parameters; the immutable campaign uses ordinary column domains and +text-protocol queries. All eight affected SQL owning suites and all 22 actual +native preparations pass after this sync. The distributed-IVF protocol fixture +now explicitly selects its required centroid route through `gpu_mode=0`, +preserving all supported/old/unknown/cancellation assertions. Its earlier +GPU-default failure reproduced on the old-native baseline. This test correction +changes no production dispatch or captured measurement. + ## Release validation record The generated development and release captures are preserved in contributor-fork diff --git a/pkg/sql/compile/vector_scan_protocol_test.go b/pkg/sql/compile/vector_scan_protocol_test.go index 3c637ff19caef..5cb89343b0563 100644 --- a/pkg/sql/compile/vector_scan_protocol_test.go +++ b/pkg/sql/compile/vector_scan_protocol_test.go @@ -282,6 +282,14 @@ func TestRequiredIVFWorkersFallbackAsWholeQuery(t *testing.T) { for _, mode := range []string{"supported", "old", "unknown", "canceled"} { t.Run(mode, func(t *testing.T) { c, client := vectorPlacementCompile(t, engine.Nodes{{Id: "a", Addr: "a:6001"}, {Id: "b", Addr: "b:6001"}}) + // This protocol fixture requires the CPU centroid route on every + // worker, including when the binary's default enables GPU dispatch. + c.proc.SetResolveVariableFunc(func(name string, _, _ bool) (any, error) { + if name == "gpu_mode" { + return int8(0), nil + } + return nil, nil + }) moruntime.ServiceRuntime(c.proc.GetService()).SetGlobalVariables(moruntime.MOProtocolVersion, defines.MORPCVersion103) client.version = defines.MORPCVersion103 typ := plan.Type{Id: int32(types.T_int64)} From 00e1cbf694042e0c6d4302772c3728da7ff3c749 Mon Sep 17 00:00:00 2001 From: bRong Njam Date: Tue, 13 Oct 2026 02:49:23 +0800 Subject: [PATCH 04/20] docs(sirius): qualify current main conflict-resolution checks --- docs/design/sirius-public-campaign.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/design/sirius-public-campaign.md b/docs/design/sirius-public-campaign.md index 9286c3d1decbb..250c1b4cf9b49 100644 --- a/docs/design/sirius-public-campaign.md +++ b/docs/design/sirius-public-campaign.md @@ -392,6 +392,13 @@ preserving all supported/old/unknown/cancellation assertions. Its earlier GPU-default failure reproduced on the old-native baseline. This test correction changes no production dispatch or captured measurement. +The conflict-resolution sync includes authoritative main `0a4db0e6a6` at +`87803799f0`. Main's updated IndexSearchScan protocol fixture already selects +CPU centroid workers and uses its current protocol version; the integration +has no remaining Go-file delta against that base. The full compiler suite and +all 22 actual native preparations pass after the sync. These checks retain +their new execution identity; the measured release capture above is unchanged. + ## Release validation record The generated development and release captures are preserved in contributor-fork From 27a4586ad504f3f52a115ec9f92aacf84143268a Mon Sep 17 00:00:00 2001 From: bRong Njam Date: Tue, 13 Oct 2026 02:38:53 +0800 Subject: [PATCH 05/20] docs(sirius): record approved execution profile optimization --- docs/design/sirius-public-campaign.md | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/docs/design/sirius-public-campaign.md b/docs/design/sirius-public-campaign.md index 250c1b4cf9b49..2b6016b16cd76 100644 --- a/docs/design/sirius-public-campaign.md +++ b/docs/design/sirius-public-campaign.md @@ -1,6 +1,6 @@ # Public Sirius acceptance campaign -Design version: 3. Owners: MatrixOne public SQL, compile/CN evidence and +Design version: 4. Owners: MatrixOne public SQL, compile/CN evidence and `optools` campaign tooling. Tracks MO #28968 and parent #28966. The user approved this design before implementation on 2026-10-09 and selected @@ -11,6 +11,27 @@ native source for the guarded release SDK. The user approved this implementation revision on 2026-10-11 after selecting one complete D PR and separate allocation profiling without a native observability prerequisite. E/F and Docker changes are excluded. +## Approved execution-profile optimization (2026-10-13) + +The owner approved the focused performance plan before implementation. This +increment makes the owned campaign's Go CPU budget explicit and records pinned +allocation granularity; production backend and service defaults do not change. +Both `serve` and `freeze` require positive `--gomaxprocs` and accept +`--host-block-bytes` of 65536 or 1048576, defaulting to 65536. New provenance +records `config.gomaxprocs` and `config.native_host_block_bytes`. `run` validates +the frozen profile before starting services and launches native and embedded +routes with the same frozen budget, independently of its ambient environment. +Historical captures retain their actual settings and controller identity; +new runs require refrozen provenance. Runner/result schema version 2 is unchanged. + +The current-host qualification profile is 16 Go CPUs and 64 KiB host blocks. +Compare code revisions at identical settings; profile/kernel observations are +separate from uninstrumented timing. Canonical SF1/SF10 correctness, streams +1/2/4, bounded input/results, cleanup and GPU evidence remain mandatory. Repeat +pinned-pool controls at the new block size, retain the 5% full-suite regression +and 10% numeric gates, and do not infer acceptance from the targeted diagnostics. +Q15 shared-CTE reuse, Q13 join changes and default cutover require separate work. + ## Contract and choice For every admitted campaign cell, the immutable dataset, canonical SQL, public From e00fb95ad78e2758c651481c88331d10f29f64ae Mon Sep 17 00:00:00 2001 From: bRong Njam Date: Tue, 13 Oct 2026 03:58:24 +0800 Subject: [PATCH 06/20] Optimize frozen Sirius campaign profiles and string IN probes --- docs/design/sirius-public-campaign.md | 6 ++- optools/sirius_campaign.py | 51 ++++++++++++------ optools/sirius_campaign_test.py | 63 +++++++++++++++++++++++ optools/sirius_campaign_v2.py | 9 ++++ optools/sirius_campaign_v2_test.py | 25 +++++++++ pkg/sql/plan/function/operator_in.go | 19 ++++--- pkg/sql/plan/function/operator_in_test.go | 43 ++++++++++++++++ 7 files changed, 194 insertions(+), 22 deletions(-) diff --git a/docs/design/sirius-public-campaign.md b/docs/design/sirius-public-campaign.md index 2b6016b16cd76..a5693ced9a13e 100644 --- a/docs/design/sirius-public-campaign.md +++ b/docs/design/sirius-public-campaign.md @@ -225,7 +225,11 @@ the generator; a matching row count alone is insufficient. Use `sirius_campaign.py freeze` with the source root, packaged service, runner, dataset directory and both load reports to capture actual source/binary/SDK, -hardware and toolchain identities. Credentials remain in private `runner.json`. +hardware and toolchain identities. Supply `--gomaxprocs 16` to both `serve` and +`freeze` for the current host. Their `--host-block-bytes` defaults to 65536; +use 1048576 explicitly for the matched block-size control. `run` replays the +frozen CPU and block settings, and does not accept ambient overrides. +Credentials remain in private `runner.json`. `run` consumes that frozen provenance and an empty output directory; add `--resource` for the separate allocation campaign. Run serially, without concurrent builds, GPU tests or dataset writes. The controller owns service diff --git a/optools/sirius_campaign.py b/optools/sirius_campaign.py index 935896ce5ca40..40da183b6b619 100644 --- a/optools/sirius_campaign.py +++ b/optools/sirius_campaign.py @@ -854,7 +854,17 @@ def write_timing_tables(output, campaign, records, summary): write_private(output / "timings.md", "\n".join(lines) + "\n") -def native_configuration(workspace): +def execution_environment(gomaxprocs): + if type(gomaxprocs) is not int or gomaxprocs <= 0: + raise v2.CampaignError("campaign Go CPU budget must be a positive integer") + environment = os.environ.copy() + environment["GOMAXPROCS"] = str(gomaxprocs) + return environment + + +def native_configuration(workspace, host_block_bytes=65536): + if type(host_block_bytes) is not int or host_block_bytes not in (65536, 1048576): + raise v2.CampaignError("unsupported campaign host block size") return f"""sirius: space: gpu: @@ -863,6 +873,7 @@ def native_configuration(workspace): host: - numa_id: -1 memory_capacity: {16 << 30} + block_size: {host_block_bytes} telemetry: output_directory: {json.dumps(str(workspace/'native-telemetry'))} """ @@ -1089,6 +1100,14 @@ def run_complete(arguments, *, provenance=None): ] if base_runner.get("session") != expected_session: raise v2.CampaignError("live campaign session differs from frozen provenance") + campaign = v2.build_campaign( + provenance, + resource=arguments.resource, + flight_profiles=[ + {"id": route, "coverage": [], "status": "N/A", "reason": "not-provisioned"} + for route in ("flight-mo", "flight-tae") + ], + ) sirius = tomllib.loads((workspace / "cn-2.toml").read_text())["cn"]["sirius"] if any( sirius.get(key) != value @@ -1100,21 +1119,12 @@ def run_complete(arguments, *, provenance=None): "cleanup-timeout": "30s", }.items() ) or Path(sirius["native-config-path"]).read_text() != native_configuration( - workspace + workspace, provenance["config"]["native_host_block_bytes"] ): raise v2.CampaignError( "live campaign configuration differs from frozen profile" ) - campaign = v2.build_campaign( - provenance, - resource=arguments.resource, - flight_profiles=[ - {"id": route, "coverage": [], "status": "N/A", "reason": "not-provisioned"} - for route in ("flight-mo", "flight-tae") - ], - ) - environment = os.environ.copy() - environment["GOMAXPROCS"] = "4" + environment = execution_environment(provenance["config"]["gomaxprocs"]) run_workspace = workspace / f"campaign-{uuid.uuid4().hex}" run_workspace.mkdir(mode=0o700) (run_workspace / "scratch").mkdir(mode=0o700) @@ -1521,13 +1531,12 @@ def excluded(specs, phase_name, base): def start_cluster(arguments): + environment = execution_environment(arguments.gomaxprocs) workspace = arguments.workspace.resolve() workspace.mkdir(mode=0o700, parents=True) native = workspace / "sirius.yaml" - write_private(native, native_configuration(workspace)) + write_private(native, native_configuration(workspace, arguments.host_block_bytes)) configs = make_configs(workspace, native, 2, arguments.port_base) - environment = os.environ.copy() - environment["GOMAXPROCS"] = "4" services = [] previous_subreaper = v2.legacy._enter_exclusive_subreaper() try: @@ -1578,6 +1587,8 @@ def freeze_provenance(arguments): """Capture actual binaries, guarded SDK, source data and local toolchain.""" import sirius_sdk + execution_environment(arguments.gomaxprocs) + native_configuration(arguments.source_root, arguments.host_block_bytes) root = arguments.source_root.resolve() packaged = json.loads((root / "lib/sirius-provenance.json").read_text()) if packaged["binary_sha256"] != sha256(arguments.service): @@ -1668,6 +1679,8 @@ def command(*args): "gdb": command("gdb", "--version").splitlines()[0], }, "config": { + "gomaxprocs": arguments.gomaxprocs, + "native_host_block_bytes": arguments.host_block_bytes, "gpu_memory_bytes": 6 << 30, "native_host_memory_bytes": 16 << 30, "input_window_bytes": 64 << 20, @@ -1702,6 +1715,10 @@ def main(argv=None): serve.add_argument("--service", type=Path, required=True) serve.add_argument("--source-root", type=Path, required=True) serve.add_argument("--port-base", type=int, default=36001) + serve.add_argument("--gomaxprocs", type=int, required=True) + serve.add_argument( + "--host-block-bytes", type=int, choices=(65536, 1048576), default=65536 + ) run = sub.add_parser("run") run.add_argument("--workspace", type=Path, required=True) run.add_argument("--service", type=Path, required=True) @@ -1724,6 +1741,10 @@ def main(argv=None): freeze.add_argument("--datasets", type=Path, required=True) freeze.add_argument("--loaded", type=Path, nargs=2, required=True) freeze.add_argument("--output", type=Path, required=True) + freeze.add_argument("--gomaxprocs", type=int, required=True) + freeze.add_argument( + "--host-block-bytes", type=int, choices=(65536, 1048576), default=65536 + ) args = parser.parse_args(argv) def interrupted(*_): diff --git a/optools/sirius_campaign_test.py b/optools/sirius_campaign_test.py index ea0fa3b720c05..ff1f6b5098f9c 100644 --- a/optools/sirius_campaign_test.py +++ b/optools/sirius_campaign_test.py @@ -36,6 +36,69 @@ class ControllerTest(unittest.TestCase): + def test_execution_environment_replays_frozen_cpu_budget(self): + with patch.dict(os.environ, {"GOMAXPROCS": "99"}): + environment = controller.execution_environment(16) + self.assertEqual(environment["GOMAXPROCS"], "16") + self.assertEqual(os.environ["GOMAXPROCS"], "99") + for value in (0, -1, True, 1.5, "16", None): + with self.subTest(value=value), self.assertRaises(v2.CampaignError): + controller.execution_environment(value) + + def test_native_configuration_records_supported_block_size(self): + workspace = Path("/tmp/campaign-profile-fixture") + self.assertIn("block_size: 65536", controller.native_configuration(workspace)) + self.assertIn( + "block_size: 1048576", controller.native_configuration(workspace, 1048576) + ) + for value in (0, -1, True, 65536.0, 32768, 2097152): + with self.subTest(value=value), self.assertRaises(v2.CampaignError): + controller.native_configuration(workspace, value) + + def test_frozen_block_drift_rejects_before_service_launch(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + workspace = root / "workspace" + workspace.mkdir() + service, runner = root / "service", root / "runner" + service.write_text("fixture service") + runner.write_text("fixture runner") + identity = provenance() + identity["service_binary_sha256"] = controller.sha256(service) + identity["runner_binary_sha256"] = controller.sha256(runner) + identity["controller_sha256"] = controller.controller_sha256() + (workspace / "runner.json").write_text( + json.dumps( + { + "session": [ + "set session mo_sirius_campaign_evidence=1", + "set session div_precision_increment=4", + "set time_zone='+00:00'", + ] + } + ) + ) + native = workspace / "sirius.yaml" + native.write_text(controller.native_configuration(workspace, 1048576)) + (workspace / "cn-2.toml").write_text( + '[cn.sirius]\nenabled=true\nbackend="embedded"\ninput-mode="mo"\n' + 'max-waiting-queries=16\ncleanup-timeout="30s"\n' + + "native-config-path=" + + json.dumps(str(native)) + + "\n" + ) + args = argparse.Namespace( + workspace=workspace, + output=root / "output", + service=service, + runner=runner, + resource=False, + ) + with patch.object(controller, "Service") as launch: + with self.assertRaisesRegex(v2.CampaignError, "configuration differs"): + controller.run_complete(args, provenance=identity) + launch.assert_not_called() + def test_service_generation_failure_retains_spawn_cleanup(self): with tempfile.TemporaryDirectory() as directory: child = MagicMock(pid=123) diff --git a/optools/sirius_campaign_v2.py b/optools/sirius_campaign_v2.py index 8ed040709bcac..285db47787f45 100644 --- a/optools/sirius_campaign_v2.py +++ b/optools/sirius_campaign_v2.py @@ -147,6 +147,8 @@ def build_campaign(provenance, *, resource=False, flight_profiles=()): ( "config", { + "gomaxprocs", + "native_host_block_bytes", "gpu_memory_bytes", "native_host_memory_bytes", "input_window_bytes", @@ -173,6 +175,13 @@ def build_campaign(provenance, *, resource=False, flight_profiles=()): "time_zone": "+00:00", }: raise CampaignError("campaign session semantics differ") + config = provenance["config"] + if type(config["gomaxprocs"]) is not int or config["gomaxprocs"] <= 0: + raise CampaignError("invalid campaign Go CPU budget") + if type(config["native_host_block_bytes"]) is not int or config[ + "native_host_block_bytes" + ] not in (65536, 1048576): + raise CampaignError("invalid campaign host block size") expected = { "gpu_memory_bytes": 6 << 30, "native_host_memory_bytes": 16 << 30, diff --git a/optools/sirius_campaign_v2_test.py b/optools/sirius_campaign_v2_test.py index adfe80ef9fd92..a708e529dc3c7 100644 --- a/optools/sirius_campaign_v2_test.py +++ b/optools/sirius_campaign_v2_test.py @@ -50,6 +50,8 @@ def provenance(): "gdb": "test", }, "config": { + "gomaxprocs": 4, + "native_host_block_bytes": 65536, "gpu_memory_bytes": 6 << 30, "native_host_memory_bytes": 16 << 30, "input_window_bytes": 64 << 20, @@ -147,6 +149,29 @@ def test_provenance_rejects_secrets_and_changed_limits(self): with self.assertRaises(campaign.CampaignError): campaign.build_campaign(value) + def test_execution_profile_requires_explicit_bounded_settings(self): + for budget, block in ((4, 1048576), (16, 65536)): + value = provenance() + value["config"].update(gomaxprocs=budget, native_host_block_bytes=block) + frozen = campaign.build_campaign(value)["provenance"]["config"] + self.assertEqual(frozen["gomaxprocs"], budget) + self.assertEqual(frozen["native_host_block_bytes"], block) + for key, values in ( + ("gomaxprocs", (None, True, 0, -1, "16", 16.0)), + ("native_host_block_bytes", (None, True, 0, 32768, 65536.0)), + ): + for setting in values: + value = provenance() + value["config"][key] = setting + with self.subTest(key=key, setting=setting), self.assertRaises( + campaign.CampaignError + ): + campaign.build_campaign(value) + value = provenance() + del value["config"][key] + with self.subTest(missing=key), self.assertRaises(campaign.CampaignError): + campaign.build_campaign(value) + def test_schedule_counts_and_sampling_separation(self): timing = campaign.build_campaign(provenance()) resources = campaign.build_campaign(provenance(), resource=True) diff --git a/pkg/sql/plan/function/operator_in.go b/pkg/sql/plan/function/operator_in.go index d423b6cdc95fd..b9b7adffc6e58 100644 --- a/pkg/sql/plan/function/operator_in.go +++ b/pkg/sql/plan/function/operator_in.go @@ -118,7 +118,7 @@ func (op *opOperatorStrIn) init(left, tuple *vector.Vector) { return } op.mp = make(map[string]bool, 1) - op.mp[op.key(v)] = true + op.mp[string(op.keyBytes(v))] = true return } @@ -129,11 +129,11 @@ func (op *opOperatorStrIn) init(left, tuple *vector.Vector) { op.hasNull = true continue } - op.mp[op.key(v)] = true + op.mp[string(op.keyBytes(v))] = true } } -func (op *opOperatorStrIn) key(value []byte) string { +func (op *opOperatorStrIn) keyBytes(value []byte) []byte { if op.padSpace { value = bytes.TrimRight(value, " ") } @@ -144,7 +144,14 @@ func (op *opOperatorStrIn) key(value []byte) string { // can collide with a valid UCA weight string. value = types.CollationKeyOrOriginal(op.typ.Charset, value) } - return string(value) + return value +} + +func (op *opOperatorStrIn) containsKey(value []byte) bool { + // A lookup borrows the normalized bytes only for this expression. Map + // insertion above retains an owned string; probes need no owned copy. + _, ok := op.mp[string(op.keyBytes(value))] + return ok } func (op *opOperatorFixedIn[T]) operatorIn(parameters []*vector.Vector, result vector.FunctionResultWrapper, proc *process.Process, length int, selectList *FunctionSelectList) error { @@ -219,7 +226,7 @@ func (op *opOperatorStrIn) operatorIn(parameters []*vector.Vector, result vector return err } } else { - _, ok := op.mp[op.key(v)] + ok := op.containsKey(v) if !ok && op.hasNull { if err := rs.Append(false, true); err != nil { return err @@ -248,7 +255,7 @@ func (op *opOperatorStrIn) operatorNotIn(parameters []*vector.Vector, result vec return err } } else { - _, ok := op.mp[op.key(v)] + ok := op.containsKey(v) if !ok && op.hasNull { if err := rs.Append(false, true); err != nil { return err diff --git a/pkg/sql/plan/function/operator_in_test.go b/pkg/sql/plan/function/operator_in_test.go index 702022629798c..c8a0ccd857c27 100644 --- a/pkg/sql/plan/function/operator_in_test.go +++ b/pkg/sql/plan/function/operator_in_test.go @@ -23,6 +23,49 @@ import ( "github.com/stretchr/testify/require" ) +func TestOperatorStrInRetainsOwnedKeys(t *testing.T) { + proc := testutil.NewProcess(t) + for _, typ := range []types.Type{types.T_varchar.ToType(), types.New(types.T_char, 64, 0)} { + t.Run(typ.Oid.String(), func(t *testing.T) { + tuple := vector.NewVec(typ) + defer tuple.Free(proc.Mp()) + value := []byte("membership key longer than inline vector storage ") + require.NoError(t, vector.AppendBytes(tuple, value, false, proc.Mp())) + op := newOpOperatorStrIn() + op.init(nil, tuple) + probe := append([]byte(nil), tuple.GetBytesAt(0)...) + // Reusing the list vector must not alter a retained map key. + tuple.GetBytesAt(0)[0] = 'X' + require.True(t, op.containsKey(probe)) + require.False(t, op.containsKey(tuple.GetBytesAt(0))) + }) + } +} + +func TestOperatorStrInBinaryProbeAllocations(t *testing.T) { + key := "membership key longer than the compiler's short-string buffer" + op := &opOperatorStrIn{typ: types.T_varchar.ToType(), mp: map[string]bool{key: true}} + probe := []byte(key) + require.Zero(t, testing.AllocsPerRun(1000, func() { + if !op.containsKey(probe) { + t.Fatal("known key was not found") + } + })) +} + +func BenchmarkOperatorStrInBinaryProbe(b *testing.B) { + key := "membership key longer than the compiler's short-string buffer" + op := &opOperatorStrIn{typ: types.T_varchar.ToType(), mp: map[string]bool{key: true}} + probe := []byte(key) + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + if !op.containsKey(probe) { + b.Fatal("known key was not found") + } + } +} + func TestOperatorInNullableListUsesThreeValuedLogic(t *testing.T) { proc := testutil.NewProcess(t) From 8fafa3ae3e74e84d3a956345c6ee039eb282c212 Mon Sep 17 00:00:00 2001 From: bRong Njam Date: Tue, 13 Oct 2026 10:39:37 +0800 Subject: [PATCH 07/20] Pin merged Sirius payload and Decimal64 groupby optimizations --- docs/design/sirius-public-campaign.md | 10 ++++++++++ third_party/sirius | 2 +- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/design/sirius-public-campaign.md b/docs/design/sirius-public-campaign.md index a5693ced9a13e..34e7f5ca0dadb 100644 --- a/docs/design/sirius-public-campaign.md +++ b/docs/design/sirius-public-campaign.md @@ -32,6 +32,16 @@ pinned-pool controls at the new block size, retain the 5% full-suite regression and 10% numeric gates, and do not infer acceptance from the targeted diagnostics. Q15 shared-CTE reuse, Q13 join changes and default cutover require separate work. +Sirius #32 merged as `87aaaa24137d20e5022e41d2b67c867b608883c7` and #33 +merged as `0cc6e484e0ee9dae3aa41919598116625d1abec8`. The consumer pins the +latter for logical input/result initialization and eligible Decimal64 grouped +SUM/AVG/COUNT. Checked global merge, final precision and AVG rounding retain +their existing implementation. Fresh guarded release qualification compares +against merged #31 on the same MO main base and frozen 16-CPU/64-KiB profile; +historical captures below retain their actual identities. Final optimization +acceptance remains pending the complete matched timing, resource, numeric and +lifecycle gates. + ## Contract and choice For every admitted campaign cell, the immutable dataset, canonical SQL, public diff --git a/third_party/sirius b/third_party/sirius index b953bb5a87788..0cc6e484e0ee9 160000 --- a/third_party/sirius +++ b/third_party/sirius @@ -1 +1 @@ -Subproject commit b953bb5a87788e17557929987208b3a0068d933b +Subproject commit 0cc6e484e0ee9dae3aa41919598116625d1abec8 From 76fd75c46968958901d4e26d78d0e432a256e525 Mon Sep 17 00:00:00 2001 From: bRong Njam Date: Tue, 13 Oct 2026 11:11:46 +0800 Subject: [PATCH 08/20] test(logservice): pass bounded range to gossip port allocator --- pkg/logservice/gossip_fixture_test.go | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/pkg/logservice/gossip_fixture_test.go b/pkg/logservice/gossip_fixture_test.go index eb662a639b29e..85b659081753e 100644 --- a/pkg/logservice/gossip_fixture_test.go +++ b/pkg/logservice/gossip_fixture_test.go @@ -127,7 +127,11 @@ func isGossipFixtureBindCollision(err error) bool { func makeGossipFixtureConfigs(t *testing.T, rt runtime.Runtime, nodeCount int) ([]Config, error) { t.Helper() return makeGossipFixtureConfigsWithAllocator(t, rt, nodeCount, func() (int, error) { - return randomPorts.allocate(probeTestPort) + first, limit, err := testPortRange() + if err != nil { + return 0, err + } + return randomPorts.allocate(first, limit, probeTestPort) }) } From c569d8b826ee66dd4b79d2aaffcc8658a51756a6 Mon Sep 17 00:00:00 2001 From: bRong Njam Date: Tue, 13 Oct 2026 11:29:12 +0800 Subject: [PATCH 09/20] docs(sirius): record merged optimization release acceptance --- docs/design/sirius-public-campaign.md | 46 +++++++++++++++++++++++++-- 1 file changed, 44 insertions(+), 2 deletions(-) diff --git a/docs/design/sirius-public-campaign.md b/docs/design/sirius-public-campaign.md index 34e7f5ca0dadb..5cb9581a0093b 100644 --- a/docs/design/sirius-public-campaign.md +++ b/docs/design/sirius-public-campaign.md @@ -39,8 +39,50 @@ SUM/AVG/COUNT. Checked global merge, final precision and AVG rounding retain their existing implementation. Fresh guarded release qualification compares against merged #31 on the same MO main base and frozen 16-CPU/64-KiB profile; historical captures below retain their actual identities. Final optimization -acceptance remains pending the complete matched timing, resource, numeric and -lifecycle gates. +acceptance requires the complete matched timing, resource, numeric and lifecycle +gates recorded below, plus CI and reviewer approval. + +The fresh optimization release capture uses MO `8fafa3ae3e`, Sirius `0cc6e484` +and guarded SDK `1de3ab8b`; its matched pre-optimization baseline uses MO +`8539955f9f`, Sirius `b953bb5a` and SDK `f65b21f4`. Both use MO main `3e50e1b32d`, +the same canonical SQL/datasets/provider, 16 Go CPUs and 64 KiB host blocks. +All 744 timing/correctness executions and 264 separate resource executions pass; +six warmed SF1/SF10 stream 1/2/4 profiles show zero unexplained growth. + +| GPU median | Matched baseline | Optimized release | +| --- | ---: | ---: | +| SF1 full suite | 1.686514 s | 1.395820 s | +| SF10 full suite | 15.091929 s | 12.556484 s | +| SF1 Q20 | 0.116503 s | 0.066711 s | +| SF10 Q20 | 1.235098 s | 0.857208 s | + +Both full-suite metrics pass the 5% regression limit; Q20 improves 42.7% and +30.6%, passing the 15% target at both scales. Q19 improves 37% at SF10. Q15 is +12% slower in this capture; shared-CTE reuse remains separate work. +All ten numeric controls pass the 10% limit on 42 retained samples per case. +One bounded balanced follow-up resolves a bimodal scalar result without +discarding the initial samples or changing inputs, timing boundaries or limits. + +Fresh merged-source GPU checks pass 39 cases/35,107 assertions; production +numeric ABI checks pass ten cases/1,964 assertions. The 96 MiB result tests +close ordered/unordered output, full-window cancellation/deadline and borrowed +lifetime. Native control normal/ASan+UBSan/TSan checks pass 93 cases/877 +assertions using unchanged merged production and the accounting-only fixture +correction in Sirius #34. Four raw sanitizer statuses remain 99, reconciled +against the exact Native A CUB and same-profile no-GPU-work pinned-pool +controls plus all six zero-growth profiles. The conversion CUDA source is +unchanged; the C++ pool-constructor change is separately identified. + +The delivery also merges main `175a645a85`: its set-operation remapping and +row-error-only changes do not affect successful canonical TPC-H paths. The +new error branch passes its targeted frontend regression. A six-line +test-only gossip allocator repair closes the inherited main compile error and +preserves the shared ephemeral-port exclusion and failure propagation. +Measured captures retain their actual identities; no source or binary is +relabeled. Public writer/compiler normal/race cleanup, owning function/compiler, +actual GPU bridge and same-process cuVS checks, all 86 tooling tests and full +SCA pass. Fail-closed artifact acceptance reports local completion, with CI and +reviewer approval remaining. Generated evidence remains outside Git. ## Contract and choice From 7f751b4f46f014993ccd39eeb80c3fbd16d71a66 Mon Sep 17 00:00:00 2001 From: bRong Njam Date: Tue, 13 Oct 2026 12:15:55 +0800 Subject: [PATCH 10/20] Pin merged Sirius fixture fix and reject ignored CPU budgets --- docs/design/sirius-public-campaign.md | 16 +++++++++++++--- optools/sirius_campaign.py | 6 ++++-- optools/sirius_campaign_test.py | 7 ++++--- optools/sirius_campaign_v2.py | 8 +++++++- optools/sirius_campaign_v2_test.py | 4 ++-- third_party/sirius | 2 +- 6 files changed, 31 insertions(+), 12 deletions(-) diff --git a/docs/design/sirius-public-campaign.md b/docs/design/sirius-public-campaign.md index 5cb9581a0093b..d8bead5923210 100644 --- a/docs/design/sirius-public-campaign.md +++ b/docs/design/sirius-public-campaign.md @@ -16,7 +16,8 @@ separate allocation profiling without a native observability prerequisite. E/F a The owner approved the focused performance plan before implementation. This increment makes the owned campaign's Go CPU budget explicit and records pinned allocation granularity; production backend and service defaults do not change. -Both `serve` and `freeze` require positive `--gomaxprocs` and accept +Both `serve` and `freeze` require `--gomaxprocs` in Go's runtime range +1 through 2147483647 and accept `--host-block-bytes` of 65536 or 1048576, defaulting to 65536. New provenance records `config.gomaxprocs` and `config.native_host_block_bytes`. `run` validates the frozen profile before starting services and launches native and embedded @@ -33,8 +34,9 @@ and 10% numeric gates, and do not infer acceptance from the targeted diagnostics Q15 shared-CTE reuse, Q13 join changes and default cutover require separate work. Sirius #32 merged as `87aaaa24137d20e5022e41d2b67c867b608883c7` and #33 -merged as `0cc6e484e0ee9dae3aa41919598116625d1abec8`. The consumer pins the -latter for logical input/result initialization and eligible Decimal64 grouped +merged as `0cc6e484e0ee9dae3aa41919598116625d1abec8`. Sirius #34 merged as +`06400cf48073e51955067bfd75c5c4defe955a72`; the consumer pins this fixture +correction on top of logical input/result initialization and eligible Decimal64 grouped SUM/AVG/COUNT. Checked global merge, final precision and AVG rounding retain their existing implementation. Fresh guarded release qualification compares against merged #31 on the same MO main base and frozen 16-CPU/64-KiB profile; @@ -84,6 +86,14 @@ actual GPU bridge and same-process cuVS checks, all 86 tooling tests and full SCA pass. Fail-closed artifact acceptance reports local completion, with CI and reviewer approval remaining. Generated evidence remains outside Git. +The deep delivery review rejects CPU budgets beyond Go's signed 32-bit range: +the runtime silently ignores those values, so accepting them would misreport +the execution profile. Boundary fixtures fail before this guard and pass after +it. The validated 16-CPU profile is unchanged. Merged #34 is tree-identical to +the tested fixture correction and changes no production, ABI, CMake or provider +input relative to measured `0cc6e484`. Its source-pin and SDK receipts are +refreshed separately; existing captures retain their original identities. + ## Contract and choice For every admitted campaign cell, the immutable dataset, canonical SQL, public diff --git a/optools/sirius_campaign.py b/optools/sirius_campaign.py index 40da183b6b619..3e703b036e053 100644 --- a/optools/sirius_campaign.py +++ b/optools/sirius_campaign.py @@ -855,8 +855,10 @@ def write_timing_tables(output, campaign, records, summary): def execution_environment(gomaxprocs): - if type(gomaxprocs) is not int or gomaxprocs <= 0: - raise v2.CampaignError("campaign Go CPU budget must be a positive integer") + if type(gomaxprocs) is not int or not 0 < gomaxprocs <= v2.MAX_GOMAXPROCS: + raise v2.CampaignError( + "campaign Go CPU budget must be between 1 and 2147483647" + ) environment = os.environ.copy() environment["GOMAXPROCS"] = str(gomaxprocs) return environment diff --git a/optools/sirius_campaign_test.py b/optools/sirius_campaign_test.py index ff1f6b5098f9c..a614b7afdb0cf 100644 --- a/optools/sirius_campaign_test.py +++ b/optools/sirius_campaign_test.py @@ -38,10 +38,11 @@ class ControllerTest(unittest.TestCase): def test_execution_environment_replays_frozen_cpu_budget(self): with patch.dict(os.environ, {"GOMAXPROCS": "99"}): - environment = controller.execution_environment(16) - self.assertEqual(environment["GOMAXPROCS"], "16") + for budget in (16, (1 << 31) - 1): + environment = controller.execution_environment(budget) + self.assertEqual(environment["GOMAXPROCS"], str(budget)) self.assertEqual(os.environ["GOMAXPROCS"], "99") - for value in (0, -1, True, 1.5, "16", None): + for value in (0, -1, True, 1.5, "16", None, 1 << 31, 10**100): with self.subTest(value=value), self.assertRaises(v2.CampaignError): controller.execution_environment(value) diff --git a/optools/sirius_campaign_v2.py b/optools/sirius_campaign_v2.py index 285db47787f45..479ca3619cd4a 100644 --- a/optools/sirius_campaign_v2.py +++ b/optools/sirius_campaign_v2.py @@ -44,6 +44,9 @@ ROW_BYTES = 1 << 20 OUTPUT_BYTES = 1 << 20 STDERR_BYTES = 64 << 10 +# The Go runtime parses GOMAXPROCS as a signed 32-bit integer. Larger values +# are ignored, so accepting one would record a CPU budget the service never uses. +MAX_GOMAXPROCS = (1 << 31) - 1 ROUTES = ( {"id": "mo-native", "backend": "native", "scan_mode": "mo"}, {"id": "embedded-mo", "backend": "embedded", "scan_mode": "mo"}, @@ -176,7 +179,10 @@ def build_campaign(provenance, *, resource=False, flight_profiles=()): }: raise CampaignError("campaign session semantics differ") config = provenance["config"] - if type(config["gomaxprocs"]) is not int or config["gomaxprocs"] <= 0: + if ( + type(config["gomaxprocs"]) is not int + or not 0 < config["gomaxprocs"] <= MAX_GOMAXPROCS + ): raise CampaignError("invalid campaign Go CPU budget") if type(config["native_host_block_bytes"]) is not int or config[ "native_host_block_bytes" diff --git a/optools/sirius_campaign_v2_test.py b/optools/sirius_campaign_v2_test.py index a708e529dc3c7..7c5222ac294af 100644 --- a/optools/sirius_campaign_v2_test.py +++ b/optools/sirius_campaign_v2_test.py @@ -150,14 +150,14 @@ def test_provenance_rejects_secrets_and_changed_limits(self): campaign.build_campaign(value) def test_execution_profile_requires_explicit_bounded_settings(self): - for budget, block in ((4, 1048576), (16, 65536)): + for budget, block in ((4, 1048576), (16, 65536), ((1 << 31) - 1, 65536)): value = provenance() value["config"].update(gomaxprocs=budget, native_host_block_bytes=block) frozen = campaign.build_campaign(value)["provenance"]["config"] self.assertEqual(frozen["gomaxprocs"], budget) self.assertEqual(frozen["native_host_block_bytes"], block) for key, values in ( - ("gomaxprocs", (None, True, 0, -1, "16", 16.0)), + ("gomaxprocs", (None, True, 0, -1, "16", 16.0, 1 << 31, 10**100)), ("native_host_block_bytes", (None, True, 0, 32768, 65536.0)), ): for setting in values: diff --git a/third_party/sirius b/third_party/sirius index 0cc6e484e0ee9..06400cf48073e 160000 --- a/third_party/sirius +++ b/third_party/sirius @@ -1 +1 @@ -Subproject commit 0cc6e484e0ee9dae3aa41919598116625d1abec8 +Subproject commit 06400cf48073e51955067bfd75c5c4defe955a72 From a76c965e0d74866707fa5cd53719ca2d31226c63 Mon Sep 17 00:00:00 2001 From: bRong Njam Date: Tue, 13 Oct 2026 12:22:48 +0800 Subject: [PATCH 11/20] docs(sirius): record deep review and merged fixture validation --- docs/design/sirius-public-campaign.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/design/sirius-public-campaign.md b/docs/design/sirius-public-campaign.md index d8bead5923210..9e9d524fe38ef 100644 --- a/docs/design/sirius-public-campaign.md +++ b/docs/design/sirius-public-campaign.md @@ -93,6 +93,12 @@ it. The validated 16-CPU profile is unchanged. Merged #34 is tree-identical to the tested fixture correction and changes no production, ABI, CMake or provider input relative to measured `0cc6e484`. Its source-pin and SDK receipts are refreshed separately; existing captures retain their original identities. +The refreshed release SDK is `878c0ec8` at merged `06400cf4`. Its guarded MO +consumer build, standard merged control suite (93 cases/877 assertions), owning +parser/function/compiler suites, all 22 native preparations and real GPU +bridge/cancellation/cuVS coexistence checks pass. Main `e5111936b9`'s +qualified-identifier parser consumers pass separately. The unchanged complete +archive passes the tightened acceptance validator; raw runs are not relabeled. ## Contract and choice From c5b77a2594c26b979f7a7250f811aa76d77be993 Mon Sep 17 00:00:00 2001 From: bRong Njam Date: Tue, 13 Oct 2026 15:14:32 +0800 Subject: [PATCH 12/20] Preserve historical Sirius campaign archive verification --- docs/design/sirius-public-campaign.md | 7 ++++- optools/sirius_acceptance_test.py | 21 ++++++++++++++ optools/sirius_campaign_v2.py | 40 +++++++++++++++++++-------- optools/sirius_campaign_v2_test.py | 37 +++++++++++++++++++++++++ 4 files changed, 92 insertions(+), 13 deletions(-) diff --git a/docs/design/sirius-public-campaign.md b/docs/design/sirius-public-campaign.md index 9e9d524fe38ef..bebee6d36e51c 100644 --- a/docs/design/sirius-public-campaign.md +++ b/docs/design/sirius-public-campaign.md @@ -23,7 +23,12 @@ records `config.gomaxprocs` and `config.native_host_block_bytes`. `run` validate the frozen profile before starting services and launches native and embedded routes with the same frozen budget, independently of its ambient environment. Historical captures retain their actual settings and controller identity; -new runs require refrozen provenance. Runner/result schema version 2 is unchanged. +new runs require refrozen provenance. Read-only verification preserves the +historical v2 configuration shape when both profile fields are absent; it neither +adds defaults nor modifies archived provenance. If either field is present, both +must be valid. New campaign construction and executor admission require both +fields, and evidence with different profile shapes cannot be combined. +Runner/result schema version 2 is unchanged. The current-host qualification profile is 16 Go CPUs and 64 KiB host blocks. Compare code revisions at identical settings; profile/kernel observations are diff --git a/optools/sirius_acceptance_test.py b/optools/sirius_acceptance_test.py index cf1b537f7305f..e08f892be67b4 100644 --- a/optools/sirius_acceptance_test.py +++ b/optools/sirius_acceptance_test.py @@ -228,6 +228,27 @@ def records(manifest, offset): class AcceptanceTest(unittest.TestCase): + def test_historical_v2_archive_preserves_missing_execution_profile(self): + inputs = evidence() + identities = [ + inputs[0]["provenance"], + inputs[2]["provenance"], + *(item["provenance"] for item in inputs[4]), + *(item["provenance"] for item in inputs[5:]), + ] + for identity in identities: + identity["config"].pop("gomaxprocs", None) + identity["config"].pop("native_host_block_bytes", None) + original = copy.deepcopy(inputs) + self.assertTrue(acceptance.validate(*inputs)["local_acceptance_complete"]) + self.assertEqual(original, inputs, "verification must not invent settings") + + inputs = evidence() + for key in ("gomaxprocs", "native_host_block_bytes"): + del inputs[0]["provenance"]["config"][key] + with self.assertRaisesRegex(campaign.CampaignError, "semantic inputs differ"): + acceptance.validate(*inputs) + def test_cli_binds_consumed_inputs_to_archive_checksums(self): inputs = evidence() with tempfile.TemporaryDirectory() as directory: diff --git a/optools/sirius_campaign_v2.py b/optools/sirius_campaign_v2.py index 479ca3619cd4a..e932cbf945fd0 100644 --- a/optools/sirius_campaign_v2.py +++ b/optools/sirius_campaign_v2.py @@ -47,6 +47,7 @@ # The Go runtime parses GOMAXPROCS as a signed 32-bit integer. Larger values # are ignored, so accepting one would record a CPU budget the service never uses. MAX_GOMAXPROCS = (1 << 31) - 1 +EXECUTION_PROFILE_FIELDS = frozenset(("gomaxprocs", "native_host_block_bytes")) ROUTES = ( {"id": "mo-native", "backend": "native", "scan_mode": "mo"}, {"id": "embedded-mo", "backend": "embedded", "scan_mode": "mo"}, @@ -94,7 +95,24 @@ def query_uuid(value): return value -def build_campaign(provenance, *, resource=False, flight_profiles=()): +def validate_execution_profile(config): + if not isinstance(config, dict) or not EXECUTION_PROFILE_FIELDS <= config.keys(): + raise CampaignError("campaign execution requires an explicit CPU and host profile") + if ( + type(config["gomaxprocs"]) is not int + or not 0 < config["gomaxprocs"] <= MAX_GOMAXPROCS + ): + raise CampaignError("invalid campaign Go CPU budget") + if ( + type(config["native_host_block_bytes"]) is not int + or config["native_host_block_bytes"] not in (65536, 1048576) + ): + raise CampaignError("invalid campaign host block size") + + +def build_campaign( + provenance, *, resource=False, flight_profiles=(), require_execution_profile=True +): required = ( "matrixone_revision", "sirius_revision", @@ -150,8 +168,6 @@ def build_campaign(provenance, *, resource=False, flight_profiles=()): ( "config", { - "gomaxprocs", - "native_host_block_bytes", "gpu_memory_bytes", "native_host_memory_bytes", "input_window_bytes", @@ -166,6 +182,11 @@ def build_campaign(provenance, *, resource=False, flight_profiles=()): ), ): value = provenance[field] + if field == "config" and isinstance(value, dict): + # V2 archives predating explicit profiles retain their original + # shape. Partial or invalid profiles never become historical data. + if require_execution_profile or EXECUTION_PROFILE_FIELDS & value.keys(): + allowed |= EXECUTION_PROFILE_FIELDS if not isinstance(value, dict) or set(value) != allowed: raise CampaignError("invalid bounded provenance fields") if len(canonical(value)) > 4096: @@ -179,15 +200,8 @@ def build_campaign(provenance, *, resource=False, flight_profiles=()): }: raise CampaignError("campaign session semantics differ") config = provenance["config"] - if ( - type(config["gomaxprocs"]) is not int - or not 0 < config["gomaxprocs"] <= MAX_GOMAXPROCS - ): - raise CampaignError("invalid campaign Go CPU budget") - if type(config["native_host_block_bytes"]) is not int or config[ - "native_host_block_bytes" - ] not in (65536, 1048576): - raise CampaignError("invalid campaign host block size") + if require_execution_profile or EXECUTION_PROFILE_FIELDS & config.keys(): + validate_execution_profile(config) expected = { "gpu_memory_bytes": 6 << 30, "native_host_memory_bytes": 16 << 30, @@ -1028,6 +1042,7 @@ def execute_campaign(campaign, executor, scratch): identities = set() produced = set() try: + validate_execution_profile(campaign["provenance"]["config"]) for spec in campaign["schedule"]: request = copy.deepcopy(spec) request["spool_budget_bytes"] = min( @@ -1116,6 +1131,7 @@ def summarize(campaign, records): campaign["provenance"], resource=campaign["resource_campaign"], flight_profiles=campaign["flight_profiles"], + require_execution_profile=False, ) if campaign != trusted: raise CampaignError("summary rejects a partial or modified acceptance schedule") diff --git a/optools/sirius_campaign_v2_test.py b/optools/sirius_campaign_v2_test.py index 7c5222ac294af..b3e349e251b73 100644 --- a/optools/sirius_campaign_v2_test.py +++ b/optools/sirius_campaign_v2_test.py @@ -184,6 +184,43 @@ def test_schedule_counts_and_sampling_separation(self): ) self.assertEqual((0, 2, 1, 3), campaign.ORDER_KEYS[2]) + def test_historical_profile_cannot_launch_an_executor(self): + manifest = campaign.build_campaign(provenance()) + for key in ("gomaxprocs", "native_host_block_bytes"): + del manifest["provenance"]["config"][key] + with self.assertRaises(campaign.CampaignError): + campaign.build_campaign(manifest["provenance"]) + calls = [] + + def executor(spec): + calls.append(spec) + return {} + + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + root.chmod(0o700) + with self.assertRaises(campaign.CampaignError): + campaign.execute_campaign(manifest, executor, campaign.Scratch(root)) + self.assertEqual([], calls, "historical verification cannot admit execution") + self.assertEqual([], list(root.iterdir())) + + def test_archive_verification_rejects_partial_or_invalid_profiles(self): + for key, values in ( + ("gomaxprocs", (None, True, 0, 1 << 31)), + ("native_host_block_bytes", (None, True, 32768, 65536.0)), + ): + for setting in values: + manifest = campaign.build_campaign(provenance()) + manifest["provenance"]["config"][key] = setting + with self.subTest(key=key, setting=setting), self.assertRaises( + campaign.CampaignError + ): + campaign.summarize(manifest, []) + manifest = campaign.build_campaign(provenance()) + del manifest["provenance"]["config"][key] + with self.subTest(missing=key), self.assertRaises(campaign.CampaignError): + campaign.summarize(manifest, []) + def test_full_value_duplicate_and_empty_comparison(self): for a, b, keys, ok in ( ([], [], [], True), From 0bd18a0c0dcab38134955dd27bc6c5827b1a5ab9 Mon Sep 17 00:00:00 2001 From: bRong Njam Date: Tue, 13 Oct 2026 15:20:20 +0800 Subject: [PATCH 13/20] Link immutable Sirius optimization review evidence --- docs/design/sirius-public-campaign.md | 45 ++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/docs/design/sirius-public-campaign.md b/docs/design/sirius-public-campaign.md index bebee6d36e51c..ac2a9d1b8a524 100644 --- a/docs/design/sirius-public-campaign.md +++ b/docs/design/sirius-public-campaign.md @@ -89,7 +89,8 @@ Measured captures retain their actual identities; no source or binary is relabeled. Public writer/compiler normal/race cleanup, owning function/compiler, actual GPU bridge and same-process cuVS checks, all 86 tooling tests and full SCA pass. Fail-closed artifact acceptance reports local completion, with CI and -reviewer approval remaining. Generated evidence remains outside Git. +reviewer approval remaining. Generated evidence is preserved separately from +this source change, in the contributor-fork evidence branch described below. The deep delivery review rejects CPU budgets beyond Go's signed 32-bit range: the runtime silently ignores those values, so accepting them would misreport @@ -105,6 +106,48 @@ bridge/cancellation/cuVS coexistence checks pass. Main `e5111936b9`'s qualified-identifier parser consumers pass separately. The unchanged complete archive passes the tightened acceptance validator; raw runs are not relabeled. +## Optimization evidence preservation + +The matched optimization archive and merged #34 qualification are preserved at +contributor-fork commit `56a5c86936fbda06d4c41dbf2ccbcb717e82901d`, +on branch `evidence/sirius-perf-30021-c5b77a2594`. The +[qualification ledger](https://github.com/aunjgr/matrixone/blob/56a5c86936fbda06d4c41dbf2ccbcb717e82901d/docs/design/evidence/sirius-perf-30021/qualification.json) +records the original baseline/candidate identities, all captured artifact +checksums, and the source/build/provider equivalence through merged `06400cf4`. +The bundle retains the 744-run candidate and matched baseline, 264 resource runs, +all 42 numeric samples per case, raw sanitizer/control logs, and merged-SDK +build, control, 22-query preparation and actual GPU bridge evidence. + +Product validation identifies MO `4e330e4219` separately from the earlier +measured `8fafa3ae3e` capture. Review fix `c5b77a2594` changes archive verification +and its tests/documentation; Go/CGo/native product code is unchanged. Its 89 +tooling tests pass, the original checksum-bound v2 archive verifies unchanged, +and new execution still requires an explicit valid profile. Neither historical +nor optimized measurements are relabeled as a new benchmark run. CI and +reviewer approval remain external gates. + +From the review-fixed source checkout, export and verify the immutable bundle: + +```sh +git fetch https://github.com/aunjgr/matrixone.git \ + refs/heads/evidence/sirius-perf-30021-c5b77a2594 +task_optimization_dir=$(mktemp -d) +git show 56a5c86936fbda06d4c41dbf2ccbcb717e82901d:docs/design/evidence/sirius-perf-30021/sirius-perf-30021.tar.gz \ + > "$task_optimization_dir/qualification.tar.gz" +printf '%s %s\n' \ + dad13b3c3fc6bf7a2aba545fbb173e0aea7e457e0b65080b66ec8fceb43583cc \ + "$task_optimization_dir/qualification.tar.gz" | sha256sum -c - +tar -xzf "$task_optimization_dir/qualification.tar.gz" -C "$task_optimization_dir" +task_optimization_root="$task_optimization_dir/sirius-perf-30021/acceptance" +python3 optools/sirius_acceptance.py \ + --timing "$task_optimization_root/timing" \ + --resource "$task_optimization_root/resource" \ + --allocations "$task_optimization_root/allocations.json" \ + --lifecycle "$task_optimization_root/lifecycle.json" \ + --sanitizer "$task_optimization_root/sanitizer.json" \ + --baselines "$task_optimization_root/numeric-baselines.json" +``` + ## Contract and choice For every admitted campaign cell, the immutable dataset, canonical SQL, public From 2173814b0c33827f4070c70a40cf223af83f1fdd Mon Sep 17 00:00:00 2001 From: Nan DENG Date: Tue, 13 Oct 2026 07:55:51 +0000 Subject: [PATCH 14/20] fix(sql): explain current typed index search scans in JSON --- docs/design/explain_format_json.md | 9 ++ pkg/sql/plan/explain/sql_json.go | 60 ++++++--- .../sql_json_index_search_schema_test.go | 124 ++++++++++++++++++ pkg/sql/plan/explain/sql_json_test.go | 57 ++++---- .../optimizer/explain_format_json.result | 14 ++ .../cases/optimizer/explain_format_json.sql | 27 +++- 6 files changed, 237 insertions(+), 54 deletions(-) create mode 100644 pkg/sql/plan/explain/sql_json_index_search_schema_test.go diff --git a/docs/design/explain_format_json.md b/docs/design/explain_format_json.md index 3c5a389fa24b8..5d1f700198f29 100644 --- a/docs/design/explain_format_json.md +++ b/docs/design/explain_format_json.md @@ -159,6 +159,15 @@ The following mappings are normative for schema version 1: `fill_value[i]=` from `FillVal`, in their original order. Empty target/value lists add no entries. Unknown modes and missing required expressions are errors; the renderer never rebuilds pre-pruning targets. +- An `INDEX_SEARCH_SCAN` uses the current typed `IndexSearchScan` contract + for vector and fulltext indexes. It retains the query payload, candidate + limit, distance bounds, included columns and pre-filters. Algorithm-owned + option bytes are preserved as `algo_options_base64`; existing plugin + EXPLAIN hooks add readable `algo_setting[i]` entries. Named algorithm + expressions (including IVF first-round limits) use `algo_expr["name"]`, + with quoted names and matching ordered name/expression arrays. Missing, + empty or duplicate expression names are errors. No protobuf fields are + invented and no options are decoded using another algorithm's schema. - Serialized scalar bytes use `serialized(type=,width=,scale=,value=0x)`. Serialized literal vectors use diff --git a/pkg/sql/plan/explain/sql_json.go b/pkg/sql/plan/explain/sql_json.go index 10dc7f9010540..4c2af59b3cc46 100644 --- a/pkg/sql/plan/explain/sql_json.go +++ b/pkg/sql/plan/explain/sql_json.go @@ -17,6 +17,7 @@ package explain import ( "bytes" "context" + "encoding/base64" "encoding/json" "math" "strconv" @@ -418,12 +419,15 @@ func (b *sqlJSONPlanBuilder) addNodeDetails(item *sqlJSONNode, node *plan.Node) if err != nil { return err } - case plan.Node_VECTOR_INDEX_SCAN: - if node.VectorIndexScan == nil || node.VectorIndexScan.Index == nil || node.VectorIndexScan.Index.IndexName == "" { - return moerr.NewInvalidInput(b.ctx, "vector index scan metadata is missing") + case plan.Node_INDEX_SEARCH_SCAN: + if node.IndexSearchScan == nil || node.IndexSearchScan.Index == nil || node.IndexSearchScan.Index.IndexName == "" { + return moerr.NewInvalidInput(b.ctx, "index search scan metadata is missing") } item.TableName = sqlJSONTableName(node) - item.Expressions, err = sqlJSONVectorIndexExpressions(b.ctx, node.VectorIndexScan, &b.textOpts) + if item.TableName == "" { + item.TableName = sqlJSONObjectRefName(node.IndexSearchScan.SourceTable) + } + item.Expressions, err = sqlJSONIndexSearchExpressions(b.ctx, node.IndexSearchScan, &b.textOpts) if err != nil { return err } @@ -621,9 +625,9 @@ func sqlJSONSampleExpressions(ctx context.Context, spec *plan.SampleFuncSpec) ([ }, nil } -func sqlJSONVectorIndexExpressions(ctx context.Context, spec *plan.VectorIndexScan, options *ExplainOptions) ([]string, error) { +func sqlJSONIndexSearchExpressions(ctx context.Context, spec *plan.IndexSearchScan, options *ExplainOptions) ([]string, error) { if spec == nil || spec.Index == nil { - return nil, moerr.NewInvalidInput(ctx, "vector index scan metadata is missing") + return nil, moerr.NewInvalidInput(ctx, "index search scan metadata is missing") } values := make([]string, 0, 16+len(spec.PreFilters)+len(spec.IncludedColumns)) values = append(values, "index="+spec.Index.IndexName) @@ -633,18 +637,26 @@ func sqlJSONVectorIndexExpressions(ctx context.Context, spec *plan.VectorIndexSc direction := spec.Direction.String() switch spec.Direction { case plan.OrderBySpec_INTERNAL, plan.OrderBySpec_ASC, plan.OrderBySpec_DESC: - // valid vector scan directions + // valid index scan directions default: - return nil, moerr.NewInvalidInputf(ctx, "vector index scan has unknown direction %d", spec.Direction) + return nil, moerr.NewInvalidInputf(ctx, "index search scan has unknown direction %d", spec.Direction) } values = append(values, "direction="+direction, - "initial_probe_count="+strconv.FormatUint(uint64(spec.InitialProbeCount), 10), - "bucket_expand_step="+strconv.FormatUint(uint64(spec.BucketExpandStep), 10), "post_filter_over_fetch="+strconv.FormatBool(spec.PostFilterOverFetch), ) - if spec.ThreadsSearch != 0 { - values = append(values, "threads_search="+strconv.FormatInt(spec.ThreadsSearch, 10)) + // Algorithm settings belong to the plugin's wire contract. Preserve every + // byte without assuming a JSON format or inventing fields on the shared + // protobuf. Use its existing EXPLAIN hook for human-readable settings. + if len(spec.AlgoOptions) != 0 { + values = append(values, "algo_options_base64="+base64.StdEncoding.EncodeToString(spec.AlgoOptions)) + } + settings, err := indexSearchScanSettings(spec) + if err != nil { + return nil, err + } + for i, setting := range settings { + values = append(values, "algo_setting["+strconv.Itoa(i)+"]="+setting) } for i, column := range spec.IncludedColumns { values = append(values, "included_column["+strconv.Itoa(i)+"]="+column) @@ -653,9 +665,8 @@ func sqlJSONVectorIndexExpressions(ctx context.Context, spec *plan.VectorIndexSc name string expr *plan.Expr }{ - {name: "query_vector", expr: spec.QueryVector}, + {name: "query_payload", expr: spec.QueryPayload}, {name: "candidate_limit", expr: spec.CandidateLimit}, - {name: "first_round_limit", expr: spec.FirstRoundLimit}, } { value, err := sqlJSONExpr(ctx, field.expr, options) if err != nil { @@ -665,6 +676,21 @@ func sqlJSONVectorIndexExpressions(ctx context.Context, spec *plan.VectorIndexSc values = append(values, field.name+"="+value) } } + if len(spec.AlgoExprNames) != len(spec.AlgoExprs) { + return nil, moerr.NewInvalidInput(ctx, "index search scan algorithm expression names do not match expressions") + } + seen := make(map[string]struct{}, len(spec.AlgoExprNames)) + for i, name := range spec.AlgoExprNames { + if _, duplicate := seen[name]; name == "" || duplicate || spec.AlgoExprs[i] == nil { + return nil, moerr.NewInvalidInput(ctx, "index search scan algorithm expression is missing or ambiguous") + } + seen[name] = struct{}{} + value, err := sqlJSONExpr(ctx, spec.AlgoExprs[i], options) + if err != nil { + return nil, err + } + values = append(values, "algo_expr["+strconv.Quote(name)+"]="+value) + } if spec.DistanceRange != nil { for _, bound := range []struct { name string @@ -678,18 +704,18 @@ func sqlJSONVectorIndexExpressions(ctx context.Context, spec *plan.VectorIndexSc continue } if bound.expr == nil { - return nil, moerr.NewInvalidInputf(ctx, "vector index scan %s expression is missing", bound.name) + return nil, moerr.NewInvalidInputf(ctx, "index search scan %s expression is missing", bound.name) } boundType := bound.boundType.String() if boundType == "" { - return nil, moerr.NewInvalidInputf(ctx, "vector index scan %s has unknown bound type %d", bound.name, bound.boundType) + return nil, moerr.NewInvalidInputf(ctx, "index search scan %s has unknown bound type %d", bound.name, bound.boundType) } value, err := sqlJSONExpr(ctx, bound.expr, options) if err != nil { return nil, err } if value == "" { - return nil, moerr.NewInvalidInputf(ctx, "vector index scan %s expression is empty", bound.name) + return nil, moerr.NewInvalidInputf(ctx, "index search scan %s expression is empty", bound.name) } values = append(values, bound.name+"_type="+boundType, bound.name+"="+value) } diff --git a/pkg/sql/plan/explain/sql_json_index_search_schema_test.go b/pkg/sql/plan/explain/sql_json_index_search_schema_test.go new file mode 100644 index 0000000000000..ea24dee05cd8d --- /dev/null +++ b/pkg/sql/plan/explain/sql_json_index_search_schema_test.go @@ -0,0 +1,124 @@ +// Copyright 2026 Matrix Origin +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package explain + +import ( + "context" + "encoding/base64" + "encoding/json" + "strings" + "testing" + + "github.com/matrixorigin/matrixone/pkg/pb/plan" + planpkg "github.com/matrixorigin/matrixone/pkg/sql/plan" + "github.com/stretchr/testify/require" +) + +func sqlJSONWireIndexNode(t *testing.T, node *plan.Node) sqlJSONNode { + t.Helper() + query := &plan.Query{StmtType: plan.Query_SELECT, Steps: []int32{0}, Nodes: []*plan.Node{node}} + wire, err := query.Marshal() + require.NoError(t, err) + var remote plan.Query + require.NoError(t, remote.Unmarshal(wire)) + raw, err := BuildSQLJSONPlan(t.Context(), &remote) + require.NoError(t, err) + var decoded sqlJSONPlan + require.NoError(t, json.Unmarshal(raw, &decoded)) + require.Len(t, decoded.MatrixOne.Nodes, 1) + return decoded.MatrixOne.Nodes[0] +} + +func TestSQLJSONIndexSearchWirePayload(t *testing.T) { + opts := []byte(`{"initial_probe_count":7,"bucket_expand_step":3,"threads_search":2}`) + node := sqlJSONWireIndexNode(t, &plan.Node{ + NodeId: 0, NodeType: plan.Node_INDEX_SEARCH_SCAN, + IndexSearchScan: &plan.IndexSearchScan{ + SourceTable: &plan.ObjectRef{SchemaName: "db", ObjName: "items"}, + Index: &plan.IndexDef{IndexName: "idx_embedding", IndexAlgo: "ivfflat"}, + DistanceFunction: "l2_distance", Direction: plan.OrderBySpec_DESC, + QueryPayload: &plan.Expr{Expr: &plan.Expr_Lit{Lit: &plan.Literal{Value: &plan.Literal_VecVal{VecVal: "[0.1,0.2]"}}}}, + CandidateLimit: planpkg.MakePlan2Uint64ConstExprWithType(12), + PostFilterOverFetch: true, AlgoOptions: opts, + AlgoExprNames: []string{"first_round_limit"}, + AlgoExprs: []*plan.Expr{planpkg.MakePlan2Uint64ConstExprWithType(4)}, + IncludedColumns: []string{"id", "embedding"}, + }, + }) + require.Equal(t, "Vector Index Scan", node.Operator) + require.Equal(t, "db.items", node.TableName) + for _, want := range []string{"index=idx_embedding", "metric=l2_distance", "direction=DESC", + "post_filter_over_fetch=true", "algo_setting[0]=NProbe: 7", "query_payload=[0.1,0.2]", + "candidate_limit=12", `algo_expr["first_round_limit"]=4`, "included_column[1]=embedding"} { + require.Contains(t, node.Expressions, want) + } + for _, value := range node.Expressions { + if encoded, ok := strings.CutPrefix(value, "algo_options_base64="); ok { + decoded, err := base64.StdEncoding.DecodeString(encoded) + require.NoError(t, err) + require.Equal(t, opts, decoded) + return + } + } + t.Fatal("missing algorithm options") +} + +func TestSQLJSONFulltextIndexSearchWireMetadata(t *testing.T) { + node := fulltextSearchScanNode() + node.IndexSearchScan.SourceTable = &plan.ObjectRef{SchemaName: "db", ObjName: "documents"} + node.IndexSearchScan.QueryPayload = planpkg.MakePlan2StringConstExprWithType("needle") + decoded := sqlJSONWireIndexNode(t, node) + require.Equal(t, "Fulltext Index Scan", decoded.Operator) + require.Equal(t, "db.documents", decoded.TableName) + require.Contains(t, decoded.Expressions, "index=ft_body") + require.Contains(t, strings.Join(decoded.Expressions, "\n"), "needle") + require.NotContains(t, strings.Join(decoded.Expressions, "\n"), "metric=") +} + +func TestSQLJSONIndexSearchOpaqueOptionsAndNamedExpressions(t *testing.T) { + opts := []byte{0, 255, 'x'} + decoded := sqlJSONWireIndexNode(t, &plan.Node{NodeType: plan.Node_INDEX_SEARCH_SCAN, + IndexSearchScan: &plan.IndexSearchScan{ + Index: &plan.IndexDef{IndexName: "future_index", IndexAlgo: "future_plugin"}, + AlgoOptions: opts, AlgoExprNames: []string{"a]\n=2"}, AlgoExprs: []*plan.Expr{sqlJSONTestInt32(1)}, + }}) + require.Contains(t, decoded.Expressions, "algo_options_base64="+base64.StdEncoding.EncodeToString(opts)) + require.Contains(t, decoded.Expressions, `algo_expr["a]\n=2"]=1`) +} + +func TestSQLJSONIndexSearchRejectsMalformedAlgorithmMetadata(t *testing.T) { + for _, tc := range []struct { + name string + names []string + exprs []*plan.Expr + }{ + {"missing name", nil, []*plan.Expr{sqlJSONTestInt32(1)}}, + {"missing expression", []string{"limit"}, nil}, + {"empty name", []string{""}, []*plan.Expr{sqlJSONTestInt32(1)}}, + {"nil expression", []string{"limit"}, []*plan.Expr{nil}}, + {"duplicate name", []string{"limit", "limit"}, []*plan.Expr{sqlJSONTestInt32(1), sqlJSONTestInt32(2)}}, + } { + t.Run(tc.name, func(t *testing.T) { + _, err := sqlJSONIndexSearchExpressions(context.Background(), &plan.IndexSearchScan{ + Index: &plan.IndexDef{IndexName: "idx"}, AlgoExprNames: tc.names, AlgoExprs: tc.exprs, + }, &ExplainOptions{Format: EXPLAIN_FORMAT_TEXT}) + require.Error(t, err) + }) + } + _, err := sqlJSONIndexSearchExpressions(t.Context(), &plan.IndexSearchScan{ + Index: &plan.IndexDef{IndexName: "idx", IndexAlgo: "ivfflat"}, AlgoOptions: []byte("{"), + }, &ExplainOptions{Format: EXPLAIN_FORMAT_TEXT}) + require.Error(t, err) +} diff --git a/pkg/sql/plan/explain/sql_json_test.go b/pkg/sql/plan/explain/sql_json_test.go index bd1866f9ae5a5..19162c101f653 100644 --- a/pkg/sql/plan/explain/sql_json_test.go +++ b/pkg/sql/plan/explain/sql_json_test.go @@ -16,7 +16,9 @@ package explain import ( "context" + "encoding/base64" "encoding/json" + "fmt" "math" "strings" "testing" @@ -577,15 +579,16 @@ func TestBuildSQLJSONPlanPreservesVectorIndexPayload(t *testing.T) { Steps: []int32{0}, Nodes: []*plan.Node{{ NodeId: 0, - NodeType: plan.Node_VECTOR_INDEX_SCAN, + NodeType: plan.Node_INDEX_SEARCH_SCAN, ObjRef: &plan.ObjectRef{SchemaName: "db", ObjName: "items"}, TableDef: &plan.TableDef{Name: "items"}, - VectorIndexScan: &plan.VectorIndexScan{ + IndexSearchScan: &plan.IndexSearchScan{ Index: &plan.IndexDef{IndexName: "idx_embedding"}, DistanceFunction: "l2_distance", - QueryVector: sqlJSONTestInt32(1), + QueryPayload: sqlJSONTestInt32(1), CandidateLimit: sqlJSONTestInt32(10), - FirstRoundLimit: sqlJSONTestInt32(4), + AlgoExprNames: []string{"first_round_limit"}, + AlgoExprs: []*plan.Expr{sqlJSONTestInt32(4)}, PreFilters: []*plan.Expr{sqlJSONTestInt32(2)}, }, }}, @@ -603,12 +606,10 @@ func TestBuildSQLJSONPlanPreservesVectorIndexPayload(t *testing.T) { "index=idx_embedding", "metric=l2_distance", "direction=INTERNAL", - "initial_probe_count=0", - "bucket_expand_step=0", "post_filter_over_fetch=false", - "query_vector=1", + "query_payload=1", "candidate_limit=10", - "first_round_limit=4", + `algo_expr["first_round_limit"]=4`, "pre_filter[0]=2", }, decoded.MatrixOne.Nodes[0].Expressions) } @@ -619,11 +620,11 @@ func TestBuildSQLJSONPlanPreservesLiteralVectorQuery(t *testing.T) { Steps: []int32{0}, Nodes: []*plan.Node{{ NodeId: 0, - NodeType: plan.Node_VECTOR_INDEX_SCAN, + NodeType: plan.Node_INDEX_SEARCH_SCAN, TableDef: &plan.TableDef{Name: "items"}, - VectorIndexScan: &plan.VectorIndexScan{ - Index: &plan.IndexDef{IndexName: "idx_embedding"}, - QueryVector: &plan.Expr{Expr: &plan.Expr_Lit{Lit: &plan.Literal{Value: &plan.Literal_VecVal{VecVal: "[1,2]"}}}}, + IndexSearchScan: &plan.IndexSearchScan{ + Index: &plan.IndexDef{IndexName: "idx_embedding"}, + QueryPayload: &plan.Expr{Expr: &plan.Expr_Lit{Lit: &plan.Literal{Value: &plan.Literal_VecVal{VecVal: "[1,2]"}}}}, }, }}, }) @@ -637,10 +638,8 @@ func TestBuildSQLJSONPlanPreservesLiteralVectorQuery(t *testing.T) { require.Equal(t, []string{ "index=idx_embedding", "direction=INTERNAL", - "initial_probe_count=0", - "bucket_expand_step=0", "post_filter_over_fetch=false", - "query_vector=[1,2]", + "query_payload=[1,2]", }, decoded.MatrixOne.Nodes[0].Expressions) } @@ -732,12 +731,11 @@ func TestBuildSQLJSONPlanPreservesVectorExecutionSettings(t *testing.T) { Steps: []int32{0}, Nodes: []*plan.Node{{ NodeId: 0, - NodeType: plan.Node_VECTOR_INDEX_SCAN, - VectorIndexScan: &plan.VectorIndexScan{ + NodeType: plan.Node_INDEX_SEARCH_SCAN, + IndexSearchScan: &plan.IndexSearchScan{ Index: &plan.IndexDef{IndexName: "idx_embedding"}, Direction: direction, - InitialProbeCount: probe, - BucketExpandStep: expand, + AlgoOptions: []byte(fmt.Sprintf(`{"initial_probe_count":%d,"bucket_expand_step":%d}`, probe, expand)), PostFilterOverFetch: overfetch, }, }}, @@ -755,11 +753,10 @@ func TestBuildSQLJSONPlanPreservesVectorExecutionSettings(t *testing.T) { low := buildExpressions(1, plan.OrderBySpec_ASC, 2, false) high := buildExpressions(16, plan.OrderBySpec_DESC, 8, true) require.NotEqual(t, low, high) - require.Contains(t, low, "initial_probe_count=1") + require.Contains(t, low, "algo_options_base64="+base64.StdEncoding.EncodeToString([]byte(`{"initial_probe_count":1,"bucket_expand_step":2}`))) require.Contains(t, low, "direction=ASC") - require.Contains(t, high, "initial_probe_count=16") + require.Contains(t, high, "algo_options_base64="+base64.StdEncoding.EncodeToString([]byte(`{"initial_probe_count":16,"bucket_expand_step":8}`))) require.Contains(t, high, "direction=DESC") - require.Contains(t, high, "bucket_expand_step=8") require.Contains(t, high, "post_filter_over_fetch=true") } @@ -770,8 +767,8 @@ func TestBuildSQLJSONPlanPreservesVectorIndexDistanceRange(t *testing.T) { Steps: []int32{0}, Nodes: []*plan.Node{{ NodeId: 0, - NodeType: plan.Node_VECTOR_INDEX_SCAN, - VectorIndexScan: &plan.VectorIndexScan{ + NodeType: plan.Node_INDEX_SEARCH_SCAN, + IndexSearchScan: &plan.IndexSearchScan{ Index: &plan.IndexDef{IndexName: "idx_embedding"}, DistanceRange: &plan.DistRange{ UpperBoundType: plan.BoundType_INCLUSIVE, @@ -797,8 +794,6 @@ func TestBuildSQLJSONPlanPreservesVectorIndexDistanceRange(t *testing.T) { require.Equal(t, []string{ "index=idx_embedding", "direction=INTERNAL", - "initial_probe_count=0", - "bucket_expand_step=0", "post_filter_over_fetch=false", "distance_upper_bound_type=INCLUSIVE", "distance_upper_bound=5", @@ -806,8 +801,6 @@ func TestBuildSQLJSONPlanPreservesVectorIndexDistanceRange(t *testing.T) { require.Equal(t, []string{ "index=idx_embedding", "direction=INTERNAL", - "initial_probe_count=0", - "bucket_expand_step=0", "post_filter_over_fetch=false", "distance_upper_bound_type=INCLUSIVE", "distance_upper_bound=50", @@ -872,10 +865,10 @@ func TestSQLJSONHelpersRejectMalformedInputsAndPreserveStatistics(t *testing.T) Steps: []int32{0}, Nodes: []*plan.Node{{ NodeId: 1, - NodeType: plan.Node_VECTOR_INDEX_SCAN, - VectorIndexScan: &plan.VectorIndexScan{ - Index: &plan.IndexDef{IndexName: "idx_embedding"}, - QueryVector: &plan.Expr{Expr: &plan.Expr_Lit{Lit: &plan.Literal{Value: &plan.Literal_Jsonval{Jsonval: "{}"}}}}, + NodeType: plan.Node_INDEX_SEARCH_SCAN, + IndexSearchScan: &plan.IndexSearchScan{ + Index: &plan.IndexDef{IndexName: "idx_embedding"}, + QueryPayload: &plan.Expr{Expr: &plan.Expr_Lit{Lit: &plan.Literal{Value: &plan.Literal_Jsonval{Jsonval: "{}"}}}}, }, }}, }) diff --git a/test/distributed/cases/optimizer/explain_format_json.result b/test/distributed/cases/optimizer/explain_format_json.result index c4833d8fb0d9e..d7f0431ffef1b 100644 --- a/test/distributed/cases/optimizer/explain_format_json.result +++ b/test/distributed/cases/optimizer/explain_format_json.result @@ -57,6 +57,20 @@ explain format=json select id, tag from vectors_include order by l2_distance(v,' EXPLAIN drop table vectors_include; set probe_limit=default; +set experimental_fulltext_index=1; +set experimental_fulltext2_index=1; +create table documents(id bigint primary key, body text); +create fulltext index ix_text on documents(body); +explain format=json select id from documents where match(body) against('needle') > 0; +EXPLAIN +drop table documents; +create table documents(id bigint primary key, body text); +create fulltext2 index ix_text on documents(body) with parser ngram; +explain format=json select id from documents where match(body) against('needle') > 0; +EXPLAIN +drop table documents; +set experimental_fulltext_index=default; +set experimental_fulltext2_index=default; create table fill_values(ts timestamp, v int); insert into fill_values values ('2024-01-01 00:00:00',1),('2024-01-01 00:00:10',3); explain format=json select c from (select _wstart as a, max(v) as b, min(v) as c from fill_values interval(ts, 5, second) fill(prev)) x; diff --git a/test/distributed/cases/optimizer/explain_format_json.sql b/test/distributed/cases/optimizer/explain_format_json.sql index 618b950c3109f..214cce484b4ef 100644 --- a/test/distributed/cases/optimizer/explain_format_json.sql +++ b/test/distributed/cases/optimizer/explain_format_json.sql @@ -50,12 +50,12 @@ create table vectors(id bigint primary key, v vecf32(3)); insert into vectors values (1,'[0,0,0]'),(2,'[1,1,1]'),(3,'[2,2,2]'); create index ix using ivfflat on vectors(v) lists=1 op_type 'vector_l2_ops'; set probe_limit=1; --- @regex("(?s)Vector Index Scan.*query_vector=0x0000803F0000803F0000803F",true) +-- @regex("(?s)Vector Index Scan.*query_payload=0x0000803F0000803F0000803F",true) explain format=json select id from vectors order by l2_distance(v,'[1,1,1]') limit 2; --- @regex("(?s)Vector Index Scan.*query_vector=0x000000400000004000000040",true) +-- @regex("(?s)Vector Index Scan.*query_payload=0x000000400000004000000040",true) explain format=json select id from vectors order by l2_distance(v,'[2,2,2]') limit 2; set probe_limit=16; --- @regex("(?s)Vector Index Scan.*initial_probe_count=16",true) +-- @regex("(?s)Vector Index Scan.*algo_setting[^=]*=NProbe: 16([^0-9]|$)",true) explain format=json select id from vectors order by l2_distance(v,'[1,1,1]') limit 2; -- @regex("(?s)Vector Index Scan.*distance_upper_bound_type=EXCLUSIVE.*distance_upper_bound=5[^0-9]",true) explain format=json select id from vectors where l2_distance(v,'[1,1,1]') < 5 order by l2_distance(v,'[1,1,1]') limit 2; @@ -75,13 +75,30 @@ explain format=json select id from vectors where l2_distance(v,'[1,1,1]') > 1 or create table vectors_include(id bigint primary key, v vecf32(3), tag varchar(10)); insert into vectors_include values (1,'[0,0,0]','zero'),(2,'[1,1,1]','one'),(3,'[2,2,2]','two'); create index ix_include using ivfflat on vectors_include(v) lists=1 op_type 'vector_l2_ops' include(tag); --- @regex("(?s)Vector Index Scan.*first_round_limit=2",true) +-- @regex("(?s)Vector Index Scan.*algo_expr[^=]*first_round_limit[^=]*=2([^0-9]|$)",true) explain format=json select id, tag from vectors_include order by l2_distance(v,'[1,1,1]') limit 2 by rank with option 'mode=include'; --- @regex("(?s)Vector Index Scan.*first_round_limit=3",true) +-- @regex("(?s)Vector Index Scan.*algo_expr[^=]*first_round_limit[^=]*=3([^0-9]|$)",true) explain format=json select id, tag from vectors_include order by l2_distance(v,'[1,1,1]') limit 3 by rank with option 'mode=include'; drop table vectors_include; set probe_limit=default; +-- INDEX_SEARCH_SCAN also carries fulltext payloads through the same protocol. +set experimental_fulltext_index=1; +set experimental_fulltext2_index=1; +create table documents(id bigint primary key, body text); +create fulltext index ix_text on documents(body); +-- @regex("(?s)Fulltext Index Scan.*query_payload=[^a-zA-Z0-9_]*needle([^a-zA-Z0-9_]|$)",true) +explain format=json select id from documents where match(body) against('needle') > 0; +drop table documents; +create table documents(id bigint primary key, body text); +create fulltext2 index ix_text on documents(body) with parser ngram; +-- @regex("(?s)Fulltext Index Scan.*query_payload=[^a-zA-Z0-9_]*needle([^a-zA-Z0-9_]|$)",true) +explain format=json select id from documents where match(body) against('needle') > 0; +drop table documents; +set experimental_fulltext_index=default; +set experimental_fulltext2_index=default; + + -- FILL preserves mode and the targets remaining after projection pruning. create table fill_values(ts timestamp, v int); insert into fill_values values ('2024-01-01 00:00:00',1),('2024-01-01 00:00:10',3); From 01def75b2445aaf41072e70690b1f7e25254c420 Mon Sep 17 00:00:00 2001 From: bRong Njam Date: Tue, 13 Oct 2026 17:33:33 +0800 Subject: [PATCH 15/20] Revert "Merge remote-tracking branch 'upstream/pr-30109' into fix/sirius-input-coalescing-release" This reverts commit 97d8c0f19e77f8b132976f3111a7a2bbbeeb9590, reversing changes made to 19834ab48835cc00074298f56d5285451118985a. --- docs/design/explain_format_json.md | 9 -- pkg/sql/plan/explain/sql_json.go | 60 +++------ .../sql_json_index_search_schema_test.go | 124 ------------------ pkg/sql/plan/explain/sql_json_test.go | 57 ++++---- .../optimizer/explain_format_json.result | 14 -- .../cases/optimizer/explain_format_json.sql | 27 +--- 6 files changed, 54 insertions(+), 237 deletions(-) delete mode 100644 pkg/sql/plan/explain/sql_json_index_search_schema_test.go diff --git a/docs/design/explain_format_json.md b/docs/design/explain_format_json.md index 5d1f700198f29..3c5a389fa24b8 100644 --- a/docs/design/explain_format_json.md +++ b/docs/design/explain_format_json.md @@ -159,15 +159,6 @@ The following mappings are normative for schema version 1: `fill_value[i]=` from `FillVal`, in their original order. Empty target/value lists add no entries. Unknown modes and missing required expressions are errors; the renderer never rebuilds pre-pruning targets. -- An `INDEX_SEARCH_SCAN` uses the current typed `IndexSearchScan` contract - for vector and fulltext indexes. It retains the query payload, candidate - limit, distance bounds, included columns and pre-filters. Algorithm-owned - option bytes are preserved as `algo_options_base64`; existing plugin - EXPLAIN hooks add readable `algo_setting[i]` entries. Named algorithm - expressions (including IVF first-round limits) use `algo_expr["name"]`, - with quoted names and matching ordered name/expression arrays. Missing, - empty or duplicate expression names are errors. No protobuf fields are - invented and no options are decoded using another algorithm's schema. - Serialized scalar bytes use `serialized(type=,width=,scale=,value=0x)`. Serialized literal vectors use diff --git a/pkg/sql/plan/explain/sql_json.go b/pkg/sql/plan/explain/sql_json.go index 4c2af59b3cc46..10dc7f9010540 100644 --- a/pkg/sql/plan/explain/sql_json.go +++ b/pkg/sql/plan/explain/sql_json.go @@ -17,7 +17,6 @@ package explain import ( "bytes" "context" - "encoding/base64" "encoding/json" "math" "strconv" @@ -419,15 +418,12 @@ func (b *sqlJSONPlanBuilder) addNodeDetails(item *sqlJSONNode, node *plan.Node) if err != nil { return err } - case plan.Node_INDEX_SEARCH_SCAN: - if node.IndexSearchScan == nil || node.IndexSearchScan.Index == nil || node.IndexSearchScan.Index.IndexName == "" { - return moerr.NewInvalidInput(b.ctx, "index search scan metadata is missing") + case plan.Node_VECTOR_INDEX_SCAN: + if node.VectorIndexScan == nil || node.VectorIndexScan.Index == nil || node.VectorIndexScan.Index.IndexName == "" { + return moerr.NewInvalidInput(b.ctx, "vector index scan metadata is missing") } item.TableName = sqlJSONTableName(node) - if item.TableName == "" { - item.TableName = sqlJSONObjectRefName(node.IndexSearchScan.SourceTable) - } - item.Expressions, err = sqlJSONIndexSearchExpressions(b.ctx, node.IndexSearchScan, &b.textOpts) + item.Expressions, err = sqlJSONVectorIndexExpressions(b.ctx, node.VectorIndexScan, &b.textOpts) if err != nil { return err } @@ -625,9 +621,9 @@ func sqlJSONSampleExpressions(ctx context.Context, spec *plan.SampleFuncSpec) ([ }, nil } -func sqlJSONIndexSearchExpressions(ctx context.Context, spec *plan.IndexSearchScan, options *ExplainOptions) ([]string, error) { +func sqlJSONVectorIndexExpressions(ctx context.Context, spec *plan.VectorIndexScan, options *ExplainOptions) ([]string, error) { if spec == nil || spec.Index == nil { - return nil, moerr.NewInvalidInput(ctx, "index search scan metadata is missing") + return nil, moerr.NewInvalidInput(ctx, "vector index scan metadata is missing") } values := make([]string, 0, 16+len(spec.PreFilters)+len(spec.IncludedColumns)) values = append(values, "index="+spec.Index.IndexName) @@ -637,26 +633,18 @@ func sqlJSONIndexSearchExpressions(ctx context.Context, spec *plan.IndexSearchSc direction := spec.Direction.String() switch spec.Direction { case plan.OrderBySpec_INTERNAL, plan.OrderBySpec_ASC, plan.OrderBySpec_DESC: - // valid index scan directions + // valid vector scan directions default: - return nil, moerr.NewInvalidInputf(ctx, "index search scan has unknown direction %d", spec.Direction) + return nil, moerr.NewInvalidInputf(ctx, "vector index scan has unknown direction %d", spec.Direction) } values = append(values, "direction="+direction, + "initial_probe_count="+strconv.FormatUint(uint64(spec.InitialProbeCount), 10), + "bucket_expand_step="+strconv.FormatUint(uint64(spec.BucketExpandStep), 10), "post_filter_over_fetch="+strconv.FormatBool(spec.PostFilterOverFetch), ) - // Algorithm settings belong to the plugin's wire contract. Preserve every - // byte without assuming a JSON format or inventing fields on the shared - // protobuf. Use its existing EXPLAIN hook for human-readable settings. - if len(spec.AlgoOptions) != 0 { - values = append(values, "algo_options_base64="+base64.StdEncoding.EncodeToString(spec.AlgoOptions)) - } - settings, err := indexSearchScanSettings(spec) - if err != nil { - return nil, err - } - for i, setting := range settings { - values = append(values, "algo_setting["+strconv.Itoa(i)+"]="+setting) + if spec.ThreadsSearch != 0 { + values = append(values, "threads_search="+strconv.FormatInt(spec.ThreadsSearch, 10)) } for i, column := range spec.IncludedColumns { values = append(values, "included_column["+strconv.Itoa(i)+"]="+column) @@ -665,8 +653,9 @@ func sqlJSONIndexSearchExpressions(ctx context.Context, spec *plan.IndexSearchSc name string expr *plan.Expr }{ - {name: "query_payload", expr: spec.QueryPayload}, + {name: "query_vector", expr: spec.QueryVector}, {name: "candidate_limit", expr: spec.CandidateLimit}, + {name: "first_round_limit", expr: spec.FirstRoundLimit}, } { value, err := sqlJSONExpr(ctx, field.expr, options) if err != nil { @@ -676,21 +665,6 @@ func sqlJSONIndexSearchExpressions(ctx context.Context, spec *plan.IndexSearchSc values = append(values, field.name+"="+value) } } - if len(spec.AlgoExprNames) != len(spec.AlgoExprs) { - return nil, moerr.NewInvalidInput(ctx, "index search scan algorithm expression names do not match expressions") - } - seen := make(map[string]struct{}, len(spec.AlgoExprNames)) - for i, name := range spec.AlgoExprNames { - if _, duplicate := seen[name]; name == "" || duplicate || spec.AlgoExprs[i] == nil { - return nil, moerr.NewInvalidInput(ctx, "index search scan algorithm expression is missing or ambiguous") - } - seen[name] = struct{}{} - value, err := sqlJSONExpr(ctx, spec.AlgoExprs[i], options) - if err != nil { - return nil, err - } - values = append(values, "algo_expr["+strconv.Quote(name)+"]="+value) - } if spec.DistanceRange != nil { for _, bound := range []struct { name string @@ -704,18 +678,18 @@ func sqlJSONIndexSearchExpressions(ctx context.Context, spec *plan.IndexSearchSc continue } if bound.expr == nil { - return nil, moerr.NewInvalidInputf(ctx, "index search scan %s expression is missing", bound.name) + return nil, moerr.NewInvalidInputf(ctx, "vector index scan %s expression is missing", bound.name) } boundType := bound.boundType.String() if boundType == "" { - return nil, moerr.NewInvalidInputf(ctx, "index search scan %s has unknown bound type %d", bound.name, bound.boundType) + return nil, moerr.NewInvalidInputf(ctx, "vector index scan %s has unknown bound type %d", bound.name, bound.boundType) } value, err := sqlJSONExpr(ctx, bound.expr, options) if err != nil { return nil, err } if value == "" { - return nil, moerr.NewInvalidInputf(ctx, "index search scan %s expression is empty", bound.name) + return nil, moerr.NewInvalidInputf(ctx, "vector index scan %s expression is empty", bound.name) } values = append(values, bound.name+"_type="+boundType, bound.name+"="+value) } diff --git a/pkg/sql/plan/explain/sql_json_index_search_schema_test.go b/pkg/sql/plan/explain/sql_json_index_search_schema_test.go deleted file mode 100644 index ea24dee05cd8d..0000000000000 --- a/pkg/sql/plan/explain/sql_json_index_search_schema_test.go +++ /dev/null @@ -1,124 +0,0 @@ -// Copyright 2026 Matrix Origin -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package explain - -import ( - "context" - "encoding/base64" - "encoding/json" - "strings" - "testing" - - "github.com/matrixorigin/matrixone/pkg/pb/plan" - planpkg "github.com/matrixorigin/matrixone/pkg/sql/plan" - "github.com/stretchr/testify/require" -) - -func sqlJSONWireIndexNode(t *testing.T, node *plan.Node) sqlJSONNode { - t.Helper() - query := &plan.Query{StmtType: plan.Query_SELECT, Steps: []int32{0}, Nodes: []*plan.Node{node}} - wire, err := query.Marshal() - require.NoError(t, err) - var remote plan.Query - require.NoError(t, remote.Unmarshal(wire)) - raw, err := BuildSQLJSONPlan(t.Context(), &remote) - require.NoError(t, err) - var decoded sqlJSONPlan - require.NoError(t, json.Unmarshal(raw, &decoded)) - require.Len(t, decoded.MatrixOne.Nodes, 1) - return decoded.MatrixOne.Nodes[0] -} - -func TestSQLJSONIndexSearchWirePayload(t *testing.T) { - opts := []byte(`{"initial_probe_count":7,"bucket_expand_step":3,"threads_search":2}`) - node := sqlJSONWireIndexNode(t, &plan.Node{ - NodeId: 0, NodeType: plan.Node_INDEX_SEARCH_SCAN, - IndexSearchScan: &plan.IndexSearchScan{ - SourceTable: &plan.ObjectRef{SchemaName: "db", ObjName: "items"}, - Index: &plan.IndexDef{IndexName: "idx_embedding", IndexAlgo: "ivfflat"}, - DistanceFunction: "l2_distance", Direction: plan.OrderBySpec_DESC, - QueryPayload: &plan.Expr{Expr: &plan.Expr_Lit{Lit: &plan.Literal{Value: &plan.Literal_VecVal{VecVal: "[0.1,0.2]"}}}}, - CandidateLimit: planpkg.MakePlan2Uint64ConstExprWithType(12), - PostFilterOverFetch: true, AlgoOptions: opts, - AlgoExprNames: []string{"first_round_limit"}, - AlgoExprs: []*plan.Expr{planpkg.MakePlan2Uint64ConstExprWithType(4)}, - IncludedColumns: []string{"id", "embedding"}, - }, - }) - require.Equal(t, "Vector Index Scan", node.Operator) - require.Equal(t, "db.items", node.TableName) - for _, want := range []string{"index=idx_embedding", "metric=l2_distance", "direction=DESC", - "post_filter_over_fetch=true", "algo_setting[0]=NProbe: 7", "query_payload=[0.1,0.2]", - "candidate_limit=12", `algo_expr["first_round_limit"]=4`, "included_column[1]=embedding"} { - require.Contains(t, node.Expressions, want) - } - for _, value := range node.Expressions { - if encoded, ok := strings.CutPrefix(value, "algo_options_base64="); ok { - decoded, err := base64.StdEncoding.DecodeString(encoded) - require.NoError(t, err) - require.Equal(t, opts, decoded) - return - } - } - t.Fatal("missing algorithm options") -} - -func TestSQLJSONFulltextIndexSearchWireMetadata(t *testing.T) { - node := fulltextSearchScanNode() - node.IndexSearchScan.SourceTable = &plan.ObjectRef{SchemaName: "db", ObjName: "documents"} - node.IndexSearchScan.QueryPayload = planpkg.MakePlan2StringConstExprWithType("needle") - decoded := sqlJSONWireIndexNode(t, node) - require.Equal(t, "Fulltext Index Scan", decoded.Operator) - require.Equal(t, "db.documents", decoded.TableName) - require.Contains(t, decoded.Expressions, "index=ft_body") - require.Contains(t, strings.Join(decoded.Expressions, "\n"), "needle") - require.NotContains(t, strings.Join(decoded.Expressions, "\n"), "metric=") -} - -func TestSQLJSONIndexSearchOpaqueOptionsAndNamedExpressions(t *testing.T) { - opts := []byte{0, 255, 'x'} - decoded := sqlJSONWireIndexNode(t, &plan.Node{NodeType: plan.Node_INDEX_SEARCH_SCAN, - IndexSearchScan: &plan.IndexSearchScan{ - Index: &plan.IndexDef{IndexName: "future_index", IndexAlgo: "future_plugin"}, - AlgoOptions: opts, AlgoExprNames: []string{"a]\n=2"}, AlgoExprs: []*plan.Expr{sqlJSONTestInt32(1)}, - }}) - require.Contains(t, decoded.Expressions, "algo_options_base64="+base64.StdEncoding.EncodeToString(opts)) - require.Contains(t, decoded.Expressions, `algo_expr["a]\n=2"]=1`) -} - -func TestSQLJSONIndexSearchRejectsMalformedAlgorithmMetadata(t *testing.T) { - for _, tc := range []struct { - name string - names []string - exprs []*plan.Expr - }{ - {"missing name", nil, []*plan.Expr{sqlJSONTestInt32(1)}}, - {"missing expression", []string{"limit"}, nil}, - {"empty name", []string{""}, []*plan.Expr{sqlJSONTestInt32(1)}}, - {"nil expression", []string{"limit"}, []*plan.Expr{nil}}, - {"duplicate name", []string{"limit", "limit"}, []*plan.Expr{sqlJSONTestInt32(1), sqlJSONTestInt32(2)}}, - } { - t.Run(tc.name, func(t *testing.T) { - _, err := sqlJSONIndexSearchExpressions(context.Background(), &plan.IndexSearchScan{ - Index: &plan.IndexDef{IndexName: "idx"}, AlgoExprNames: tc.names, AlgoExprs: tc.exprs, - }, &ExplainOptions{Format: EXPLAIN_FORMAT_TEXT}) - require.Error(t, err) - }) - } - _, err := sqlJSONIndexSearchExpressions(t.Context(), &plan.IndexSearchScan{ - Index: &plan.IndexDef{IndexName: "idx", IndexAlgo: "ivfflat"}, AlgoOptions: []byte("{"), - }, &ExplainOptions{Format: EXPLAIN_FORMAT_TEXT}) - require.Error(t, err) -} diff --git a/pkg/sql/plan/explain/sql_json_test.go b/pkg/sql/plan/explain/sql_json_test.go index 19162c101f653..bd1866f9ae5a5 100644 --- a/pkg/sql/plan/explain/sql_json_test.go +++ b/pkg/sql/plan/explain/sql_json_test.go @@ -16,9 +16,7 @@ package explain import ( "context" - "encoding/base64" "encoding/json" - "fmt" "math" "strings" "testing" @@ -579,16 +577,15 @@ func TestBuildSQLJSONPlanPreservesVectorIndexPayload(t *testing.T) { Steps: []int32{0}, Nodes: []*plan.Node{{ NodeId: 0, - NodeType: plan.Node_INDEX_SEARCH_SCAN, + NodeType: plan.Node_VECTOR_INDEX_SCAN, ObjRef: &plan.ObjectRef{SchemaName: "db", ObjName: "items"}, TableDef: &plan.TableDef{Name: "items"}, - IndexSearchScan: &plan.IndexSearchScan{ + VectorIndexScan: &plan.VectorIndexScan{ Index: &plan.IndexDef{IndexName: "idx_embedding"}, DistanceFunction: "l2_distance", - QueryPayload: sqlJSONTestInt32(1), + QueryVector: sqlJSONTestInt32(1), CandidateLimit: sqlJSONTestInt32(10), - AlgoExprNames: []string{"first_round_limit"}, - AlgoExprs: []*plan.Expr{sqlJSONTestInt32(4)}, + FirstRoundLimit: sqlJSONTestInt32(4), PreFilters: []*plan.Expr{sqlJSONTestInt32(2)}, }, }}, @@ -606,10 +603,12 @@ func TestBuildSQLJSONPlanPreservesVectorIndexPayload(t *testing.T) { "index=idx_embedding", "metric=l2_distance", "direction=INTERNAL", + "initial_probe_count=0", + "bucket_expand_step=0", "post_filter_over_fetch=false", - "query_payload=1", + "query_vector=1", "candidate_limit=10", - `algo_expr["first_round_limit"]=4`, + "first_round_limit=4", "pre_filter[0]=2", }, decoded.MatrixOne.Nodes[0].Expressions) } @@ -620,11 +619,11 @@ func TestBuildSQLJSONPlanPreservesLiteralVectorQuery(t *testing.T) { Steps: []int32{0}, Nodes: []*plan.Node{{ NodeId: 0, - NodeType: plan.Node_INDEX_SEARCH_SCAN, + NodeType: plan.Node_VECTOR_INDEX_SCAN, TableDef: &plan.TableDef{Name: "items"}, - IndexSearchScan: &plan.IndexSearchScan{ - Index: &plan.IndexDef{IndexName: "idx_embedding"}, - QueryPayload: &plan.Expr{Expr: &plan.Expr_Lit{Lit: &plan.Literal{Value: &plan.Literal_VecVal{VecVal: "[1,2]"}}}}, + VectorIndexScan: &plan.VectorIndexScan{ + Index: &plan.IndexDef{IndexName: "idx_embedding"}, + QueryVector: &plan.Expr{Expr: &plan.Expr_Lit{Lit: &plan.Literal{Value: &plan.Literal_VecVal{VecVal: "[1,2]"}}}}, }, }}, }) @@ -638,8 +637,10 @@ func TestBuildSQLJSONPlanPreservesLiteralVectorQuery(t *testing.T) { require.Equal(t, []string{ "index=idx_embedding", "direction=INTERNAL", + "initial_probe_count=0", + "bucket_expand_step=0", "post_filter_over_fetch=false", - "query_payload=[1,2]", + "query_vector=[1,2]", }, decoded.MatrixOne.Nodes[0].Expressions) } @@ -731,11 +732,12 @@ func TestBuildSQLJSONPlanPreservesVectorExecutionSettings(t *testing.T) { Steps: []int32{0}, Nodes: []*plan.Node{{ NodeId: 0, - NodeType: plan.Node_INDEX_SEARCH_SCAN, - IndexSearchScan: &plan.IndexSearchScan{ + NodeType: plan.Node_VECTOR_INDEX_SCAN, + VectorIndexScan: &plan.VectorIndexScan{ Index: &plan.IndexDef{IndexName: "idx_embedding"}, Direction: direction, - AlgoOptions: []byte(fmt.Sprintf(`{"initial_probe_count":%d,"bucket_expand_step":%d}`, probe, expand)), + InitialProbeCount: probe, + BucketExpandStep: expand, PostFilterOverFetch: overfetch, }, }}, @@ -753,10 +755,11 @@ func TestBuildSQLJSONPlanPreservesVectorExecutionSettings(t *testing.T) { low := buildExpressions(1, plan.OrderBySpec_ASC, 2, false) high := buildExpressions(16, plan.OrderBySpec_DESC, 8, true) require.NotEqual(t, low, high) - require.Contains(t, low, "algo_options_base64="+base64.StdEncoding.EncodeToString([]byte(`{"initial_probe_count":1,"bucket_expand_step":2}`))) + require.Contains(t, low, "initial_probe_count=1") require.Contains(t, low, "direction=ASC") - require.Contains(t, high, "algo_options_base64="+base64.StdEncoding.EncodeToString([]byte(`{"initial_probe_count":16,"bucket_expand_step":8}`))) + require.Contains(t, high, "initial_probe_count=16") require.Contains(t, high, "direction=DESC") + require.Contains(t, high, "bucket_expand_step=8") require.Contains(t, high, "post_filter_over_fetch=true") } @@ -767,8 +770,8 @@ func TestBuildSQLJSONPlanPreservesVectorIndexDistanceRange(t *testing.T) { Steps: []int32{0}, Nodes: []*plan.Node{{ NodeId: 0, - NodeType: plan.Node_INDEX_SEARCH_SCAN, - IndexSearchScan: &plan.IndexSearchScan{ + NodeType: plan.Node_VECTOR_INDEX_SCAN, + VectorIndexScan: &plan.VectorIndexScan{ Index: &plan.IndexDef{IndexName: "idx_embedding"}, DistanceRange: &plan.DistRange{ UpperBoundType: plan.BoundType_INCLUSIVE, @@ -794,6 +797,8 @@ func TestBuildSQLJSONPlanPreservesVectorIndexDistanceRange(t *testing.T) { require.Equal(t, []string{ "index=idx_embedding", "direction=INTERNAL", + "initial_probe_count=0", + "bucket_expand_step=0", "post_filter_over_fetch=false", "distance_upper_bound_type=INCLUSIVE", "distance_upper_bound=5", @@ -801,6 +806,8 @@ func TestBuildSQLJSONPlanPreservesVectorIndexDistanceRange(t *testing.T) { require.Equal(t, []string{ "index=idx_embedding", "direction=INTERNAL", + "initial_probe_count=0", + "bucket_expand_step=0", "post_filter_over_fetch=false", "distance_upper_bound_type=INCLUSIVE", "distance_upper_bound=50", @@ -865,10 +872,10 @@ func TestSQLJSONHelpersRejectMalformedInputsAndPreserveStatistics(t *testing.T) Steps: []int32{0}, Nodes: []*plan.Node{{ NodeId: 1, - NodeType: plan.Node_INDEX_SEARCH_SCAN, - IndexSearchScan: &plan.IndexSearchScan{ - Index: &plan.IndexDef{IndexName: "idx_embedding"}, - QueryPayload: &plan.Expr{Expr: &plan.Expr_Lit{Lit: &plan.Literal{Value: &plan.Literal_Jsonval{Jsonval: "{}"}}}}, + NodeType: plan.Node_VECTOR_INDEX_SCAN, + VectorIndexScan: &plan.VectorIndexScan{ + Index: &plan.IndexDef{IndexName: "idx_embedding"}, + QueryVector: &plan.Expr{Expr: &plan.Expr_Lit{Lit: &plan.Literal{Value: &plan.Literal_Jsonval{Jsonval: "{}"}}}}, }, }}, }) diff --git a/test/distributed/cases/optimizer/explain_format_json.result b/test/distributed/cases/optimizer/explain_format_json.result index d7f0431ffef1b..c4833d8fb0d9e 100644 --- a/test/distributed/cases/optimizer/explain_format_json.result +++ b/test/distributed/cases/optimizer/explain_format_json.result @@ -57,20 +57,6 @@ explain format=json select id, tag from vectors_include order by l2_distance(v,' EXPLAIN drop table vectors_include; set probe_limit=default; -set experimental_fulltext_index=1; -set experimental_fulltext2_index=1; -create table documents(id bigint primary key, body text); -create fulltext index ix_text on documents(body); -explain format=json select id from documents where match(body) against('needle') > 0; -EXPLAIN -drop table documents; -create table documents(id bigint primary key, body text); -create fulltext2 index ix_text on documents(body) with parser ngram; -explain format=json select id from documents where match(body) against('needle') > 0; -EXPLAIN -drop table documents; -set experimental_fulltext_index=default; -set experimental_fulltext2_index=default; create table fill_values(ts timestamp, v int); insert into fill_values values ('2024-01-01 00:00:00',1),('2024-01-01 00:00:10',3); explain format=json select c from (select _wstart as a, max(v) as b, min(v) as c from fill_values interval(ts, 5, second) fill(prev)) x; diff --git a/test/distributed/cases/optimizer/explain_format_json.sql b/test/distributed/cases/optimizer/explain_format_json.sql index 214cce484b4ef..618b950c3109f 100644 --- a/test/distributed/cases/optimizer/explain_format_json.sql +++ b/test/distributed/cases/optimizer/explain_format_json.sql @@ -50,12 +50,12 @@ create table vectors(id bigint primary key, v vecf32(3)); insert into vectors values (1,'[0,0,0]'),(2,'[1,1,1]'),(3,'[2,2,2]'); create index ix using ivfflat on vectors(v) lists=1 op_type 'vector_l2_ops'; set probe_limit=1; --- @regex("(?s)Vector Index Scan.*query_payload=0x0000803F0000803F0000803F",true) +-- @regex("(?s)Vector Index Scan.*query_vector=0x0000803F0000803F0000803F",true) explain format=json select id from vectors order by l2_distance(v,'[1,1,1]') limit 2; --- @regex("(?s)Vector Index Scan.*query_payload=0x000000400000004000000040",true) +-- @regex("(?s)Vector Index Scan.*query_vector=0x000000400000004000000040",true) explain format=json select id from vectors order by l2_distance(v,'[2,2,2]') limit 2; set probe_limit=16; --- @regex("(?s)Vector Index Scan.*algo_setting[^=]*=NProbe: 16([^0-9]|$)",true) +-- @regex("(?s)Vector Index Scan.*initial_probe_count=16",true) explain format=json select id from vectors order by l2_distance(v,'[1,1,1]') limit 2; -- @regex("(?s)Vector Index Scan.*distance_upper_bound_type=EXCLUSIVE.*distance_upper_bound=5[^0-9]",true) explain format=json select id from vectors where l2_distance(v,'[1,1,1]') < 5 order by l2_distance(v,'[1,1,1]') limit 2; @@ -75,30 +75,13 @@ explain format=json select id from vectors where l2_distance(v,'[1,1,1]') > 1 or create table vectors_include(id bigint primary key, v vecf32(3), tag varchar(10)); insert into vectors_include values (1,'[0,0,0]','zero'),(2,'[1,1,1]','one'),(3,'[2,2,2]','two'); create index ix_include using ivfflat on vectors_include(v) lists=1 op_type 'vector_l2_ops' include(tag); --- @regex("(?s)Vector Index Scan.*algo_expr[^=]*first_round_limit[^=]*=2([^0-9]|$)",true) +-- @regex("(?s)Vector Index Scan.*first_round_limit=2",true) explain format=json select id, tag from vectors_include order by l2_distance(v,'[1,1,1]') limit 2 by rank with option 'mode=include'; --- @regex("(?s)Vector Index Scan.*algo_expr[^=]*first_round_limit[^=]*=3([^0-9]|$)",true) +-- @regex("(?s)Vector Index Scan.*first_round_limit=3",true) explain format=json select id, tag from vectors_include order by l2_distance(v,'[1,1,1]') limit 3 by rank with option 'mode=include'; drop table vectors_include; set probe_limit=default; --- INDEX_SEARCH_SCAN also carries fulltext payloads through the same protocol. -set experimental_fulltext_index=1; -set experimental_fulltext2_index=1; -create table documents(id bigint primary key, body text); -create fulltext index ix_text on documents(body); --- @regex("(?s)Fulltext Index Scan.*query_payload=[^a-zA-Z0-9_]*needle([^a-zA-Z0-9_]|$)",true) -explain format=json select id from documents where match(body) against('needle') > 0; -drop table documents; -create table documents(id bigint primary key, body text); -create fulltext2 index ix_text on documents(body) with parser ngram; --- @regex("(?s)Fulltext Index Scan.*query_payload=[^a-zA-Z0-9_]*needle([^a-zA-Z0-9_]|$)",true) -explain format=json select id from documents where match(body) against('needle') > 0; -drop table documents; -set experimental_fulltext_index=default; -set experimental_fulltext2_index=default; - - -- FILL preserves mode and the targets remaining after projection pruning. create table fill_values(ts timestamp, v int); insert into fill_values values ('2024-01-01 00:00:00',1),('2024-01-01 00:00:10',3); From d0302953df0a4b3858cf89666a2336f3ded93a99 Mon Sep 17 00:00:00 2001 From: bRong Njam Date: Tue, 13 Oct 2026 18:36:23 +0800 Subject: [PATCH 16/20] fix(explain): migrate JSON consumers to IndexSearchScan --- docs/design/explain_format_json.md | 7 + pkg/sql/plan/explain/sql_json.go | 60 +++++--- pkg/sql/plan/explain/sql_json_test.go | 134 +++++++++++++----- .../cases/optimizer/explain_format_json.sql | 10 +- 4 files changed, 156 insertions(+), 55 deletions(-) diff --git a/docs/design/explain_format_json.md b/docs/design/explain_format_json.md index 3c5a389fa24b8..da2cf2db7d8ff 100644 --- a/docs/design/explain_format_json.md +++ b/docs/design/explain_format_json.md @@ -159,6 +159,13 @@ The following mappings are normative for schema version 1: `fill_value[i]=` from `FillVal`, in their original order. Empty target/value lists add no entries. Unknown modes and missing required expressions are errors; the renderer never rebuilds pre-pruning targets. +- An `INDEX_SEARCH_SCAN` uses the current typed `IndexSearchScan` contract. + Its query expression is `query_payload`, and named algorithm expressions + use `algo_expr["name"]`. Algorithm-owned option bytes are retained as + `algo_options_base64`; the existing plugin EXPLAIN hooks add readable + `algo_setting[i]` entries. The name and expression arrays must match, with + nonempty, unique names and nonnil expressions. Vector and fulltext scans + share this contract; algorithm options stay opaque to the SQL layer. - Serialized scalar bytes use `serialized(type=,width=,scale=,value=0x)`. Serialized literal vectors use diff --git a/pkg/sql/plan/explain/sql_json.go b/pkg/sql/plan/explain/sql_json.go index 10dc7f9010540..4c2af59b3cc46 100644 --- a/pkg/sql/plan/explain/sql_json.go +++ b/pkg/sql/plan/explain/sql_json.go @@ -17,6 +17,7 @@ package explain import ( "bytes" "context" + "encoding/base64" "encoding/json" "math" "strconv" @@ -418,12 +419,15 @@ func (b *sqlJSONPlanBuilder) addNodeDetails(item *sqlJSONNode, node *plan.Node) if err != nil { return err } - case plan.Node_VECTOR_INDEX_SCAN: - if node.VectorIndexScan == nil || node.VectorIndexScan.Index == nil || node.VectorIndexScan.Index.IndexName == "" { - return moerr.NewInvalidInput(b.ctx, "vector index scan metadata is missing") + case plan.Node_INDEX_SEARCH_SCAN: + if node.IndexSearchScan == nil || node.IndexSearchScan.Index == nil || node.IndexSearchScan.Index.IndexName == "" { + return moerr.NewInvalidInput(b.ctx, "index search scan metadata is missing") } item.TableName = sqlJSONTableName(node) - item.Expressions, err = sqlJSONVectorIndexExpressions(b.ctx, node.VectorIndexScan, &b.textOpts) + if item.TableName == "" { + item.TableName = sqlJSONObjectRefName(node.IndexSearchScan.SourceTable) + } + item.Expressions, err = sqlJSONIndexSearchExpressions(b.ctx, node.IndexSearchScan, &b.textOpts) if err != nil { return err } @@ -621,9 +625,9 @@ func sqlJSONSampleExpressions(ctx context.Context, spec *plan.SampleFuncSpec) ([ }, nil } -func sqlJSONVectorIndexExpressions(ctx context.Context, spec *plan.VectorIndexScan, options *ExplainOptions) ([]string, error) { +func sqlJSONIndexSearchExpressions(ctx context.Context, spec *plan.IndexSearchScan, options *ExplainOptions) ([]string, error) { if spec == nil || spec.Index == nil { - return nil, moerr.NewInvalidInput(ctx, "vector index scan metadata is missing") + return nil, moerr.NewInvalidInput(ctx, "index search scan metadata is missing") } values := make([]string, 0, 16+len(spec.PreFilters)+len(spec.IncludedColumns)) values = append(values, "index="+spec.Index.IndexName) @@ -633,18 +637,26 @@ func sqlJSONVectorIndexExpressions(ctx context.Context, spec *plan.VectorIndexSc direction := spec.Direction.String() switch spec.Direction { case plan.OrderBySpec_INTERNAL, plan.OrderBySpec_ASC, plan.OrderBySpec_DESC: - // valid vector scan directions + // valid index scan directions default: - return nil, moerr.NewInvalidInputf(ctx, "vector index scan has unknown direction %d", spec.Direction) + return nil, moerr.NewInvalidInputf(ctx, "index search scan has unknown direction %d", spec.Direction) } values = append(values, "direction="+direction, - "initial_probe_count="+strconv.FormatUint(uint64(spec.InitialProbeCount), 10), - "bucket_expand_step="+strconv.FormatUint(uint64(spec.BucketExpandStep), 10), "post_filter_over_fetch="+strconv.FormatBool(spec.PostFilterOverFetch), ) - if spec.ThreadsSearch != 0 { - values = append(values, "threads_search="+strconv.FormatInt(spec.ThreadsSearch, 10)) + // Algorithm settings belong to the plugin's wire contract. Preserve every + // byte without assuming a JSON format or inventing fields on the shared + // protobuf. Use its existing EXPLAIN hook for human-readable settings. + if len(spec.AlgoOptions) != 0 { + values = append(values, "algo_options_base64="+base64.StdEncoding.EncodeToString(spec.AlgoOptions)) + } + settings, err := indexSearchScanSettings(spec) + if err != nil { + return nil, err + } + for i, setting := range settings { + values = append(values, "algo_setting["+strconv.Itoa(i)+"]="+setting) } for i, column := range spec.IncludedColumns { values = append(values, "included_column["+strconv.Itoa(i)+"]="+column) @@ -653,9 +665,8 @@ func sqlJSONVectorIndexExpressions(ctx context.Context, spec *plan.VectorIndexSc name string expr *plan.Expr }{ - {name: "query_vector", expr: spec.QueryVector}, + {name: "query_payload", expr: spec.QueryPayload}, {name: "candidate_limit", expr: spec.CandidateLimit}, - {name: "first_round_limit", expr: spec.FirstRoundLimit}, } { value, err := sqlJSONExpr(ctx, field.expr, options) if err != nil { @@ -665,6 +676,21 @@ func sqlJSONVectorIndexExpressions(ctx context.Context, spec *plan.VectorIndexSc values = append(values, field.name+"="+value) } } + if len(spec.AlgoExprNames) != len(spec.AlgoExprs) { + return nil, moerr.NewInvalidInput(ctx, "index search scan algorithm expression names do not match expressions") + } + seen := make(map[string]struct{}, len(spec.AlgoExprNames)) + for i, name := range spec.AlgoExprNames { + if _, duplicate := seen[name]; name == "" || duplicate || spec.AlgoExprs[i] == nil { + return nil, moerr.NewInvalidInput(ctx, "index search scan algorithm expression is missing or ambiguous") + } + seen[name] = struct{}{} + value, err := sqlJSONExpr(ctx, spec.AlgoExprs[i], options) + if err != nil { + return nil, err + } + values = append(values, "algo_expr["+strconv.Quote(name)+"]="+value) + } if spec.DistanceRange != nil { for _, bound := range []struct { name string @@ -678,18 +704,18 @@ func sqlJSONVectorIndexExpressions(ctx context.Context, spec *plan.VectorIndexSc continue } if bound.expr == nil { - return nil, moerr.NewInvalidInputf(ctx, "vector index scan %s expression is missing", bound.name) + return nil, moerr.NewInvalidInputf(ctx, "index search scan %s expression is missing", bound.name) } boundType := bound.boundType.String() if boundType == "" { - return nil, moerr.NewInvalidInputf(ctx, "vector index scan %s has unknown bound type %d", bound.name, bound.boundType) + return nil, moerr.NewInvalidInputf(ctx, "index search scan %s has unknown bound type %d", bound.name, bound.boundType) } value, err := sqlJSONExpr(ctx, bound.expr, options) if err != nil { return nil, err } if value == "" { - return nil, moerr.NewInvalidInputf(ctx, "vector index scan %s expression is empty", bound.name) + return nil, moerr.NewInvalidInputf(ctx, "index search scan %s expression is empty", bound.name) } values = append(values, bound.name+"_type="+boundType, bound.name+"="+value) } diff --git a/pkg/sql/plan/explain/sql_json_test.go b/pkg/sql/plan/explain/sql_json_test.go index bd1866f9ae5a5..3bb227113c748 100644 --- a/pkg/sql/plan/explain/sql_json_test.go +++ b/pkg/sql/plan/explain/sql_json_test.go @@ -16,7 +16,9 @@ package explain import ( "context" + "encoding/base64" "encoding/json" + "fmt" "math" "strings" "testing" @@ -577,15 +579,16 @@ func TestBuildSQLJSONPlanPreservesVectorIndexPayload(t *testing.T) { Steps: []int32{0}, Nodes: []*plan.Node{{ NodeId: 0, - NodeType: plan.Node_VECTOR_INDEX_SCAN, + NodeType: plan.Node_INDEX_SEARCH_SCAN, ObjRef: &plan.ObjectRef{SchemaName: "db", ObjName: "items"}, TableDef: &plan.TableDef{Name: "items"}, - VectorIndexScan: &plan.VectorIndexScan{ + IndexSearchScan: &plan.IndexSearchScan{ Index: &plan.IndexDef{IndexName: "idx_embedding"}, DistanceFunction: "l2_distance", - QueryVector: sqlJSONTestInt32(1), + QueryPayload: sqlJSONTestInt32(1), CandidateLimit: sqlJSONTestInt32(10), - FirstRoundLimit: sqlJSONTestInt32(4), + AlgoExprNames: []string{"first_round_limit"}, + AlgoExprs: []*plan.Expr{sqlJSONTestInt32(4)}, PreFilters: []*plan.Expr{sqlJSONTestInt32(2)}, }, }}, @@ -603,12 +606,10 @@ func TestBuildSQLJSONPlanPreservesVectorIndexPayload(t *testing.T) { "index=idx_embedding", "metric=l2_distance", "direction=INTERNAL", - "initial_probe_count=0", - "bucket_expand_step=0", "post_filter_over_fetch=false", - "query_vector=1", + "query_payload=1", "candidate_limit=10", - "first_round_limit=4", + `algo_expr["first_round_limit"]=4`, "pre_filter[0]=2", }, decoded.MatrixOne.Nodes[0].Expressions) } @@ -619,11 +620,11 @@ func TestBuildSQLJSONPlanPreservesLiteralVectorQuery(t *testing.T) { Steps: []int32{0}, Nodes: []*plan.Node{{ NodeId: 0, - NodeType: plan.Node_VECTOR_INDEX_SCAN, + NodeType: plan.Node_INDEX_SEARCH_SCAN, TableDef: &plan.TableDef{Name: "items"}, - VectorIndexScan: &plan.VectorIndexScan{ - Index: &plan.IndexDef{IndexName: "idx_embedding"}, - QueryVector: &plan.Expr{Expr: &plan.Expr_Lit{Lit: &plan.Literal{Value: &plan.Literal_VecVal{VecVal: "[1,2]"}}}}, + IndexSearchScan: &plan.IndexSearchScan{ + Index: &plan.IndexDef{IndexName: "idx_embedding"}, + QueryPayload: &plan.Expr{Expr: &plan.Expr_Lit{Lit: &plan.Literal{Value: &plan.Literal_VecVal{VecVal: "[1,2]"}}}}, }, }}, }) @@ -637,10 +638,8 @@ func TestBuildSQLJSONPlanPreservesLiteralVectorQuery(t *testing.T) { require.Equal(t, []string{ "index=idx_embedding", "direction=INTERNAL", - "initial_probe_count=0", - "bucket_expand_step=0", "post_filter_over_fetch=false", - "query_vector=[1,2]", + "query_payload=[1,2]", }, decoded.MatrixOne.Nodes[0].Expressions) } @@ -732,12 +731,11 @@ func TestBuildSQLJSONPlanPreservesVectorExecutionSettings(t *testing.T) { Steps: []int32{0}, Nodes: []*plan.Node{{ NodeId: 0, - NodeType: plan.Node_VECTOR_INDEX_SCAN, - VectorIndexScan: &plan.VectorIndexScan{ - Index: &plan.IndexDef{IndexName: "idx_embedding"}, + NodeType: plan.Node_INDEX_SEARCH_SCAN, + IndexSearchScan: &plan.IndexSearchScan{ + Index: &plan.IndexDef{IndexName: "idx_embedding", IndexAlgo: "ivfflat"}, Direction: direction, - InitialProbeCount: probe, - BucketExpandStep: expand, + AlgoOptions: []byte(fmt.Sprintf(`{"initial_probe_count":%d,"bucket_expand_step":%d}`, probe, expand)), PostFilterOverFetch: overfetch, }, }}, @@ -755,11 +753,12 @@ func TestBuildSQLJSONPlanPreservesVectorExecutionSettings(t *testing.T) { low := buildExpressions(1, plan.OrderBySpec_ASC, 2, false) high := buildExpressions(16, plan.OrderBySpec_DESC, 8, true) require.NotEqual(t, low, high) - require.Contains(t, low, "initial_probe_count=1") + require.Contains(t, low, "algo_options_base64="+base64.StdEncoding.EncodeToString([]byte(`{"initial_probe_count":1,"bucket_expand_step":2}`))) + require.Contains(t, low, "algo_setting[0]=NProbe: 1") require.Contains(t, low, "direction=ASC") - require.Contains(t, high, "initial_probe_count=16") + require.Contains(t, high, "algo_options_base64="+base64.StdEncoding.EncodeToString([]byte(`{"initial_probe_count":16,"bucket_expand_step":8}`))) + require.Contains(t, high, "algo_setting[0]=NProbe: 16") require.Contains(t, high, "direction=DESC") - require.Contains(t, high, "bucket_expand_step=8") require.Contains(t, high, "post_filter_over_fetch=true") } @@ -770,8 +769,8 @@ func TestBuildSQLJSONPlanPreservesVectorIndexDistanceRange(t *testing.T) { Steps: []int32{0}, Nodes: []*plan.Node{{ NodeId: 0, - NodeType: plan.Node_VECTOR_INDEX_SCAN, - VectorIndexScan: &plan.VectorIndexScan{ + NodeType: plan.Node_INDEX_SEARCH_SCAN, + IndexSearchScan: &plan.IndexSearchScan{ Index: &plan.IndexDef{IndexName: "idx_embedding"}, DistanceRange: &plan.DistRange{ UpperBoundType: plan.BoundType_INCLUSIVE, @@ -797,8 +796,6 @@ func TestBuildSQLJSONPlanPreservesVectorIndexDistanceRange(t *testing.T) { require.Equal(t, []string{ "index=idx_embedding", "direction=INTERNAL", - "initial_probe_count=0", - "bucket_expand_step=0", "post_filter_over_fetch=false", "distance_upper_bound_type=INCLUSIVE", "distance_upper_bound=5", @@ -806,8 +803,6 @@ func TestBuildSQLJSONPlanPreservesVectorIndexDistanceRange(t *testing.T) { require.Equal(t, []string{ "index=idx_embedding", "direction=INTERNAL", - "initial_probe_count=0", - "bucket_expand_step=0", "post_filter_over_fetch=false", "distance_upper_bound_type=INCLUSIVE", "distance_upper_bound=50", @@ -872,10 +867,10 @@ func TestSQLJSONHelpersRejectMalformedInputsAndPreserveStatistics(t *testing.T) Steps: []int32{0}, Nodes: []*plan.Node{{ NodeId: 1, - NodeType: plan.Node_VECTOR_INDEX_SCAN, - VectorIndexScan: &plan.VectorIndexScan{ - Index: &plan.IndexDef{IndexName: "idx_embedding"}, - QueryVector: &plan.Expr{Expr: &plan.Expr_Lit{Lit: &plan.Literal{Value: &plan.Literal_Jsonval{Jsonval: "{}"}}}}, + NodeType: plan.Node_INDEX_SEARCH_SCAN, + IndexSearchScan: &plan.IndexSearchScan{ + Index: &plan.IndexDef{IndexName: "idx_embedding"}, + QueryPayload: &plan.Expr{Expr: &plan.Expr_Lit{Lit: &plan.Literal{Value: &plan.Literal_Jsonval{Jsonval: "{}"}}}}, }, }}, }) @@ -906,3 +901,76 @@ func TestSQLJSONTableAndTargetNameFallbacks(t *testing.T) { require.Equal(t, []*plan.Expr{sqlJSONTestInt32(1), sqlJSONTestInt32(2)}, sqlJSONLimitExpressions(&plan.Node{Limit: sqlJSONTestInt32(1), Offset: sqlJSONTestInt32(2)})) require.Empty(t, sqlJSONLimitExpressions(nil)) } + +func TestBuildSQLJSONPlanPreservesIndexSearchWireMetadata(t *testing.T) { + for _, tc := range []struct { + algo string + payload *plan.Expr + options []byte + operator string + want string + }{ + {"ivfflat", &plan.Expr{Expr: &plan.Expr_Lit{Lit: &plan.Literal{Value: &plan.Literal_VecVal{VecVal: "[1,2]"}}}}, []byte(`{"initial_probe_count":7,"bucket_expand_step":3,"threads_search":2}`), "Vector Index Scan", "algo_setting[0]=NProbe: 7"}, + {"fulltext", planpkg.MakePlan2StringConstExprWithType("needle"), nil, "Fulltext Index Scan", "needle"}, + {"fulltext2", planpkg.MakePlan2StringConstExprWithType("needle"), nil, "Fulltext Index Scan", "needle"}, + {"future_plugin", sqlJSONTestInt32(1), []byte{0, 255, 'x'}, "Vector Index Scan", "query_payload=1"}, + } { + t.Run(tc.algo, func(t *testing.T) { + query := &plan.Query{StmtType: plan.Query_SELECT, Steps: []int32{0}, Nodes: []*plan.Node{{ + NodeType: plan.Node_INDEX_SEARCH_SCAN, + IndexSearchScan: &plan.IndexSearchScan{ + SourceTable: &plan.ObjectRef{SchemaName: "db", ObjName: "items"}, + Index: &plan.IndexDef{IndexName: "idx", IndexAlgo: tc.algo}, + QueryPayload: tc.payload, + AlgoOptions: tc.options, + AlgoExprNames: []string{"first_round_limit"}, + AlgoExprs: []*plan.Expr{sqlJSONTestInt32(4)}, + CandidateLimit: sqlJSONTestInt32(12), + }, + }}} + wire, err := query.Marshal() + require.NoError(t, err) + var remote plan.Query + require.NoError(t, remote.Unmarshal(wire)) + raw, err := BuildSQLJSONPlan(t.Context(), &remote) + require.NoError(t, err) + var decoded sqlJSONPlan + require.NoError(t, json.Unmarshal(raw, &decoded)) + require.Len(t, decoded.MatrixOne.Nodes, 1) + node := decoded.MatrixOne.Nodes[0] + require.Equal(t, tc.operator, node.Operator) + require.Equal(t, "db.items", node.TableName) + require.Contains(t, strings.Join(node.Expressions, "\n"), tc.want) + require.Contains(t, node.Expressions, "candidate_limit=12") + require.Contains(t, node.Expressions, `algo_expr["first_round_limit"]=4`) + if len(tc.options) != 0 { + require.Contains(t, node.Expressions, "algo_options_base64="+base64.StdEncoding.EncodeToString(tc.options)) + } + }) + } +} + +func TestSQLJSONIndexSearchRejectsMalformedAlgorithmMetadata(t *testing.T) { + for _, tc := range []struct { + name string + names []string + exprs []*plan.Expr + }{ + {"missing name", nil, []*plan.Expr{sqlJSONTestInt32(1)}}, + {"missing expression", []string{"limit"}, nil}, + {"empty name", []string{""}, []*plan.Expr{sqlJSONTestInt32(1)}}, + {"nil expression", []string{"limit"}, []*plan.Expr{nil}}, + {"duplicate name", []string{"limit", "limit"}, []*plan.Expr{sqlJSONTestInt32(1), sqlJSONTestInt32(2)}}, + } { + t.Run(tc.name, func(t *testing.T) { + _, err := sqlJSONIndexSearchExpressions(t.Context(), &plan.IndexSearchScan{ + Index: &plan.IndexDef{IndexName: "idx"}, AlgoExprNames: tc.names, AlgoExprs: tc.exprs, + }, &ExplainOptions{Format: EXPLAIN_FORMAT_TEXT}) + require.Error(t, err) + }) + } + _, err := sqlJSONIndexSearchExpressions(t.Context(), &plan.IndexSearchScan{ + Index: &plan.IndexDef{IndexName: "idx", IndexAlgo: "ivfflat"}, AlgoOptions: []byte("{"), + }, &ExplainOptions{Format: EXPLAIN_FORMAT_TEXT}) + require.Error(t, err) +} diff --git a/test/distributed/cases/optimizer/explain_format_json.sql b/test/distributed/cases/optimizer/explain_format_json.sql index 618b950c3109f..75dc50cd9dfbd 100644 --- a/test/distributed/cases/optimizer/explain_format_json.sql +++ b/test/distributed/cases/optimizer/explain_format_json.sql @@ -50,12 +50,12 @@ create table vectors(id bigint primary key, v vecf32(3)); insert into vectors values (1,'[0,0,0]'),(2,'[1,1,1]'),(3,'[2,2,2]'); create index ix using ivfflat on vectors(v) lists=1 op_type 'vector_l2_ops'; set probe_limit=1; --- @regex("(?s)Vector Index Scan.*query_vector=0x0000803F0000803F0000803F",true) +-- @regex("(?s)Vector Index Scan.*query_payload=0x0000803F0000803F0000803F",true) explain format=json select id from vectors order by l2_distance(v,'[1,1,1]') limit 2; --- @regex("(?s)Vector Index Scan.*query_vector=0x000000400000004000000040",true) +-- @regex("(?s)Vector Index Scan.*query_payload=0x000000400000004000000040",true) explain format=json select id from vectors order by l2_distance(v,'[2,2,2]') limit 2; set probe_limit=16; --- @regex("(?s)Vector Index Scan.*initial_probe_count=16",true) +-- @regex("(?s)Vector Index Scan.*algo_setting[^=]*=NProbe: 16([^0-9]|$)",true) explain format=json select id from vectors order by l2_distance(v,'[1,1,1]') limit 2; -- @regex("(?s)Vector Index Scan.*distance_upper_bound_type=EXCLUSIVE.*distance_upper_bound=5[^0-9]",true) explain format=json select id from vectors where l2_distance(v,'[1,1,1]') < 5 order by l2_distance(v,'[1,1,1]') limit 2; @@ -75,9 +75,9 @@ explain format=json select id from vectors where l2_distance(v,'[1,1,1]') > 1 or create table vectors_include(id bigint primary key, v vecf32(3), tag varchar(10)); insert into vectors_include values (1,'[0,0,0]','zero'),(2,'[1,1,1]','one'),(3,'[2,2,2]','two'); create index ix_include using ivfflat on vectors_include(v) lists=1 op_type 'vector_l2_ops' include(tag); --- @regex("(?s)Vector Index Scan.*first_round_limit=2",true) +-- @regex("(?s)Vector Index Scan.*algo_expr[^=]*first_round_limit[^=]*=2([^0-9]|$)",true) explain format=json select id, tag from vectors_include order by l2_distance(v,'[1,1,1]') limit 2 by rank with option 'mode=include'; --- @regex("(?s)Vector Index Scan.*first_round_limit=3",true) +-- @regex("(?s)Vector Index Scan.*algo_expr[^=]*first_round_limit[^=]*=3([^0-9]|$)",true) explain format=json select id, tag from vectors_include order by l2_distance(v,'[1,1,1]') limit 3 by rank with option 'mode=include'; drop table vectors_include; set probe_limit=default; From 717edc4ab68a64cdecb84988e0b14e9be1447bd2 Mon Sep 17 00:00:00 2001 From: bRong Njam Date: Tue, 13 Oct 2026 23:06:52 +0800 Subject: [PATCH 17/20] Clarify prepared qualification evidence and pending publication --- docs/design/sirius-public-campaign.md | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/docs/design/sirius-public-campaign.md b/docs/design/sirius-public-campaign.md index ac2a9d1b8a524..39d232e35c890 100644 --- a/docs/design/sirius-public-campaign.md +++ b/docs/design/sirius-public-campaign.md @@ -89,8 +89,9 @@ Measured captures retain their actual identities; no source or binary is relabeled. Public writer/compiler normal/race cleanup, owning function/compiler, actual GPU bridge and same-process cuVS checks, all 86 tooling tests and full SCA pass. Fail-closed artifact acceptance reports local completion, with CI and -reviewer approval remaining. Generated evidence is preserved separately from -this source change, in the contributor-fork evidence branch described below. +reviewer approval remaining. Generated evidence is prepared separately from +this source change in the local evidence branch described below. Publication +remains pending explicit approval. The deep delivery review rejects CPU budgets beyond Go's signed 32-bit range: the runtime silently ignores those values, so accepting them would misreport @@ -108,10 +109,11 @@ archive passes the tightened acceptance validator; raw runs are not relabeled. ## Optimization evidence preservation -The matched optimization archive and merged #34 qualification are preserved at -contributor-fork commit `56a5c86936fbda06d4c41dbf2ccbcb717e82901d`, -on branch `evidence/sirius-perf-30021-c5b77a2594`. The -[qualification ledger](https://github.com/aunjgr/matrixone/blob/56a5c86936fbda06d4c41dbf2ccbcb717e82901d/docs/design/evidence/sirius-perf-30021/qualification.json) +The matched optimization archive and merged #34 qualification are prepared at +local commit `56a5c86936fbda06d4c41dbf2ccbcb717e82901d`, +on local branch `evidence/sirius-perf-30021-c5b77a2594`. Publication to +`aunjgr/matrixone` is pending explicit authorization; the branch and archive +are not yet available for remote review. The prepared qualification ledger records the original baseline/candidate identities, all captured artifact checksums, and the source/build/provider equivalence through merged `06400cf4`. The bundle retains the 744-run candidate and matched baseline, 264 resource runs, @@ -126,7 +128,18 @@ and new execution still requires an explicit valid profile. Neither historical nor optimized measurements are relabeled as a new benchmark run. CI and reviewer approval remain external gates. -From the review-fixed source checkout, export and verify the immutable bundle: +Current delivery qualification for MO `195457c2fd` and main `8586fa0871` is +prepared separately at local evidence commit +`7a241f4138dcc01872636e2e55f3fadeaf828310`. Owning function/EXPLAIN suites, +string/collation ownership controls under race, all 22 native preparations, +GPU bridge/cancellation/cuVS checks, the source/SDK guarded service build and +full SCA pass. The current EXPLAIN BVT passes 115/115 twice with teardown. +These delivery checks preserve the original performance-capture identities; +they do not claim a new benchmark or passing remote coverage result. The +review's required evidence-access gate remains open until publication. + +After the evidence branch is published, export and verify the immutable bundle +from the review-fixed source checkout: ```sh git fetch https://github.com/aunjgr/matrixone.git \ From 59741ad63ac0c159376bf7c8af92e0ebf65d1c76 Mon Sep 17 00:00:00 2001 From: bRong Njam Date: Wed, 14 Oct 2026 01:49:32 +0800 Subject: [PATCH 18/20] Link the published Sirius qualification evidence --- docs/design/sirius-public-campaign.md | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/docs/design/sirius-public-campaign.md b/docs/design/sirius-public-campaign.md index 39d232e35c890..cbbf7af71c324 100644 --- a/docs/design/sirius-public-campaign.md +++ b/docs/design/sirius-public-campaign.md @@ -89,9 +89,8 @@ Measured captures retain their actual identities; no source or binary is relabeled. Public writer/compiler normal/race cleanup, owning function/compiler, actual GPU bridge and same-process cuVS checks, all 86 tooling tests and full SCA pass. Fail-closed artifact acceptance reports local completion, with CI and -reviewer approval remaining. Generated evidence is prepared separately from -this source change in the local evidence branch described below. Publication -remains pending explicit approval. +reviewer approval remaining. Generated evidence is published separately from +this source change in the contributor-fork evidence branch described below. The deep delivery review rejects CPU budgets beyond Go's signed 32-bit range: the runtime silently ignores those values, so accepting them would misreport @@ -109,11 +108,10 @@ archive passes the tightened acceptance validator; raw runs are not relabeled. ## Optimization evidence preservation -The matched optimization archive and merged #34 qualification are prepared at -local commit `56a5c86936fbda06d4c41dbf2ccbcb717e82901d`, -on local branch `evidence/sirius-perf-30021-c5b77a2594`. Publication to -`aunjgr/matrixone` is pending explicit authorization; the branch and archive -are not yet available for remote review. The prepared qualification ledger +The matched optimization archive and merged #34 qualification are published at +immutable commit `56a5c86936fbda06d4c41dbf2ccbcb717e82901d`, +on [evidence branch `evidence/sirius-perf-30021-c5b77a2594`](https://github.com/aunjgr/matrixone/tree/evidence/sirius-perf-30021-c5b77a2594). +The [qualification ledger](https://github.com/aunjgr/matrixone/blob/7a241f4138dcc01872636e2e55f3fadeaf828310/docs/design/evidence/sirius-perf-30021/qualification.json) records the original baseline/candidate identities, all captured artifact checksums, and the source/build/provider equivalence through merged `06400cf4`. The bundle retains the 744-run candidate and matched baseline, 264 resource runs, @@ -129,17 +127,18 @@ nor optimized measurements are relabeled as a new benchmark run. CI and reviewer approval remain external gates. Current delivery qualification for MO `195457c2fd` and main `8586fa0871` is -prepared separately at local evidence commit +published separately at immutable evidence commit `7a241f4138dcc01872636e2e55f3fadeaf828310`. Owning function/EXPLAIN suites, string/collation ownership controls under race, all 22 native preparations, GPU bridge/cancellation/cuVS checks, the source/SDK guarded service build and full SCA pass. The current EXPLAIN BVT passes 115/115 twice with teardown. These delivery checks preserve the original performance-capture identities; they do not claim a new benchmark or passing remote coverage result. The -review's required evidence-access gate remains open until publication. +remote fetch verifies all three archive hashes and all 133 recorded file +checksums. The downloaded benchmark archive passes acceptance with +`local_acceptance_complete=true`; the evidence-access review thread is resolved. -After the evidence branch is published, export and verify the immutable bundle -from the review-fixed source checkout: +From the source checkout, export and verify the immutable bundle: ```sh git fetch https://github.com/aunjgr/matrixone.git \ From 5dc0b238ed66f49fac83d8b892d9c6ecd9e3ce92 Mon Sep 17 00:00:00 2001 From: bRong Njam Date: Wed, 14 Oct 2026 23:28:00 +0800 Subject: [PATCH 19/20] test(optools): wait for process-pool readiness without phase timers --- optools/ut_embedded_schedule_test.go | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/optools/ut_embedded_schedule_test.go b/optools/ut_embedded_schedule_test.go index 9f1b859eff130..cac0eb0833a5e 100644 --- a/optools/ut_embedded_schedule_test.go +++ b/optools/ut_embedded_schedule_test.go @@ -257,14 +257,16 @@ trap cleanup_pool_driver EXIT ( trap release_pool_children EXIT trap 'exit 143' TERM INT - read -r -t 3 first first_pid <&8 || exit 100 - read -r -t 3 second second_pid <&8 || exit 101 + # The harness deadline bounds the whole operation. Precompilation may still + # be running here, so worker readiness must not start a separate short timer. + read -r first first_pid <&8 || exit 100 + read -r second second_pid <&8 || exit 101 [[ "$first $second" == 'a b' || "$first $second" == 'b a' ]] || exit 102 [[ "$first_pid" != "$second_pid" ]] || exit 103 kill -0 "$first_pid" && kill -0 "$second_pid" || exit 104 [[ -e "$CASE_DIR/active-a" && -e "$CASE_DIR/active-b" && ! -e "$CASE_DIR/executed-c" ]] || exit 105 printf 'release\n' >&10 - read -r -t 3 third third_pid <&8 || exit 106 + read -r third third_pid <&8 || exit 106 [[ "$third" == c && "$third_pid" != "$first_pid" && "$third_pid" != "$second_pid" ]] || exit 107 kill -0 "$third_pid" || exit 108 [[ ! -e "$CASE_DIR/active-a" && -e "$CASE_DIR/active-b" && -e "$CASE_DIR/active-c" ]] || exit 109 @@ -275,7 +277,10 @@ UT_EMBEDDED_PACKAGE_PARALLEL=2 run_embedded_tests "$scope" || status=$? driver_status=0 wait "$pool_driver_pid" || driver_status=$? pool_driver_pid="" -[[ "$status" == 0 && "$driver_status" == 0 ]] || exit 90 +[[ "$status" == 0 && "$driver_status" == 0 ]] || { + printf 'embedded execution status=%s pool driver status=%s\n' "$status" "$driver_status" >&2 + exit 90 +} [[ "$(grep -c '^pool=2 ' "$CASE_DIR/pool-events")" == 3 ]] || exit 91 # Indexed execution checkpoints exclude the outer command and compile events. awk ' From 5187f92147c50412c26255f257e0dff86c2f9389 Mon Sep 17 00:00:00 2001 From: bRong Njam Date: Wed, 14 Oct 2026 23:36:24 +0800 Subject: [PATCH 20/20] docs(sirius): record CI fixture correction and main qualification --- docs/design/sirius-public-campaign.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/design/sirius-public-campaign.md b/docs/design/sirius-public-campaign.md index cbbf7af71c324..d061f56ec1982 100644 --- a/docs/design/sirius-public-campaign.md +++ b/docs/design/sirius-public-campaign.md @@ -554,6 +554,25 @@ their new execution identity; the measured release capture above is unchanged. ## Release validation record +### CI process-pool readiness correction (2026-10-14) + +The standard race UT retry at `778c61e4fc` failed in +`TestEmbeddedPrebuiltExecutionUsesBoundedProcessPool`. Its driver applied a +three-second worker-readiness timeout while precompilation was still running. +A controlled four-second compilation hold reproduces driver status 100; the +corrected fixture passes the same hold using FIFO readiness and the existing +fifteen-second outer harness deadline. All concurrency, identity, checkpoint +ordering and cleanup assertions remain. All 286 tooling cases pass normally +and under race, and the incremental tooling static check passes. + +Main `cd1eaa7a8a` and its subsequent ROLLUP correction `c3c4581595` are merged +without conflicts. Current SQL/function/parser/EXPLAIN/planner/metadata/exporter/ +compiler/frontend owning evidence covers 10,327 roots and 37,727 named passes. +All 22 actual native preparations and GPU data/cancellation/cuVS controls pass +with unchanged native pin `06400cf4` and SDK fingerprint `878c0ec8`. The final +planner group is rerun after the ROLLUP merge; the other owning inputs remain +unchanged. Retained timing/resource captures keep their original identities. + The generated development and release captures are preserved in contributor-fork tag `evidence/sirius-d-4230ce3d14`, at commit `4230ce3d143e77a8b4be83feb47bd26858a8d459`. The