Skip to content

feat: add bounded native Sirius CGo bridge - #29203

Draft
aunjgr wants to merge 58 commits into
matrixorigin:mainfrom
aunjgr:feature/28966-native-cgo-bridge
Draft

aunjgr wants to merge 58 commits into
matrixorigin:mainfrom
aunjgr:feature/28966-native-cgo-bridge

Conversation

@aunjgr

@aunjgr aunjgr commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

What type of PR is this?

  • API-change
  • BUG
  • Improvement
  • Documentation
  • Feature
  • Test and CI
  • Code Refactoring

Which issue(s) this PR fixes:

Refs #28966. This intermediate PR does not close the migration issue.

Review notes

This PR remains draft. The CN-facing SiriusInput currently exposes only Push, which acquires native credit after the producer has materialized vectors; design section 5 requires credit before MO allocation/copy. The pre-allocation admission API, publication failure, cancellation, and lease release ownership need design review before this PR leaves draft.

Sirius #23 merged at e2e2f08f; this PR pins that public commit at third_party/sirius. Its tree matches the fully tested Sirius PR head. MO #29254 (Pixi GPU provider) and #29209 (MO-reader-only bindings) are merged. This branch includes authoritative MO main through e5f70bedc4.

At the earlier bridge head 3208d50034, a frozen Sirius mo Pixi build produced and release-packaged the combined MO/cuVS/Sirius binary; verify-package checked its exact submodule SHA and runtime closure. The native bridge integration passed normal and race runs, including the two-stream cuVS → Sirius → cuVS same-process test. Focused CN/compiler Sirius tests, all 16 SDK packaging tests, and the full MO pre-push static/SCA gate passed under Pixi Go 1.26.5. The Sirius tree passed all 3,627 C++ unit cases plus native control, binding, GPU, and two-stream result suites before its content-identical merge. No Docker image or sidecar service was used.

At current head 25caa13128, the MO CUDA sample Makefile and legacy Claude test entry points are byte-for-byte identical to authoritative main. A provenance audit found the optional /usr/local/cuda references in upstream Sirius and inherited MO tooling, so this PR does not change them or its merged Sirius pin. The supported MO/Sirius embedding build still resolves CUDA through the frozen Pixi prefix; the NVIDIA driver remains host-provided. After the latest-main merge, the GPU-toolchain and runtime-image tests, wrapper-tag and native-build contract tests, make config, and the full MO pre-push SCA gate passed. The combined binary was not rebuilt after the main merge; the earlier head's runtime validation is the available evidence.

The license-header scan excludes only the external Sirius submodule, which carries its own license. The bridge README documents submodule initialization, Pixi build, CN configuration, package verification, and troubleshooting.

This PR delivers the opt-in CGo/service bridge only. Embedded public SQL MO-reader/result wiring, exact-numeric parity, all-22 TPC-H validation, default cutover, and sidecar retirement remain separate milestones. Flight stays the default; direct-TAE input is deferred.

What this PR does / why we need it:

Add an opt-in CGo bridge and CN service owner for statically embedded Sirius, pinned as a MatrixOne submodule. Bound native query, input, and result ownership across cancellation and GPU work; verify the SDK against the exact submodule commit and package its runtime dependencies beside the MO binary. Keep Flight as the default backend and reject unavailable embedded execution explicitly.

@aunjgr

aunjgr commented Sep 21, 2026

Copy link
Copy Markdown
Contributor Author

Release SDK refresh completed after Sirius #19 merged: source revision cd8030981f89982b317c868f89c6a9d0435328e1, clean release mode, 72 hashed SDK inputs and 41 relocated runtime libraries. The packaged MO binary starts and the real scalar/empty+constant-NULL/cancellation bridge GPU tests pass from the relocated bundle. The PR remains draft only for the separate combined gpu+sirius/cuVS same-process gate on MO’s pinned GPU toolchain.

This branch had an error being deployed

1 failed deployment
ci — 25caa131 Deployed Sep 29, 2026 by aunjgr via Matrixone Coverage execution / Coverage #56232
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/api-change kind/documentation Improvements or additions to documentation kind/feature kind/test-ci size/XXL Denotes a PR that changes 2000+ lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants