feat(python-sdk): add managed file queries and Arrow analytics

[dengn]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

Closes #30050
Closes #30051
Closes #30052
Closes #30053

Tracks parent features #30048 and #30049; parent design acceptance remains open.

What this PR does / why we need it:

Feature and requirements

Add an optional synchronous analytics layer for Parquet/CSV/JSONL exploration through native external tables, atomic DataFrame/Arrow uploads through native IPC LOAD, and bounded Arrow RecordBatch results through a real unbuffered PyMySQL cursor. Share checked scalar mapping, strict transaction settings and explicit resource ownership; PyArrow remains an optional analytics dependency.

Sub-feature	Design and acceptance cases
#30050 File inference/multi-file relations	Design
#30051 Ownership, import and recovery	Design
#30052 Atomic DataFrame/Arrow upload	Design
#30053 Bounded Arrow results	Design

Current behavior and review fixes

• Bounded file expansion/inference, union_by_name, Hive partitions, remote filesystem/path mappings and context-managed relation cleanup. Reject CSV field loss, unsafe promotions and timestamp truncation.
• Create/append with one native Arrow LOAD transaction and bounded shared staging. Keep UTC, strict SQL and TN PK checks through COMMIT. Unknown acknowledgements are reported without automatic retries; acknowledged commits followed by cleanup failures retain committed=True.
• Reserve external-table UUIDs before CREATE/COMMIT. Cleanup stays bound to the original engine; definite collisions preserve the existing object. Failed cleanup retains exact pending names and a retryable scoped owner on the primary exception. No prefix reaper or CREATE retry.
• TIME clock conversion now constructs a microsecond source array before safely casting to seconds/milliseconds/microseconds/nanoseconds. Default files.query() preserves inferred Parquet time32(ms); fractional narrowing fails and releases the connection. Unspecified SQL TIME still defaults to duration(us) for negative/>24h values.
• The runnable example and RSS benchmark generate unique table names and only DROP after acknowledged successful upload. A forced collision preserves the foreign table and row 42. Workers receive the parent's exact table name. The example uses an owned temporary Parquet file.

Fresh validation (2026-10-18)

Python 3.12 / PyArrow 26.0.0 / SQLAlchemy 1.4.54, 2.0.55 and 2.1.5:

• Full SDK offline + analytics collection on each family: 1071 passed + 86 subtests, with 187 native/environment-dependent cases explicitly skipped in the final matrix. These skips are not online acceptance.
• TIME counterexamples: 5 failed, 1 passed before the fix; all six pass after it. Actual example/benchmark entry points reproduce unsafe cleanup on the old code; all 8 ownership/collision/success/later-failure cases pass after correction.
• 37 targeted cases passed with the isolated native MatrixOne service: four TIME output units, default Parquet milliseconds/NULLs, rejected fractional narrowing with pool recovery, eight ownership models, eight network fault/control cases, six TIME conversion models and eight CLI ownership cases. Native server built from aa4b6ed. Packet tests observe real CREATE/COMMIT acknowledgements and preserve independent catalog/control evidence.
• Black, isort, fatal Flake8 and diff checks pass.

Earlier source-build qualification (PyArrow 18/21, 192 analytics cases, dual CN, local moto S3, short soak, constrained memory and six SIGKILL/restart barriers) remains historical evidence in the retained record. Those campaigns were not rerun for this follow-up. Test commands and prerequisites.

CI for this commit: https://github.com/matrixorigin/matrixone/actions/runs/39125091928 (passed: six SDK offline cells and CI Required). Kernel/online jobs follow SDK-only scope. Design re-review, feature acceptance and formal Request Changes remain pending; no approval or merge is claimed.

Limits

Uses ordinary external catalog tables, not private SQL temporary tables or new SQL table functions. Arrow LOAD needs a shared client/CN path; there is no Flight or zero-copy output. Batch size bounds rows, not cell bytes. Network timeouts are inactivity bounds, not total deadlines or server KILL. Source preflight is not an atomic snapshot; crashes/unknown commits require scoped recovery. Local source builds, same-host dual CN and local S3 do not certify a released binary, cloud deployment or multi-day endurance. Integrate #30074's complete connection policy before landing this draft.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[XuPeng-SH]

REQUEST_CHANGES — design-phase gate failed on an unclosed external-table ownership transition. Reviewed all four versioned designs, their acceptance matrices, the change map and causal construction/cleanup paths. This is a new cross-owner persistent/streaming feature; design review is mandatory, and all four documents currently mark acceptance pending. The inline counterexample must be closed in the proposal before implementation approval. A narrow fault-injection probe through the actual manager/cleanup code confirms that a modeled committed CREATE with a lost acknowledgement leaves the generated table unregistered, whereas the normal control issues CREATE and DROP. This is not a new native lost-ack run: schema and commit-boundary dependencies are injected. Existing native upload COMMIT faults and the pre-second-CREATE rejection test do not cover this construction boundary. Preserve owned names before ambiguous side effects and retain scoped recovery/cleanup evidence without blind retry or a new reaper. Full downstream implementation/test review is gated by this design failure; no claim that all 21 files have passed implementation review. No source edits, subagents or merge.

[XuPeng-SH]

[P2] Register table ownership before CREATE can have an uncertain outcome

The table is added only after engine.begin().exit returns. If CREATE EXTERNAL TABLE commits but its COMMIT acknowledgement is lost, this append is never reached; construction enters relation.close() with no record of this UUID. The caller receives only the failure and never obtains the relation, so a normal network error can leave an ordinary catalog table with no cleanup/recovery owner. The SIGKILL limitation does not cover this ordinary exception path.

A fault-injection probe through the real read_files/FileRelation path produces CREATE+DROP normally and CREATE with no DROP after the modeled post-commit acknowledgement failure. Existing packet fault tests cover upload_arrow, while partial_relation_creation_cleanup fails before the second CREATE. Reserve the uniquely owned name before ambiguous side effects, attempt cleanup on a fresh usable connection, and preserve the scoped name/cleanup evidence if outcome cannot be confirmed. Amend the lifecycle design and extend the native commit-ack control to this entry point.

[XuPeng-SH]

mo-bot review · mo-bot-xupeng · done

Commit: c7159569c1607ad32b1810d8e79ff9debf5e344a

Review completed. Read the assessment.

[XuPeng-SH]

mo-bot review · mo-bot-xupeng · review

Commit: c7159569c1607ad32b1810d8e79ff9debf5e344a

Assessment: request_changes

Inspected the complete 22-file delta and four designs. The construction-ownership fix passes focused models, but TIME conversion corrupts supported values and the runnable example can delete a pre-existing table.

Model: gpt-6-astra / xhigh

Session: 01a14d34-4e51-79a3-a62d-6c9c16279dc8 · slot 0 · generation 0

• P2 clients/python/matrixone/analytics.py:374: Read a Parquet time32('ms') column through files.query(), or request time32('s'/'ms') or time64('ns') in analytics.query's result schema.
  Impact: _time_microseconds produces microseconds, but pa.array interprets those integers in the requested unit. Thus 00:00:02 becomes 2,000,000 milliseconds instead of 2,000. Fractional values also bypass checked narrowing. Construct a microsecond-typed source array before safely casting to the requested type.
  Evidence: Precision probes reproduce three unit mismatches and accepted lossy narrowing; the microsecond control passes. test_review_default_time.py reproduces the default Parquet-query failure through real SDK/Arrow code with an injected TIME transport. Evidence: default-time.log and checks/1792312493030524337-3.json.

• P2 clients/python/examples/example_35_arrow_analytics.py:46: Run the example against a database containing analytics_example. Its default create=True upload rejects the existing table.
  Impact: The finally block nevertheless drops analytics_example, deleting an object the example never created. Use a unique owned name and distinguish collision rejection from owned creation before cleanup. benchmark_analytics.py:128 repeats this fixed-name cleanup pattern.
  Evidence: The actual example.main, exercised with a SQLite-backed CREATE/DROP boundary, deletes a pre-existing table containing row 42 after CREATE fails. The owned-table cleanup control passes. Evidence: example-final.log and checks/17923124224484690-5.json; the authoritative receipt mapping is retained in review-evidence.json.

Validation:

• Preflight passed. Python 3.13.5, PyArrow 21.0.0 and SQLAlchemy 2.0.55: 22 focused ownership/inference/type tests passed without skips, including eight construction/collision cases. Evidence: ownership.log and checks/1792312354401521984-3.json.

• The prior lost-CREATE-ack concern is addressed by reservation-before-CREATE, collision exclusion and retryable scoped cleanup in source and focused models.

• Retained regression probes include neighboring controls. Initial example harness failures are distinguished from product failures. Additional upload-cast observations were not promoted to findings because their contract is less explicit.

• Commands, source fingerprints, terminal receipts, JUnit outcomes and logs were reconciled in review-evidence.json under /home/xupeng/.local/state/mo-bot/jobs/fe876c8964b0f3af2635f952/attempt-1. Source remains unchanged.

• Frozen evidence records six successful exact-head SDK offline CI cells. Online/kernel checks were skipped; local_validation contains no reusable worker gates.

Limitations:

• Local probes use injected database boundaries with real SDK/Arrow execution, or SQLite for example ownership. Native MatrixOne, S3, multi-CN, soak and crash campaigns were not rerun.

• Historical author reports reference unavailable raw XML/native artifacts; their native and performance outcomes were not independently reconciled to this candidate.

• The four design documents still mark acceptance pending. This assessment does not grant design acceptance or full native integration approval.

[XuPeng-SH]

mo-bot review · mo-bot-xupeng · done

Commit: 84792be374c46591eb249986ac33c05ebb6f13b7

Review completed. Read the assessment.

[XuPeng-SH]

mo-bot review · mo-bot-xupeng · review

Commit: 84792be374c46591eb249986ac33c05ebb6f13b7

Assessment: request_changes

The prior TIME-conversion and example-cleanup counterexamples now pass focused checks. However, mandatory design acceptance remains open; full implementation approval is gated.

Model: gpt-6-astra / xhigh

Session: 01a14d34-4e51-79a3-a62d-6c9c16279dc8 · slot 0 · generation 0

• P2 clients/python/docs/design/analytics-file-lifecycle.md:71: This new analytics feature adds an 887-line production module and crosses persistent catalog, shared staging, transaction and streaming ownership boundaries.
  Impact: The repository requires a distinct design-first approval before implementation acceptance. Record a traceable approval decision for the exact four design revisions, then complete downstream implementation review; passing tests cannot substitute for this gate.
  Evidence: feature-design-review.md sections 1–2 and 4 explicitly require this gate. All four designs state that design review and merge acceptance are pending: lifecycle:71, inference:68, upload:63 and stream:57. The current PR description also states that parent design acceptance remains open; supplied reviews contain no subsequent approval.

Validation:

• 15 focused tests passed, with seven native cases deliberately deselected: six TIME conversion cases, eight CLI ownership cases and the retained default-Parquet TIME counterexample. fixes.log/fixes.xml match checks/1792326171177093328-3.json.

• Reconciled retained receipts, logs and JUnit results for 22 ownership/inference/type models, verifying relevant source and test inputs remain unchanged. Audit receipt: checks/1792326176819707215-4.json.

• Preflight and final evidence audit passed; source remained unchanged. Four designs and the complete follow-up delta were reviewed. Frozen evidence records six successful offline SDK CI cells.

• Scope and evidence mapping: /home/xupeng/.local/state/mo-bot/jobs/9ba21d667bd0ade7f7afc99f/attempt-1/review-record.json.

Limitations:

• Full downstream implementation, broader test architecture, performance and security review remain pending under the mandatory design gate.

• Focused checks use real Arrow with injected database boundaries or SQLite ownership operations; they do not establish native MatrixOne acceptance.

• Current native, S3, multi-CN, soak and crash claims were not independently reconciled. local_validation is empty, historical raw XML artifacts were unavailable, and online CI was skipped.

[aptend]

Reviewed exact head 84792be374c46591eb249986ac33c05ebb6f13b7 against base 8ed73ff81531db6ec72cdb4eb791b7889d3b44b6. REQUEST_CHANGES for one independently reproduced persistent-ownership defect.

[P2] Freeze the schema as part of external-table cleanup ownership — clients/python/matrixone/analytics.py:516.

FileRelation retains the engine and UUID table name, but its DROP is unqualified. A pooled connection's current database can change independently of its engine URL. With a single-connection pool:

c = Client(..., database="a", pool_size=1, max_overflow=0)
r = c.analytics.read_parquet(path)
# n is the generated mo_py_files_<uuid> table in a.
# An independent connection may create b.n containing sentinel row 42.
c.execute("USE b")
r.close()

The close returns successfully and clears _tables / sets _closed=True, yet independent catalog checks show a.n still exists. If b.n exists, that unrelated table is deleted. Without the b.n control, DROP IF EXISTS still falsely retires ownership while leaking a.n. No network fault, crash, UUID collision during CREATE, or client reconnect is required.

This reproduced against a freshly built native CN from this exact head. A second public-API run repeated three pairs: all three no-USE controls dropped only a.n and preserved b.n; all three USE cases leaked a.n, deleted b.n and reported closed/no pending ownership.

Capture the actual schema on the creating connection before ambiguous CREATE/COMMIT, retain schema+table as the object identity, and qualify owned relation references/cleanup with it. Retaining the original engine alone is insufficient; reading only its configured URL is also insufficient if that session was already switched. Add USE/default-schema reuse and same-name foreign-table controls alongside existing lost-ack/reconnect cases.

Review scope and evidence:

• Complete 24-file implementation/test/doc review, all 3 reviews, 3 discussion comments, inline comment and unresolved/outdated thread; also the latest 7-file correction and both parent contracts.
• Independently reviewed all four exact-head designs before implementation validation: inference fa33bb723e8ba3ad5fd1478f906aca0d0d95b77b, lifecycle c1e14cb0f045c7ac301491cafe0240f56a5041cb, upload d09b12635bc608aeaf36e203996f896eb80221a1, stream 6bf0728c639db06bee783dde1b572bb383786d91. The ownership identity needs the concrete correction above. This verdict is not based on pending approval labels/text.
• Prior construction reservation, TIME-rescaling/precision and CLI collision findings pass fresh applicable tests.
• Python 3.13.5 / Arrow 26.0.0 / SQLAlchemy 2.0.55 / PyMySQL 1.2.3: full SDK offline + analytics collection 1071 passed + 86 subtests, 187 explicit environment skips.
• Fresh native single-CN analytics run: 204 passed, 3 failed, 16 skipped. Lost CREATE/COMMIT acknowledgements, collision preservation, strict import/upload rollback, stream timeout/pool recovery and TIME cases ran. The three permission-suite failures are server rejection of tenant SQL-file paths (only sys moadmin allowed); no permission bypass attempted, and these are not counted as passing. Multi-CN/S3 were skipped. Test endpoint literals were remapped only to this harness-owned CN; no existing local service was used.
• Exact-head CI has six successful offline cells; online/kernel jobs were skipped. Historical S3/dual-CN/soak/SIGKILL evidence was inspected but not relabeled fresh. No full native-acceptance, performance or cloud certification.

Unhappy-path audit	Closure
Q1 ownership	Staging and stream owners are explicit; persistent relation owner is missing its schema component.
Q2 cleanup/failure	Lost-ack/rollback/cleanup-state controls checked; normal close can delete the wrong object and discard recovery evidence.
Q3 reuse/restart/growth	Real pooled-session database reuse reproduces this finding; early-close/FD/concurrency controls exercised; crash/soak campaigns not rerun.

No production source changes or pushes. All counterexample databases were removed in finally and the isolated native fixtures closed.

[aptend]

[P2] Qualify DROP with the schema captured when this object was created. Keeping the original engine does not pin a pooled session's current database. Reproduced on this exact head with Client(database="a", pool_size=1, max_overflow=0): create a file relation, execute the public client.execute("USE b"), then relation.close(). This unqualified DROP runs in b, while the UUID-owned external table remains in a. Close nevertheless removes its reservation and reports closed, so even retrying close cannot reclaim it.

With an independently created same-name b table containing row 42, close deletes that unrelated table as well. Three fresh native pairs reproduce this: no-USE controls preserve b and delete a; all USE cases preserve a and delete b. Capture actual schema+table before CREATE/COMMIT uncertainty and use that frozen qualified identity for relation SQL and DROP; add default-database reuse and foreign-object controls. Full reproduction/evidence is in the review body.