Skip to content

Refactor: use upstream udf-runner-cpp base for Python VM - #6

Open
tkilias wants to merge 28 commits into
mainfrom
refactoring/use-upstream-udf-runner-base
Open

Refactor: use upstream udf-runner-cpp base for Python VM#6
tkilias wants to merge 28 commits into
mainfrom
refactoring/use-upstream-udf-runner-base

Conversation

@tkilias

@tkilias tkilias commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR replaces the locally vendored UDF client transport/base with the upstream exasol/udf-runner-cpp v1 base Bazel module. It retains only the Python VM and the minimal local adapter needed to instantiate it.

What changed

  • Pins exaudfclient_base through Bzlmod to upstream commit b3a8c204a2555b373c66110d5e8301a374b32a6a, using udf-runner-cpp/v1/base as the module root.
  • Replaces the local common UDF client implementation with a small Python-only executable, argument parser, and VM factory.
  • Keeps the Python VM, SWIG bindings, DataFrame extension, and Python repository rules.
  • Removes the locally vendored transport implementation, loader, protocol binding implementation, streaming VM, benchmark VM, their build configuration, and obsolete diagrams.
  • Preserves the deployed interface: /exaudf/exaudfclient and lang=python.
  • Makes every emulator test image fetch zmqcontainer.proto directly from the pinned upstream commit, then generate its Python protobuf binding with protoc; the local schema fixture is removed.

Implementation plan completed

  1. Consume the upstream v1 base with a pinned git_override, instead of the local base module.
  2. Keep Python-specific sources locally and relink them only to the upstream base headers/dependency collections, preserving linker namespace isolation for ZeroMQ and Protobuf.
  3. Restrict accepted runner language arguments to lang=python and remove local streaming/benchmark VM support.
  4. Preserve normal and static client artifacts, wrapper behavior, the DataFrame extension runtime path, and Python flavor packaging.
  5. Pin each emulator Dockerfile schema download to the same upstream commit as the Bzlmod dependency, removing the duplicate local protobuf schema.

Validation

  • Passed git diff --check.
  • Passed bazel mod graph.
  • Confirmed Bzlmod resolution with bazel mod show_repo exaudfclient_base; it resolves the requested remote, commit, and udf-runner-cpp/v1/base strip prefix.
  • Confirmed the raw schema URL is byte-identical to the schema at the pinned upstream revision.
  • Confirmed there are no remaining local schema fixture references or local streaming/benchmark VM references.

Known limitation

A full local compile was not run: this workstation has Bazel 9.2, while the pinned upstream v1 base targets Bazel 7.2.1. Bazel 9 rejects an upstream legacy built-in rule before analyzing the local Python target. The project container/CI toolchain uses the supported Bazel 7 line.

Original prompt

use https://github.com/exasol/udf-runner-cpp/tree/main/udf-runner-cpp/v1 instead local implementation of udfclient, only preserve the python vm

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 31, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-08-31T10:03:40.749836Z 3e7762a PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3e7762a73c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

RUN mkdir /exasol_emulator
COPY emulator/ /exasol_emulator
COPY /exaudfclient/base/exaudflib/zmqcontainer.proto /exasol_emulator
COPY /exaudfclient/base/zmqcontainer.proto /exasol_emulator

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Restore the complete protocol fixture

The newly selected fixture ends inside exascript_response without a closing brace (and also drops the ping and call response fields used by emulator/exasolution.py). Consequently, the immediately following protoc command exits with Reached end of input in message definition (missing '}'), so every base_test_build_run image updated to copy this file fails to build.

Useful? React with 👍 / 👎.

Comment thread exaudfclient/exa_vm_factory.cc
bazel build --lockfile_mode=off --config no-tty -c dbg --config python --config fast-binary --verbose_failures
working-directory: ./exaudfclient/

tests:

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we probably can remove this for the moment, because we don't have python cpp tests

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this comment was resolved but not fixed. Currently this job executes the bazel test from udf-runner-cpp and we don't have bazel tests for udf-runner-python, so we can remove this job.

"replace_swig_import_helper.py"])

load("//:variables.bzl", "VM_ENABLED_DEFINES")
+load("@rules_cc//cc:defs.bzl", "cc_library")

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this we should fix in udf-runner-cpp

Comment thread exaudfclient/exaudfclient.template.sh Outdated
Co-authored-by: Torsten Kilias <tkilias@users.noreply.github.com>
Comment thread .github/workflows/check_bazel_tests.yml
Comment thread flavors/template-Exasol-all-python-3.10/flavor_base/build_run/Dockerfile Outdated
Comment thread flavors/template-Exasol-all-python-3.12/flavor_base/build_run/Dockerfile Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be removed I guess

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants