Ship a C++ SDK in the wheel

[shoumikhin]

The wheel ships the runtime, kernels, delegate, thread pool and profiler as separate shared
libraries, but nothing outside Python can use them, because the installed CMake package names
none of them. A C++ application would have to hard-code paths into
the wheel's private layout.

The headers have the same gap. The wheel installs only the subset a custom-operator build needs,
which leaves out extension/module, the entry point the documentation tells C++ callers to use. So
the wheel ships the libraries to run a model and no way to call them.

Name each shipped library as a CMake component, so find_package locates them, and ship the
headers a caller needs. A component is just a name a consumer can ask for, and CMake reports a
missing one while configuring rather than at link time.

find_package(executorch 1.5 REQUIRED COMPONENTS kernels_optimized)
target_link_libraries(my_app PRIVATE executorch::runtime
                                     executorch::kernels_optimized)

component	library it resolves to
executorch::runtime	libexecutorch.so
executorch::kernels_optimized	libexecutorch_kernels_optimized.so
executorch::backend_xnnpack	libexecutorch_backend_xnnpack.so
executorch::threadpool	libexecutorch_threadpool.so
executorch::etdump	libexecutorch_etdump.so

Each component records where the wheel keeps its libraries, so an application built against it
finds them without the caller setting a library search path.

Headers include the module and tensor entry points, the CPU kernel helpers, the allocator and data
loader concrete classes Module's constructors take, the profiler entry points, and the
FlatTensorDataMap and MergedDataMap types plus the .ptd file header a caller writing a .ptd needs.

These headers are part of the published artifact, so the wheel workflows build on a change to one.
Without that a header change reaches a nightly wheel with no wheel build having run on it. They are
listed as whole trees rather than as the directories the copy list names, so growing that list
cannot quietly leave a header outside the filter. Measured over the last 147 commits on main: the
whole trees would newly build a wheel on four of them, the exact directories on one.

CMake 3.28 or newer gets these targets. Older versions do not, because they write the $ORIGIN
marker (the "look next to me" token in a library search path) incorrectly:

3.24.3, 3.27.9   Makefiles double the dollar sign, Ninja drops the name
3.28.4, 3.31.8   both write the token correctly

That would produce a target that runs where it was built and fails once the application is copied
elsewhere, so no target is defined below 3.28. Those versions get plain variables instead:
EXECUTORCH_LIBRARIES with the runtime and every shipped library by path, plus
EXECUTORCH_INCLUDE_DIRS, EXECUTORCH_COMPILE_DEFINITIONS and EXECUTORCH_CXX_STANDARD. All four
are needed, because an imported target carries the definitions and the C++ standard along with the
library and a plain path carries neither. Linking the libraries alone stops at
#error "You need C++17 to compile ExecuTorch".

ET_USE_THREADPOOL is added to EXECUTORCH_COMPILE_DEFINITIONS on the pre-3.28 route when the
thread pool library ships. Without it the runtime header supplies a local inline serial fallback
for parallel_for, so a consumer following the documented recipe linked the thread pool library
and still ran serial code with no diagnostic.

Built the wheel, installed it into a clean environment, and built a C++ application against the
installed wheel alone:

• the application links the runtime, runs a model, and matches eager PyTorch, and still runs after
  being copied away from the wheel.
• asking for a component the wheel does not ship fails while configuring, naming the component.
• a version request is honoured, including ranges.
• shipped headers can be included on their own, and one entry point per shipped component also
  links against the shipped libraries. A small number are exempt because they need something outside
  the package: a Windows shim, a test framework, or a header that says in its own text not to
  include it directly. The exempt list is compiled too, so an entry that starts working is reported
  rather than left in place.
• the thread pool probe compiles with ET_USE_THREADPOOL, on both the modern-CMake route (from the
  runtime target) and the pre-3.28 route (from EXECUTORCH_COMPILE_DEFINITIONS). Without it the
  header supplies a local inline definition and the probe linked identically whether or not the
  library was on the link line, so it could not detect the component being dropped. Measured both
  ways.
• an application's runtime search path is recorded as DT_RUNPATH, not the older DT_RPATH. That
  matters because DT_RPATH is searched ahead of LD_LIBRARY_PATH and is inherited by
  dependencies, so a consumer could not point a locally built or instrumented runtime at their
  application. Verified by shadowing the runtime through LD_LIBRARY_PATH and watching the loader
  pick it up, which DT_RPATH ignores.
• on real CMake 3.24 and 3.27, an application configures, builds and runs through the variables.
  Measured what each one contributes, with the consumer pinned to C++14 so its own standard does not
  hide the package's requirement: linking EXECUTORCH_LIBRARIES alone fails on a missing header,
  adding the include directories and definitions then fails on the C++ standard, and applying
  EXECUTORCH_CXX_STANDARD builds and loads a model. The kernels also need scoped retention there,
  because a registration-only library exports nothing the application references and the linker
  drops it, which showed up as "Missing operator" at run time rather than as a link error. The
  smoke test now runs the same shape automatically when EXECUTORCH_PRE_328_CMAKE points at an
  older cmake binary, so a future change on the fallback path fails a check rather than only
  showing up on the first user with older cmake.
• find_package succeeds when the interpreter on PATH is not the one the wheel was built for. The
  extension's own file name carries its suffix, so asking a different interpreter for it reported a
  complete install as not found.

Ran on Linux x86_64 and aarch64. The macOS wheel keeps the fused extension and ships no separate
libraries, so these checks do not apply there and its smoke test does not run them.

[shoumikhin]

Stack from ghstack (oldest at bottom):

• Name the Python extension after what it contains #21816
• Ship linkable libraries in the macOS wheel #21771
• Ship the OpenVINO delegate as its own library #21770
• Build and publish CUDA wheels #21668
• Ship the CUDA delegate in the wheel #21645
• Ship the quantized kernels as their own library #21642
• -> Ship a C++ SDK in the wheel #21639
• Split the runtime out of the Python bindings extension #21610

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21639

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⏳ 1 Pending, 3 Unrelated Failures

As of commit ed9732b with merge base 46f9696 ():

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• pull / test-models-linux (add_mul, portable, linux.2xlarge) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
• pull / unittest / macos / macos-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Apple / build-demo-ios / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 65

This comment was automatically generated by Dr. CI and updates every 15 minutes.