Skip to content

CI: g6f instances - #709

Open
swahtz wants to merge 10 commits into
mainfrom
ci/g6f_instances
Open

CI: g6f instances#709
swahtz wants to merge 10 commits into
mainfrom
ci/g6f_instances

Conversation

@swahtz

@swahtz swahtz commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Moves all three pull-request test workflows onto fractional-GPU (vGPU) EC2 instances and
adds the build option those instances require. Note: A successful run on the g6f instance type run off of this branch can be found here: https://github.com/openvdb/fvdb-core/actions/runs/30422551926

cudaMallocAsync depends on GPU unified memory, which is not available on a vGPU slice —
it fails at runtime with CUDA error 801: operation not supported. NanoVDB selects between
cudaMallocAsync and plain cudaMalloc at compile time via NANOVDB_USE_SYNC_CUDA_MALLOC,
so testing on fractional GPUs requires compiling with that macro defined.

New build option

  • FVDB_USE_SYNC_CUDA_MALLOC CMake option (default OFF, so the default build is unchanged)
    in src/cmake/get_nanovdb.cmake, which sets NANOVDB_USE_SYNC_CUDA_MALLOC through
    add_compile_definitions. The macro is consumed in a header by an inline function, so it
    has to be visible to every C++/CUDA translation unit — a per-target definition would give
    the library, bindings, tests, and benchmarks inconsistent inline bodies.
  • ./build.sh sync_cuda_malloc build modifier that forwards the CMake define, documented in
    the build-modifier list in README.md.

CI runner configuration

  • fgpu_ami added to .github/versions.json, exposed by load-versions.yml as the new
    aws-fgpu-ami and aws-fgpu-az-config outputs, following the existing CPU/GPU pattern of
    fanning the AMI across every subnet for availability.
  • tests.yml, cu128.yml, and cu130.yml now start g6f.2xlarge (8 vCPU, 1/4 L4) test
    runners instead of g6.xlarge (4 vCPU, full L4), and build with sync_cuda_malloc.

Restoring async-allocator coverage

Because every PR now tests the synchronous allocator, nightly-publish.yml gains a GPU test
stage on a full g6.xlarge that installs the published nightly wheel — built with the default
asynchronous allocator — and runs the Python unit tests once a night. This is purely additive:
the wheel build matrix is untouched, so no build times or artifacts change. A new
cuda-default-patch output pins the test container to the same CUDA patch version as the
build container rather than hardcoding it.

The nightly stage runs only the Python unit tests, not the gtests, since the gtests already
run on every PR and exercise the same kernels.

Relates to #73, which reports this exact CUDA error 801 failure on an OVX/vGPU VM. The new
sync_cuda_malloc modifier gives affected users a supported way to build for those systems.

swahtz added 9 commits July 21, 2026 16:55
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
…UDA_MALLOC and setup tests.yml to use this

Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
…s that we might not catch in the PR CI that disables async malloc

Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz swahtz self-assigned this Jul 29, 2026
@swahtz swahtz added the CI Issues related to the Github actions CI/CD. For build issues use CMake/Build label Jul 29, 2026
build.sh ctest located the build root with

    find build -name CMakeCache.txt -type f -print -quit

which stops at the first match. find walks directories in filesystem
order, so the match was whichever cache the filesystem happened to hand
back first, and every dependency CPM/FetchContent configures leaves a
CMakeCache.txt of its own under _deps: nanovdb-subbuild, benchmark-build,
blosc-build and googletest-build each have one alongside the real cache.

When a dependency won, /src was appended to its path and the gtest job
aborted before running a single test:

    Error: No CTestTestfile.cmake found in build/cp312-cp312-linux_x86_64
    -Release/_deps/nanovdb-subbuild/src

Prune _deps from the search and accept a candidate only if it actually
contains src/CTestTestfile.cmake, so the result no longer depends on
directory ordering. A tree with both Debug and Release configured now
also selects whichever was built with tests enabled, rather than an
arbitrary one.

The ordering dependence dates back to #418 and went unnoticed because
the real cache usually came back first.

Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
swahtz marked this pull request as ready for review July 29, 2026 10:54
@swahtz
swahtz requested review from a team as code owners July 29, 2026 10:54
@swahtz
swahtz requested review from blackencino and harrism July 29, 2026 10:54
@swahtz swahtz changed the title CI/g6f instances CI: g6f instances Jul 31, 2026
@swahtz swahtz added this to the v0.6 milestone Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI Issues related to the Github actions CI/CD. For build issues use CMake/Build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant