CI: g6f instances - #709
Open
swahtz wants to merge 10 commits into
Open
Conversation
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
…UDA_MALLOC and setup tests.yml to use this Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
…s that we might not catch in the PR CI that disables async malloc Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
build.sh ctest located the build root with
find build -name CMakeCache.txt -type f -print -quit
which stops at the first match. find walks directories in filesystem
order, so the match was whichever cache the filesystem happened to hand
back first, and every dependency CPM/FetchContent configures leaves a
CMakeCache.txt of its own under _deps: nanovdb-subbuild, benchmark-build,
blosc-build and googletest-build each have one alongside the real cache.
When a dependency won, /src was appended to its path and the gtest job
aborted before running a single test:
Error: No CTestTestfile.cmake found in build/cp312-cp312-linux_x86_64
-Release/_deps/nanovdb-subbuild/src
Prune _deps from the search and accept a candidate only if it actually
contains src/CTestTestfile.cmake, so the result no longer depends on
directory ordering. A tree with both Debug and Release configured now
also selects whichever was built with tests enabled, rather than an
arbitrary one.
The ordering dependence dates back to #418 and went unnoticed because
the real cache usually came back first.
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Moves all three pull-request test workflows onto fractional-GPU (vGPU) EC2 instances and
adds the build option those instances require. Note: A successful run on the g6f instance type run off of this branch can be found here: https://github.com/openvdb/fvdb-core/actions/runs/30422551926
cudaMallocAsyncdepends on GPU unified memory, which is not available on a vGPU slice —it fails at runtime with
CUDA error 801: operation not supported. NanoVDB selects betweencudaMallocAsyncand plaincudaMallocat compile time viaNANOVDB_USE_SYNC_CUDA_MALLOC,so testing on fractional GPUs requires compiling with that macro defined.
New build option
FVDB_USE_SYNC_CUDA_MALLOCCMake option (defaultOFF, so the default build is unchanged)in
src/cmake/get_nanovdb.cmake, which setsNANOVDB_USE_SYNC_CUDA_MALLOCthroughadd_compile_definitions. The macro is consumed in a header by an inline function, so ithas to be visible to every C++/CUDA translation unit — a per-target definition would give
the library, bindings, tests, and benchmarks inconsistent inline bodies.
./build.sh sync_cuda_mallocbuild modifier that forwards the CMake define, documented inthe build-modifier list in
README.md.CI runner configuration
fgpu_amiadded to.github/versions.json, exposed byload-versions.ymlas the newaws-fgpu-amiandaws-fgpu-az-configoutputs, following the existing CPU/GPU pattern offanning the AMI across every subnet for availability.
tests.yml,cu128.yml, andcu130.ymlnow startg6f.2xlarge(8 vCPU, 1/4 L4) testrunners instead of
g6.xlarge(4 vCPU, full L4), and build withsync_cuda_malloc.Restoring async-allocator coverage
Because every PR now tests the synchronous allocator,
nightly-publish.ymlgains a GPU teststage on a full
g6.xlargethat installs the published nightly wheel — built with the defaultasynchronous allocator — and runs the Python unit tests once a night. This is purely additive:
the wheel build matrix is untouched, so no build times or artifacts change. A new
cuda-default-patchoutput pins the test container to the same CUDA patch version as thebuild container rather than hardcoding it.
The nightly stage runs only the Python unit tests, not the gtests, since the gtests already
run on every PR and exercise the same kernels.
Relates to #73, which reports this exact
CUDA error 801failure on an OVX/vGPU VM. The newsync_cuda_mallocmodifier gives affected users a supported way to build for those systems.