refactor: drop the CUDA devel base and fix cold-start failures - #2
Merged
Conversation
Orchestrators pull this image, so the terms it ships under and the support it does not carry both need to be explicit: the server inside is daydream's StreamDiffusion, Apache-2.0, pinned and unmodified, with its LICENSE at /src/LICENSE and no NOTICE to propagate. Model weights are fetched at runtime under their own terms. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing in this image compiles against CUDA. torch carries its runtime in the cu128 wheels, and streamdiffusion.tools.install-tensorrt is pure pip: it installs the tensorrt and nvidia-cudnn-cu12 wheels and reads torch.version.cuda, never nvcc. Upstream's devel base is one commit from Dec 2023, when xformers and stable-fast still built native extensions, and this image installs neither. Without a system CUDA tree the loader needs the wheels' lib dirs, so register them with ldconfig and assert the stack loads at build time rather than on the first stream. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
insightface arrives with the ipadapter extra and is sdist-only, so it compiles a Cython/C++ extension at install time. That compiler, not any CUDA tooling, is what the devel base was actually supplying. Install it for the one step that needs it and purge it in the same layer, so the app keeps every extra it had and the image still ships no toolchain. Measured: 24.9 GB to 14.6 GB. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The volume mounts over /models at start, so /models/engines cannot be created at build time. Without it the engines symlink dangles, and Path.mkdir(exist_ok=True) re-raises on a dangling symlink, so the first stream on a clean checkout dies with FileExistsError: 'engines'. Predates the slim base: any cold start without ./models/engines already on the host hits it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The wheels ship libcudart.so.12 but not the plain libcudart.so, which is a dev symlink the CUDA devel base used to provide. Parts of the TensorRT path dlopen the bare name, and ldconfig does not help because its cache is keyed on SONAMEs. Recreate the symlinks the -dev packages gave us and assert the bare-name dlopen at build time. Verified end to end: engines compile and the pipeline returns frames. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The comments added with the slim base ran four to eight lines where the surrounding ones run one to three. Keep the reasoning that is not recoverable from the code and drop the rest; the detail lives in the commit messages and the PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Say what the example demonstrates and point at the live runner docs, rather than only stating what it is not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
polygraphy globs libcudart.so* across LD_LIBRARY_PATH, /usr/local/cuda/ lib64, /usr/lib and /lib, and only falls back to dlopening the bare "libcudart.so" when that search finds nothing. The devel base satisfied the search with /usr/local/cuda/lib64; slim satisfies it by naming the wheel dirs. Recreating 33 dev symlinks and an ldconfig entry was treating the fallback as if it were the primary path. The assert now constructs polygraphy's Cuda(), which is the consumer that has to work, and catches a python bump moving the hardcoded paths. Verified with a cold engine compile: 3 engines built, frames returned. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Same guarantee in two lines instead of seven, and one layer fewer. The ENV moves above the install, which nothing before it reads. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebuilds the app image on
python:3.11-sliminstead ofnvidia/cuda:12.8.1-cudnn-devel-ubuntu22.04, and fixes two failures that stopped a clean checkout from ever reaching a frame.24.9 GB to 14.6 GB, verified end to end on an RTX 3090: engines compile, the pipeline returns restyled frames, and a warm start reaches frames in 2s from the cached engines.
Why the devel base was not needed
Nothing here compiles against CUDA. torch carries its runtime in the cu128 wheels, and
streamdiffusion.tools.install-tensorrtis pure pip: it installs thetensorrtandnvidia-cudnn-cu12wheels and readstorch.version.cuda, never nvcc. Upstream's devel base is a single commit from Dec 2023, when xformers and stable-fast still built native extensions; this image installs neither.It matches the shipped examples too.
realtime-transcriptiononrunner-app-examplesmain is a GPU app onpython:3.12-slim, getting cuBLAS and cuDNN from wheels with the driver arriving through the container runtime. No shipped example uses a CUDA base.What the devel base was actually providing
Two things, both now handled explicitly and asserted at build time rather than inherited by accident:
g++, forinsightface(via theipadapterextra), which is sdist-only and compiles a Cython extension. Installed for that one step and purged in the same layer, so the app keeps every extra and the image still ships no toolchain.libcudart.so.12but not the plainlibcudart.so, and parts of the TensorRT path dlopen the bare name.ldconfigalone does not help, since its cache is keyed on SONAMEs.Independent bug fix
127fa62is not related to the base change and affectsmaintoday: the volume mounts over/modelsat start, so/models/enginescannot be created at build time. Theenginessymlink then dangles, andPath.mkdir(exist_ok=True)re-raises on a dangling symlink, so the first stream on any cold checkout dies withFileExistsError: 'engines'. It is a separate commit touching only theCMDhunk if you would rather land it on its own.Also
Adds a License and attribution section recording the Apache-2.0 payload and the example-only status, plus OCI image labels. Image size figures in
README.mdandbuild.ymlupdated to the measured 14.6 GB.