Repository navigation
nemotron omni: video input through the RADIO tower - #940
Conversation
A request's image tokens were capped at 16,384 on the command line. A model with a 262,144-token window and an engine that encodes an image as its prompt pass reaches it (no rows buffer for the whole request) can take more, up to the window; the per-image budget is still the total over the request's images.
cwschroeder
left a comment
cwschroeder
left a comment
There was a problem hiding this comment.
CPU review of the video delta at 83c810da3f719c8e6239d46c48587e53c0cc9a17, above #938, against main 46a08c18. Two integration issues are reproduced below.
Marker patch and standalone capacity reproducer. The marker fix is commit f49514e2f44081457a016beb3764fcf21d1bd948; its tests cover both literal spellings and empty/identical aliases. All 372 server CPU tests pass. The separate capacity diagnostic produces two passing controls and two failing capacity assertions in both Debug and Fast.
With the marker patch, the four exact No GPU test commands, whole-tree Zig format check and whitespace check pass locally on macOS arm64 with Zig 0.17.0. The full lean gate remains red: engine_api.zig has 602 lines. I reproduced that on the unchanged PR head; it is also the current remote No GPU failure (run 38901869867). The marker patch adds no further lean problems. Desktop, mobile and format checks on the original PR head are green.
The capacity fixture exercises host preparation/planning, with reduced pixel payload and enough context to isolate the rows limit. It initializes no CUDA context, device buffer or tower. This review does not qualify #938's private CUDA underlay or a GPU/checkpoint run.
| const s = try text.tokenString(a, id); | ||
| if (s.len > 0) try out.append(a, s); | ||
| } | ||
| if (v.video) |o| if (o.video_text.len > 0) try out.append(a, o.video_text); // user text that spells it stays text |
There was a problem hiding this comment.
[P2] Shield the actual video token spelling as well as its alias
RADIO's mark_ids contains only image/open/close, so appending <video> leaves <SPECIAL_30> unshielded. A user quoting that spelling in ordinary text encodes video_token: expand then refuses a text-only request with “1 video placeholders for 0 video parts”. Beside one real video it counts two placeholders. The CPU regression fails on this head and passes when markerText also includes text.tokenString(o.video_token). The linked patch preserves ordinary user text and the template's genuine marker; it also covers empty and identical aliases.
| .group_open = x.marks.open, | ||
| .group_close = x.marks.close, | ||
| .markers = &x.mark_ids, | ||
| .video_tokens = o.video_tokens, |
There was a problem hiding this comment.
[P2] Bound the combined video/image rows to the Super-VL buffer
This offer exposes the default 16384 video tokens, while Nemotron Omni initializes its shared rows buffer from the default 13312 image_tokens. Actual video preparation and RADIO planning yield 16384 rows for a square 64-second source. A 52-second video fits alone at 13312 rows, but adding one minimal image gives 13568. The server budgets images and videos separately, and the Super-VL lane passes their combined list to Vision.begin, which returns ImageRows before tower execution. Please enforce a combined capacity bound or size the buffer accordingly; clamping video alone leaves the mixed case. The linked CPU fixture includes fitting controls and reproduces both mismatches with sufficient context. Bolt's piecewise path is separate.
Bolt's config.json is a nemotron_h_omni wrapper around a nemotron_h_puzzle text model: per-layer expert counts, IQ2_XXS experts, NVFP4 Mamba projections, Q8_0 attention. The serve path reads its model type as nemotron_h_puzzle when an omni config wraps one, and the CUDA host opens it with a lane backend over libbolt.so. - families/nemotron/bolt_native.zig: the backend (prompt pieces, shared verify windows for up to 16 streams, rollback, the LiLiCorr drafts, keyed draws, prompt-cache copies, images through the RADIO tower). - zig build bolt -Dnvcc -Dbolt-src: builds libbolt.so beside the server; the backend loads it from there and checks its ABI. - prompt cache rules: a family may set snap_lookahead (Bolt has no head reading ahead: 0). - docs/recipes/nemotron-3.5-bolt.md: build, limits and receipts.
83c810d to
6a1b40f
Compare
The server budgets a request's images (--vision-image-tokens) and videos (--vision-video-tokens) apart, but the Omni lane plans them together into one rows buffer that Vision.init sized from the image budget alone. A 64-second video (16,384 rows) or a 52-second video beside one image (13,568) therefore failed with ImageRows before the tower ran. Limits gains video_tokens; the rows buffer holds image_tokens + video_tokens when the checkpoint has a video projection, and the video offer never exceeds the budget the buffer was sized for. Bolt encodes the images each prompt piece crosses, so it keeps both totals as server limits only. The test pins the property the sizing relies on: a prepared video never holds more visual tokens than its budget. Reported by cwschroeder in the review of ashhart#940.
|
Thanks for the review, @cwschroeder, and for the reproducer: both findings were real and are fixed on this branch ( 1. Marker shielding. I applied your 2. Shared rows capacity. Fixed in The new test pins the property the sizing relies on: a prepared video never holds more visual tokens than the budget it was given (budgets 16,384, 5,000 and 600), and the smaller budgets fill to within one group. I did not run your Checks on this head (macOS is not available to me; Linux, Zig 0.17.0): |
cwschroeder
left a comment
cwschroeder
left a comment
There was a problem hiding this comment.
Both reviewed issues are addressed at 07af207d. The adopted marker commit c790c00a preserves the offered patch: all three changed server files are byte-identical to f49514e2. The current server CPU suite passes 377/377 tests.
I rechecked the Super VL budget fix with an updated CPU diagnostic. It copies the actual Vision.init capacity expression and allocation argument, then exercises the real offer, image/video preparation and row planner. All six cases pass in Debug and Fast, including the 64-second video plus image (16,640 rows within 29,696) and a reserved video budget of 512 that clamps the offer. The old budget logic fails four of these invariants while the two original controls pass.
Source review confirms both limits reach the allocation and the offer cannot exceed the reserved video budget. All four current-head CI checks are green; the earlier whole-tree lean blocker is also resolved.
This approval closes the reviewed marker/default-and-mixed Super VL budget findings. The diagnostic does not initialize the loader, tower, row allocation or CUDA context, and supplies no tower/checkpoint numerical or performance qualification. Bolt's separate piecewise path and its non-default budget combinations were not exercised by this diagnostic.
cwschroeder
left a comment
cwschroeder
left a comment
There was a problem hiding this comment.
Follow-up to my earlier review at 07af207d: a separate Bolt case appears with low, independent image/video budgets. The previously checked Super VL and marker cases remain fixed.
One-line reservation patch and CPU reproduction, candidate cca9f21c. Three video cases return ImageRows on this head; three controls pass. With the patch, all seven tests pass in Debug and Fast, including 72 reservation boundary cases. The fixture calls the actual CLI, offer and RADIO video preparation, and copies the exact allocation expressions and complete imageRuns method; tower/synchronization are inert probes.
Whole-tree format, whitespace and lean comparison pass (517/517, zero new). The actual Bolt.open body also compiles for Linux arm64 without execution. Source review found no issue in the patch. This candidate has not run the complete workflow suite or remote CI, and the CPU fixture does not qualify libbolt, CUDA numerics, full mixed-media/chunk coverage or throughput.
| var problem: []const u8 = ""; | ||
| if (o.vision) |lim| { | ||
| // the rows buffer holds one prompt piece's images; a request's token totals are the server's limit alone | ||
| tower = (try vision.Vision.init(gpa, io, ctx.d, dir, .{ .image_tokens = @min(lim.image_tokens, image_rows), .max_images = lim.max_images }, &problem)) orelse { |
There was a problem hiding this comment.
[P2] Reserve video rows in Bolt's bounded prompt-piece pool
--vision-image-tokens 16 is accepted independently of the default 16,384 video tokens. This initializer reserves only 16 rows, then the following assignments expose the full video budget. A one-second square clip prepares a 256-row group, so imageRuns returns ImageRows before the tower call. With image budget 256, an eight-second clip fails on a later group in its first piece. The server budgets video parts independently (images.zig:247-260), so the image budget does not reject that video beforehand.
The offered patch passes a video reservation capped to the space left below image_rows. Vision.init includes it only for a checkpoint with a temporal projection; image-only allocations stay unchanged, and the pool remains capped at 16,384 rows.
response_format, the guided_* fields and tool_choice "required" or a named function were refused with HTTP 400: the backend offered no masked draws, so the engine advertised no structured output, and the server could not read an omni checkpoint's vocab_size (it sits in llm_config). - bolt_native: verify redraws the rows of a masked window among the tokens their bits allow (rows x words, the grammar's layout), first_masked draws the first token again under the first mask, and held() gives the grammar the drafts the drafter holds. A masked row with fewer allowed tokens than candidates is drawn from the allowed ones only. - libbolt (bolt_mask_rows, C ABI 2): a masked variant of the two-stage GPU top-k, which treats a disallowed token as minus infinity; only the masked rows are recomputed. - server/vocab.zig: the vocabulary size comes from text_config, llm_config or the top level, in that order. - The libbolt binding moves to bolt_api.zig, which keeps bolt_native.zig under 600 lines.
A Nemotron Omni checkpoint with a video projection (video_embedder, video_temporal_patch_size 2) takes videos: frames sampled at 2 a second (1 above 720 rows, at most 128), sized by the processor's video rule, two to a tubelet whose patches sit side by side, and labelled "Frame i sampled at t seconds and frame j sampled at u seconds: " before each pair as the checkpoint's processor writes them. - radio: video grid rule, tubelet weights and projection, sampling and packing (video.zig), its tests. - engine_api: VideoOffer.video_text (the template's own placeholder, swapped for video_token before encoding, since "<video>" tokenizes by its neighbours) and PreparedVideo.labels (words before each group, no trailing time). - server: the offer's max_videos limit replaces a constant; --vision-max-videos; user text that spells the video placeholder stays text.
The server budgets a request's images (--vision-image-tokens) and videos (--vision-video-tokens) apart, but the Omni lane plans them together into one rows buffer that Vision.init sized from the image budget alone. A 64-second video (16,384 rows) or a 52-second video beside one image (13,568) therefore failed with ImageRows before the tower ran. Limits gains video_tokens; the rows buffer holds image_tokens + video_tokens when the checkpoint has a video projection, and the video offer never exceeds the budget the buffer was sized for. Bolt encodes the images each prompt piece crosses, so it keeps both totals as server limits only. The test pins the property the sizing relies on: a prepared video never holds more visual tokens than its budget. Reported by cwschroeder in the review of ashhart#940.
07af207 to
4c5f174
Compare
|
Thank you, @MiaAI-Lab, the shared RADIO video preparation and marker fixes are useful additions to the native media path. We checked 4c5f174. Super VL now reserves both budgets, but bolt_native.zig:458-465 still allocates from image tokens alone before advertising video tokens. With a 16-token image budget, a normal video pair can exceed the actual row pool. We can repair that bounded-pool accounting and add low-independent-budget/mixed-media regressions at landing. After the combined Bolt base, we will qualify the video delta on it and Super VL separately. Please retain paired-frame references plus video drafted/plain, resumed/fresh and concurrent/solo cases; the current descriptions and inherited text checks leave those device gates open. Posted by Ash's AI Assistant Synthena |
|
Thank you, @MiaAI-Lab, for video, @cwschroeder for marker shielding, and @BobClawblaw for the Nemotron foundation. 1.1.0 includes #940 ( Posted by Ash's AI Assistant Synthena |
The server budgets a request's images (--vision-image-tokens) and videos (--vision-video-tokens) apart, but the Omni lane plans them together into one rows buffer that Vision.init sized from the image budget alone. A 64-second video (16,384 rows) or a 52-second video beside one image (13,568) therefore failed with ImageRows before the tower ran. Limits gains video_tokens; the rows buffer holds image_tokens + video_tokens when the checkpoint has a video projection, and the video offer never exceeds the budget the buffer was sized for. Bolt encodes the images each prompt piece crosses, so it keeps both totals as server limits only. The test pins the property the sizing relies on: a prepared video never holds more visual tokens than its budget. Reported by cwschroeder in the review of ashhart#940.
…he reserved video token shielded in user text ashhart#940 samples a video's frames into groups the RADIO tower reads as images, each with its time and the words before it, and budgets video tokens apart from image tokens. It joins our vision offer beside the Qwen, Flash Next and GLM towers: Nemotron Omni's video comes from its own prepare, bounded by the serve flags' video and frame counts, and the template's `<video>` becomes the reserved token before encoding. With it, our checks: the video config, frame rate, frame indices, dimensions and grid are refused before any decode or cast, prompt spans and row products before any device use, and every owner unwinds on a failed allocation. The video types stay in engine_vision.zig with ashhart#940's labels and video text added; its video_offer.zig is not taken. Test names lose their review labels and the video step's filter becomes optional.
3 participants
Refs #937. Stacked on #938: this branch contains that PR's two commits; review the last commit only (
nemotron omni: video input through the RADIO tower). I will rebase when #938 moves.Restacked. This branch now sits on #938's new tip (
a6660d38, which adds structured output); the video commits are unchanged apart from a doc conflict resolved indocs/recipes/nemotron-3.5-bolt.md. CI is green on4c5f1748.An earlier push of this branch failed CI's lean check:
zig/src/core/engine_api.zigwent to 602 lines (limit 600; the check counts new problems per file, so an offsetting fix elsewhere does not count). The video offer types (VideoSource,FrameSink,PreparedVideo,VideoOffer) now live inzig/src/core/video_offer.zig, re-exported byengine_api.zigunder the same names, which is 568 lines; no other file changed meaning.What this changes
Video input for a Nemotron Omni checkpoint that has a video projection (
video_embedder,video_temporal_patch_size2), through the RADIO tower and the existing video server path. Written and tested on Bolt; Super VL has the same tower and I have no Super VL weights to run it on.families/radio: the processor's video grid rule (_compute_target_patches_video, checked against its outputs for eleven sizes), the tubelet weights and projection (a pair of frames' patches side by side throughvideo_embedder), andvideo.zig: frames sampled at 2 a second (1 a second above 720 rows, at most 128 there), spread over the clip past the cap, paired (a lone last frame with itself), with the words the processor writes before each pair ("Frame 1 sampled at 0.00 seconds and frame 2 sampled at 0.50 seconds: ", a newline before later pairs).video_tokenis<SPECIAL_30>, the checkpoint's first spare special token.engine_api/video_offer.zig:VideoOffer.video_text(the text the template writes for a video, swapped forvideo_token's own text before encoding) andPreparedVideo.labels(words before each group, in place of the trailing time). The tokenizer has no<video>token and splits it by its neighbours (<video>,><,>\n), so matching ids would refuse valid requests. User text that spells it stays text, as for image markers.--vision-max-videos(1 to 16; the engine'smax_videosreplaces the constant 2 inimages.zig).Receipt
tools/zig/build_ffmpeg.sh) andzig build video; libjpeg-turbo 3.1.4.1 for JPEG.--alone/--serialwere not run on video requests. On Bolt, measured on the video code as submitted: a clip of a square moving left to right is described as moving from left to right; 4 videos and 50 images in one chat (30,536 prompt tokens, 29 s) answered correctly; a 130 s 720p clip is 36,558 tokens, 36.7 s. The same prompt expands to the same token count as the recipe's earlier implementation (30,536). Text-only requests do not reach this code; the concurrent-equals-solo and drafted-equals-plain results are in nemotron: serve NVIDIA Nemotron 3.5 Bolt (nemotron_h_puzzle) on CUDA #938.zig build test(added: video grids for eleven sizes against the processor's outputs; pairs, a lone last frame, hashes, labels and a too-small token budget through a fake source; the Omni video config; the offer'smax_videos).zig fmt --check build.zig zig: clean.lean_new.pyagainst the base: 517 problems, 517 on the base, 0 new. The tip builds and passes the host suite.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo: shown for text in nemotron: serve NVIDIA Nemotron 3.5 Bolt (nemotron_h_puzzle) on CUDA #938, not run for video requests.