Nvidia Jetson Orin related questions

[furrysalamander]

While attempting to compile for the Jetson Orin, I ran into the following build errors:

In file included from /usr/include/c++/11/bits/stl_algobase.h:64,
                 from /usr/include/c++/11/bits/char_traits.h:39,
                 from /usr/include/c++/11/string:40,
                 from /usr/include/c++/11/stdexcept:39,
                 from /usr/include/c++/11/system_error:41,
                 from /usr/include/c++/11/bits/fs_fwd.h:35,
                 from /usr/include/c++/11/filesystem:44,
                 from /home/mike/source/audio.cpp/include/engine/framework/io/json.h:4,
                 from /home/mike/source/audio.cpp/src/framework/io/json.cpp:1:
/usr/include/c++/11/bits/stl_pair.h: In instantiation of ‘struct std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value>’:
/usr/include/c++/11/ext/aligned_buffer.h:91:28:   required from ‘struct __gnu_cxx::__aligned_buffer<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value> >’
/usr/include/c++/11/bits/hashtable_policy.h:234:43:   required from ‘struct std::__detail::_Hash_node_value_base<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value> >’
/usr/include/c++/11/bits/hashtable_policy.h:268:12:   required from ‘struct std::__detail::_Hash_node_value<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value>, true>’
/usr/include/c++/11/bits/hashtable_policy.h:277:12:   required from ‘struct std::__detail::_Hash_node<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value>, true>’
/usr/include/c++/11/bits/hashtable_policy.h:1812:13:   required from ‘struct std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value>, true> > >’
/usr/include/c++/11/bits/hashtable_policy.h:811:13:   required from ‘struct std::__detail::_Insert_base<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value>, std::allocator<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char> >, std::hash<std::__cxx11::basic_string<char> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >’
/usr/include/c++/11/bits/hashtable_policy.h:1004:12:   required from ‘struct std::__detail::_Insert<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value>, std::allocator<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char> >, std::hash<std::__cxx11::basic_string<char> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, false>’
/usr/include/c++/11/bits/hashtable.h:180:11:   required from ‘class std::_Hashtable<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value>, std::allocator<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char> >, std::hash<std::__cxx11::basic_string<char> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >’
/usr/include/c++/11/bits/unordered_map.h:105:18:   required from ‘class std::unordered_map<std::__cxx11::basic_string<char>, engine::io::json::Value>’
/home/mike/source/audio.cpp/include/engine/framework/io/json.h:62:12:   required from here
/usr/include/c++/11/bits/stl_pair.h:218:11: error: ‘std::pair<_T1, _T2>::second’ has incomplete type
  218 |        _T2 second;                ///< The second member
      |           ^~~~~~
/home/mike/source/audio.cpp/include/engine/framework/io/json.h:14:7: note: forward declaration of ‘class engine::io::json::Value’
   14 | class Value {
      |        ^~~~~

In file included from /usr/include/c++/11/bits/stl_algobase.h:64,
                 from /usr/include/c++/11/bits/char_traits.h:39,
                 from /usr/include/c++/11/string:40,
                 from /usr/include/c++/11/stdexcept:39,
                 from /usr/include/c++/11/system_error:41,
                 from /usr/include/c++/11/bits/fs_fwd.h:35,
                 from /usr/include/c++/11/filesystem:44,
                 from /home/mike/source/audio.cpp/include/engine/framework/io/config.h:3,
                 from /home/mike/source/audio.cpp/src/framework/io/config.cpp:1:
/usr/include/c++/11/bits/stl_pair.h: In instantiation of ‘struct std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value>’:
/usr/include/c++/11/ext/aligned_buffer.h:91:28:   required from ‘struct __gnu_cxx::__aligned_buffer<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value> >’
/usr/include/c++/11/bits/hashtable_policy.h:234:43:   required from ‘struct std::__detail::_Hash_node_value_base<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value> >’
/usr/include/c++/11/bits/hashtable_policy.h:268:12:   required from ‘struct std::__detail::_Hash_node_value<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value>, true>’
/usr/include/c++/11/bits/hashtable_policy.h:277:12:   required from ‘struct std::__detail::_Hash_node<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value>, true>’
/usr/include/c++/11/bits/hashtable_policy.h:1812:13:   required from ‘struct std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value>, true> > >’
/usr/include/c++/11/bits/hashtable_policy.h:811:13:   required from ‘struct std::__detail::_Insert_base<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value>, std::allocator<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char> >, std::hash<std::__cxx11::basic_string<char> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >’
/usr/include/c++/11/bits/hashtable_policy.h:1004:12:   required from ‘struct std::__detail::_Insert<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value>, std::allocator<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char> >, std::hash<std::__cxx11::basic_string<char> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, false>’
/usr/include/c++/11/bits/hashtable.h:180:11:   required from ‘class std::_Hashtable<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value>, std::allocator<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char> >, std::hash<std::__cxx11::basic_string<char> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >’
/usr/include/c++/11/bits/unordered_map.h:105:18:   required from ‘class std::unordered_map<std::__cxx11::basic_string<char>, engine::io::json::Value>’
/home/mike/source/audio.cpp/include/engine/framework/io/json.h:62:12:   required from here
/usr/include/c++/11/bits/stl_pair.h:218:11: error: ‘std::pair<_T1, _T2>::second’ has incomplete type
  218 |        _T2 second;                ///< The second member
      |           ^~~~~~
/home/mike/source/audio.cpp/include/engine/framework/io/json.h:14:7: note: forward declaration of ‘class engine::io::json::Value’
   14 | class Value {
      |        ^~~~~

In file included from /home/mike/source/audio.cpp/src/framework/sampling/noise.cpp:1:
/home/mike/source/audio.cpp/include/engine/framework/sampling/noise.h:8:42: error: ‘size_t’ was not declared in this scope; did you mean ‘std::size_t’?
    8 | std::vector<float> generate_normal_noise(size_t count, uint32_t seed, float scale = 1.0F);
      |                                          ^~~~~~
      |                                          std::size_t
/usr/include/aarch64-linux-gnu/c++/11/bits/c++config.h:280:33: note: ‘std::size_t’ declared here
  280 |   typedef __SIZE_TYPE__         size_t;
      |                                 ^~~~~~
In file included from /home/mike/source/audio.cpp/src/framework/sampling/noise.cpp:1:
/home/mike/source/audio.cpp/include/engine/framework/sampling/noise.h:8:65: error: expected primary-expression before ‘seed’
    8 | std::vector<float> generate_normal_noise(size_t count, uint32_t seed, float scale = 1.0F);
      |                                                                 ^~~~
/home/mike/source/audio.cpp/include/engine/framework/sampling/noise.h:8:71: error: expected primary-expression before ‘float’
    8 | std::vector<float> generate_normal_noise(size_t count, uint32_t seed, float scale = 1.0F);
      |                                                                       ^~~~~

In file included from /usr/include/c++/11/utility:70,
                 from /usr/include/c++/11/array:38,
                 from /home/mike/source/audio.cpp/include/engine/framework/core/module.h:5,
                 from /home/mike/source/audio.cpp/include/engine/framework/assets/tensor_source.h:3,
                 from /home/mike/source/audio.cpp/include/engine/framework/assets/resource_bundle.h:3,
                 from /home/mike/source/audio.cpp/src/framework/assets/resource_bundle.cpp:1:
/usr/include/c++/11/bits/stl_pair.h: In instantiation of ‘struct std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value>’:
/usr/include/c++/11/ext/aligned_buffer.h:91:28:   required from ‘struct __gnu_cxx::__aligned_buffer<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value> >’
/usr/include/c++/11/bits/hashtable_policy.h:234:43:   required from ‘struct std::__detail::_Hash_node_value_base<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value> >’
/usr/include/c++/11/bits/hashtable_policy.h:268:12:   required from ‘struct std::__detail::_Hash_node_value<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value>, true>’
/usr/include/c++/11/bits/hashtable_policy.h:277:12:   required from ‘struct std::__detail::_Hash_node<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value>, true>’
/usr/include/c++/11/bits/hashtable_policy.h:1812:13:   required from ‘struct std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value>, true> > >’
/usr/include/c++/11/bits/hashtable_policy.h:811:13:   required from ‘struct std::__detail::_Insert_base<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value>, std::allocator<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char> >, std::hash<std::__cxx11::basic_string<char> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >’
/usr/include/c++/11/bits/hashtable_policy.h:1004:12:   required from ‘struct std::__detail::_Insert<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value>, std::allocator<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char> >, std::hash<std::__cxx11::basic_string<char> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, false>’
/usr/include/c++/11/bits/hashtable.h:180:11:   required from ‘class std::_Hashtable<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value>, std::allocator<std::pair<const std::__cxx11::basic_string<char>, engine::io::json::Value> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char> >, std::hash<std::__cxx11::basic_string<char> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >’
/usr/include/c++/11/bits/unordered_map.h:105:18:   required from ‘class std::unordered_map<std::__cxx11::basic_string<char>, engine::io::json::Value>’
/home/mike/source/audio.cpp/include/engine/framework/io/json.h:62:12:   required from here
/usr/include/c++/11/bits/stl_pair.h:218:11: error: ‘std::pair<_T1, _T2>::second’ has incomplete type
  218 |        _T2 second;                ///< The second member
      |           ^~~~~~
/home/mike/source/audio.cpp/include/engine/framework/io/json.h:14:7: note: forward declaration of ‘class engine::io::json::Value’
   14 | class Value {
      |        ^~~~~

Plus a few others after this for similar items. I used deepseek v4 flash to work through them, and then I was able to build all the way. I have not yet fully validated the audio from the server. It's getting late and hopefully I'll get around to it sometime this week. I'm attaching the patch file for the changes I made in case it's helpful at all.

0001-fix-incomplete-type-errors.patch

[furrysalamander]

I can use the CLI, but the server tried to allocate 240GB of memory and crashed. Other than that, the output is excellent and I'm very impressed. Can't wait to continue playing around with this, especially once I can have it generate w/o doing the cuda graph warmups between each run!

[0xShug0]

@furrysalamander Thank you for the info and patch! Which model and what server config did you use? I will try to reproduce the server issue.

[0xShug0]

One possible reason is the server config have multiple entries? In this case the server will try to load all models. I just pushed a quick change to make server lazy loading model based on request (however once loaded the model will be kept in memory).

[0xShug0]

@furrysalamander patch merged. I will leave the issue open for now.

[Hi5808]

Followed up on this from the Reddit thread asking for Jetson Orin help — ran the full 40-family catalog (not just one smoke test) on both boards, current main (audio.cpp aec444c, ggml 55eab3c), JetPack 7.2, CUDA 13.2, no patch needed anywhere — the fix in this thread already covers the aarch64 compile issue on both SKUs.

Orin NX 16GB: 40/40 families ran clean, zero failures.

Orin Nano 8GB: 33/40 clean (after one real fix, see below). Initially 31/40 — 9 failures. 5 have a large RAM gap vs. the NX's usage for the same family (higgs_audio_tts, fish_audio, vevo2, glm_tts, miocodec) — expected capacity ceiling on an 8GB board. The other 4 crashed well before reaching anywhere near the memory the same model needed on the NX, so I dug into the actual stderr:

• dots_tts and vibevoice both failed at cuMemAddressReserve trying to reserve a 32GB virtual address range for ggml's CUDA VMM pool allocator (CUDA_POOL_VMM_MAX_SIZE = 1ull << 35 in ggml-cuda.cu) — fails on the Nano's smaller unified-memory address space even though it's never meant to be physically backed. -DGGML_CUDA_NO_VMM=ON (already a real option in ggml's own CMakeLists.txt, just not surfaced through audio.cpp's ENGINE_* wrappers) fixes both — confirmed with a rebuild + re-test. Might be worth exposing as an ENGINE_* option, or defaulting it on for Jetson targets, since it looks like a safe no-op elsewhere.
• index_tts2: separate issue, NvMapMemAllocInternalTagged failed on a small 234.73MB allocation — Tegra NVMAP fragmentation, not VMM-related (confirmed by re-testing with GGML_CUDA_NO_VMM=ON, fails identically). Still open.
• supertonic: a single 8192MB allocation request, which just exceeds the Nano's entire ~7.5GB pool outright — a real capacity ceiling, just an atomic one rather than gradual. Not fixable short of a smaller quantization.

All failures were clean non-zero exits or SIGABRT — no garbled/corrupted output anywhere on either board, at any point.

Also, for anyone else on a bare Nano image: cmake isn't installed by default and there's no root — the official Kitware prebuilt aarch64 tarball works fine as a no-sudo fix. Build needs an explicit -DCMAKE_CUDA_ARCHITECTURES=87-real — audio.cpp's own enable_language(CUDA) call preempts ggml's arch-fallback logic, so leaving it unset silently builds Turing-only kernels instead of the Ampere-tuned ones Orin needs.

Happy to share the full write-up/raw data if useful.

[0xShug0]

@Hi5808 Thank you so much for the comprehensive testing and for sharing the data! A full write-up would be greatly appreciated. You can simply share a link to your own repo or website where the documentation is hosted, or submit a PR to add the documentation directly to audio.cpp if you’d like.

I'll look into the issues you mentioned.

For the build: does it require any modifications to the existing build scripts? I’m wondering whether we need a dedicated build script for Orin.

I’ll keep this issue open as a reference for Orin builds and testing.

[Hi5808]

Thanks! Full write-up (with all raw benchmark CSVs and the scripts I used) is here: https://github.com/Hi5808/research-papers/blob/main/audiocpp-jetson-orin/research_30_audiocpp_jetson_orin_bringup.md

On the build question: no modifications to any existing script or source file were needed — the standard CMake invocation from docs/build/linux.md works as-is on Orin. It's purely a matter of flag values, not a different process:

cmake -S . -B build \
    -DENGINE_ENABLE_CUDA=ON \
    -DCUDAToolkit_ROOT=/usr/local/cuda \
    -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \
    -DCMAKE_CUDA_ARCHITECTURES=87-real \
    -DGGML_CUDA_NO_VMM=ON
cmake --build build -j$(nproc) --target audiocpp_cli

The two Orin-specific bits: 87-real for the architecture (worth calling out explicitly in docs, since the current example list — 86/89/120/61 — never mentions 87, and leaving it unset can silently build Turing-only kernels instead of the Ampere-tuned ones Orin needs, since your top-level enable_language(CUDA) call preempts ggml's own arch-fallback logic before it ever runs), and GGML_CUDA_NO_VMM=ON to sidestep the Nano's 32GB VMM address-reservation failure from my earlier comment.

So I don't think a dedicated build script is needed — a short "Jetson Orin" section in docs/build/linux.md with that snippet would cover it. Happy to put together a small PR for that if useful, rather than a whole separate script.

[0xShug0]

So I don't think a dedicated build script is needed — a short "Jetson Orin" section in docs/build/linux.md with that snippet would cover it. Happy to put together a small PR for that if useful, rather than a whole separate script.

That sounds perfect! Thanks for offering a PR!

[0xShug0]

@Hi5808 I’m wondering if you’d like to check out this branch https://github.com/0xShug0/audio.cpp/tree/preview/vibevoice-optimizations. I specifically optimized VibeVoice for edge devices, and it can now run on an iPhone using less than 3 GB of memory. Demo video is here https://x.com/0xShug0/status/2084826106671124972/video/1.

[Hi5808]

Small correction to my earlier comment — I speculated GGML_CUDA_NO_VMM=ON "looks like a safe no-op elsewhere" on the NX, but that was untested at the time. Actually A/B tested it just now: bs_roformer on the NX, 3 runs each way, same commit — 18.66s mean with VMM on vs. 19.01s with it off, a real ~1.9% slowdown (consistent across all 3 runs each way, not noise).

So it's not free — worth gating on board rather than applying unconditionally. I've updated my own build script to only add the flag when total system RAM is under ~10GB (a reliable Nano-vs-NX proxy). Wanted to flag this before it ends up in docs somewhere as "always safe to enable."

[Hi5808]

One more update — dug into index_tts2's NvMapMemAllocInternalTagged failure a bit further. Rebooted the Nano (clearing whatever kernel-level NVMAP allocator state had built up over the earlier multi-hour benchmark session) and ran index_tts2 as the very first CUDA workload since boot, same VMM-enabled build as before.

It succeeded — real audio output, no error. So this one was boot-persistent NVMAP fragmentation, not a hard per-model limit — a reboot is a real, practical workaround if anyone hits it in production.

Updated tally: Orin Nano 34/40, with only one genuinely unfixable case left (supertonic's single 8GB allocation, which just exceeds the board's entire memory pool — no flag or reboot changes that). Paper and raw data updated at the same link as before.