Skip to content

backends-qualcomm.md

Latest commit

 

History

History
758 lines (561 loc) · 28.6 KB

backends-qualcomm.md

File metadata and controls

758 lines (561 loc) · 28.6 KB
Raw

Qualcomm AI Engine Backend

In this tutorial we will walk you through the process of getting started to build ExecuTorch for Qualcomm AI Engine Direct and running a model on it.

Qualcomm AI Engine Direct is also referred to as QNN in the source and documentation.

::::{grid} 2 :::{grid-item-card} What you will learn in this tutorial: :class-card: card-prerequisites

What's Qualcomm AI Engine Direct?

Qualcomm AI Engine Direct is designed to provide unified, low-level APIs for AI development.

Developers can interact with various accelerators on Qualcomm SoCs with these set of APIs, including Kryo CPU, Adreno GPU, and Hexagon processors. More details can be found here.

Currently, this ExecuTorch Backend can delegate AI computations to Hexagon processors and Adreno GPU through Qualcomm AI Engine Direct APIs.

Prerequisites (Hardware and Software)

Host OS

The QNN Backend is verified on the following host operating systems:

  • Ubuntu 22.04 LTS (x64)
  • CentOS Stream 9
  • Windows 10 / 11 (x64)
  • Windows 10 / 11 (ARM64) with Qualcomm NPU
  • Windows Subsystem for Linux (WSL) with Ubuntu 22.04

In general, we verify the backend on the same OS versions that the QNN SDK is officially validated against.
The exact supported versions are documented in the QNN SDK.

Windows (x64 / ARM64) Setup

To build on native Windows platforms, the MSVC toolchain must be installed. The required MSVC Build Tools can be installed through Visual Studio Installer.

For installation instructions, refer to the official Microsoft Visual Studio Downloads page.

Windows (WSL) Setup

To install Ubuntu 22.04 on WSL, run the following command in PowerShell or Windows Terminal:

wsl --install -d ubuntu 22.04

This command will install WSL and set up Ubuntu 22.04 as the default Linux distribution.

For more details and troubleshooting, refer to the official Microsoft WSL installation guide: Install WSL | Microsoft Learn.

Hardware:

The QNN backend runs on Qualcomm SoCs (Systems on Chips) across two device families:

  • Android / Linux devices — connected over adb. This example is verified with SM8550 and SM8450.
  • Windows on ARM64 (WoA) devices — This example is verified with SC8380XP (Qualcomm Snapdragon X Elite).

The target SoC must be one of those listed in the QcomChipset enum; see qc_schema.py.

Software:

Qualcomm AI Engine Direct SDK

  • Click the "Get Software" button to download the latest version of the QNN SDK.
  • Although newer versions are available, we have verified and recommend using QNN 2.37.0 for stability.
  • You can download it directly from the following link: QNN 2.37.0

The directory with installed Qualcomm AI Engine Direct SDK looks like:

├── benchmarks
├── bin
├── docs
├── examples
├── include
├── lib
├── LICENSE.pdf
├── NOTICE.txt
├── NOTICE_WINDOWS.txt
├── QNN_NOTICE.txt
├── QNN_README.txt
├── QNN_ReleaseNotes.txt
├── ReleaseNotes.txt
├── ReleaseNotesWindows.txt
├── sdk.yaml
└── share

On Android / Linux devices:

  • Follow ExecuTorch recommended Python version.
  • A compiler to compile AOT parts, e.g., the GCC compiler comes with Ubuntu LTS. g++ version need to be 13 or higher.
  • Android NDK. This example is verified with NDK 26c.
  • (Optional) Target toolchain for linux embedded platform.

On Windows on ARM64 (WoA) devices:

  • Install the AMD64 version of Python to run AOT compilation under x64 emulation. This is required because certain Python modules used in the AOT workflow do not currently provide ARM64 prebuilt wheels.
  • MSVC Build Tools.

Setting up your developer environment

Conventions

$QNN_SDK_ROOT refers to the root of Qualcomm AI Engine Direct SDK, i.e., the directory containing QNN_README.txt.

$ANDROID_NDK_ROOT refers to the root of Android NDK.

$EXECUTORCH_ROOT refers to the root of executorch git repository.

Setup QNN SDK paths and environment variables

For Linux platform:

source $QNN_SDK_ROOT/bin/envsetup.sh

This sets up LD_LIBRARY_PATH and other required variables for the QNN SDK tools and libraries.

For Windows platform:

& "$env:QNN_SDK_ROOT\bin\envsetup.ps1"

Setup PYTHONPATH for ExecuTorch Python APIs

For Linux platform:

export PYTHONPATH=$EXECUTORCH_ROOT/..:$PYTHONPATH

For Windows platform:

$env:PYTHONPATH="$env:EXECUTORCH_ROOT\..;$env:PYTHONPATH"

Build

On Linux platform, an example script for the below building instructions is build.sh. We recommend to use the script because the ExecuTorch build-command can change from time to time. The above script is actively used. It is updated more frequently than this tutorial. An example usage is

cd $EXECUTORCH_ROOT
# Android target
./backends/qualcomm/scripts/build.sh
# (Optional) Linux embedded target
./backends/qualcomm/scripts/build.sh --enable_linux_embedded
# Android target for release build
./backends/qualcomm/scripts/build.sh --release

On Windows platform, use the PowerShell script build.ps1 for the building instructions. Both Windows x64 and ARM64 architectures are supported. Here's the example usage

cd $env:EXECUTORCH_ROOT
# Generate both Windows x64 and ARM64 target libraries
.\backends\qualcomm\scripts\build.ps1 -Release
# Generate only Windows x64 target libraries
.\backends\qualcomm\scripts\build.ps1 -SkipArm64Windows -Release
# Generate only Windows ARM64 target libraries
.\backends\qualcomm\scripts\build.ps1 -SkipX86Windows -Release

Notes

The script supports building both x64 and cross-compiling ARM64 target artifacts on Windows x64 host. After the build completes, the ARM64 libraries and executables can be copied to a Windows on Snapdragon (WoS) device using scp. This allows a .pte generated on Windows x64 host to be executed on WoS device.

Deploying and running on device

AOT compile a model

Refer to this script for the exact flow. We use deeplab-v3-resnet101 as an example in this tutorial. Run below commands to compile on Linux platform:

cd $EXECUTORCH_ROOT

python -m examples.qualcomm.scripts.deeplab_v3 --build_folder build-android --soc_model SM8550 --compile_only --download

For Windows x64 and ARM64 platforms, run the following commands for the AOT compilation:

cd $env:EXECUTORCH_ROOT
python -m examples.qualcomm.scripts.deeplab_v3 --build_folder build-x86_64-windows --soc_model SC8380XP --compile_only --download

Notes

AOT compilation on Windows on ARM64 (WoA) device currently relies on an AMD64 Python environment running under x64 emulation, since some AOT dependencies are not yet distributed as ARM64 prebuilt wheels.

You might see something like below:

Completed stage: Finalizing Graph Sequence (8966 us)
Starting stage: Completion
Completed stage: Completion (1388 us)
[INFO] [Qnn ExecuTorch]: Destroy Qnn context
[INFO] [Qnn ExecuTorch]: Destroy Qnn device
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend

The compiled model is ./deeplab_v3/dlv3_qnn.pte.

Note that the model is compiled for specific backend (e.g., HTP), so you can specify the target backend via --backend gpu or --backend lpai. If not specified, it will be default to HTP.

Test model inference on Linux x64 host with QNN HTP emulator / QNN LPAI emulator

Before deploying a model to a physical device, inference execution can be tested and validated on a Linux x64 host using the HTP / LPAI emulator.

To run the HTP emulator / LPAI emulator, the dynamic linker needs to access QNN libraries and libqnn_executorch_backend.so. We set the below two paths to LD_LIBRARY_PATH environment variable:

  1. $QNN_SDK_ROOT/lib/x86_64-linux-clang/
  2. $EXECUTORCH_ROOT/build-x86/lib/

The first path is for QNN libraries including emulator. It has been configured in the AOT compilation section.

The second path is for libqnn_executorch_backend.so.

So, we can run ./deeplab_v3/dlv3_qnn.pte by:

cd $EXECUTORCH_ROOT
export LD_LIBRARY_PATH=$EXECUTORCH_ROOT/build-x86/lib/:$LD_LIBRARY_PATH
build-x86/examples/qualcomm/executor_runner/qnn_executor_runner --model_path ./deeplab_v3/dlv3_qnn.pte

We should see some outputs like the below. Note that the emulator can take some time to finish.

I 00:00:00.174364 executorch:qnn_executor_runner.cpp:416] Method loaded.
E 00:00:00.179250 executorch:method.cpp:1373] Output 0 is memory planned, or is a constant. Cannot override the existing data pointer.
I 00:00:00.179264 executorch:qnn_executor_runner.cpp:473] ignoring error from set_output_data_ptr(): 0x2
E 00:00:00.183296 executorch:method.cpp:1373] Output 1 is memory planned, or is a constant. Cannot override the existing data pointer.
I 00:00:00.183305 executorch:qnn_executor_runner.cpp:473] ignoring error from set_output_data_ptr(): 0x2
I 00:00:00.183310 executorch:qnn_executor_runner.cpp:479] Inputs prepared.
I 00:00:00.184008 executorch:qnn_executor_runner.cpp:684] Input list not provided. Inputs prepared with default values set.
I 00:01:19.663283 executorch:qnn_executor_runner.cpp:695] Model executed successfully.
I 00:01:19.663299 executorch:qnn_executor_runner.cpp:698] Perform 0 inferences for warming up
I 00:01:53.881349 executorch:qnn_executor_runner.cpp:715] 1 inferences took 34218.046000 ms, avg 34218.046000 ms
I 00:01:53.881426 executorch:qnn_executor_runner.cpp:727] Write etdump to etdump.etdp, Size = 576
[INFO] [Qnn ExecuTorch]: Destroy Qnn context
[INFO] [Qnn ExecuTorch]: Destroy Qnn device
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend

Test model inference on Windows x64 host with QNN HTP emulator / QNN LPAI emulator

Unlike Linux, which set LD_LIBRARY_PATH to access shared libraries, Windows uses the $env:PATH environment variable. To enable runtime loading of qnn_executorch_backend.dll, ensure that it is discoverable by the Windows DLL loader.

This can be achieved by either:

  • Placing qnn_executorch_backend.dll in the same directory as qnn_executor_runner.exe; or
  • Adding the directory containing qnn_executorch_backend.dll to $env:PATH environment variable.

The generated artifacts can be found at:

  • $env:EXECUTORCH_ROOT\build-x86_64-windows\examples\qualcomm\executor_runner\Release\qnn_executor_runner.exe
  • $env:EXECUTORCH_ROOT\build-x86_64-windows\backends\qualcomm\Release\qnn_executorch_backend.dll

To add the directory containing qnn_executorch_backend.dll to the $env:PATH environment variable:

$env:PATH="$env:EXECUTORCH_ROOT\build-x86_64-windows\backends\qualcomm\Release;$env:PATH"

Once configured, qnn_executorch_backend.dll will be accessed by qnn_executor_runner.exe at runtime.

To test the model inference on Windows x64 host with QNN HTP emulator / QNN LPAI emulator:

cd $env:EXECUTORCH_ROOT\build-x86_64-windows\examples\qualcomm\executor_runner\Release
.\qnn_executor_runner.exe --model_path $env:EXECUTORCH_ROOT\deeplab_v3\dlv3_qnn.pte

Run model inference on Android smartphone with Qualcomm SoCs

Step 1. We need to push required QNN libraries to the device.

# make sure you have write-permission on below path.
DEVICE_DIR=/data/local/tmp/executorch_qualcomm_tutorial/
adb shell "mkdir -p ${DEVICE_DIR}"

For HTP

adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnHtp.so ${DEVICE_DIR}
adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnSystem.so ${DEVICE_DIR}
adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnHtpV69Stub.so ${DEVICE_DIR}
adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnHtpV73Stub.so ${DEVICE_DIR}
adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnHtpV75Stub.so ${DEVICE_DIR}
adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnHtpV79Stub.so ${DEVICE_DIR}
adb push ${QNN_SDK_ROOT}/lib/hexagon-v69/unsigned/libQnnHtpV69Skel.so ${DEVICE_DIR}
adb push ${QNN_SDK_ROOT}/lib/hexagon-v73/unsigned/libQnnHtpV73Skel.so ${DEVICE_DIR}
adb push ${QNN_SDK_ROOT}/lib/hexagon-v75/unsigned/libQnnHtpV75Skel.so ${DEVICE_DIR}
adb push ${QNN_SDK_ROOT}/lib/hexagon-v79/unsigned/libQnnHtpV79Skel.so ${DEVICE_DIR}

For GPU

adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnGpu.so ${DEVICE_DIR}

For LPAI

adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnLpai.so ${DEVICE_DIR}
adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnLpaiStub.so ${DEVICE_DIR}
adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnSystem.so ${DEVICE_DIR}
# Make sure the skel lib is signed for LPAI backend.
adb push ${QNN_SDK_ROOT}/lib/lpai-v6/signed/libQnnLpaiSkel.so ${DEVICE_DIR}

Step 2. We also need to indicate dynamic linkers on Android and Hexagon where to find these libraries by setting ADSP_LIBRARY_PATH and LD_LIBRARY_PATH. So, we can run qnn_executor_runner like

adb push ./deeplab_v3/dlv3_qnn.pte ${DEVICE_DIR}
adb push ${EXECUTORCH_ROOT}/build-android/examples/qualcomm/executor_runner/qnn_executor_runner ${DEVICE_DIR}
adb push ${EXECUTORCH_ROOT}/build-android/backends/qualcomm/libqnn_executorch_backend.so ${DEVICE_DIR}
adb shell "cd ${DEVICE_DIR} \
           && export LD_LIBRARY_PATH=${DEVICE_DIR} \
           && export ADSP_LIBRARY_PATH=${DEVICE_DIR} \
           && ./qnn_executor_runner --model_path ./dlv3_qnn.pte"

You should see something like below:

I 00:00:00.257354 executorch:qnn_executor_runner.cpp:213] Method loaded.
I 00:00:00.323502 executorch:qnn_executor_runner.cpp:262] ignoring error from set_output_data_ptr(): 0x2
I 00:00:00.357496 executorch:qnn_executor_runner.cpp:262] ignoring error from set_output_data_ptr(): 0x2
I 00:00:00.357555 executorch:qnn_executor_runner.cpp:265] Inputs prepared.
I 00:00:00.364824 executorch:qnn_executor_runner.cpp:414] Model executed successfully.
I 00:00:00.364875 executorch:qnn_executor_runner.cpp:425] Write etdump to etdump.etdp, Size = 424
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend parameters
[INFO] [Qnn ExecuTorch]: Destroy Qnn context
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend

The model is merely executed. If we want to feed real inputs and get model outputs, we can use

cd $EXECUTORCH_ROOT
# android
python -m examples.qualcomm.scripts.deeplab_v3 --build_folder build-android --soc_model SM8550 --download --device <device_serial>
# (optional) linux embedded
python -m examples.qualcomm.scripts.deeplab_v3 --build_folder build-oe-linux --soc_model SXR1230P --download --device <device_serial> --target aarch64-oe-linux-gcc9.3

The <device_serial> can be found by adb devices command.

After the above command, pre-processed inputs and outputs are put in $EXECUTORCH_ROOT/deeplab_v3 and $EXECUTORCH_ROOT/deeplab_v3/outputs folder.

The command-line arguments are written in utils.py. The model, inputs, and output location are passed to qnn_executorch_runner by --model_path, --input_list_path, and --output_folder_path.

Run model inference on Windows on Snapdragon (WoS) with Qualcomm SoCs

Before running inference on Windows on Snapdragon (WoS) with Qualcomm SoCs, ensure that qnn_executorch_backend.dll and all required QNN libraries are discoverable by the Windows loader. This can be achieved by either:

  • Copying qnn_executorch_backend.dll and the required QNN libraries into the same directory as qnn_executor_runner.exe; or
  • Adding the directories containing these libraries to the $env:PATH environment variable.

The generated artifacts can be found at:

  • $env:EXECUTORCH_ROOT\build-arm64-windows\examples\qualcomm\executor_runner\Release\qnn_executor_runner.exe
  • $env:EXECUTORCH_ROOT\build-arm64-windows\backends\qualcomm\Release\qnn_executorch_backend.dll

Depending on the selected QNN backend, the corresponding QNN libraries can be found under:

# For HTP
$env:QNN_SDK_ROOT\lib\aarch64-windows-msvc\QnnHtp.dll
$env:QNN_SDK_ROOT\lib\aarch64-windows-msvc\QnnSystem.dll
$env:QNN_SDK_ROOT\lib\aarch64-windows-msvc\QnnHtpV69Stub.dll
$env:QNN_SDK_ROOT\lib\aarch64-windows-msvc\QnnHtpV73Stub.dll
$env:QNN_SDK_ROOT\lib\aarch64-windows-msvc\QnnHtpV75Stub.dll
$env:QNN_SDK_ROOT\lib\aarch64-windows-msvc\QnnHtpV79Stub.dll
$env:QNN_SDK_ROOT\lib\hexagon-v69\unsigned\libQnnHtpV69Skel.so
$env:QNN_SDK_ROOT\lib\hexagon-v73\unsigned\libQnnHtpV73Skel.so
$env:QNN_SDK_ROOT\lib\hexagon-v75\unsigned\libQnnHtpV75Skel.so
$env:QNN_SDK_ROOT\lib\hexagon-v79\unsigned\libQnnHtpV79Skel.so
# For GPU
$env:QNN_SDK_ROOT\lib\aarch64-windows-msvc\QnnGpu.dll
# For LPAI
$env:QNN_SDK_ROOT\lib\aarch64-windows-msvc\QnnLpai.dll
$env:QNN_SDK_ROOT\lib\aarch64-windows-msvc\QnnLpaiStub.dll
$env:QNN_SDK_ROOT\lib\aarch64-windows-msvc\QnnSystem.dll
# Make sure the skel lib is signed for LPAI backend.
$env:QNN_SDK_ROOT\lib\lpai-v6\signed\libQnnLpaiSkel.so

Once configured, qnn_executorch_backend.dll and the required QNN libraries can be accessed by qnn_executor_runner.exe at runtime.

To test the model inference on Windows on Snapdragon (WoS) with Qualcomm SoCs:

cd $env:EXECUTORCH_ROOT
.\qnn_executor_runner.exe --model_path .\deeplab_v3\dlv3_qnn.pte

Run Android LlamaDemo with QNN backend

$DEMO_APP refers to the root of the executorch android demo, i.e., the directory containing build.gradle.kts.

Step 1: Rebuild ExecuTorch AAR

# Build the AAR
cd $EXECUTORCH_ROOT
export BUILD_AAR_DIR=$EXECUTORCH_ROOT/aar-out
./scripts/build_android_library.sh

Step 2: Copy AAR to Android Project

cp $EXECUTORCH_ROOT/aar-out/executorch.aar \
   $DEMO_APP/app/libs/executorch.aar

Step 3: Build Android APK

cd $DEMO_APP
./gradlew clean assembleDebug -PuseLocalAar=true

Step 4: Install on Device

adb install -r app/build/outputs/apk/debug/app-debug.apk

Step 5: Push model

adb shell mkdir -p /data/local/tmp/llama
adb push model.pte /data/local/tmp/llama
adb push tokenizer.bin /data/local/tmp/llama

Step 6: Run the Llama Demo

  • Open the App on Android
  • Select QUALCOMM backend
  • Select model.pte Model
  • Select tokenizer.bin Tokenizer
  • Select Model Type
  • Click LOAD MODEL
  • It should show Successfully loaded model.

Verification Steps

Step 1. Verify AAR Contains Your Changes

# Check for debug strings in the AAR
unzip -p $DEMO_APP/app/libs/executorch.aar jni/arm64-v8a/libexecutorch.so | \
  strings | grep "QNN"   # Replace "QNN" with your actual debug string if needed

If found, your changes are in the AAR.

Step 2. Verify APK Contains Correct Libraries

# Check QNN library version in APK
cd $DEMO_APP
unzip -l app/build/outputs/apk/debug/app-debug.apk | grep "libQnnHtp.so"

Expected size for QNN 2.37.0: ~2,465,440 bytes

Step 3. Monitor Logs During Model Loading

adb logcat -c
adb logcat | grep -E "ExecuTorch"

Common Issues and Solutions

Issue 1: Error 18 (InvalidArgument)

  • Cause: Wrong parameter order in Runner constructor or missing QNN config

  • Solution: Check $EXECUTORCH_ROOT/examples/qualcomm/oss_scripts/llama/runner/runner.h for the correct constructor signature.

Issue 2: Error 1 (Internal) with QNN API Version Mismatch

  • Symptoms:

    W [Qnn ExecuTorch]: Qnn API version 2.33.0 is mismatched
E [Qnn ExecuTorch]: Using newer context binary on old SDK
E [Qnn ExecuTorch]: Can't create context from binary. Error 5000

  • Cause: Model compiled with QNN SDK version X but APK uses QNN runtime version Y

  • Solution:

    • Update build.gradle.kts with matching QNN runtime version

    Note: The version numbers below (2.33.0 and 2.37.0) are examples only. Please check for the latest compatible QNN runtime version or match your QNN SDK version to avoid API mismatches.

    Before:

    implementation("com.qualcomm.qti:qnn-runtime:2.33.0")

    After:

    implementation("com.qualcomm.qti:qnn-runtime:2.37.0")
    • Or recompile model with matching QNN SDK version
Issue 3: Native Code Changes Not Applied

  • Symptoms:

    • Debug logs don't appear
    • Behavior doesn't change

  • Cause:

    • Gradle using Maven dependency instead of local AAR

  • Solution:

    • Always build with -PuseLocalAar=true flag
Issue 4: Logs Not Appearing

  • Cause: Wrong logging tag filter

  • Solution: QNN uses "ExecuTorch" tag:

    adb logcat | grep "ExecuTorch"

Supported model list

Please refer to $EXECUTORCH_ROOT/examples/qualcomm/scripts/ and $EXECUTORCH_ROOT/examples/qualcomm/oss_scripts/ to the list of supported models.

Each script demonstrates:

  • Model export (torch.export)
  • Quantization (PTQ/QAT)
  • Lowering and compilation to QNN delegate

Deployment on device or HTP emulator

How to Support a Custom Model in HTP Backend

Step-by-Step Implementation Guide

Please reference the simple example and more complicated examples for reference

Step 1: Prepare Your Model

import torch

# Initialize your custom model
model = YourModelClass().eval()  # Your custom PyTorch model

# Create example inputs (adjust shape as needed)
example_inputs = (torch.randn(1, 3, 224, 224),)  # Example input tensor

Step 2: [Optional] Quantize Your Model

Choose between quantization approaches, post training quantization (PTQ) or quantization aware training (QAT):

from executorch.backends.qualcomm.quantizer.quantizer import QnnQuantizer
from torchao.quantization.pt2e.quantize_pt2e import prepare_pt2e, prepare_qat_pt2e, convert_pt2e

quantizer = QnnQuantizer()
m = torch.export.export(model, example_inputs, strict=True).module()

# PTQ (Post-Training Quantization)
if quantization_type == "ptq":
    prepared_model = prepare_pt2e(m, quantizer)
    # Calibration loop would go here
    prepared_model(*example_inputs)

# QAT (Quantization-Aware Training)
elif quantization_type == "qat":
    prepared_model = prepare_qat_pt2e(m, quantizer)
    # Training loop would go here
    for _ in range(training_steps):
        prepared_model(*example_inputs)

# Convert to quantized model
quantized_model = convert_pt2e(prepared_model)

The QNNQuantizer is configurable, with the default setting being 8a8w. For advanced users, refer to the QnnQuantizer documentation for details.

Supported Quantization Schemes
  • 8a8w (default)
  • 16a16w
  • 16a8w
  • 16a4w
  • 16a4w_block
Customization Options
  • Per-node annotation: Use custom_quant_annotations.
  • Per-module (nn.Module) annotation: Use submodule_qconfig_list.
Additional Features
  • Node exclusion: Discard specific nodes via discard_nodes.
  • Blockwise quantization: Configure block sizes with block_size_map.

For practical examples, see test_qnn_delegate.py.

Step 3: Configure Compile Specs

During this step, you will need to specify the target SoC, data type, and other QNN compiler spec.

from executorch.backends.qualcomm.utils.utils import (
    generate_qnn_executorch_compiler_spec,
    generate_htp_compiler_spec,
    QcomChipset,
    to_edge_transform_and_lower_to_qnn,
)

# HTP Compiler Configuration
backend_options = generate_htp_compiler_spec(
    use_fp16=not quantized,  # False for quantized models
)

# QNN Compiler Spec
compile_spec = generate_qnn_executorch_compiler_spec(
    soc_model=QcomChipset.SM8650,  # Your target SoC
    backend_options=backend_options,
)

Step 4: Lower and Export the Model

# Lower to QNN backend
delegated_program = to_edge_transform_and_lower_to_qnn(
    quantized_model if quantized else model,
    example_inputs,
    compile_spec
)

# Export to ExecuTorch format
executorch_program = delegated_program.to_executorch()

# Save the compiled model
model_name = "custom_model_qnn.pte"
with open(model_name, "wb") as f:
    f.write(executorch_program.buffer)
print(f"Model successfully exported to {model_name}")

Deep Dive

Partitioner API

The QnnPartitioner identifies and groups supported subgraphs for execution on the QNN backend.
It uses QnnOperatorSupport to check node-level compatibility with the Qualcomm backend via QNN SDK APIs.

The partitioner tags supported nodes with a delegation_tag and handles constants, buffers, and mutable states appropriately. Please checkout QNNPartitioner for the latest changes. It mostly supports the following 4 inputs, and only compile spec is required

class QnnPartitioner(Partitioner):
    """
    QnnPartitioner identifies subgraphs that can be lowered to QNN backend, by tagging nodes for delegation,
    and manages special cases such as mutable buffers and consumed constants.
    """

    def __init__(
        self,
        compiler_specs: List[CompileSpec],
        skip_node_id_set: set = None,
        skip_node_op_set: set = None,
        skip_mutable_buffer: bool = False,
    ):
        ...

Quantization

Quantization in the QNN backend supports multiple data bit-widths and training modes (PTQ/QAT). The QnnQuantizer defines quantization configurations and annotations compatible with Qualcomm hardware.

Supported schemes include:

  • 8a8w (default)
  • 16a16w
  • 16a8w
  • 16a4w
  • 16a4w_block

Highlights:

  • QuantDtype enumerates bit-width combinations for activations and weights.
  • ModuleQConfig manages per-layer quantization behavior and observers.
  • QnnQuantizer integrates with PT2E prepare/convert flow to annotate and quantize models.

Supports:

  • Per-channel and per-block quantization

  • Custom quant annotation via custom_quant_annotations

  • Skipping specific nodes or ops

  • Per-module customization via submodule_qconfig_list

For details, see: backends/qualcomm/quantizer/quantizer.py

Operator Support

The full operator support matrix is tracked and frequently updated in the ExecuTorch repository.

It lists:

  • Supported PyTorch ops (aten.*, custom ops)
  • Planned ops
  • Deprecated ops

This matrix directly corresponds to the implementations in: executorch/backends/qualcomm/builders/node_visitors/*.py

Custom Ops Support

The QNN backend supports custom PyTorch operators with the op package mechanism. See the custom op tutorial for the full end-to-end flow. It covers:

  • Defining a custom PyTorch op (single-output and multi-output)
  • Writing and building a QNN op package (XML and Op Implementation)
  • Registering the op package with ExecuTorch via QnnCustomOpPackageBuilder
  • Annotating custom ops for quantization via CustomOpsQuantAnnotator / IOQuantConfig

FAQ

If you encounter any issues while reproducing the tutorial, please file a github issue on ExecuTorch repo and tag use #qcom_aisw tag

Debugging tips