[winml optimize] QNN/NPU and OpenVINO/NPU cannot serialize compiled nodes

[fangyangci]

Summary

Standalone optimization with QNN/NPU and OpenVINO/NPU fails when ORT tries to serialize a graph containing compiled nodes.

Priority: P1, confirmed by the release driver on 2026-09-23.
Bugbash reference: 2026-09-22 / BUG-001.
Status: Observed in the first test round; no fix or post-fix regression is recorded. This issue is based on retained test evidence, not a new test run.

Environment

• Test date: 2026-09-22; findings triaged on 2026-09-23.
• CLI: winml-cli 0.4.0, installed from a local wheel. The exact source commit/build provenance is not recorded in the test evidence.
• Hardware: Snapdragon X Elite X1E80100, Qualcomm Hexagon NPU.
• OS: Windows 11 26H1, build 28000.2956, ARM64.
• Python: 3.11.15, workspace virtual environment.
• EP: QNNExecutionProvider from WinML Catalog, package 2.2480.49.0.
• ONNX Runtime: 1.27.1.202607110137, Windows ML distribution.
• ONNX: 1.18.0; PyTorch: 2.14.0; Transformers: 4.57.6.
• Model: catalog microsoft/resnet-50, image-classification, pixel_values FP32 [1, 3, 224, 224].
• Quantization, where applicable: w8a16, uint8 weights / uint16 activations, 10 calibration samples.

Reproduction

Run in PowerShell with winml-cli 0.4.0 installed. Use a new output directory on each attempt.

$results = '.\repro-qnn-optimize'
New-Item -ItemType Directory -Path $results | Out-Null
uv run winml export -m microsoft/resnet-50 -o $results\resnet.onnx
uv run winml optimize -m $results\resnet.onnx --ep qnn --device npu -o $results\resnet-opt.onnx

Additional reproduction: OpenVINO/NPU (2026-09-23)

The same failure has also been reported when optimizing a freshly exported ResNet-50 ONNX with the OpenVINO EP on NPU:

winml export -m Microsoft/resnet-50 -o ".\resnet.onnx"
winml optimize -m ".\resnet.onnx" --ep openvino --device npu -o ".\resnet-opt.onnx"

The reporter observed the same compiled-node serialization error; adding --disable-ort-graph-optimization allows the command to complete. The OpenVINO report does not include a full traceback, version list, or saved logs, so exact environment details remain to be collected.

Expected result

Optimize an exported model for QNN/NPU or OpenVINO/NPU and save the output ONNX.

Actual result

For QNN/NPU, the standalone optimize command exits with code 1. Reproduced twice with the retained exported model. OpenVINO/NPU was separately reported to fail with the same error.

Unable to serialize model as it contains compiled nodes.
Please disable any execution providers which generate compiled nodes.

Workaround

Adding --disable-ort-graph-optimization permits output:

uv run winml optimize -m $results\resnet.onnx --ep qnn --device npu --disable-ort-graph-optimization -o $results\resnet-opt-no-ort.onnx

This bypasses the failing optimization pass; it does not establish that the default optimize path works.

Scope and evidence

The full catalog build command has a separate successful record. This issue covers the standalone default optimize path with QNN/NPU and the additionally reported OpenVINO/NPU case; it is not a claim that every build path fails. The exported input passed ONNX checker.

The following evidence is retained locally by the reporter; these filenames are an index, not uploaded attachments:

• optimize-qnn.stderr.log
• optimize-qnn-repro.stderr.log
• optimize-qnn-no-ort.stdout.log

Likely shared failure path and CLI follow-up

The ort_graph pipe sets SessionOptions.optimized_model_filepath and creates an InferenceSession with the requested EP. ORT then reports that it cannot serialize a graph containing compiled nodes. The identical error and the working bypass point to the same class of limitation for QNN and OpenVINO; the provider-specific details for the OpenVINO case have not yet been independently traced. Upstream ORT has an OpenVINO report of this error: microsoft/onnxruntime#12804.

For winml-cli, consider detecting this failure and reporting an actionable message that suggests --disable-ort-graph-optimization (or optimizing the raw ONNX without a compiling EP before deployment). Do not silently disable the pass, since that changes the requested optimization behavior.