NXP backend: remove #22179 workaround for QAT channels-last segfault - #2
Closed
JakeStevens wants to merge 190 commits into
Closed
JakeStevens wants to merge 190 commits into
JakeStevens wants to merge 190 commits into
Conversation
## Summary `constant_prop_pass` folds every `call_function` node whose arguments are all constants. Ops that draw from the RNG take only sizes as arguments, so they qualify: a model returning `x + torch.rand(4)` came out of the pass with the draw frozen into `_prop_tensor_constant0`, and returned the same value on every call. @JakeStevens spotted this while reviewing pytorch#22391 (repro there). The pass already runs in the Qualcomm and Samsung backends and in `quant_fusion_pass`, so the fix stands on its own. Skip nodes that `torch.fx.Node.is_impure()` reports as impure. That covers ops tagged `nondeterministic_seeded` (`rand`, `randn`, `bernoulli`, `dropout`, ...), mutable schemas and side-effectful functions, and is the same check `eliminate_dead_code` uses to decide what it must keep. ## Test plan New `test_constant_prop_pass_skips_nondeterministic_ops` in `exir/tests/test_passes.py`: after the pass one `aten.rand` node remains, no constant was added, and two calls give different outputs. It fails on main with `0 != 1`. `python -m unittest executorch.exir.tests.test_passes -k constant_prop`: 15 tests pass (torch 2.13.0, macOS arm64).
Differential Revision: D109546129 Pull Request resolved: pytorch#20496
Updated the workflow to label external PRs with a community label and refined the exclusion list for authors and bots. We already have a label - *community: contribution*. Just updating the workflow which adds community contributed PRs to the Project Board to have this label.
…h#22419) ### The problem Attention on rank-3 tensors exports without complaint and then fails when you run the model: ``` MLX execute failed: [scaled_dot_product_attention] input with shape (2,16,64) expected to be rank 4 ``` PyTorch accepts rank 2, 3, 4 and 5 here. The fused kernel takes rank 4 only, and has other requirements besides. The handler matched the operator by name and checked none of them, so it claimed calls the kernel cannot run. ### Adapting the shapes that can be adapted `ExpandDimsNode` → `SdpaNode` → `SqueezeNode`, as suggested in review, so rank 3 keeps the fused kernel. Rank 2 needs it too and takes two added dimensions. The dimensions go at the front. For a rank-3 input the first dimension is already the head one, so inserting in the middle moves it into the batch slot, which misaligns masks and grouped heads. Admitting a lower rank also reaches code that was only ever given rank 4. The grouped key and value unwrap looks for a repeat on dimension 1, which is the head dimension at rank 4 and the key sequence below it, so at rank 3 it would absorb a repeat carrying real keys. Without a mask the result still matches, because a duplicated key and its value give the same weighted sum, which is what makes this easy to miss. With a causal mask it is off by 5.1. Both unwraps now run only at rank 4. ### Three shapes left to decompose Decomposed calls still run on this backend, which the tests assert rather than assume. | Call | Today | Here | | --- | --- | --- | | Rank 5 and above | fails at execute | decomposes, 3.6e-07 | | Unequal batch at rank 4 | fails at execute | decomposes, 2.4e-07 | | Causal at rank 2 or 3, query shorter than key | not reachable | decomposes, 4.2e-07 | The causal one needs a word. Torch anchors a causal mask at the top left and MLX at the bottom right, so the two disagree whenever the lengths differ, silently. Rank 4 already reaches the kernel today and keeps its current behaviour: fixing that needs either a change to what the speech example computes or a fix to an off-by-one in the decomposed path, so it is filed separately as pytorch#22426. What this change does is avoid opening two more ranks onto the same trap. ### Declining a call is not free Preservation from decomposition is requested per operator, so one claimed call keeps the operator whole for a declined one in the same graph, and that call is then neither lowered nor decomposed: ``` RuntimeError: Missing out variants: {'aten::scaled_dot_product_attention'} ``` Give the whole operator back when any of its calls is unsupported. The framework does offer a per-node filter that would keep the other calls fused. It is deliberately not used, and the docstring says why: on an attention block that reshapes its output, with a declined call in the same graph, that path fails to lower at all with `Cannot view a tensor with shape (1,16,4,16) and strides (1024,16,256,1)`. It trades the fusion cost for a hard failure on a very ordinary shape. The cost of the coarser choice is real and written down: one declined call unfuses the operator's other calls in that graph. This half is not specific to attention. Two calls to `torch.roll` in one graph, one supported and one not, already fail at execute today for the same reason. ### Test plan Fifteen partitioner tests. **Eleven fail before this change**, all fifteen pass after. They assert on serialized nodes, so they check which path a call took rather than only that it answered, and the rejection cases also assert the work stayed on this backend, so they cannot pass when nothing is delegated at all. A rank-3 case was added to the operator suite too, which runs the compiled runtime. Ran every combination on an Apple Silicon Mac against eager: plain, causal, explicit scale, float mask, boolean mask, grouped-query attention, batch size 1, float16 and bfloat16, ranks 2 through 5. - rank 3 matches to 3.6e-07 and stays fused in all seven variants; rank 2 to 3.6e-07 - rank 5 decomposes at 3.6e-07 where it previously failed; unequal batch from failing to 2.4e-07 - the grouped key case measured both ways: absorbed and correct at rank 4, and off by 5.1 at rank 3 with a causal mask if absorbed, which is what the guard prevents - a zero key head count used to divide by zero and abort the whole export; it now declines that node - a mixed supported and unsupported graph exports and runs; two rolls in one graph go from failing at execute to matching eager exactly Exported the speech example and compared it against the same export without this change: **identical serialized node counts and an identical greedy token sequence over twelve decode steps**, so that model is unaffected. Ran the neighbouring backend test files, 71 tests, all passing. Co-authored-by: PyTorch Bot <pytorchbot@users.noreply.github.com>
Fixes pytorch#22426. ### The problem Attention with `is_causal=True` returns wrong values, with nothing raised, whenever the query is shorter than the keys: | shape | error against eager | | --- | --- | | query 1, keys 16 | **2.7 to 3.4** | | query 16, keys 16 | 3.6e-07 | The two libraries anchor a causal mask at opposite corners. PyTorch puts it at the top left, so the query at row `i` attends to keys `0..i`. MLX puts it at the bottom right, aligning the last query with the last key. They agree only when the lengths are equal, and nothing checked that. This is the decode step of any model that generates a token at a time, so it is not a corner case. ### The fix Slice the keys and values to the query length when the lengths are not provably equal, then let the kernel apply its own mask to what is now a square problem. That is what PyTorch computes: a row past the last query never reads a later key, so dropping those keys changes nothing. Measured over query lengths 1 to 63 against key lengths 2 to 512, the sliced form and the original agree **exactly** in float32. A slice before attention is also what the cache-aware path in this file already does, so no mask tensor is built and the fused kernel is kept. A causal query **longer** than its keys is left to decompose instead. PyTorch clamps each row to the keys that exist, and neither the kernel's own mask nor a slice reproduces that. ### Why the speech example changes too This backend already knew about the conflict. `mlx::custom_sdpa` exists for cached attention and takes a start position so the kernel gets a square problem, and seven models here use it. The speech example does not. It slices the cache itself and then asks for a causal mask over the result, which in PyTorch means **the new token sees only the first cached key**. It has been getting attention over the whole cache because the kernel read its request the other way round, so it depends on the behaviour this change corrects. Moving it onto the same path as the other models makes it ask for what it wants. Worth stating plainly: that example's own eager math was already 3.3 away from what it intended at a decode step, so this is not a regression being introduced, it is one being removed. ### Test plan Two cases added to the operator suite, a query of 1 and of 6 against 32 keys, which run the compiled runtime and compare against eager. **Both fail before this change on numerics and pass after.** Ran the shapes on an Apple Silicon Mac against eager: - query 1, 2 and 6 against longer keys, float32 and bfloat16: now exact or within 4.2e-07, previously off by around 3 - equal lengths and non-causal calls untouched, fused kernel kept in every case, no mask tensor built - a causal query longer than its keys decomposes and matches eager to 3.6e-07, previously off by 2.7 - equal-length causal attention serializes to a **byte-identical program** before and after, so that path is bit-for-bit unchanged Exported the speech model and compared its logits against HuggingFace's own decoder at a cached decode step: unchanged from before this change, and the top token still matches. Ran the neighbouring backend test files, all passing. Co-authored-by: PyTorch Bot <pytorchbot@users.noreply.github.com>
Adds timing instrumentation to the batched runner, which previously reported nothing. metrics.h defines two tiers: GenerationMetrics, published on GenerationHandle beside finish_reason(), covering the submit→first-step→first-token→end timeline, token counts, and an inter-token latency summary; and EngineMetrics, owned by the runner and read after shutdown(), covering step counts and latency, decode/prefill sequences and tokens, admitted-versus-ready decode sequences, and TTFT rollup. cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani
…orch#22482) Pin the mypy job to the repository's Torch 2.13 stack before installing timm, and include the Torch pin in its pip cache key. This makes it deterministic instead of depending on order of resolution. Adapt Arm shape and dtype checks to the SymInt and SymBool annotations exposed by newer Torch releases. cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani
Differential Revision: D118475079 Pull Request resolved: pytorch#22442
Follow-up polish on the causal slicing added in pytorch#22443. No behaviour change for any shape that worked before. ### Read the query length at build time where it is known The slice that shortens the keys for a causal call always emitted a node to read the query length at run time. For a decode step that length is a constant known when the program is built, so the node was pure overhead on exactly the path this slicing exists to speed up. There is already a helper for this, used by the cached attention handler a few hundred lines down in the same file: it returns a literal for a concrete dimension and emits a size node only for a symbolic one. | Call | Before | After | | --- | --- | --- | | query 1 against 32 keys, static | size node, 2336 bytes | no size node, **2288 bytes** | | query 6 against 32 keys, static | size node, 2336 bytes | no size node, **2288 bytes** | | dynamic query length | size node | size node, unchanged | ### Two comments promised less than the code delivered The guard that declines causal attention also declines a query length that **cannot be compared** at build time, two unrelated dynamic dimensions for instance, and the comment above it described only the case where the query is genuinely longer. It now says both, because the second is the case a reader is more likely to meet. ### A half-wired test knob The key length knob added to the attention test reached the query, key and value but not either mask branch, so a case combining a mask with an unequal key length built a mask of the wrong width. No shipped case does that today, so nothing was failing, but the next person to use it would have hit it. ### Test plan Causal attention at every shape the operator suite covers, run against eager on an Apple Silicon Mac: query 1, 2 and 6 against longer keys, equal lengths, non-causal, float32 and bfloat16. All match to 4.8e-07 or exactly, 8 of 8 cases. A dynamic query length bounded by the keys keeps its size node, exports, and matches eager to 1.2e-07. Building the test mask both ways in eager torch: with the key length it runs, and with the query length it raises the size mismatch this removes. Exported the speech model: unchanged, 8 fused attention nodes and no size nodes. Ran the neighbouring backend test files, all passing. Co-authored-by: PyTorch Bot <pytorchbot@users.noreply.github.com>
### Summary * Open Model with zero-copy using dmabufheap * Improve model loading with zero-copy * Remove unncessary comments or headers. Change file name cc @SS-JIA @digantdesai @kimishpatel --------- Signed-off-by: Jiseong oh <jiseong.oh@samsung.com> Signed-off-by: Jiseong Oh <jiseong.oh@samsung.com> Co-authored-by: Hoon Choi <hoon98.choi@samsung.com>
As titled
…22502) ### Summary - Add documentation for QNN ExecuTorch on Windows, including build, test, and inference instructions. - Add support for the `SC8380XP` SoC model. Refer to the [Supported Snapdragon devices](https://docs.qualcomm.com/doc/80-63442-10/topic/QNN_general_overview.html#supported-snapdragon-devices) for details. ### Test plan ```powershell python -m examples.qualcomm.scripts.deeplab_v3 --build_folder build-x86_64-windows --soc_model SC8380XP --compile_only --download ```
### Summary Extend `QuantizationRecipe` with QAT support and four ordered pass hooks, then wire them into `QuantizeStage`. `QuantizationRecipe` gains `is_qat`, `train_fn`, `calibration_inputs_fn`, and four pass-list fields (`pre_prepare_passes`, `post_prepare_passes`, `pre_convert_passes`, `post_convert_passes`). When `is_qat=True`, `QuantizeStage` calls `prepare_qat_pt2e` instead of `prepare_pt2e`, invokes `train_fn` on the prepared model, and skips calibration. When `is_qat=False` (default), PTQ proceeds as before, but callers may now supply a `calibration_inputs_fn` to supply their own calibration data. If omitted, the existing example-inputs fallback is used. The four pass hooks are applied at the appropriate points in both flows and compose cleanly with `_combine_recipes`. `_combine_recipes` is overhauled to handle all previously dropped fields: the four pass lists are concatenated, `is_qat` must agree across all combined recipes, at most one `train_fn` is allowed, and multiple `calibration_inputs_fn` values are chained into a single factory. The `strict`, `mode`, `pipeline_stages`, and `source_transform_in_place` scalar fields are now also validated for agreement and propagated to the combined recipe. `edge_manager_transform_passes` from `LoweringRecipe` is similarly merged. ### Test Plan ``` pytest -q export/tests/test_export_recipe.py export/tests/test_export_stages.py ```
…h#21241) ### Summary This pass replaces `dim_order_clone` and `to_dim_order_copy` nodes that consume model inputs by a sequence of permute nodes according to the proposal in pytorch#19299. Fixes pytorch#20095. ### Test plan `pytest backends/transforms/test/test_replace_channels_last_input_clones.py`
TOSA section 2.13 permits PRO-FP casts from signed integer types to fp16 or fp32. Cast int8 comparison inputs to fp16 and int16 inputs to fp32, where every value is exactly representable. This allows the comparisons to be lowered without changing their results. Signed-off-by: Sebastian Larsson <sebastian.larsson@arm.com>
Normalize unbatched max_pool2d inputs from [C, H, W] to [1, C, H, W] before decomposition, then restore the rank-3 output. Enable rank-3 support checks and add FP, INT, and VGF test coverage, including quantized VGF execution. Change-Id: Ib8852053b5248601d59685669523079fa19c0955 Signed-off-by: Yufeng Shi <yufeng.shi@arm.com>
- move test_tosa_* to tosa_dialect folder - move comparison operators out of binary operators - add aten->tosa for comparison operators Signed-off-by: Saoirse Stewart <saoirse.stewart@arm.com>
Pull Request resolved: pytorch#22481 Support runtime-varying `arange` bounds, fractional and negative steps, and symbolic `clamp` bounds for integer and floating-point tensors. Extend buffer `index.Tensor` to gather on any axis while preserving multidimensional index shapes. Authored with Codex. ghstack-source-id: 423931774 @exported-using-ghexport Differential Revision: [D118507314](https://our.internmc.facebook.com/intern/diff/D118507314/)
Pass `transpose=True` to PyTorch's batch norm fusion helper for transposed convolutions. Fuse grouped transposed convolutions one group at a time and restore the grouped parameter layout before decomposition. Add tests for unequal input and output channel counts, grouped fusion with and without bias, and non-affine batch norm. Authored with Codex. Change-Id: I7fbd523fa1627ce5d198aea5adbd083c91c0635f Signed-off-by: Yufeng Shi <yufeng.shi@arm.com>
Route the EXIR pre-decomposition hook through an Arm pass pipeline. Store compile specs on concrete partitioners so the pipeline can be configured consistently for TOSA, Ethos-U, and VGF. Keep the pipeline a no-op until backend-specific passes are registered, and update generated partitioner docs and public API manifests. Assisted by Codex. Change-Id: I1963bad577714907e5ba22eed765333d9816308c Signed-off-by: Yufeng Shi <yufeng.shi@arm.com>
Enable legal static ReflectionPad1d, ReflectionPad2d, and
ReflectionPad3d cases proven through Vela and Corstone-300.
Reflection padding mirrors edge values when extending a tensor:
Original: [A B C D]
Pad 2 each side: [C B | A B C D | C B]
Cover ranks, padding shapes, batching, and quantization modes, and
leave dynamic shapes undelegated when padding validity cannot be
proven at compile time.
Authored with Codex.
Change-Id: Ib0b833c2c866b13e504920671303ee22ceac01b6
Signed-off-by: Per Held <per.held@arm.com>
Pull Request resolved: pytorch#22496 Dense generation stamps replace allocation-heavy `std::unordered_set` update tracking while preserving recursive `ValueList` semantics. Authored with Codex. ghstack-source-id: 423933048 @exported-using-ghexport Differential Revision: [D118543865](https://our.internmc.facebook.com/intern/diff/D118543865/)
Pull Request resolved: pytorch#22497 Prioritize read dependencies before write-only outputs so active nodes short-circuit without changing resize semantics. Authored with Codex. ghstack-source-id: 423933065 @exported-using-ghexport Differential Revision: [D118543866](https://our.internmc.facebook.com/intern/diff/D118543866/)
Pull Request resolved: pytorch#22498 Inline direct generation-stamp lookup while keeping rare nested ValueList traversal out of line. Authored with Codex. ghstack-source-id: 423933067 @exported-using-ghexport Differential Revision: [D118543873](https://our.internmc.facebook.com/intern/diff/D118543873/)
Pull Request resolved: pytorch#22499 Resolve resize-invariant shader names during graph construction while preserving dynamic workgroups and the M==1 quantize gate. Authored with Codex. ghstack-source-id: 423933078 @exported-using-ghexport Differential Revision: [D118543874](https://our.internmc.facebook.com/intern/diff/D118543874/)
…ytorch#22476) Differential Revision: D118365125 Pull Request resolved: pytorch#22476
Differential Revision: D116686805 Pull Request resolved: pytorch#22282
### Summary The Samsung model test job runs every test under `backends/samsung/test/models`. One of those tests, the MobileBert fine-tuning test, always errors out before it does any work: ``` ERROR: setUpClass (test_mobilebert_finetuning.Test_Milestone_MobileBertFinetune) AttributeError: type object 'Test_Milestone_MobileBertFinetune' has no attribute 'model_cache_dir' ``` One error fails the whole job, so the other ten Samsung model tests passing does not help. The job only runs the tests when a Samsung device is available, which is why this looks like it comes and goes. There were three separate problems stacked in the test setup, and each one was hidden behind the one before it: 1. `setUpClass` read `cls.model_cache_dir`, which is not defined anywhere. 2. It passed that value to `patch_mobilebert_finetuning()`, which takes no arguments. Python only checks the number of arguments when the call happens, so this could not be seen until the first problem was gone. 3. The setup replaced `load_tokenizer` with a copy of itself that left out the model name, so `AutoTokenizer.from_pretrained()` had nothing to load. The replacement tokenizer loader was the same as the real one except for the missing model name, so it could only ever do less than the real one. Removing the setup that installed it fixes all three problems at once and lets the test use the real loader. The test now has the same shape as the other ten tests in the same directory, which build a model and check it with no extra setup. ### Test plan Ran the real test collection and setup for this file before and after the change, with the heavy third party dependencies stubbed out so only the setup path was under test. - Before: reproduces the same `AttributeError` seen in CI, and never reaches the test body. - After: setup succeeds and the test body is reached. Also checked that `model_cache_dir`, `setUpClass` and the now unused `AutoTokenizer` import are all gone, and that the file still compiles.
…2541) Fixes pytorch#21611 ## The problem, in plain terms Five shared libraries in the Linux wheel search three directories that exist on nobody's machine: ``` /lib/intel64 /lib/intel64_win /lib/win-x64 ``` Two of them name a Windows layout, in a Linux wheel. They come from PyTorch. Its exported CMake package creates a `caffe2::mkl` imported target with a hardcoded list of link directories, and linking torch brings them in even though this project never asks for MKL: ``` ${MKL_ROOT}/lib ${MKL_ROOT}/lib/intel64 ${MKL_ROOT}/lib/intel64_win ${MKL_ROOT}/lib/win-x64 ``` `MKL_ROOT` resolves to nothing here, so what the linker records is left anchored at the filesystem root. Packaging copies the built libraries out of the build tree rather than installing them, so whatever the linker recorded ships as is. The bare `/lib` form does not survive, because CMake filters its own implicit link directories out of the link line, which is why three entries appear rather than four. This only affects Linux x86_64. The aarch64 nightly records no absolute entries at all, since MKL is not found there, and the macOS and Windows wheels record none either. ## Why it is worth fixing Nothing needs those directories. No shipped library names an MKL or OpenMP runtime among its dependencies, so nothing resolves through them. They are not merely untidy either. They sit ahead of the relative entries packaging appends, and the loader searches in order, so a user who happens to have a matching directory resolves a library from there instead of from the one the wheel installed. That is the same shadowing the release check already rejects a CUDA toolkit prefix for. ## What changed Two small pieces. Packaging now drops these entries, in the same place the other unusable ones are already dropped. The match is deliberately narrow: only the exact `/lib/<arch>` form that an empty prefix produces. A real MKL installation spells the same arch directory below a prefix, as `/opt/intel/mkl/lib/intel64`, and that is a directory the environment genuinely provides, so it is kept. The release check that rejects absolute search paths listed these three as allowed. That is why they shipped while a check whose whole purpose is rejecting absolute paths reported the wheel clean. It now rejects the empty prefix form specifically, and still accepts a real installation's prefixed directory, so the two halves agree rather than contradict each other. ## Test plan New unit tests in `.ci/scripts/tests/test_runtime_path_filter.py`. They read the functions out of `setup.py`, so they exercise the code that ships rather than a copy, and they run on every pull request through the existing unit test job. They need no wheel build, which is what the previous check could not manage: it could only see this after a full build, and only on the platform that built one. ``` pytest .ci/scripts/tests/test_runtime_path_filter.py 27 passed ``` They cover both directions, because a filter that satisfies either one alone is wrong: - the three unresolved entries are dropped - a real MKL installation and ordinary system directories are kept - every relative entry on a shipped library survives, so a library can still find its siblings - the absolute torch directory is kept when it is a library's only route to torch, which is what stops this becoming a blanket rule that breaks importing I checked by mutation that each part of the fix is load bearing: | what I broke | tests that failed | | --- | ---: | | removed the packaging filter | 4 | | removed the release check's maths rejection | 3 | | removed its build directory branch | 2 | | removed its catch-all rejection | 1 | | severed the call that applies it to a shipped library | 1 | | reordered its guards so a build directory is accepted | 1 | | narrowed what it accepts, refusing a real install | 3 | | made the packaging filter reject an ordinary system directory | 1 | | added an arch to the shared constant, untaught to the check | 2 | The release check's per-entry decision is a small module level function, so the unit test calls the same code the wheel check runs and compares the reason it returns. Asserting only that a path was rejected was not enough: the check rejects every absolute path it does not recognise, so an unknown arch passed for the wrong reason and two of its three branches could each be deleted on their own. I also measured the published artifacts directly, reading the recorded search paths out of every shared library in each wheel: | wheel | libraries with a search path | unusable entries | | --- | ---: | ---: | | 1.4.1 release, CPython 3.12 | 6 | 0 | | current nightly, CPU, CPython 3.12 | 18 | 15 | | current nightly, CUDA, CPython 3.13 | 21 | 15 | | current nightly, CPU, aarch64 | 16 | 0 | The 1.4.1 release predates the code changed here. Both nightly rows show the same 15 entries across the same five libraries, which is what this removes. Reading those same libraries' declared dependencies shows no MKL or OpenMP runtime in any of them. ## Scope This covers the first of the two things the issue suggests: not shipping a runtime search path a user cannot use. It does not do the second, which is to resolve MKL through something the wheel or its declared dependencies own. That one belongs upstream, since the directory list is set in PyTorch's own CMake package, and its own comment there marks it as a hack. Nothing in this wheel links MKL, so there is no dependency here left to redirect. ## What I did not verify No wheel was built for this. The change is to the filter those builds already run, and the tests exercise it directly. The link line that emits the paths is unchanged; the entries are dropped at packaging, the same way the CUDA toolkit and torch directories already are. Co-authored-by: PyTorch Bot <pytorchbot@users.noreply.github.com>
## Ulterior Motive Keep large model weights off heap until a backend chooses their destination and lifetime. ## Rationale **What**: Split safetensors metadata parsing from payload access. Add metadata-only lookup, owned acquisition, direct destination loading, package identity, and checksum verification. **Why**: Eager extraction duplicates model-sized weights and prevents direct placement into backend-managed memory. ## Details ```text Package::load(path) | +-- open ZIP directory +-- read program.ptg +-- read safetensors prefix + JSON only | +-- constant_info(key) no payload read +-- acquire_constant(key) new OwnedBytes +-- load_constant_into() caller destination +-- verify_constants() streamed CRC check ``` - Package identity survives moves. Alias lookups return canonical owner keys, enabling cross-instance upload accounting. - Prefix and checksum scratch arrays are value-initialized, satisfying the blocking clang-tidy signal. - `ptn_inspector` uses the path-based `Package::load` API after the eager `load_file` API is removed. - New package-test ZIP handles use RAII on constructor failures. Authored with Codex. Differential Revision: [D119396097](https://our.internmc.facebook.com/intern/diff/D119396097/) ghstack-source-id: 427858773 Pull-Request: pytorch#22703
## Ulterior Motive
Define backend-neutral execution boundaries before landing the first Vulkan
implementation.
## Rationale
**What**: Add pure abstract `EngineContext` and `EngineExecutable` interfaces.
**Why**: Native runtime can load methods and weights but needs a stable contract
for backend compilation and execution.
## Details
```text
process scope compiled region scope
EngineContext ----------------compile--> EngineExecutable
| |
+-- backend/device identity +-- input/output metadata
+-- shared device state +-- set_input()
+-- kernel/pipeline reuse +-- execute()
+-- get_output()
```
`EngineContext` must outlive its executables. `compile()` currently accepts a
whole `Method` for full delegation. Unsupported graphs fail with exceptions
instead of a separate `can_run()` boolean. Polymorphic types are immovable and
noncopyable; out-of-line destructors anchor vtables.
Authored with Claude Code.
Differential Revision: [D118480656](https://our.internmc.facebook.com/intern/diff/D118480656/)
ghstack-source-id: 427858783
Pull-Request: pytorch#22704
…ect (pytorch#22410) ### Summary `add_embedding_legacy_node()` and both `index_select` node builders pass `nullptr` as their resizing logic: ```cpp // Resize Args {}, // Resizing Logic nullptr)); ``` even though `resize_embedding_node`, `resize_index_select_channel_node` and the local `resize_fn` are already defined immediately above them and do the right thing. Under dynamic shapes the outputs keep the extents they were built with (the upper bound) instead of tracking the real input sizes, so consumers read them at the wrong size. `register_index_select()` also does not set `supports_resize`, so this patch sets it now that the op honours resize. ### Fix Pass the resize functions that already exist. Three one-line changes plus the registry flag. ### How it surfaced A TTS model on Adreno 840 produced wrong outputs after an unrelated change moved a partition boundary. The embedding output had been keeping its upper-bound extents all along; previously a CPU round-trip happened to re-establish the correct shape downstream, so the defect was masked. Once the gather stayed on the GPU the stale extents reached the consumer. Verified on a Galaxy S26 Ultra: with these wired up, the affected sub-models return correct shapes and match their CPU references (cosine >= 0.999 at sequence lengths well below the dynamic bound) where before they were wrong at every length except the bound itself. Note this class of bug is hard to see with `executor_runner`, which can only run at the dynamic upper bound -- exactly the one shape where a missing resize is harmless.
…ch#22432) ### Summary `get_symmetric_quantization_config` says "per-token input quantization" in its `is_dynamic` branch, but the spec it builds is `per_tensor_affine`. The spec is correct: the kernel behind `et_vk.linear_q8ta_q8csw` takes a single input scale and zero point (`QuantizedLinear.cpp`, `input_quant_config(8, kPerTensor, ...)`). Only the comment is wrong. It is worth saying accurately, because the difference is large on encoder-style models. Mean cosine of the embedding against the fp32 eager model on `all-mpnet-base-v2` / `multi-qa-mpnet-base-dot-v1`: | config | cosine | | --- | --- | | `is_dynamic=True, weight_bits=8` | 0.910 / 0.948 | | `is_dynamic=False, weight_bits=8` | 0.9993 / 0.9993 | | torchao per-token 8da8w, same models | 0.998 / 0.998 | Dynamic int8 is not the problem; the per-tensor scale is. A per-token variant for int8 weights would be the real fix, and pytorch#22431 tracks that. This PR only makes the current behaviour discoverable, and points at `is_dynamic=False` for callers that care about fidelity. Related to pytorch#22431 ### Test plan Comment and docstring only, no functional change. `black --check` and `flake8` clean on the touched file.
### Summary Update the development version on `main` from `1.5.0` to `1.6.0` after cutting the `release/1.5` branch. ### Test plan - verified `version.txt` contains `1.6.0` - `git diff --check`
pytorch#22661) Summary: Before, we didn't support this case (ex shape(4, 4, 2) + shape(4, 3)) Now we do. Reviewed By: ethansfng Differential Revision: D119425194 Pull Request resolved: pytorch#22661
…ytorch#22711) Part of pytorch#22686 ### Problem The `executorch` SwiftPM product declared only the C++ standard library. The image processor inside it calls into Accelerate and Core Image, so a package that depends on `executorch` alone and links with `-all_load` fails to link. I reproduced it against the published prebuilt frameworks: 16 missing symbols, 8 from vImage, which lives in Accelerate, and 8 from Core Image. On iOS the Core Image half is usually hidden, because an app that imports UIKit, SwiftUI or AVFoundation already pulls Core Image in. On macOS, and on an iOS app that imports only Foundation, both halves fail. Until now each app had to name these frameworks itself, which is a detail the package should own. ### Change Declare Accelerate, Core Graphics, Core Image, Core Video and Foundation on the `executorch` product. All five are used by the image processor. For most consumers Accelerate and Core Image are the two that fix the link, because Core Graphics, Core Video and Foundation already arrive as link hints, either from the Swift files in the framework or from the consumer's own code when it is built with modules enabled, which is the default. Turn modules off in an Objective-C consumer and those hints disappear, and then Core Graphics and Core Video are needed explicitly too. Declaring all five covers that case as well and lets the product state what it actually uses. ### Note for reviewers The manifest that prebuilt-binary users get is generated from `Package.swift.template` on the `swiftpm` branch, not from this file, and it carries the same `executorch` entry with only `c++`. This change alone therefore does not reach anyone using the prebuilt frameworks. I will send the matching change against that branch as a follow up. That is why this says "Part of" rather than "Fixes": the issue should stay open until the template change lands too. ### Test plan - `swift package --disable-sandbox dump-package` parses, and both the release and debug `executorch` targets carry the five frameworks plus libc++. - Built a package that depends only on the `executorch` product with `-all_load` against the prebuilt frameworks. It links and runs with this change. Removing just the added block brings back the 16 missing symbols, so the check fails without the fix. - The Apple builds on this PR pass. The demo app jobs do not run here, because they need secrets a fork pull request cannot use, so nothing in CI currently links a real app against this product. A test that would catch this needs a consumer that links `executorch` and nothing else. Co-authored-by: PyTorch Bot <pytorchbot@users.noreply.github.com>
…rch#22638) Pool backend-generated helper constants per TOSA basic block using dtype, shape, and exact serialized bytes as the key. Implement the pool in a TosaSerializer subclass. Helper operands use the tensor returned by addConst, making reuse safe. Keep graph-owned constants unpooled because later lowering refers to their FX names. This covers model parameters, buffers, lifted constants, and CONST_SHAPE outputs, while preserving packed FP4 serialization. Deduplicate identical static CONST_SHAPE nodes in FX, where their uses can safely be rewritten to the first occurrence. Representative TOSA-FP operator comparisons: | Model | Before | After | Reduction | Bytes saved | |-------------|-------:|------:|------------:|------------:| | SD3 | 1,518 | 1,038 | 480 (31.6%) | 89,068 | | InceptionV3 | 673 | 456 | 217 (32.2%) | 42,092 | | Conformer | 488 | 365 | 123 (25.2%) | 22,852 | This change was authored with assistance from Codex. Change-Id: I4de44b034aff33cbf2888801238cc48df5a9124d Signed-off-by: Yufeng Shi <yufeng.shi@arm.com>
Generalize index.Tensor decomposition to treat leading full-slice dimensions as the TOSA GATHER batch dimension. Reshape values to [P, K, C], expand linearized indices to [P, W], and restore the PyTorch output shape without transposing the data tensor. Update support checks to accept leading None entries while rejecting interleaved indexing, and add tests for TOSA FP, TOSA INT, and VGF. Change-Id: I657dd1876c7ca2632b951bc517f8960a10242548 Signed-off-by: Yufeng Shi <yufeng.shi@arm.com>
) ### Summary Enable and refactor MLPerf Tiny Keyword Spotting tests. ### Test plan tests can be manually run using `pytest -c /dev/null backends/nxp/tests/` cc @robert-kalmar @JakeStevens @digantdesai @rascani @roman-janik-nxp
Lower isinf and isnan to TOSA-supported comparisons for floating-point profiles. Keep quantized floating-point inputs on CPU because integer quantization cannot preserve NaN or infinity. Change-Id: Ib53653b0ebb1c5497a7db3932945a1b9c5060f0f Signed-off-by: Sebastian Larsson <sebastian.larsson@arm.com>
Change-Id: I3eaa54c7b794f583214bb252015e38960ae9be45 Signed-off-by: Sebastian Larsson <sebastian.larsson@arm.com>
Expect no CPU isinf calls after FP and VGF partitioning. Account for four fewer delegate calls now that isinf is lowered inside delegates. Keep the TOSA INT fallback expectations unchanged. Signed-off-by: Sebastian Larsson <sebastian.larsson@arm.com> Change-Id: Ib52c3a9e8bf55d9b8b1ef600a7e7d0fc694e34a7
Signed-off-by: Sebastian Larsson <sebastian.larsson@arm.com>
… Linux **Problem** On the Cortex-A path, execute() created the driver objects (the network and the DMA buffers, including a full copy of the weights) on every inference. **Fix** Create them once in platform_init() and keep them for the lifetime of the loaded method. Measured on the Corstone-1000 FVP: 53-77% less CPU time in execute() per inference. Signed-off-by: Youngsik Yang <vacu9708@gmail.com>
Properly handle platform setup failures by propagating platform initialization failures and releasing the execution handle. Signed-off-by: Youngsik Yang <vacu9708@gmail.com>
Capture VGF coverage checker output during pre-push and print it only when the check fails. Keep strict attribution diagnostics available for direct checker use while making successful pushes concise. Signed-off-by: Per Held <per.held@arm.com> Assisted-by: Codex Change-Id: I4246d3f4ef07e4ec3dd3ace3292fc9b098a97931
…n supported by the device. (pytorch#22727) Enable Vulkan BF16 shader support in the VGF backend when supported by the device. This fixes validation errors for BF16 workloads by enabling VK_KHR_shader_bfloat16 and shaderBFloat16Type. cc @SS-JIA @manuelcandales @digantdesai @cbilgin @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani Signed-off-by: Elena Zhelezina <elena.zhelezina@arm.com>
…torch#22733) In automatic generation of supported ops we don't take into account registry of custom ops. This PR fixes this. cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani Signed-off-by: Elena Zhelezina <elena.zhelezina@arm.com>
Boolean sums produce int64 outputs, so the TOSA partitioner rejects them before lowering. Cast boolean inputs to int32 and run the sum with an explicit int32 dtype. Cast the result back to the original output dtype to preserve model semantics. Signed-off-by: Sebastian Larsson <sebastian.larsson@arm.com>
### Summary Addresses the first half of pytorch#21950. `_fft_r2c.out` only had an optimized (pocketfft-backed) kernel. A program that leaves it undelegated builds fine and writes a `.pte`, but a runtime carrying only the portable kernels cannot load it. The failure surfaces as `0x14 OperatorMissing` from `load_method`, long after the export reported success. `_fft_r2c.out` and `_fft_c2r.out` are two of only three ops in `optimized.yaml` with no counterpart in the portable `functions.yaml`: ``` portable ops: 206 optimized ops: 23 in optimized but NOT in portable (3): _fft_c2r.out _fft_r2c.out linear.out ``` The third, `linear.out`, ExecuTorch decomposes anyway, so the FFT pair is the real gap. ### Approach This is a direct O(n^2) evaluation of the transform sum, not a fast Fourier transform, in keeping with the portable library's role as the dependency-free reference; `kernels/optimized` keeps the asymptotically faster pocketfft path for anyone who links it. Audio front-ends transforming a few hundred points per frame, which is where this op tends to show up, are the intended case. Two details worth calling out: - **Multi-dimensional transforms** run the real transform along the last requested dimension and complex transforms along the rest, matching pocketfft's multi-axis `r2c`. A complex pass has to read a whole line before overwriting it; lines up to 128 elements (2 KB for double) use a stack buffer and longer ones ask the runtime for temporary memory. The single-dimension case, which is what `torch.fft.rfft` lowers to, needs no line buffer at all. - **The quarter-turn twiddle factors are returned exactly** rather than through `cos`/`sin`, so a real input's Nyquist bin comes out with a zero imaginary part instead of rounding noise around 1e-16. That is what pocketfft produces and what the existing tests expect. ### Test plan Registers the shared `op_fft_r2c_test` for `portable` alongside `aten` and `optimized`. All five cases pass against both kernel libraries: ``` $ ./cmake-out/kernels/test/portable_kernels_test --gtest_filter='OpFftR2c*' [ PASSED ] 5 tests. $ ./cmake-out/kernels/test/optimized_kernels_test --gtest_filter='OpFftR2c*' [ PASSED ] 5 tests. ``` End to end on the model from the issue (`torch.fft.rfft(x).abs().pow(2)` over `[8, 512]`, exported with no partitioner so the op stays on the CPU), run through `executor_runner` built with portable kernels only: | | result | | --- | --- | | before | exit 134 (abort on load) | | after | exit 0, output `[262144., 1.2e-27, 2.4e-27, ...]` | which is the correct `abs(rfft(ones))**2`: `512**2` in bin 0 and zero elsewhere. Accuracy against numpy, since the tests above use exactly representable values: | case | max abs error | | --- | --- | | `rfft`, length 5 (odd, no Nyquist bin, no quarter-turn twiddles) | 2.5e-15 | | `rfft`, 4x512 | 1.8e-13 (1.4e-15 relative to the largest output) | | `rfft2`, 6x8 | 1.8e-14 | I wanted to add the length-5 case as a regular test, but `tensors_are_close()` falls through to a bitwise `memcmp` for complex dtypes, so a complex comparison cannot have a tolerance and only exactly representable expected values can pass. That is why the existing cases all use small integers. Happy to fix that separately if it would be welcome. ### Not addressed here The issue also asks for lowering to fail when an op is left undelegated and no kernel exists, rather than emitting a `.pte` that dies at load. That one is a change to the AOT/runtime contract, since the export side does not know which kernel library the runtime will link (portable, optimized, or a selective build), so it needs its own discussion rather than riding along here. cc @larryliu0820 @manuelcandales @JakeStevens Co-authored-by: Jacob Stevens <stevens.jacob1492@gmail.com>
…gfault What: test_mlperf_tiny_classification_mse_cpu_vs_npu used the Python edge reference instead of the portable-kernel C++ reference for the QAT plus channels-last variant, because that configuration segfaulted in the portable kernels. Why: the crash is fixed on current main. The reporter's stack predates two out-of-bounds fixes in the portable dequantize path that this model exercises at runtime (16 per-channel dequantize ops): pytorch#21517 fixed an out-of-bounds traversal for non-contiguous (channels-last) inputs, and pytorch#21773 fixed misreading int32 zero points as int64. The int32 zero points are QAT-only: QAT emits int32 bias zero points while PTQ emits int64, matching the issue's QAT-only signature. Verification: exported the exact failing configuration (QAT, channels-last, 15-epoch training, dataset calibration, NXP edge passes) and ran it with the portable-kernel executor_runner. It runs cleanly and its outputs bit-match the eager quantized reference. The quantized_kernels_test suite passes 74/74, including the regression tests from both fixes. lintrunner reports no issues on the touched file. The NXP SDK-gated test itself was not run here (no SDK); NXP CI will exercise it. Fixes pytorch#22179 Authored with AI assistance (Muse Code).
Owner
Author
|
Opened by mistake against the fork; the real PR is pytorch#22746. |
JakeStevens
had a problem deploying
to
upload-benchmark-results
September 13, 2026 15:35 — with 
GitHub Actions
Failure
GitHub Actions
Failure
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What: test_mlperf_tiny_classification_mse_cpu_vs_npu used the Python edge reference instead of the portable-kernel C++ reference for the QAT plus channels-last variant, because that configuration segfaulted in the portable kernels.
Why: the crash is fixed on current main (issue pytorch#22179). The reporter's stack predates two out-of-bounds fixes in the portable dequantize path that this model exercises at runtime (16 per-channel dequantize ops): pytorch#21517 fixed an out-of-bounds traversal for non-contiguous (channels-last) inputs, and pytorch#21773 fixed misreading int32 zero points as int64. The int32 zero points are QAT-only: QAT emits int32 bias zero points while PTQ emits int64, matching the issue's QAT-only signature.
Verification: exported the exact failing configuration (QAT, channels-last, 15-epoch training, dataset calibration, NXP edge passes) and ran it with the portable-kernel executor_runner. It runs cleanly and its outputs bit-match the eager quantized reference. The quantized_kernels_test suite passes 74/74, including the regression tests from both fixes. lintrunner reports no issues on the touched file. The NXP SDK-gated test itself was not run here (no SDK); NXP CI will exercise it.
Fixes pytorch#22179
Authored with AI assistance (Muse Code).