Libtorch backend

[gluefox]

🚀 The feature, motivation and pitch

There are rumours of a libtorch backend for executorch here: https://dev-discuss.pytorch.org/t/re-what-is-the-correct-future-proof-way-of-deploying-a-pytorch-python-model-in-c-for-inference/3433

Is there a way to access those builds so we can run our models against it?

Alternatives

No response

Additional context

No response

RFC (Optional)

No response

cc @cccclai

[shoumikhin]

Thanks for asking, and sorry for the confusion. There is no libtorch backend build to download. The forum discussion refers to ExecuTorch's ATen mode, which is only partly implemented and is not available in any public build today.

For CPU inference, some of what you want already works. If you export without adding a hardware backend, you get a model file that holds your weights and a list of plain operators, with no GPU or CPU specific code inside it. The same file can go to different machines, and nothing needs to be compiled for the target. Two conditions apply. Your model has to use operators that ExecuTorch supports, and each target needs an ExecuTorch runtime built with those operators in it. Speed also depends on which CPU kernels that runtime is built with, so it can be slower than PyTorch.

Set model to your model in eval mode and example_input to a representative input tensor.

import torch
from executorch.exir import to_edge_transform_and_lower

exported = torch.export.export(model, (example_input,))
program = to_edge_transform_and_lower(exported).to_executorch()
with open("model.pte", "wb") as f:
    program.write_to_file(f)

For CUDA, one file that runs on several GPU architectures is a known gap. #17666 describes the request. #22198 proposes support for several GPU architectures in one file, with a PTX fallback. It is open and not merged yet, so it is the thing to watch rather than something you can use now.

To point you at the right path, is CPU-only execution an option for you, or do you need GPU support? If you need GPU, which GPUs and which operating systems?

[gluefox]

Hey, thank you a lot for the in-depth explanation.

The CPU path is definitely interesting and the example seems to be straightforward. Having a arch-agnostic CUDA path is also a huge step forward. However, there will remain a big gap between the legacy TorchScript workflow. We are deploying models to all major OS (including Mac Silicon and Windows / Linux CUDA systems). With libtorch / torchscript one single artifact can be served everywhere (at the cost of some performance, but that is ok in our case).

In summary, I assume the ATen mode would be able to handle all available backends and become TorchScripts successor. If there is more information on it or ways to test this mode we will happily do it!

[Gasoonjia]

@gluefox thanks for your question. Right now we are activately working on native backend (https://github.com/pytorch/executorch/tree/main/backends/native) which will support same-pte cross-os scenerio.
cc. @digantdesai @SS-JIA @metascroy

[digantdesai]

With libtorch / torchscript one single artifact can be served everywhere

We do not currently have a exact substitute for this legacy flow, unfortunately.

That said, we understand the appeal of portable model artifact generated from any torch.nn.Module. To add to what @shoumikhin and @Gasoonjia said, (1) ET ATen mode is not-well-maintained in the repository, its status is KTLO ATM, and (2) even with Native, we would still be going through torch.export i.e. no full torchscript support.

[gluefox]

(2) even with Native, we would still be going through torch.export i.e. no full torchscript support
Going through torch.export would be fine I think.

(1) ET ATen mode is not-well-maintained in the repository, its status is KTLO ATM,
and
Right now we are activately working on native backend
seem contradictory, or are those two different things?

[Gasoonjia]

ATen mode and Native Backend are totally different scenerio.

ET ATen mode is a operator mode that we can directly leverage ATen operator instead of our own portable operator or lowering to delegate for calculation. It was introduced previously to mitigate the early stage operator coverage gap. With the development of ExecuTorch portable library and backend coverage, it is less and less used recently.
ET Native Backend is an incoming first party delegate in ExecuTorch, which tries to unified user experience with single PTE across different operator system.