Add migration guide from torchvision to torchcodec

README.md

@@ -1,12 +1,12 @@
-[**Installation**](#installing-torchcodec) | [**Simple Example**](#using-torchcodec) | [**Detailed Example**](https://meta-pytorch.org/torchcodec/stable/generated_examples/) | [**Documentation**](https://meta-pytorch.org/torchcodec) | [**Contributing**](CONTRIBUTING.md) | [**License**](#license)
+[**Installation**](#installing-torchcodec) | [**Documentation**](https://meta-pytorch.org/torchcodec) | [**Contributing**](CONTRIBUTING.md) | [**License**](#license)
 
 # TorchCodec
 
-TorchCodec is a Python library for decoding video and audio data into PyTorch
-tensors, on CPU and CUDA GPU. It also supports video and audio encoding on CPU!
-It aims to be fast, easy to use, and well integrated
-into the PyTorch ecosystem.  If you want to use PyTorch to train ML models on
-videos and audio, or run inference, TorchCodec is how you turn these into data.
+TorchCodec is a PyTorch-native library for decoding and encoding media: videos,
+audio, and images, on CPU and CUDA GPU. It aims to be fast, easy to
+use, and well integrated into the PyTorch ecosystem. If you want to use PyTorch
+to train ML models on videos, audio, or images, or run inference, TorchCodec is
+how you turn these into tensors, and back.
 
 We achieve these capabilities through:
 
@@ -16,7 +16,7 @@ We achieve these capabilities through:
   installed. FFmpeg is a mature library with broad coverage available on most
   systems. It is, however, not easy to use. TorchCodec abstracts FFmpeg's
   complexity to ensure it is used correctly and efficiently. (FFmpeg is
-  optional, and the image decoders don't need it: see [Installing
+  optional, and the image decoders and encoders don't need it: see [Installing
   TorchCodec](#installing-torchcodec).)
 * Returning data as PyTorch tensors, ready to be fed into PyTorch transforms
   or used directly to train models.
@@ -70,7 +70,7 @@ the `VideoDecoder`:
 ffmpeg -f lavfi -i testsrc2=size=640x400:duration=10:rate=25 /tmp/output_video.mp4
 ```
 
-#### Encoding
+#### Video and Audio Encoding
 
 ```python
 from torchcodec.encoders import Encoder
@@ -90,6 +90,22 @@ with encoder.open_file("output.mp4"):
     # ...
 ```
 
+#### Image Decoding and Encoding
+
+```python
+from torchcodec.decoders import decode_image, decode_jpeg
+from torchcodec.encoders import JpegEncoder
+
+# JPEG, PNG, WebP, GIF, AVIF and HEIC, with the format detected automatically.
+image = decode_image("path/to/image.jpg")  # uint8 tensor of shape [C, H, W]
+
+# Or use the format-specific decoders, e.g. to decode JPEGs on GPU:
+image = decode_jpeg("path/to/image.jpg", device="cuda")
+
+# JPEG and PNG encoding. JPEGEncoder also supports CUDA encoding!
+JpegEncoder(image).to_file("output.jpg")  # also .to_tensor() and .to_file_like()
+```
+
 ## Installing TorchCodec
 
 1. Install FFmpeg, if it's not already installed. TorchCodec supports all major
@@ -109,9 +125,9 @@ with encoder.open_file("output.mp4"):
 
    > **Note:** FFmpeg is an *optional* dependency. It is needed for video
    > and audio decoding and encoding (`VideoDecoder`, `AudioDecoder`,
-   > `VideoEncoder`, `AudioEncoder`, etc.). The image decoders
-   > (`decode_image`, `decode_jpeg`, `decode_png`, etc.)
-   > do **not** require FFmpeg, so if you only need image decoding you can skip
+   > `VideoEncoder`, `AudioEncoder`, etc.). The image decoders and encoders
+   > (`decode_image`, `decode_jpeg`, `JpegEncoder`, `PngEncoder`, etc.)
+   > do **not** require FFmpeg, so if you only need images you can skip
    > this step.
 
 2. Install PyTorch and TorchCodec:
@@ -128,15 +144,15 @@ with encoder.open_file("output.mp4"):
 
 ### CUDA support
 
-CUDA-enabled wheels are installed by default on Linux. For Windows, you'll need
-to pass `--index-url` as described below.
+On CUDA GPUs, TorchCodec supports decoding and encoding of videos and jpeg
+images. CUDA-enabled wheels are installed by default on Linux. For Windows,
+you'll need to pass `--index-url` as described below.
+
 
-Make sure you have a GPU with NVDEC hardware that can decode the format you
-want. Refer to Nvidia's GPU support matrix
+For video, make sure you have a GPU with NVDEC and NVENC hardware that supports
+the formats you want. Refer to Nvidia's GPU support matrix
 [here](https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new).
 
-You will need the `libnvrtc` CUDA library, which is usually part of the CUDA
-Toolkit.
 
 To select a specific CUDA Toolkit version, use `--index-url`. Make sure to
 install the corresponding PyTorch version as well (refer to the
@@ -147,12 +163,16 @@ install the corresponding PyTorch version as well (refer to the
 pip install torch torchcodec --index-url=https://download.pytorch.org/whl/cu130
 ```
 
-Make sure your FFmpeg has NVDEC support:
+Make sure your FFmpeg has NVDEC and NVENC support:
 
 ```bash
 ffmpeg -decoders | grep -i nvidia
 # This should show a line like this:
 # V..... h264_cuvid           Nvidia CUVID H264 decoder (codec h264)
+
+ffmpeg -encoders | grep -i nvidia
+# This should show a line like this:
+# V....D h264_nvenc           NVIDIA NVENC H.264 encoder (codec h264)
 ```
 
 To check that FFmpeg libraries work with NVDEC correctly you can decode a

docs/source/conf.py

@@ -64,9 +64,9 @@ def __init__(self, src_dir):
         self.src_dir = src_dir
 
     def __call__(self, filename):
-        # We have two top-level galleries, one for decoding examples and one for
-        # encoding examples. We define the example order within each gallery
-        # individually.
+        # We have three top-level galleries: decoding examples, encoding
+        # examples, and migration guides. We define the example order within
+        # each gallery individually.
         if "examples/decoding" in self.src_dir:
             order = [
                 "basic_example.py",
@@ -82,14 +82,18 @@ def __call__(self, filename):
                 "transforms.py",
                 "hdr_decoding.py",
             ]
-        else:
-            assert "examples/encoding" in self.src_dir
+        elif "examples/encoding" in self.src_dir:
             order = [
                 "image_encoding.py",
                 "audio_encoding.py",
                 "video_encoding.py",
                 "multi_stream_encoding.py",
             ]
+        else:
+            assert "examples/migration" in self.src_dir
+            order = [
+                "torchvision_migration.py",
+            ]
 
         try:
             return order.index(filename)

docs/source/index.rst

@@ -1,20 +1,21 @@
 Welcome to the TorchCodec documentation!
 ========================================
 
-TorchCodec is a Python library for decoding video and audio data into PyTorch
-tensors, on CPU and CUDA GPU. It also supports audio and video encoding!
-It aims to be fast, easy to use, and well integrated into the PyTorch ecosystem.
-If you want to use PyTorch to train ML models on videos and audio, TorchCodec is
-how you turn these into data.
+TorchCodec is a PyTorch-native library for decoding and encoding media: videos,
+audio, and images, on CPU and CUDA GPU. It aims to be fast, easy to
+use, and well integrated into the PyTorch ecosystem. If you want to use PyTorch
+to train ML models on videos, audio, or images, TorchCodec is how you turn these
+into tensors, and back.
 
 We achieve these capabilities through:
 
 * Pythonic APIs that mirror Python and PyTorch conventions.
-* Relying on `FFmpeg <https://www.ffmpeg.org/>`_ to do the decoding / encoding.
-  TorchCodec uses the version of FFmpeg you already have installed. FFmpeg is a
-  mature library with broad coverage available on most systems. It is, however,
-  not easy to use.  TorchCodec abstracts FFmpeg's complexity to ensure it is
-  used correctly and efficiently.
+* Relying on `FFmpeg <https://www.ffmpeg.org/>`_ to do the video and audio
+  decoding / encoding. TorchCodec uses the version of FFmpeg you already have
+  installed. FFmpeg is a mature library with broad coverage available on most
+  systems. It is, however, not easy to use.  TorchCodec abstracts FFmpeg's
+  complexity to ensure it is used correctly and efficiently. FFmpeg is optional:
+  the image decoders and encoders don't need it.
 * Returning data as PyTorch tensors, ready to be fed into PyTorch transforms
   or used directly to train models.
 
@@ -140,6 +141,19 @@ Encoding
         How to encode audio samples into an audio file
 
 
+Migrating from torchvision
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. grid:: 3
+
+     .. grid-item-card:: :octicon:`file-code;1em`
+        Migrating from torchvision
+        :link: generated_examples/migration/torchvision_migration.html
+        :link-type: url
+
+        How to port ``torchvision.io`` image decoding and encoding code
+
+
 .. toctree::
    :maxdepth: 1
    :hidden:

examples/decoding/image_decoding.py

@@ -21,6 +21,13 @@
 - :func:`~torchcodec.decoders.decode_gif`
 - :func:`~torchcodec.decoders.decode_avif`
 - :func:`~torchcodec.decoders.decode_heic`
+
+.. note::
+
+    These decoders supersede the ones from ``torchvision.io``: they are more
+    robust and support more features. See
+    :ref:`sphx_glr_generated_examples_migration_torchvision_migration.py` for a
+    migration guide.
 """
 
 # %%

examples/encoding/image_encoding.py

@@ -12,6 +12,13 @@
 In this example, we'll learn how to encode an image tensor to JPEG or PNG using
 the :class:`~torchcodec.encoders.JpegEncoder` and
 :class:`~torchcodec.encoders.PngEncoder` classes.
+
+.. note::
+
+    These encoders supersede the ones from ``torchvision.io``: they are more
+    robust and support more features. See
+    :ref:`sphx_glr_generated_examples_migration_torchvision_migration.py` for a
+    migration guide.
 """
 
 # %%

examples/migration/README.rst

@@ -0,0 +1,2 @@
+Migrating from torchvision
+--------------------------

examples/migration/torchvision_migration.py

@@ -0,0 +1,150 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""
+========================================
+Migrating from TorchVision to TorchCodec
+========================================
+
+The image decoders and encoders of ``torchvision.io`` now live in torchcodec.
+This is a short guide to porting your code over. Everything you could do with
+``torchvision.io`` you can do with TorchCodec, usually with a very similar call.
+And TorchCodec supports many more features. To learn more about the image
+decoding and encoding features of TorchCodec, refer to the
+:ref:`image decoding <sphx_glr_generated_examples_decoding_image_decoding.py>`
+and
+:ref:`image encoding <sphx_glr_generated_examples_encoding_image_encoding.py>`
+tutorials.
+
+TL;DR
+-----
+
+- ``decode_image(x)`` -> ``decode_image(x)``, but watch out for the
+  :ref:`changed defaults <decoding_defaults>`
+- ``decode_jpeg(x, device="cuda")`` -> ``decode_jpeg(x, device="cuda")``, same
+  caveat
+- ``read_file(path)`` -> not needed, pass ``path`` to the decoder
+- ``encode_jpeg(img, quality)`` -> ``JpegEncoder(img).to_tensor(quality=...)``
+- ``write_jpeg(img, path, quality)`` -> ``JpegEncoder(img).to_file(path, quality=...)``
+- ``encode_png(img, level)`` -> ``PngEncoder(img).to_tensor(compression_level=...)``
+- ``write_png(img, path, level)`` -> ``PngEncoder(img).to_file(path, compression_level=...)``
+- ``write_file(path, encoded)`` -> not needed, use ``to_file``
+
+The rest of this guide goes over these one by one.
+"""
+
+# %%
+# A bit of boilerplate first: let's make up some encoded image bytes to play
+# with, by encoding a random image.
+import torch
+
+from torchcodec.encoders import JpegEncoder, PngEncoder
+
+raw_image_bytes = JpegEncoder(
+    torch.randint(0, 256, (3, 256, 256), dtype=torch.uint8)
+).to_tensor()
+
+# %%
+# Decoding
+# --------
+#
+# ``torchvision.io.decode_image`` becomes
+# :func:`torchcodec.decoders.decode_image`. Both accept raw encoded bytes, a
+# tensor of encoded bytes, or a path to a file:
+#
+# .. code-block:: python
+#
+#     # Before
+#     from torchvision.io import decode_image
+#     image = decode_image("image.jpg")
+#
+#     # After
+#     from torchcodec.decoders import decode_image
+#     image = decode_image("image.jpg")
+#
+# The format-specific decoders map over one-to-one as well:
+# ``decode_jpeg``, ``decode_png``, ``decode_webp``, ``decode_gif``, and
+# torchcodec adds ``decode_avif`` and ``decode_heic`` without needing the
+# separate ``torchvision-extra-decoders`` package.
+
+from torchcodec.decoders import decode_image
+
+image = decode_image(raw_image_bytes)
+print(f"{image.shape = }, {image.dtype = }")
+
+# %%
+# ``torchvision.io.read_file`` has no equivalent, and you don't need one: pass
+# the path (a ``str`` or a ``pathlib.Path``) straight to the decoder.
+#
+# .. code-block:: python
+#
+#     # Before
+#     from torchvision.io import decode_image, read_file
+#     image = decode_image(read_file("image.jpg"))
+#
+#     # After
+#     from torchcodec.decoders import decode_image
+#     image = decode_image("image.jpg")
+
+# %%
+# .. _decoding_defaults:
+#
+# A few decoding defaults changed
+# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#
+# - ``mode`` now defaults to ``"RGB"`` instead of ``"UNCHANGED"``. If you were
+#   relying on the source's own channel layout, pass ``mode="UNCHANGED"``.
+# - The output is always ``torch.uint8`` by default, even for 16-bit sources.
+#   To get torchvision's behaviour, where the dtype follows the source, pass
+#   ``output_dtype="auto"``.
+# - The ``apply_exif_orientation`` parameter is gone: EXIF orientation is
+#   always applied.
+
+print(f"{decode_image(raw_image_bytes, mode='GRAY').shape = }")
+print(f"{decode_image(raw_image_bytes, output_dtype=torch.uint16).dtype = }")
+
+# %%
+# Encoding
+# --------
+#
+# The encoding functions became classes: instantiate an encoder with the image,
+# then choose where the encoded bytes should go.
+#
+# .. code-block:: python
+#
+#     # Before
+#     from torchvision.io import encode_jpeg, write_jpeg
+#     encoded = encode_jpeg(image, quality=80)  # to a tensor
+#     write_jpeg(image, "image.jpg", quality=80)  # to a file
+#
+#     # After
+#     from torchcodec.encoders import JpegEncoder
+#     encoded = JpegEncoder(image).to_tensor(quality=80)  # to a tensor
+#     JpegEncoder(image).to_file("image.jpg", quality=80)  # to a file
+#
+# PNG works the same way with :class:`~torchcodec.encoders.PngEncoder` and
+# ``compression_level``:
+
+print(f"{JpegEncoder(image).to_tensor(quality=80).shape = }")
+print(f"{PngEncoder(image).to_tensor(compression_level=6).shape = }")
+
+# %%
+# There is no batch equivalent to ``encode_jpeg(list_of_images)``: an encoder
+# takes a single image, so encode a batch with a plain Python loop. You're not
+# losing any speed:
+#
+# .. code-block:: python
+#
+#     encoded = [JpegEncoder(image).to_tensor() for image in images]
+
+# %%
+# Encoders also support a third destination that torchvision didn't have: a
+# file-like object, i.e. anything with ``write`` and ``seek``.
+import io
+
+buffer = io.BytesIO()
+JpegEncoder(image).to_file_like(buffer)
+print(f"{len(buffer.getvalue()) = }")