diff --git a/api-reference-v2/migrate-from-v1.mdx b/api-reference-v2/migrate-from-v1.mdx index b958c9a39..909cf1dbf 100644 --- a/api-reference-v2/migrate-from-v1.mdx +++ b/api-reference-v2/migrate-from-v1.mdx @@ -35,7 +35,7 @@ Most resources keep the same concept but move to a new path. The following table v2 generalizes path parameters: where v1 used resource-specific names such as `{podId}`, `{endpointId}`, `{networkVolumeId}`, `{templateId}`, and `{containerRegistryAuthId}`, v2 uses a single generic `{id}` parameter across resources. - Billing paths don't map by name. In v1, `/billing/endpoints` returns Serverless billing history; in v2, Serverless billing moves to `/v2/billing/serverless`. The v2 path `/v2/billing/endpoints` is a different, new resource — Public Endpoint billing history — so update your Serverless billing calls to the new path rather than assuming the old one carries over. + Billing paths don't map by name. In v1, `/billing/endpoints` returns Serverless billing history; in v2, Serverless billing moves to `/v2/billing/serverless`. The v2 path `/v2/billing/endpoints` is a different, new resource: Public Endpoint billing history. Update your Serverless billing calls to the new path rather than assuming the old one carries over. ## Consolidated Pod lifecycle @@ -114,7 +114,7 @@ The v2 wrapper key matches the resource: `GET /v2/pods` returns `{"pods":[...]}` In v1, errors return a simple `{"message":"..."}` object. In v2, errors follow the RFC 9457 problem format with required `title`, `status`, and `detail` fields, plus an optional `errors` array of validation strings. -As in v1, Runpod returns a `403` when a valid API key lacks access to the requested resource—but in v2 that response now uses the problem format shown here. +As in v1, Runpod returns a `403` when a valid API key lacks access to the requested resource. In v2, that response now uses the problem format shown here. @@ -136,7 +136,7 @@ As in v1, Runpod returns a `403` when a valid API key lacks access to the reques ## New in v2 -- The v2 API adds capabilities that have no v1 equivalent. +The v2 API adds capabilities that have no v1 equivalent. - Catalog endpoints let you browse available compute without provisioning it: `GET /v2/catalog/gpus` and `/gpus/{id}`, `/cpus` and `/cpus/{id}`, and `/datacenters` and `/datacenters/{id}`. See [List GPU types](/api-reference-v2/catalog/list-gpu-types). diff --git a/flash/apps/build-app.mdx b/flash/apps/build-app.mdx index dfac947d4..1ae0982d0 100644 --- a/flash/apps/build-app.mdx +++ b/flash/apps/build-app.mdx @@ -119,7 +119,7 @@ Besides starting the API server, `flash dev` also starts an interactive API expl To run endpoint functions in the explorer: 1. Expand one of the functions under **GPU Workers** or **CPU Workers**. -2. Click **Try it out** and then **Execute**. +2. Select **Try it out**, then **Execute**. You'll get a response from your workers right in the explorer. diff --git a/flash/cli/build.mdx b/flash/cli/build.mdx index fd9972dde..199e221a0 100644 --- a/flash/cli/build.mdx +++ b/flash/cli/build.mdx @@ -80,7 +80,7 @@ If you use other environment file variants like `.env.dev` or `.env.staging`, ad Local (non-pip) modules that your endpoints import are bundled automatically, provided they pass the ignore filter: your `.gitignore` plus the built-in patterns listed above. During the build, Flash resolves the transitive local-import closure of every shipped Python file and checks it against the files the ignore filter already selected. -If shipped code imports a local module that an ignore rule excludes (for example, a `test_*.py` sibling or a file under `tests/`), the build fails with a `LocalModuleResolutionError` instead of silently overriding your ignore rules or shipping a broken artifact. Flash names each excluded file and the file that imports it: +If shipped code imports a local module that an ignore rule excludes (for example, a `test_*.py` sibling or a file under `tests/`), the build fails with a `LocalModuleResolutionError`. Failing early is deliberate: Flash won't silently override your ignore rules or ship a broken artifact. The error names each excluded file and the file that imports it: ``` Shipped code imports local modules that the build ignore rules (.gitignore or built-in defaults) exclude: @@ -89,7 +89,7 @@ Shipped code imports local modules that the build ignore rules (.gitignore or bu Shipping them would silently override a deliberate exclusion, and omitting them would break the worker with ModuleNotFoundError. Remove the matching ignore pattern or stop importing these modules from shipped code. ``` -If an `@Endpoint` file has a local import Flash can't resolve at all (a broken relative import, or a file outside your project root), the build also fails with a clear error. A file that fails resolution but doesn't define an endpoint is skipped with a warning and the build continues. For details on how local imports are resolved and bundled, see [Import local modules](/flash/create-endpoints#import-local-modules). +If an `@Endpoint` file has a local import Flash can't resolve at all (for example, a broken relative import or a file outside your project root), the build fails with a clear error. If a file that doesn't define an endpoint fails resolution, Flash skips it with a warning and continues the build. For details on how local imports are resolved and bundled, see [Import local modules](/flash/create-endpoints#import-local-modules). ## Build artifacts @@ -208,6 +208,6 @@ ls .flash/.build/ -Most users should use `flash deploy` instead, which runs build and deploy in one step. Use `flash build` when you need more control or want to inspect the artifact. +In most cases, use `flash deploy` instead, which runs build and deploy in one step. Use `flash build` when you need more control or want to inspect the artifact. diff --git a/flash/create-endpoints.mdx b/flash/create-endpoints.mdx index b2ce31eae..3672e1085 100644 --- a/flash/create-endpoints.mdx +++ b/flash/create-endpoints.mdx @@ -244,13 +244,13 @@ async def process_video(video_data): ## Import local modules -Your endpoint can import local (non-pip) Python modules that live alongside it in your project, such as a sibling `utils.py` file or a `helpers/` package. Flash detects these imports, follows them transitively, and ships the module source to the worker for you, so an import like `import utils` or `from helpers import load` works remotely with no extra configuration. +Your endpoint can import local (non-pip) Python modules that live alongside it in your project, such as a sibling `utils.py` file or a `helpers/` package. Flash detects these imports, follows them transitively, and ships the module source to the worker for you. Imports like `import utils` or `from helpers import load` work remotely with no extra configuration. Flash resolves local imports whether they appear at the top of the file or inside the function body, and it supports absolute imports (`import utils`), relative imports (`from . import helpers`), and dynamic imports with a literal name (`importlib.import_module("plugin")`). It also pulls in the `__init__.py` files for any packages you import. Flash can't resolve dynamic imports whose module name is computed at runtime, so it emits a warning, and you're responsible for making those modules available on the worker. -Flash bundles only local project files. Standard library modules are already present in the worker image, and pip packages must still be declared through the `dependencies` parameter. This applies transitively: if a bundled local module imports a pip package at its top level, that package must still be declared in the `dependencies` of any endpoint that uses the module. +Flash bundles only local project files. Standard library modules are already present in the worker image, and pip packages must still be declared through the `dependencies` parameter. This applies transitively. If a bundled local module imports a pip package at its top level, declare that package in the `dependencies` of any endpoint that uses the module. -On `flash build` and `flash deploy`, local modules are bundled when they pass the ignore filter, and importing a local module that an ignore rule excludes (or one Flash can't resolve) fails the build. See [Local modules and the ignore filter](/flash/cli/build#local-modules-and-the-ignore-filter) for details. +On `flash build` and `flash deploy`, Flash bundles local modules that pass the ignore filter. Importing a local module that an ignore rule excludes (or one Flash can't resolve) fails the build. See [Local modules and the ignore filter](/flash/cli/build#local-modules-and-the-ignore-filter) for details. ### Live execution size limit diff --git a/flash/overview.mdx b/flash/overview.mdx index 887582a87..f8380ddd5 100644 --- a/flash/overview.mdx +++ b/flash/overview.mdx @@ -50,7 +50,7 @@ Flash requires a Runpod account with a verified email address. ### Install Flash -Flash requires [Python 3.10, 3.11, 3.12, or 3.13](https://www.python.org/downloads/) and runs natively on macOS and Linux. Windows users can run Flash through [WSL2](/flash/windows-wsl2). +Flash requires [Python 3.10, 3.11, 3.12, or 3.13](https://www.python.org/downloads/) and runs natively on macOS and Linux. On Windows, run Flash through [WSL2](/flash/windows-wsl2). Install Flash using `pip` or `uv`: @@ -104,7 +104,7 @@ flash --help ## Limitations -- Flash runs natively on macOS and Linux. Windows users can run Flash through [WSL2](/flash/windows-wsl2). +- Flash runs natively on macOS and Linux. On Windows, run Flash through [WSL2](/flash/windows-wsl2). - CPU endpoints are restricted to the `EU-RO-1` datacenter. GPU endpoints can deploy to [multiple datacenters](/flash/configuration/parameters#datacenter). - Flash can rapidly scale workers across multiple endpoints, and you may hit your maximum worker threshold quickly. Contact [Runpod support](https://www.runpod.io/contact) to increase your account's capacity if needed. diff --git a/flash/quickstart.mdx b/flash/quickstart.mdx index 21124cb73..e479be3f7 100644 --- a/flash/quickstart.mdx +++ b/flash/quickstart.mdx @@ -16,7 +16,7 @@ This quickstart gets you running GPU workloads on Runpod in minutes. You'll exec ## Step 1: Install Flash -Flash runs natively on macOS and Linux. Windows users can run Flash through [WSL2](/flash/windows-wsl2). +Flash runs natively on macOS and Linux. On Windows, run Flash through [WSL2](/flash/windows-wsl2). Create a virtual environment and install Flash using [uv](https://docs.astral.sh/uv/): diff --git a/flash/troubleshooting.mdx b/flash/troubleshooting.mdx index cb6da5a08..d2bcc005c 100644 --- a/flash/troubleshooting.mdx +++ b/flash/troubleshooting.mdx @@ -35,7 +35,7 @@ Available levels: `DEBUG`, `INFO`, `WARNING`, `ERROR`. View detailed metrics and logs in the [Runpod console](https://console.runpod.io/serverless): 1. Navigate to the **Serverless** section. -2. Click on your endpoint to view: +2. Select your endpoint to view: - Active workers and queue depth. - Request history and job status. - Worker logs and execution details. @@ -48,7 +48,7 @@ Access detailed logs for specific workers: 1. Go to the [Serverless console](https://console.runpod.io/serverless). 2. Select your endpoint. -3. Click on a worker to view its logs. +3. Select a worker to view its logs. Logs include dependency installation output, function execution output (print statements, errors), and system-level messages. diff --git a/instant-clusters/ray-vllm.mdx b/instant-clusters/ray-vllm.mdx index 29f46c897..d4c3496c0 100644 --- a/instant-clusters/ray-vllm.mdx +++ b/instant-clusters/ray-vllm.mdx @@ -5,7 +5,7 @@ description: "Run distributed inference across multiple nodes using Ray and vLLM tag: BETA --- -This tutorial shows how to use Instant Clusters with Ray to run distributed inference on large language models. By combining Ray's cluster management with vLLM's tensor and pipeline parallelism, you can serve models that exceed the memory of a single node — for example, a 70B parameter model across multiple 8×H100 pods. +This tutorial shows how to use Instant Clusters with Ray to run distributed inference on large language models. By combining Ray's cluster management with vLLM's tensor and pipeline parallelism, you can serve models that exceed the memory of a single node. For example, you can serve a 70B parameter model across multiple 8×H100 Pods. Ray handles the cluster topology; vLLM uses it to split the model across GPUs both within each node (tensor parallelism) and across nodes (pipeline parallelism). @@ -17,8 +17,8 @@ Ray handles the cluster topology; vLLM uses it to split the model across GPUs bo ## Requirements -- A Runpod account with sufficient credits for a multi-node cluster -- Basic familiarity with large language model inference and distributed GPU setups +- A Runpod account with sufficient credits for a multi-node cluster. +- Basic familiarity with large language model inference and distributed GPU setups. --- @@ -26,10 +26,10 @@ Ray handles the cluster topology; vLLM uses it to split the model across GPUs bo 1. Open the [Instant Clusters page](https://console.runpod.io/instant-clusters). 2. Click **Create Cluster**. -3. Name your cluster and configure it. For this walkthrough, set **Pod Count** to **2** and select **8× H100 SXM GPUs** per pod. Use the **Runpod PyTorch** template as your base image. +3. Name your cluster and configure it. For this walkthrough, set **Pod Count** to **2** and select **8× H100 SXM GPUs** per Pod. Use the **Runpod PyTorch** template as your base image. - Increase `/dev/shm` when configuring your pod. The default (64 MB) is too small for large tensor-parallel workloads. Set it to at least 8 GB. In the pod configuration, add the environment variable `MALLOC_ARENA_MAX=1` and set `--shm-size` to `8g` in your Docker run options. + Increase `/dev/shm` when configuring your Pod. The default (64 MB) is too small for large tensor-parallel workloads. Set it to at least 8 GB. In the Pod configuration, add the environment variable `MALLOC_ARENA_MAX=1` and set `--shm-size` to `8g` in your Docker run options. 4. Click **Deploy Cluster**. You are redirected to the Instant Clusters page. @@ -38,9 +38,9 @@ Ray handles the cluster topology; vLLM uses it to split the model across GPUs bo ## Step 2: Start the Ray head on pod-0 -The first pod (`CLUSTERNAME-pod-0`) runs the Ray head node. All other pods connect to it as workers. +The first Pod (`CLUSTERNAME-pod-0`) runs the Ray head node. All other Pods connect to it as workers. -1. Click your cluster to expand the pod list. +1. Click your cluster to expand the Pod list. 2. Click **CLUSTERNAME-pod-0**, then click **Connect → Web Terminal**. 3. In the terminal, clone the reference scripts: @@ -69,16 +69,16 @@ The first pod (`CLUSTERNAME-pod-0`) runs the Ray head node. All other pods conne ``` - `RAY_NODE_IP_ADDRESS` and `VLLM_HOST_IP` must be set to the pod's internal network IP — not `0.0.0.0`. Setting them prevents Ray and vLLM from binding to the wrong interface on multi-NIC pods. + `RAY_NODE_IP_ADDRESS` and `VLLM_HOST_IP` must be set to the Pod's internal network IP, not `0.0.0.0`. Setting them prevents Ray and vLLM from binding to the wrong interface on multi-NIC Pods. --- -## Step 3: Join the worker pods to the cluster +## Step 3: Join the worker Pods to the cluster -Repeat this for each remaining pod in the cluster (`pod-1`, `pod-2`, …). +Repeat this for each remaining Pod in the cluster (`pod-1`, `pod-2`, …). -1. In the Instant Clusters page, click the next pod and open its **Web Terminal**. +1. In the Instant Clusters page, click the next Pod and open its **Web Terminal**. 2. Clone the same scripts: ```bash @@ -111,7 +111,7 @@ Repeat this for each remaining pod in the cluster (`pod-1`, `pod-2`, …). --num-gpus=$NUM_TRAINERS ``` - `$MASTER_ADDR` is injected automatically by Runpod into all pods in the cluster — it resolves to `pod-0`. + Runpod injects `$MASTER_ADDR` automatically into all Pods in the cluster. It resolves to `pod-0`. --- @@ -123,7 +123,7 @@ Run this on `pod-0` to confirm all nodes have joined: ray status ``` -Expected output for a two-pod cluster with 8 GPUs each: +Expected output for a two-Pod cluster with 8 GPUs each: ``` ======== Autoscaler status: ... ======== @@ -198,37 +198,37 @@ When you are done, return to the [Instant Clusters page](https://console.runpod. ## Environment variables reference -Runpod injects these environment variables into every pod in the cluster. The startup scripts rely on them. +Runpod injects these environment variables into every Pod in the cluster. The startup scripts rely on them. | Variable | Description | |---|---| | `MASTER_ADDR` | Hostname of `pod-0`, the Ray head node | | `MASTER_PORT` | Port for inter-node communication (default: `29500`) | -| `NUM_NODES` | Total number of pods in the cluster | -| `NUM_TRAINERS` | Number of GPUs per pod | -| `NODE_RANK` | Index of this pod (`0` for head, `1+` for workers) | +| `NUM_NODES` | Total number of Pods in the cluster | +| `NUM_TRAINERS` | Number of GPUs per Pod | +| `NODE_RANK` | Index of this Pod (`0` for head, `1+` for workers) | --- ## Common issues **Ray workers don't join** -Confirm `$MASTER_ADDR` resolves from each worker pod. Run `ping $MASTER_ADDR` in a worker terminal. If it fails, the cluster network may still be initializing — wait 30 seconds and try again. +Confirm `$MASTER_ADDR` resolves from each worker Pod. Run `ping $MASTER_ADDR` in a worker terminal. If it fails, the cluster network may still be initializing. Wait 30 seconds and try again. **vLLM OOM during model load** -Check that `/dev/shm` is large enough (at least 8 GB for 70B models). Also verify that `--tensor-parallel-size` matches the number of GPUs per node — a mismatch causes uneven shard sizes. +Check that `/dev/shm` is large enough (at least 8 GB for 70B models). Also verify that `--tensor-parallel-size` matches the number of GPUs per node. A mismatch causes uneven shard sizes. **`VLLM_HOST_IP` binding error** -This error occurs when vLLM tries to bind to `0.0.0.0` on a pod with multiple network interfaces. Make sure `VLLM_HOST_IP` is set to the internal IP (`hostname -I | awk '{print $1}'`) before starting the server. +This error occurs when vLLM tries to bind to `0.0.0.0` on a Pod with multiple network interfaces. Make sure `VLLM_HOST_IP` is set to the internal IP (`hostname -I | awk '{print $1}'`) before starting the server. **Stale Ray cluster after restart** -If you restart a pod, Ray does not automatically rejoin the cluster. Rerun `head.sh` on `pod-0` first, then `worker.sh` on all other pods. +If you restart a Pod, Ray does not automatically rejoin the cluster. Rerun `head.sh` on `pod-0` first, then `worker.sh` on all other Pods. --- ## Next steps - Adapt the serve script to load your own model from a [GlobalStore](/storage/globalstore) or [Network Volume](/storage/network-volumes) mount. -- Scale up by increasing the pod count and adjusting `--pipeline-parallel-size` accordingly. +- Scale up by increasing the Pod count and adjusting `--pipeline-parallel-size` accordingly. - Try [Axolotl on an Instant Cluster](/instant-clusters/axolotl) for distributed fine-tuning. - Review the [Instant Cluster configuration reference](/instant-clusters/configuration) for full details on environment variables and networking. diff --git a/release-notes.mdx b/release-notes.mdx index 568b7ad70..966e120d7 100644 --- a/release-notes.mdx +++ b/release-notes.mdx @@ -11,7 +11,7 @@ rss: true **August 18, 2026** -

New Release [REST API v2](/api-reference-v2/overview)

REST API v2 is now generally available. v2 moves to a new base URL (`https://api.runpod.io/v2`), reorganizes resource paths, standardizes request and response shapes, and adds new capabilities including catalog endpoints, pod log streaming, and Serverless observability. See the [migration guide](/api-reference-v2/migrate-from-v1) to move your existing integrations. +

New Release [REST API v2](/api-reference-v2/overview)

REST API v2 is now generally available. v2 moves to a new base URL (`https://api.runpod.io/v2`), reorganizes resource paths, and standardizes request and response shapes. It also adds new capabilities including catalog endpoints, Pod log streaming, and Serverless observability. See the [migration guide](/api-reference-v2/migrate-from-v1) to move your existing integrations.

Deprecation REST API v1

REST API v1 will be retired on **November 15, 2026**. Migrate your integrations to [REST API v2](/api-reference-v2/overview) before that date. See the [migration guide](/api-reference-v2/migrate-from-v1) to get started. diff --git a/serverless/overview.mdx b/serverless/overview.mdx index 2b93b0359..c56539097 100644 --- a/serverless/overview.mdx +++ b/serverless/overview.mdx @@ -119,7 +119,7 @@ When using load balancing endpoints, you can define your own custom API endpoint ### Fitness checks and preflight checks -Validate your worker's environment at startup before it takes traffic. Fitness checks, also known as preflight checks, run registered checks in order before a worker begins processing jobs, catching issues like missing GPUs, unloaded models, or bad configuration before any request reaches your worker. +Validate your worker's environment at startup before it takes traffic. Fitness checks (also known as preflight checks) run registered checks in order before a worker begins processing jobs. This catches issues like missing GPUs, unloaded models, or bad configuration before any request reaches your worker. [Learn more about fitness checks and preflight checks](/serverless/development/fitness-checks)