High-performance distributed training for LLMs — RL, SFT, MoE, and beyond.

🚀 Installation · ⚡ Quick Start · 📚 Documentation

---

🔍 Overview

XoRL is a distributed training framework designed for large language models with composable parallelism and flexible training modes.

The XoRL stack consists of three repos:

Repo	Description
xorl	Distributed training framework — local SFT/pretraining and server-mode RL training
xorl-client	Lightweight Python SDK for driving the xorl training server (forward/backward, optimizer steps, checkpointing, sampling)
xorl-sglang	Fork of SGLang with weight-sync APIs, MoE routing export, and numerical alignment for online RL

Two training modes:

• Local — torchrun-based training for offline SFT and pretraining
• Server — REST API-driven training for online RL loops where xorl-client drives the training loop and xorl-sglang serves inference

Parallelism strategies — compose across validated combinations; some adapter, virtual-pipeline, and weight-sync combinations have explicit restrictions:

Strategy	Description
FSDP2	Fully sharded data parallelism (PyTorch native)
Tensor Parallel	Column/row weight sharding across GPUs
Pipeline Parallel	Six schedules, including 1F1B, interleaved 1F1B, zero-bubble, and V-style variants
Context Parallel	Ring attention + Ulysses sequence parallel
Expert Parallel	MoE expert sharding via DeepEP

Fine-tuning methods — full weights, LoRA, and QLoRA (int4/nvfp4/block_fp8), all FSDP2-compatible.

---

🚀 Installation

git clone --recurse-submodules git@github.com:togethercomputer/xorl.git
cd xorl

Already cloned without --recurse-submodules? Run git submodule update --init --recursive

Option A: uv (recommended)

uv sync
source .venv/bin/activate

Option B: conda

conda create -n xorl python=3.12
conda activate xorl
pip install -e .

Submodules

The repo includes two git submodules under submodules/ (needed for server / online RL training):

• xorl-client — Lightweight Python SDK (no PyTorch dependency) for driving the xorl training server. Provides ServiceClient, TrainingClient, SamplingClient, and RestClient with async-first APIFuture semantics, automatic request ordering, and Tinker API compatibility.
• xorl-sglang — XoRL's fork of SGLang with NCCL and P2P weight sync, MoE route export (R3), and architecture-resolved numerical programs for online RL. The pinned revision does not include the sparse-delta receiver.

The default install already includes xorl-client from its public repository. To develop the client submodule in place, install its editable checkout:

pip install -e submodules/xorl-client

Do not install the xorl-sglang submodule into the default PyTorch 2.12 environment. For a single environment containing XoRL, xorl-client, and xorl-sglang, use the alternate pyproject.sglang.toml profile, which pins the compatible PyTorch 2.11/CUDA 13 stack:

uv:

cp pyproject.sglang.toml pyproject.toml
UV_PROJECT_ENVIRONMENT=.venv-sglang uv sync
source .venv-sglang/bin/activate

conda:

conda create -n xorl-sglang python=3.12
conda activate xorl-sglang
cp pyproject.sglang.toml pyproject.toml
pip install -e . -e "submodules/xorl-sglang/python[all]"

Note: Copying the alternate manifest replaces the tracked pyproject.toml; uv sync also generates the ignored local uv.lock for this profile. Do this in a clean checkout, restore pyproject.toml, and do not add the generated lock with unrelated changes. The separate .venv-sglang keeps this profile isolated from the default .venv. The default profile uses PyTorch 2.12.1/CUDA 13.2 and Triton 3.7.1, while the combined profile uses PyTorch 2.11/CUDA 13 and Triton 3.6.0. Both use FlashAttention 4.

See the installation guide for full setup including optional dependencies (DeepEP, Flash Attention).

⚡ Quick Start

# Local training on 8 GPUs
torchrun --nproc_per_node=8 -m xorl.cli.train examples/local/dummy/configs/full/qwen3_8b.yaml

See the quick start guide for more examples including MoE, server training, and LoRA.

---

📚 Documentation

Topic	Link
Parallelism	Overview
MoE & DeepEP	MoE docs
LoRA / QLoRA	Adapters
Server training	Server docs
Config reference	Local · Server

---

🧠 Supported Models

Model	Type	HuggingFace ID
Qwen3	Dense	Qwen/Qwen3-8B, Qwen/Qwen3-32B, ...
Qwen3-MoE	Mixture-of-Experts	Qwen/Qwen3-30B-A3B, Qwen/Qwen3-235B-A22B, ...
Qwen3.5	Dense	Qwen/Qwen3.5-7B, ...
Qwen3.5-MoE	Mixture-of-Experts	Qwen/Qwen3.5-35B-A3B, Qwen/Qwen3.5-397B-A17B, ...
GLM-5	Mixture-of-Experts	zai-org/GLM-5.2-FP8
DeepSeek V4	Hybrid-attention MoE	deepseek-ai/DeepSeek-V4-Flash

Models are loaded directly from HuggingFace checkpoints — no preprocessing needed. See the supported models page for details.

---

🤝 Contributing

See CONTRIBUTING.md for development setup, coding conventions, and how to run tests.