English | 简体中文

Reconciling Parallel Decoding with Precise Visual Grounding at Flash Speed

Demo Video · Quick Start · Documentation · Citation

📰 News

• 2026-10-14: GroundAnything and GroundAnything-VLM model weights are now available on ModelScope.
• 2026-10-14: GroundAnything received strong community recognition, with 74 upvotes on Hugging Face Daily Papers.
• 2026-10-04: Released the project webpage.
• 2026-10-03: Released the source code, deployment and batch-annotation guides, and full-suite evaluation workflows.
• 2026-10-01: We released the GroundAnything and GroundAnything-VLM model weights on Hugging Face.
• 2026-09-30: The GroundAnything paper is available on arXiv.

🧭 Contents

Highlights · Demo · Models · Installation · Deployment Options · Quick Start · vLLM Deployment · Batch Annotation · Tasks and Output Format · Method and Inference Infrastructure · Evaluation · Training · Results · Documentation · License · Citation · Acknowledgement

✨ Highlights

• Unlocking diffusion for broad visual grounding. We introduce GroundAnything, a 4B foundation model that unifies diverse grounding tasks with precise localization and parallel decoding, trained through grounding pretraining, AR-to-diffusion conversion with joint objectives, supervised fine-tuning, and GRPO-based reinforcement learning.
• State-of-the-art grounding at 4B scale. Across 30 benchmarks, GroundAnything surpasses the prior overall state of the art among similarly sized AR models and outperforms the larger Qwen3.7-Max. GroundAnything-VLM establishes a new overall state of the art at this scale and remains competitive with GPT-6 Astra.
• Systematic acceleration studies. We analyze decoding strategies in detail, compare with MTP-based generation, and evaluate progressive infrastructure optimizations for practical deployment.

🎬 Demo

▶ Watch the demo

▶ Parallel Decoding

🤗 Models

Checkpoint	Generation	Download	Evaluation mode
GroundAnything	Entropy-guided diffusion; optional self-speculation	DLM weights	GAM
GroundAnything-VLM	Autoregressive generation	VLM weights	GAM

GroundAnything's main results use entropy-guided decoding. GroundAnything-VLM is a separate autoregressive checkpoint. Both use the GAM evaluation mode.

🛠️ Installation

The serving workflows below target Linux x86_64 and Python 3.12. The serving setup installs the bundled custom SGLang engine and its pinned dependencies. The supplied serving profile uses Torch 2.9.1, Transformers 5.5.4, Triton 3.5.1, sgl-kernel 0.3.20, and FlashInfer 0.5.3.

git clone https://github.com/groundingpi/GroundAnything.git
cd GroundAnything
python3 -m pip install -r requirements.txt huggingface_hub

Already have a source checkout? Start with cd GroundAnything. Run subsequent commands from the repository root.

requirements.txt installs the lightweight HTTP client, visualization tools, and setup dependencies. Serving, training, and evaluation each use their own environment; pip install -r requirements.txt alone does not install the model runtime. The client does not load weights and requires no Torch installation.

Tested accelerators: NVIDIA B300, B200, H200, H800, and PPU. Use the matching runtime for each accelerator. See Environment Setup for installation details.

🧩 Deployment Options

Model	Backend	Use case	Guide
GroundAnything (DLM)	Custom SGLang	Entropy-guided parallel or self-speculative decoding	DLM quick start
GroundAnything-VLM	Custom SGLang	Autoregressive grounding with the existing service	VLM quick start
GroundAnything-VLM	vLLM + Transformers backend	An alternative autoregressive serving route	GPU / PPU deployment

Use the repository's patched engines and launchers. The new vLLM adapter is for GroundAnything-VLM only and has CPU-level checks; end-to-end accelerator validation is pending. DLM decoding stays on SGLang.

All routes expose an OpenAI-compatible image + text API and produce structured visual grounding. Prompts cover referring expressions, object localization, text-region grounding, document layout, and point localization. See Tasks and Output Format for the prompt and coordinate contract. For processing an image collection, start with Batch Annotation.

🚀 Quick Start

⚡ GroundAnything: entropy-guided decoding

Download the complete DLM release and start the service:

hf download GroundingPI/GroundAnything --local-dir weights/dlm_bundle
python3 run.py setup serve
python3 run.py serve --decoder denoise

The published checkpoint is already a complete model bundle, so it does not require prepare-model. The supplied SGLang installer is a CUDA/GPU recipe; PPU requires its matching platform runtime.

🚀 GroundAnything: self-speculative decoding

Stop the existing DLM service, then reuse the same weights with:

python3 run.py serve --decoder speculative

🔁 GroundAnything-VLM: autoregressive decoding

Download the separate autoregressive checkpoint. If the serving environment has not yet been prepared, run python3 run.py setup serve first.

hf download GroundingPI/GroundAnything-VLM --local-dir weights/vlm
python3 run.py serve --config configs/release/vlm_sglang.yaml

Service	Base URL	Model ID
GroundAnything, either DLM decoder	http://127.0.0.1:8101/v1	groundinganything
GroundAnything-VLM	http://127.0.0.1:8102/v1	groundinganything-vlm

🎯 Run a prediction

Keep the service running. In a second terminal, use the environment where you installed requirements.txt and replace your_image.jpg with your image:

from grounding_anything import GroundingAnything, visualize

client = GroundingAnything(
    base_url="http://127.0.0.1:8101/v1",
    model="groundinganything",
)

# Referring-expression grounding
result = client.predict("your_image.jpg", "the red car", task="bbox")
print(result.to_dict())
if result.valid:
    visualize("your_image.jpg", result).save("prediction.png")

# Point localization
point = client.predict(
    "your_image.jpg", "the center of the red car", task="point"
)
print(point.to_dict())

For GroundAnything-VLM, use the same client with base_url="http://127.0.0.1:8102/v1" and model="groundinganything-vlm".

The command-line example saves result.json and, for valid output, prediction.png. Choose a new output directory for each run:

python3 examples/predict.py \
  --image your_image.jpg --phrase "the red car" \
  --task bbox --output outputs/car

For the VLM service, append --base-url http://127.0.0.1:8102/v1 --model groundinganything-vlm.

Client parameters and return values

Interface	Parameters
GroundingAnything(...)	base_url, model, optional api_key, timeout (default: 120 seconds)
predict(...)	Image path (JPEG, PNG, WebP), referring phrase, task="bbox" or "point", max_tokens (default: 4096)
visualize(...)	Image path and a valid result; returns a Pillow image

result.to_dict() contains task, predictions, raw_output, finish_reason, usage, parse_error, and valid. Parsed predictions contain labels and coordinates on a 0–999 grid. The visualizer converts them to image pixels. Truncated or malformed responses have valid=False and retain their raw output for inspection.

See Examples and the client implementation for more usage details.

⚡ vLLM Deployment

GroundAnything-VLM also has an opt-in vLLM Transformers-backend adapter for autoregressive grounding. Its dedicated .venv-vllm environment uses Linux x86_64 / Python 3.12, a preinstalled accelerator-compatible Torch / vLLM 0.18.x runtime, and the bundled Transformers 5.7.0 fork. The default SGLang environment remains separate.

For NVIDIA GPUs, run from the repository root:

hf download GroundingPI/GroundAnything-VLM --local-dir weights/vlm
python3 run.py setup vllm --platform gpu
python3 run.py serve --engine vllm --platform gpu

For PPU, run inside the matching vendor runtime image and replace gpu with ppu in both setup and serving commands. Skip setup if .venv-vllm is already prepared. Use the same --venv on both commands to select another fresh environment.

The API is http://127.0.0.1:8102/v1, with model ID groundinganything-vlm. Stop an existing SGLang VLM service on that port before starting vLLM, or choose a different port in the vLLM release configuration. Check readiness in another terminal:

curl --fail http://127.0.0.1:8102/v1/models

Defaults are BF16, eager execution, TP=1, one active sequence, 16,384 context tokens, and one image per request. Custom requests must include skip_special_tokens: false and spaces_between_special_tokens: false. Evaluate with GAM mode using configs/eval/vlm_vllm.yaml.

GroundAnything's DLM entropy-guided and self-speculative decoders use the custom SGLang backend; the vLLM route accepts only the separate VLM checkpoint. This new adapter has CPU-level checks; end-to-end accelerator validation is pending. See the vLLM Deployment Guide for GPU/PPU setup, a complete image request, and configuration details.

🗂️ Batch Annotation

Use the JSONL batch guide to annotate image collections with per-image grounding prompts. The batch example saves raw responses, parsed coordinates, completion status, and usage; successful items can be skipped when resuming the same inputs and request configuration.

With a service running and your input manifest prepared:

python3 examples/batch_predict.py \
  --input requests.jsonl --output predictions.jsonl \
  --base-url http://127.0.0.1:8101/v1 --model groundinganything

The supplied serving profiles use one active sequence. Batch processing here means sequential image requests with durable output. For multiple accelerators, split the manifest across independent service replicas and use a separate output file per worker. Inspect truncated or invalid results before using predictions as annotations.

🎯 Tasks and Output Format

The models accept one image and a text instruction. The convenience client's predict() method constructs referring-box and referring-point prompts. For other tasks, send the corresponding prompt to the service's OpenAI-compatible /v1/chat/completions endpoint.

Task	Prompt
Object / dense grounding	Locate all the instances that match the following categories: car</c>person.
Referring boxes	Locate the target referred to by the following description: the red car.
Object points	Point to: car</c>person.
Referring points	Point to the target referred to by the following description: the red car.
OCR	OCR task detect all the text in box format.
Document layout	Detect all document layout elements that match the following categories: title</c>text.
GUI grounding	Point to the UI element to click for the following instruction: open the settings menu.
Visual prompting	Provide reference boxes in the native spatial-token format, then request similar objects.

Visual-prompt example

Given reference boxes <|box_start|><100><200><500><650><|box_end|> indicating one or more objects, find all similar objects in the image and output their bounding boxes.

All released checkpoints use the GAM spatial-token protocol, with integer coordinates from 0 to 999. Boxes contain (x1, y1, x2, y2); points contain (x, y). A missing target is represented by None.

<|object_ref_start|>car<|object_ref_end|><|box_start|><100><200><500><650><|box_end|>
<|object_ref_start|>car center<|object_ref_end|><|box_start|><300><425><|box_end|>
<|object_ref_start|>absent object<|object_ref_end|><|box_start|>None<|box_end|>

Use the checkpoint's tokenizer, processor, and chat template. Custom HTTP requests should keep skip_special_tokens=false, preserve adjacent coordinate tokens without added spaces, and send the image as a base64 data URI in an image_url content part alongside the text prompt.

⚙️ Method and Inference Infrastructure

The variants share a MoonViT-V2 (Kimi K3) visual encoder, a multimodal projector, and a Qwen3-4B language architecture.

⚡ Entropy-guided decoding

The release recipe uses block size 32, sub-block size 4, and entropy threshold 0.8. Each physical block contains a known anchor and 31 masked positions.

After image/query prefill, GroundAnything denoises response blocks with bidirectional attention and caches the completed prefix. Sub-blocks are processed from left to right. Our default Entropy-Guided Decoding commits masked positions with $H_j\leq\tau$, where $H_j=-\sum_v p_j(v)\log p_j(v)$ is the entropy of the unmodified token distribution. If none qualifies, the lowest-entropy position is committed to ensure progress. Committed tokens remain fixed, and a causal pass constructs the completed block's cache without AR verification.

🚀 Self-speculative decoding

The shared weights also support diffusion drafting with causal verification, accepting the longest matching prefix. Verification accepts only the longest consecutive draft prefix agreeing with the causal predictions. The first disagreement ends acceptance; later coincidental matches are discarded. Rejected suffix states are removed from the cache.

Self-speculation verifies against the converted model's causal branch. Exact greedy verification preserves that branch's outputs, not necessarily the outputs of the separately trained GroundAnything-VLM.

The documented --decoder speculative route uses linear drafting and greedy verification.

⚙️ SGLang, CUDA Graph, and FP8

The implementation progresses from Native PyTorch (Eager) to SGLang (Eager), CUDA Graph replay, and selective FP8 execution. Denoising and causal passes retain their respective attention and cache semantics. FP8 applies to eligible language-model linear operations, with the remaining components kept in BF16. These changes reduce execution overhead or arithmetic cost; quantization can still alter logits and decoding decisions.

The default DLM service uses BF16, Triton attention, eager execution, one GPU, one active request, and two queued requests. Client concurrency queues requests; use independent replicas for higher service concurrency.

Execution option	Purpose	Supplied default
SGLang eager	Integrated scheduling, attention kernels, and cache ownership	Enabled
CUDA Graph replay	Reduce repeated GPU launch overhead for compatible captured work	Disabled; studied in the paper
Selective FP8	Accelerate eligible language-model linear operations	Disabled; default inference remains BF16

CUDA Graph implementation details

The optional implementation captures fixed-shape block work. The FlashInfer path uses persistent attention masks and device-resident buffer updates; the Triton verifier uses separate draft/verification metadata and input buffers with shared model parameters and KV storage. The supported capture case is B32 with one request and no tensor, pipeline, or data parallel expansion. Prefill and unsupported shapes use eager execution.

Graph replay preserves the active decoding algorithm. Selective FP8 is a separate numerical configuration and may change logits and token decisions. The paper compares these infrastructure optimizations within the same decoding mode. The release launcher does not expose a --cuda-graph switch.

Use the bundled engine and serving profile. Installing upstream SGLang alone does not provide these model and decoding integrations. See Inference for configuration and the native reference route.

📈 Evaluation

The GroundAnything evaluation suite covers the 30 benchmarks reported in the paper. The Evaluation Guide provides the data link, path setup, the exact 30-task selection, input validation, and full-suite execution.

The shared evaluator supports 7 modes:

Mode	Supported models
GAM	GroundingPI, GroundAnything, GroundAnything-VLM
VLM	Generic vision-language baselines
REXOMNI	Rex-Omni
LOCATEANYTHING	LocateAnything
GROUNDINGDINO	GroundingDINO through a compatible service
DLM	Legacy diffusion checkpoints using the GAM protocol
RLV2	Legacy RL checkpoints using the GAM protocol

Use GAM mode for all three released checkpoints. Start the model service and configure the data paths before running:

python3 run.py setup eval
# Select the route matching the service that is already running.
python3 run.py eval --decoder denoise
python3 run.py eval --decoder speculative
python3 run.py eval --config configs/eval/vlm.yaml

The shipped recipes are 8-sample smoke tests for selected tasks. For the complete 30-benchmark suite, follow the full evaluation walkthrough: it selects the paper's task list, uses limit: null, and writes results under a fresh run_id. Evaluation connects to an existing service and does not start or switch its decoder.

🏋️ Training

Training runs in its own environment. Prepare the complete model files, tokenizer, validated input caches, and manifests before launching. The supplied training recipe targets H800-class CUDA GPUs, with a pinned CUDA toolchain and source-built FlashAttention. See Environment Setup for the matching runtime and dependencies.

🧩 Choose a training stage

Stage	Native configuration	Launch configuration
General	general.yaml	general_train.yaml
Specialist	specialist.yaml	specialist_train.yaml
SFT 1	sft1.yaml	sft1_train.yaml
SFT 2	sft2.yaml	sft2_train.yaml

Set the base-model path, matching tokenizer, prepared inputs, output directory, batch size, and sequence length in the native configuration. Set the distributed topology and initial checkpoint in the launch configuration. Keep the training mode as DLM; GAM is the evaluation mode.

The supplied stage templates target 8 nodes × 8 devices. Adjust nnodes, nproc_per_node, node_rank, master_addr, and master_port for your cluster, and keep the runtime checks and packing manifest consistent with the chosen world size and batch size.

python3 run.py setup train

# Inspect the configured training command before launching.
.venv-train/bin/python scripts/run.py configs/release/sft1_train.yaml --dry-run

# Run on each node with its corresponding launch configuration.
python3 run.py train --config configs/release/sft1_train.yaml

Training consumes prepared caches and manifests for sampling, length/image checks, response lengths, and packing. These inputs must be prepared before launch; a generic JSONL-to-training-cache converter is not included. See Data Preparation.

🔄 Initialize or resume a checkpoint

Launch setting	Behavior
checkpoint.initial_dlm_checkpoint	Load DLM weights for a new stage with a new optimizer and scheduler
checkpoint.resume_from_checkpoint	Resume a complete training checkpoint, including optimizer, scheduler, and training state

Choose one of these settings. To fine-tune the released DLM, use the complete weights/dlm_bundle as the initial DLM checkpoint and set the native model.path to the matching autoregressive base model, with its tokenizer. To resume SFT, update its launch YAML and rerun python3 run.py train --config ...; the top-level --resume shortcut selects the smoke-test recipe.

The Training Guide also covers optional reinforcement learning. After training, export a fresh complete inference bundle using the Inference Guide. The published Hugging Face DLM bundle is already prepared for inference.

📊 Results

Paper and supplementary material.

📚 Documentation

Guide	Contents
Environment Setup	Workflow environments and platform prerequisites
Inference	Serving, model preparation, configuration, and reference backend
vLLM Deployment	GPU / PPU setup, container starting point, and image API requests
Batch Annotation	Resumable JSONL predictions for image collections
Examples	Image prediction, JSON output, and visualization
Evaluation	Dataset setup, paper benchmark suite, execution, and results
Training	Training recipes, distributed settings, and checkpoints
Data Preparation	Input formats and local preparation requirements
Third-party Sources	Bundled frameworks and provenance

GroundAnything/
├── run.py                  # Workflow launcher
├── grounding_anything/       # Lightweight HTTP client and visualization
├── configs/                # Serving, training, and evaluation recipes
├── infer/                  # Model services and decoding
├── train/                  # Training workflows
├── eval/                   # Tasks, prompts, requests, and metrics
├── models/                 # Model definitions
├── examples/               # Prediction examples
├── environments/           # Environment installers
└── docs/                   # Detailed guides

Use python3 run.py --help to inspect the command-line interface. Full model workflows run from the source tree; the Python package installs the independent HTTP client.

📜 License

Original project contributions are available under the Apache License 2.0, with no additional restrictions imposed by this project. This grant covers only rights held by the contributing authors.

Third-party material retains its applicable licenses, including the Kimi K3 License for Kimi-derived material and applicable derivative works. These upstream conditions remain in force. See Third-party Notices for component attribution and the released model's license scope for the model package.

📖 Citation

If this work supports your research, please cite:

@misc{yu2026groundanythingreconcilingparalleldecoding,
  title = {{GroundAnything}: Reconciling Parallel Decoding with Precise Visual Grounding at Flash Speed},
  author = {Qize Yu and Lianrui Fan and Bowen Ping and Xini Ding and Zetian Song and Junbo Niu and Kaixuan Wang and Tianxing Chen and Yue Chen and Minghua He and Yuran Wang and Jie Huang and Haojun Zhang and Min Chen and Hao Li and Wenxuan Song and Ruihai Wu and Xianming Liu and Shilong Liu and Shuchang Zhou and Ping Luo and Shiyu Huang},
  year = {2026},
  eprint = {2609.39600},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  url = {https://arxiv.org/abs/2609.39600},
}

🙏 Acknowledgement

We thank the teams behind Rex-Omni and LocateAnything for sharing their work and open-source implementations.