[earakely-scale]

What

Train on a bundle's tasks with verl, through verl's own agent loop. verl 0.9 runs multi-turn tool rollouts in verl.experimental.agent_loop: per row, an agent loop generates on verl's vLLM, parses the model's tool calls, adds their answers to the context and returns the tokens, the loss mask and, optionally, the rollout's reward_score, in which case verl skips its reward function. A custom loop is registered through actor_rollout_ref.rollout.agent.agent_loop_config_path, and each row picks it by agent_name.

agent_env.training.verl, behind a new verl extra (openenv, pyarrow, transformers; verl, vLLM and torch stay the trainer's to pin).

The writer

python -m agent_env.training.verl URL OUT writes the tasks of a bundle's OpenEnv server (agent_env.openenv.build_app):

• OUT/<split>.parquet: a row per task, with an explicit schema. A row has:
  • the task's system prompt and first prompt;
  • agent_name: agentenv and data_source: agentenv/<env>;
  • extra_info with the split, index, task id and the row's tools as JSON.
• OUT/agent_loop.yaml: registers the loop with the server's URL, the deadlines, attempts and the token's variable.

A row's tools come from playing one task of each toolset, as the NeMo Gym writer does; both writers share _openenv.tasks_and_tools.

Its options:

• --token-var NAME names the environment variable holding the token a server asks for. The writer reads it now and the loop in every verl worker, so the token is in no file.
• --tokenizer MODEL counts each row's prompt with its tools as the model's chat template renders it. Pass --chat-template-kwargs with verl's data.apply_chat_template_kwargs so the count matches verl's rendering.
• --max-prompt-length N refuses rows over N tokens before anything is written. verl would cut such a prompt from the left, losing the system prompt and the tools, and its own overlong filter doesn't count a row's tools.

The loop

agent_env.training.verl.loop.AgentEnvAgentLoop is verl's ToolAgentLoop with the row's tools in place of verl's configured ones, so verl's tokenization, tool parsers, turn caps and truncation are its own.

• Tools: each parsed call is a call on the row's episode, through the Session layer the TRL helper and the NeMo Gym server use. verl's parsers get typed tool schemas (qwen3_coder converts arguments by them), and the chat template renders the row's full JSON Schema.
• Reply and reward: the model's last reply, without special tokens, ends the turn it left open. The episode's reward is the rollout's reward_score.
• A rollout the env fails (a deploy, a call past its deadline, a server that went away) is played again from the start on a new episode, a fresh sample of the same policy.
  • It's replayed up to attempts times, but none starts once reset_timeout has passed since the rollout began.
  • One that fails every time raises RolloutFailed. verl's V1 trainer (trainer.use_v1, the default on 0.9.1 and main) records a session that raised as failed and leaves its trajectory out of the loss, its group's GRPO baseline (padding gets its own uid) and the eval reward.
  • With trainer.v1.sampler.sync_refill_failed_groups=True, a group left with no trajectories is replaced by a fresh prompt instead of stopping the run.
• A restarted env server delays rollouts rather than failing them. Opening its session, a rollout waits out a server at its limit or refusing connections (Shared.waits_for_server) within reset_timeout. The writer read the server first, so a refusal then is a server coming back; a server that refuses the token fails at once.
• Configuration errors raise, and the V1 trainer logs each one and leaves it out:
  • a first prompt over rollout.prompt_length (refused rather than cut);
  • an episode that ends unscored without failing;
  • a loop instance run twice.
• A cut env answer is warned of once a tool, when it's over multi_turn.max_tool_response_length, which verl truncates silently (256 characters by default).
• Scores, attempts and seconds: a scored rollout's reward_extra_info holds each score by the step id that recorded it (score/<step>), its attempts, and its seconds generating (model_seconds) and in the env (env_seconds: waiting for a session, deploying, calling tools, ending turns, scoring). verl's V1 trainer averages them per eval as val-aux/<data_source>/<key>/mean@N and writes them per rollout to trainer.validation_data_dir. It doesn't read reward_extra_info in training steps, nor dump it to rollout_data_dir, so GDPO's gdpo_reward_keys reach it only in verl's deprecated trainer.
• verl internals it leans on are named in the class docstring, and CI checks them on verl 0.9.1 and main.

The env server

build_app(token=...) requires Authorization: Bearer <token> on every request and WebSocket but /health: a request without it gets 401, and a WebSocket is closed before it opens. That lets the env server run where other machines, verl's workers on any node, reach it. Without a token, anyone who reaches the server deploys and drives its envs. The session layer sends the token (Shared.token, opened, read_tasks, probe).

Test infrastructure

tst/unit/test_optional_extras_stay_optional.py: verl's distributions are all trl's (pyarrow comes with datasets, and transformers), so the per-extra check has nothing to block for it. _COVERED records that and a test asserts it; the no-extras check covers the module.

Tests

• Unit tier:
  • tst/unit/training/verl_test.py, the writer on the fake bundle: the rows of each split and the loop config; tools per toolset, each probed once; an empty split keeping a row's columns; a server that asks for a token, read with the one its variable holds, with the token in no file and a wrong one refused; and what's refused before the server is read.
  • tst/unit/openenv/app_test.py: a server given a token refuses HTTP and WebSocket without it, but answers /health.
  • tst/unit/training/sessions_test.py: a session waits out a restarting server only when asked, and never one that refuses the token.
• verl tier (tst/verl/loop_test.py, make verl-test): AgentEnvAgentLoop.run on verl itself, with Qwen3's tokenizer and a stand-in rollout server, on the fake bundle. Env failures come from the fake env and a broken server, not from patching Session. It covers:
  • a played row's reward, reply and reward_extra_info (its keys, the attempt count after a replay, env seconds covering the scoring), through hermes and qwen3_coder;
  • typed parameter conversion;
  • a replay on a new episode, and none once the time is up;
  • RolloutFailed when every attempt fails;
  • a prompt that doesn't fit, an unscored episode, and a loop run twice;
  • a token server played with and without its token;
  • the cut-answer warning;
  • the writer's prompt count with and without enable_thinking, and its refusal over the limit;
  • the config naming the loop.
• .github/workflows/verl.yml runs the verl tier on verl 0.9.1 and main with CPU torch; it allows no skips and isn't a required check.

GPU runs

GRPO on the GridSim v1 bundle through AgentEnvAgentLoop.

Setup:

• Model: the Qwen3-4B SFT warm start, LoRA r32/α64 on every linear layer, lr 2e-5 cosine with 3 warmup steps, thinking off, the hermes parser.
• Rollouts: 32 a step (4 days × 8, temperature 1.1), at most 4 assistant turns.
• Evals: the 200 eval days at steps 0, 20 and 40.
• Training: verl main in verlai/verl:uv.cu130.dev5 (torch 2.13, vLLM 0.29) on one L40S (g6e.4xlarge), FSDP actor and vLLM rollout sharing it, verl's V1 trainer.

Round 2, on this PR's final code. The env server ran on its own VM (c7i.4xlarge): build_app(..., token=...) on 0.0.0.0:8200, reachable only inside a temporary VPC. The trainer wrote the rows from it over the network:

python -m agent_env.training.verl http://<env-vm>:8200 data --token-var AGENTENV_TOKEN \
  --tokenizer Qwen/Qwen3-4B --max-prompt-length 3072
wrote agent_loop.yaml, all.parquet, eval.parquet, train.parquet to data
the longest prompt is 1165 tokens with its tools

The verl tier passed there too (13 tests on verl main in the CUDA image).

Eval reward, 200 eval days	Step 0	Step 20	Step 40	Train reward, first 10 → last 20 steps
Round 1, one machine (c531443)	0.524	0.586	0.601	0.585 → 0.697
Round 2, run C, env server on its own VM with a token	0.522	0.588	0.600	0.588 → 0.694

Run C had no failed or replayed rollouts and no cut answers; it took about 75 s a step.

Failures injected in round 2:

• An env server restart between episodes: rollouts opening sessions waited it out.
• Two SIGKILLs while sessions were opening: the handshake drop is transient, so they were waited out too, and steps 13, 17 and 24 just ran 6, 36 and 54 s longer. No rollout was lost and training continued.
• Run B, attempts: 1, a graceful stop 8 s into a step's episodes:
  • The rollouts mid-episode failed (submit_schedule: ConnectionClosedError: received 1012 (service restart)) and raised RolloutFailed.
  • verl left them out and refilled 4 groups that had no trajectories left (rollout_failure/evicted_samples: 4).
  • Steps 1–3 trained normally.
• The first round-2 run lost its step-40 eval. My SIGKILLs orphaned the killed servers' env containers, and 125 of them ran the env VM out of memory, so the kernel killed the server.
  • Every eval rollout waited reset_timeout (600 s) for it, then raised RolloutFailed.
  • verl V1's validation then failed on an eval with no trajectories at all (ValueError: Received an empty list as keys).
  • That run's 40 training steps had finished, with evals 0.524 and 0.585 at steps 0 and 20; run C repeated it cleanly.

Round 3, on 39791fa. One g6e.4xlarge, the env servers on the same machine. Runs A (8 steps, sync) and B (8 steps, colocate_async) on the GridSim rows and SFT model above; run D on a two-turn test bundle (four questions about a three-email inbox, a regex agent_prompt_response_verifier a turn, answer-1 and answer-2, served with reward= their mean), on Qwen3.5 with multi_turn.format=qwen3_coder.

• reward_extra_info in verl's metrics (run A's step-0 eval, 200 days): val-core/agentenv/gridsim-v1/reward/mean@1: 0.5345, val-aux/.../score/grade/mean@1: 0.5345, attempts 1.0, model_seconds 49.6, env_seconds 71.5. With 200 rollouts and 64 sessions, most of the env seconds are waiting for a session.

• Where a step's time goes (run A, median 75 s): generation 42.7, old log-probs 7, actor update 23, weight sync 2.5. A probe of 32 GridSim episodes at once, no model, put a rollout's env time at about 25 s: open and deploy 18–20 s median, a tool call 0.7 s, scoring 4 s. So the deploy is most of the env's share; the page's limits say so. verl V1's replay buffer polls every 2 s, which is why generation times come in 2 s steps.

• colocate_async against sync:

  GridSim, 8 GRPO steps	sync (A)	colocate_async (B)
  Step, median	75.4 s	66.0 s
  Generation, median	42.7 s	32.8 s
  All 8 steps	651 s	531 s
  Train reward, mean	0.58	0.61
  Failed or replayed rollouts	0	0

  The loop needs nothing for it: rollouts still running at a step's end pause their generation while the actor trains, their episodes open, and go on with the new weights (trajectory_staleness_worst 1, aborted_ratio 0). It keeps num_warmup_batches (1) more batch in flight, so the server holds twice the sessions: 64 here.

• Multi-turn and qwen3_coder (run D): Qwen3.5-4B's step-0 eval scored 0.75, score/answer-1 1.0 and score/answer-2 0.5, in 7 turns on average, so the parser's calls reached the episode and end_turn handed over the second turn. Qwen3.5-2B trained 5 GRPO steps through the loop (eval 0.375 → 0.5, no failed rollouts); its per-turn scores show it never called end_turn, so never reached turn 2, which the reward alone doesn't show.

• Found on the way, neither in the loop:

  • The mail env runs in a container with a compose network per episode, and 32 episodes at once exhausted Docker's default address pools (all predefined address pools have been fully subnetted). The loop replayed the failed deploys, then raised RolloutFailed for the 2 rollouts that failed all 3 attempts, and verl left them out. Widening default-address-pools in daemon.json fixed it; the docs page has it under limits. GridSim's env doesn't take a network per episode.
  • Training Qwen3.5-4B on one L40S ran out of GPU memory with each setting tried, first on its 248k-token vocabulary's logits; the 2B fit.

verl settings this needs (the round-2 command line):

actor_rollout_ref.rollout.multi_turn.enable=True actor_rollout_ref.rollout.multi_turn.format=hermes
actor_rollout_ref.rollout.multi_turn.max_assistant_turns=4
actor_rollout_ref.rollout.multi_turn.max_tool_response_length=8192   # verl's default 256 cuts env answers
actor_rollout_ref.rollout.agent.agent_loop_config_path=data/agent_loop.yaml
actor_rollout_ref.rollout.agent.default_agent_loop=agentenv
trainer.v1.sampler.sync_refill_failed_groups=True   # else a step whose rollouts all failed stops the run
actor_rollout_ref.rollout.layered_summon=False      # LoRA on one GPU: FSDP refuses CPU offload with NO_SHARD
data.return_raw_chat=True +data.apply_chat_template_kwargs.enable_thinking=False

The token reaches every Ray worker through the environment the trainer starts in.

Known limits:

• An eval in which every rollout failed stops verl V1's validation. That only happens with the env server unreachable for longer than reset_timeout. It's upstream behaviour the loop can't avoid, since verl doesn't tell a loop whether it's in an eval.
• Hard-killing the env server leaves its episodes' env containers running. Stopping it with SIGTERM tears them down.
• verl 0.9.1's and main's uv.locks point flash-attn at a wheelhouse URL that now 404s, so uv sync --extra fsdp fails; verl's image works. The CPU tier runs on PyPI's verl 0.9.1.
• Docker's default address pools hold about 30 networks, so a local-Docker env server can't run more than about 30 episodes of a containerized env at once until they're widened.
• Spend: about $23.50 for the three rounds; every AWS resource is torn down.

🤖 Generated with Claude Code

Confidence Score: 5/5

The changes appear safe to merge; no new blocking issue was found.

Summary

Adds training through verl’s agent loop, with task datasets, episode rewards, retries, and optional server authentication.

• verl gets task rows with the prompts and tools needed to play each episode.
• verl rollouts play task tools in fresh AgentEnv episodes and return episode scores.
• OpenEnv servers can require a bearer token for remote requests and episode sessions.

_{Reviews (8) · Last reviewed commit: "feat(training): a verl rollout's scores,..." · Reviewed by Greptile}