[earakely-scale]

Summary

TRL's GRPOTrainer and AsyncGRPO can now train on a bundle served by agent_env.openenv.build_app, through TRL's own environment_factory, with no changes to TRL.

from trl import GRPOConfig, GRPOTrainer
from agent_env.training.trl import AgentEnv      # pip install 'agentenv-framework[trl]'

env = AgentEnv("http://localhost:8000")          # or AgentEnv.local("./bundle", max_concurrent_envs=64)
trainer = GRPOTrainer(model="Qwen/Qwen3-4B", args=GRPOConfig(max_tool_calling_iterations=12),
                      train_dataset=env.dataset, environment_factory=env.factory)

• agent_env/training/trl.py, behind a new trl extra (agentenv-framework[openenv] and datasets). TRL itself stays the trainer's to pin. agent_env/training/ is where later trainers' modules go, each behind its own extra.
• env.dataset has one row per task of the split, read from the server's Task API: the prompt as chat messages, plus split and index. env.dataset_for(split) reads any other split.
• env.factory is a generated class. Its methods are the turn's tools, typed and documented from their JSON Schema (ported from the private agentenv-trl's AgentEnvSpec), plus end_turn.
  • Each instance holds one session on the server and plays one episode at a time over OpenEnv's WebSocket (GenericEnvClient).
  • reset returns at once, since GRPOTrainer resets one rollout at a time. The first tool call or get_reward waits for the deploy.
  • end_turn answers with the next turn's prompt, so multi-turn tasks work as they are.
  • Turns the model leaves open end with an empty reply when the reward is asked for.
  • end_turn waits up to the reset deadline rather than a tool's, since the last one runs the task's verifiers.
• The environment owns the reward through get_reward, which both trainers read, so no reward_funcs are needed. It returns None when the episode ended in an error, didn't match its row, or a tool missed its deadline. TRL turns None into NaN. Instances are reused across rollouts, as TRL's pool reuses them.
• Fails early, with the fix in the message:
  • no server at the URL;
  • an unknown split;
  • a tool or argument name TRL can't call;
  • a server at its session limit. The session opens at reset, and the error names MAX_CONCURRENT_ENVS / max_concurrent_envs=.
• Multi-environment mode: the Task API now names each task's toolset: the played agent's envs, its role, the task's modify_env_tool_access changes for that role, and whether its turns offer bash. Building factory plays one task per toolset, one at a time, to list its tools. Tasks with different toolsets get one class each (factory is then a dict), and every row carries TRL's environment column.
• AgentEnv.local(bundle, ...) serves build_app in process on a loopback port. A trainer that exits or crashes ends the episodes it left running and stops that server, using the same exit hook as agentenv-trl.
• build_app's app construction is split into _app(served, ...), so the unit tests serve hand-built tasks through the same server.
• The dependency tests now allow an extra that includes another (trl includes openenv). A new check confirms that importing agent_env.training.trl without the extra names [trl].
• THIRD_PARTY_NOTICES.md adds the five runtime packages the extra brings: datasets, dill, multiprocess, pyarrow and xxhash. Every license link resolves.

OpenEnv 0.10 requires fastmcp<4 and so mcp 1. A trainer with OpenEnv installed therefore runs vLLM 0.29 or older, which TRL 1.16 allows. The README says so. Once OpenEnv releases FastMCP 4 and agent-env supports mcp 2, the helper runs on vLLM 0.30+ unchanged.

Tests

• tst/unit/training/trl_test.py: 18 tests against the real OpenEnv server over the Episode API's fake steps, driven the way TRL drives it:
  • the dataset and the class shape;
  • a full rollout on an environment that is reused;
  • multi-turn tasks;
  • episodes left unscored;
  • a tool that misses its deadline, and an end_turn that waits past it for a slow verifier;
  • capacity, and no server;
  • multi-environment mode;
  • a tool name TRL can't call;
  • local() with teardown at close and at exit;
  • the tool-schema methods.
• tst/integration/training/trl_test.py on local Docker, 2 tests, 46 s:
  • rollouts on one reused environment score as the items verifier scores them (1.0, then 0.0), and a two-turn task hands over its second prompt;
  • a task whose agent works in its sandbox gets a +bash toolset of its own, plays through bash, and scores 1.0. Nothing is left running.
• Full unit tier: 11,124 passed, 66 skipped.
• Checked by hand: transformers 5.21's get_json_schema, the function TRL uses to show tools to the model, renders PortSim's real tools through these methods exactly as it did for agentenv-trl.
• tst/integration/openenv: 6 of 7 pass locally. The other fails building the echo test agent's linux/amd64 image on a Mac without buildx. That happens after its OpenEnv checks pass, and it's unrelated to this change.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; all three previous findings are addressed.

Summary

Adds TRL training on bundles served through OpenEnv, with task datasets, generated tool classes, owned rewards, and an optional local server.

• TRL can now train on tasks served by an OpenEnv server.
• OpenEnv bundle and hand-built tasks use the same app setup.

Diagram

sequenceDiagram
    participant Trainer as TRL trainer
    participant Env as Generated environment
    participant Server as OpenEnv server
    participant Episode as Task episode
    Trainer->>Env: reset(dataset row)
    Env->>Server: Start task
    Server->>Episode: Deploy and open turn
    Trainer->>Env: Call tools / end_turn
    Env->>Server: WebSocket requests
    Server->>Episode: Run tools and finish turns
    Episode-->>Env: Final reward
    Trainer->>Env: get_reward()
    Env-->>Trainer: Reward

Loading

_{Reviews (2) · Last reviewed commit: "fix(training): end_turn waits for the sc..." · Reviewed by Greptile}

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pypi/​datasets@​5.1.0

View full report