Description
In a multi-step ALFWorld rollout with history enabled, the accumulated conversation approaches max_model_len. Starting from a later environment step, vLLM returns an extremely short completion containing only "go".
The workflow does not raise an exception and continues until max_env_steps. The truncated responses are stored as normal Experience objects and may enter the training pipeline.
This was initially observed in debug mode, but debug and normal training use the same rollout model and workflow execution path, so the issue can also affect normal training.
Observed behavior
Configuration:
max_model_len: 20480
max_prompt_tokens: 16384
max_response_tokens: 512
max_env_steps: 30
enable_history: true
Observed trajectory:
step 0–16: normal responses
step 17–29: response_text == "go"
final env_done: 0.0
final env_rounds: 30
The "go" responses are stored as normal experiences without a truncation marker.
Root cause
The vLLM request output provides finish_reason and stop_reason, but vLLMRolloutModel.generate() currently constructs Experience objects without propagating those fields.
In particular, when:
output.outputs[i].finish_reason == "length"
the resulting experience is not marked as:
truncate_status = "response_truncated"
Consequently:
The workflow cannot distinguish a length-truncated response from a normal response.
MaskResponseTruncatedOperator cannot detect the sample.
A multi-step rollout may continue after truncation.
The resulting failed rollout may enter OPD/RL training as valid data.
Are You Willing to Fix This Issue?
- Yes, I am willing to fix this issue!
Description
In a multi-step ALFWorld rollout with history enabled, the accumulated conversation approaches max_model_len. Starting from a later environment step, vLLM returns an extremely short completion containing only "go".
The workflow does not raise an exception and continues until max_env_steps. The truncated responses are stored as normal Experience objects and may enter the training pipeline.
This was initially observed in debug mode, but debug and normal training use the same rollout model and workflow execution path, so the issue can also affect normal training.
Observed behavior
Configuration:
max_model_len: 20480
max_prompt_tokens: 16384
max_response_tokens: 512
max_env_steps: 30
enable_history: true
Observed trajectory:
step 0–16: normal responses
step 17–29: response_text == "go"
final env_done: 0.0
final env_rounds: 30
The "go" responses are stored as normal experiences without a truncation marker.
Root cause
The vLLM request output provides finish_reason and stop_reason, but vLLMRolloutModel.generate() currently constructs Experience objects without propagating those fields.
In particular, when:
output.outputs[i].finish_reason == "length"
the resulting experience is not marked as:
truncate_status = "response_truncated"
Consequently:
The workflow cannot distinguish a length-truncated response from a normal response.
MaskResponseTruncatedOperator cannot detect the sample.
A multi-step rollout may continue after truncation.
The resulting failed rollout may enter OPD/RL training as valid data.
Are You Willing to Fix This Issue?