The running curve is strange and different to the paper?

[outongyiLv]

Hello author, thanks for the great work! I follow the step to run StepPO on the env: alfworld

My setup

• Model: Qwen/Qwen3-4B-Instruct-2507
• GPUs: 4 × 80G
• Data: unzipped from your provided alfworld.zip
• using the scripts in /StepPO/examples/run_alfworld_step_adv.sh

but I found the curve is super strange as followed:

Validation reward trajectory

step	valid_seen	valid_unseen
5	12.1%	4.5%
40	32.9%	29.1%
90	79.3%	76.1%
95	82.1%	79.9% ← peak
105	52.1%	53.7%
115	30.0%	29.1%
125	37.1%	35.8%

Other metrics at step 125

• actor/kl_loss climbed from 0.2 (step 1) → 4.9 (step 125), monotonically increasing
• critic/score/mean (train reward) still rising to 0.058 → looks like overfitting
• actor/pg_clipfrac ≈ 0.3, 30% of updates hitting the clip boundary

Questions

Can you tell me the step I followed is right? and the curve in training is also right? but now the maximize curve is about 70/80% and the testing curve is decresing, but the result in paper is about 85%/92%?

I am looking forward to your response, thanks!

[Melmaphother]

Hi, thanks for sharing the detailed results.

The training trajectory before around 100 steps looks reasonable. The sudden performance drop afterward suggests that the training dynamics became unstable after this point, rather than an issue with the earlier training process.

Could you try resuming the training from the checkpoint around step 95 or 100? Due to the variance of RL training, the resumed run may avoid this collapse and continue improving. If the KL divergence increases rapidly again, reducing the learning rate or strengthening the KL regularization may also help.