Really like that every rollout gets its own sandbox — isolation is the part most agent-training setups skimp on.

We run coding agents in production harnesses, and the failure mode we see most isn't wrong code — it's non-terminating turns: hand an agent an open-ended objective and it can think in a loop for hours (one of ours burned its full daily inference cap doing exactly that this morning; it's the failure class we built ThumbGate around).

Does the reward setup here penalize rollouts that never emit a final answer, or do you hard-cap steps in the env? Curious because we ended up putting runaway detection at the harness layer, outside the model — prompt-level bounds kept getting reasoned around.

The loop-owning flip is the right call, and the honest paragraph near the end is the interesting one.

Three things change between the run that works locally and the run that works remotely: 4B to 8B, local subprocess to remote sandbox, 110 steps to 10. So "moving to Qwen3-8B was what made it click" is not separable from the sandbox move yet. The 4B remote arm at 110 steps is the one that would separate them.

But there is a mechanism in your own config I would check before spending that GPU time.

train_turn_fn=has_tool_call reinforces action turns, not prose. That makes the number of trained tokens in a trajectory scale with how many tool calls it made. A trajectory that solves in 3 calls contributes 3 turns. One that flails through 40 contributes 40.

Now put that next to the failure you describe: reward climbs, then collapses, with the agent spamming tool calls without ever solving. The failure mode and the token-weight maximiser are the same behaviour. If the group-relative advantage lands per trained token rather than per trajectory, a spam rollout carries an order of magnitude more gradient weight than a clean solve at equal reward, and the collapse is self-reinforcing rather than incidental.

So: is the advantage normalised per trajectory or per trained token on this path?

One thing that cuts your way, though. You call the 10-step run noisy and it is stronger than you are giving it credit for. With 32 prompts and your own reward_std near 0.45, the standard error of a step mean is about 0.08, so 0.27 to 0.71 is roughly five and a half standard errors, before even counting multiple completions per prompt. The weak part is not the size of the move, it is that first-versus-last is the noisiest possible summary of a 10-point series. A fitted slope with a CI across all 10 steps would probably read better than the endpoints do.

Is trained-token count per trajectory logged in trackio anywhere? That is the series I would pull first if the 4B collapse repeats.