--- license: apache-2.0 tags: - reinforcement-learning - on-policy-distillation - llm-agents - scienceworld language: - en --- # ftb-sciworld-repro A self-contained, runnable bundle for training **five multi-turn on-policy distillation methods on ScienceWorld** with a Qwen3-1.7B student and a Qwen3-32B teacher: | method | what it does | | --- | --- | | **OPD** | vanilla on-policy distillation: the student runs whole episodes alone, the teacher scores its tokens afterwards | | **Guided-OPD** | per-turn coin flip picks teacher or student; teacher turns get SFT cross-entropy, student turns get the OPD reverse-KL | | **TCOD-B2F** | replays a gold-action prefix, student takes over near the end; the takeover point walks backwards over training | | **TCOD-F2B** | student always starts at turn 1, but episodes are truncated short and grow over training | | **FTB** | FutureBridge: inserts a validated teacher "bridge" at the highest-disagreement student turn | Everything needed to run is in here except the model weights, the task data and the Python dependencies, which `scripts/setup.sh` fetches. ## Requirements - **8 GPUs with ≥ 80 GB each, on one node.** The teacher is dense Qwen3-32B at TP2 — roughly 32 GB of weights per device before any KV cache — so 40 GB cards cannot hold it at this parallelism. The split is 4 student rollout + 2 teacher + 2 FSDP trainer. - Linux with CUDA and NCCL - Python 3.10 - **Java 17+ on PATH** — ScienceWorld starts a JVM per episode - ~80 GB free disk for models, plus ~4 GB per saved checkpoint (4 per run) ## Quick start ```bash hf download SeanWang0027/ftb-sciworld-repro --local-dir ftb-sciworld-repro cd ftb-sciworld-repro # If the node already has a working Trinity/TCOD environment: bash scripts/setup.sh # If it does not, also install the pinned dependency set: bash scripts/setup.sh --with-deps # Reuse models already on the node instead of downloading ~68 GB: bash scripts/setup.sh --models /path/to/models # expects Qwen3-1.7B, Qwen3-32B # Then, one method at a time (each wants all 8 GPUs): bash scripts/run.sh ftb bash scripts/run.sh opd bash scripts/run.sh guided_opd bash scripts/run.sh tcod_f2b bash scripts/run.sh tcod_b2f ``` Behind a scheduler: `sbatch -A -p --export=ALL,METHOD=ftb scripts/run_slurm.sbatch`. (The Hub does not preserve the executable bit, so invoke the scripts as `bash scripts/...` — as above — rather than `./scripts/...`.) `setup.sh` is idempotent — re-run it after a partial failure. It pip-installs the vendored `tcod/` (with `--no-deps` unless you pass `--with-deps`), copies the FutureBridge overlay over it, **verifies all five workflow classes import**, puts the models under `./models`, downloads the task split to `./data/scienceworld`, and creates `./outputs/`. A run is 200 training steps, batch 16 tasks × up to 30 environment turns, with a checkpoint every 50 steps under `outputs/checkpoints/FutureBridge-OPD//global_step_*/`. Expect hours to a day per method; B2F and FTB are slowest (they boot two ScienceWorld JVMs per training task for gold-path replay). ## What is upstream and what is not This matters if you are reproducing published numbers. **Upstream, redistributed unmodified** (both Apache-2.0, see `NOTICE`): - `tcod/` — [kokolerk/TCOD](https://github.com/kokolerk/TCOD) at commit `17a8af2`, exported with `git archive` - `overlay/` — [ChenChiShui/FutureBridge-OPD](https://github.com/ChenChiShui/FutureBridge-OPD) `source/` at commit `bfcdf91` - `configs/ftb.yaml` — verbatim from that release **Added here:** `configs/{opd,guided_opd,tcod_b2f,tcod_f2b}.yaml`, `configs/bench.yaml.tmpl`, everything in `scripts/`, this README, `NOTICE`. The FutureBridge release ships ScienceWorld configs for FutureBridge and its ablations only. For the four baselines it ships the *workflow implementations* but no configuration — instead its `configs/README.md` gives a "Baseline workflow mappings" table saying to swap `default_workflow_type` and keep the paper settings. The four baseline YAMLs here are a reconstruction from that table: every shared setting is identical to `ftb.yaml`, and the deltas are the workflow class, the method's own `workflow_args`, run/buffer names, the monitor backend (tensorboard rather than wandb), and — for Guided-OPD only — the `mix` policy-loss block. **Judgement calls the upstream table does not determine**, in case you want to change them: | setting | value here | basis | | --- | --- | --- | | B2F / F2B `checkpoint_steps` | 5 / 6 | upstream TCOD's own `TCOD_examples/scienceworld/tcod_{b2f,f2b}.yaml` | | Guided-OPD `beta_start/end`, `curriculum_ratio` | 1.0 / 0.0 / 0.8 | defaults in `guided_opd_workflow.py`'s own schedule function | | Guided-OPD **`mu`** | **0.5** | weighs the forward-KL (teacher-turn SFT) and reverse-KL (student-turn OPD) terms equally. `MIXPolicyLossFn`'s own default is **0.1**. Nothing in the release fixes this value — treat it as a hyperparameter, not a reproduction. | | `mix` batch-size args | derived from the 8-GPU split | not specified upstream | ## Reporting results **The upstream protocol does not evaluate.** Every released config sets `eval_interval: 9999` and `eval_on_startup: false`, so the declared eval task set never fires, and no evaluation entry point is shipped. Its `configs/README.md` says to "run the configured evaluation task set once after training" but provides no mechanism. What the release's own analysis code reads is the **training** rollout metric — `analysis/plot_training_dynamics.py` maps `rollout/env_done/mean` to the panel labelled "Completion Rate". So the faithful reproduction is: run the five methods and plot `rollout/env_done/mean` against step. One caveat if you compare those curves across methods: the training rollout is not produced by the same policy in each arm. OPD's trajectories are 100% student; B2F's begin with a gold-action prefix; Guided-OPD's contain teacher turns with probability β (≈1 early in training); FTB's contain both. Early-training completion is therefore partly the teacher's or the gold path's work, not the student's. `scripts/eval.sh` is provided as an **addition, not a reproduction**: it scores a run's checkpoints on the held-out 1,308-task test split through a single shared workflow, so the arms are measured under identical conditions with no teacher intervention. See the header of that script for why it forces one workflow for all five methods. Either way the metric is `env_done`. **There is no score metric anywhere**, here or upstream: ScienceWorld's 0–100 partial credit is computed and stored on `Experience.reward`, which never reaches the metrics dictionary that the monitor aggregates. And `env_done` counts any terminating episode — ScienceWorld also terminates on *failure*, by driving the score negative — so it is not a clean success rate. ## Known properties worth knowing before you interpret runs These are properties of the upstream implementation, not bugs introduced here. None of them are patched in this bundle. **1. The prompt context accumulates, and overflow stops the agent.** Four of the five workflows (all but Guided-OPD, which keeps a 10-turn window) send the entire growing chat history every turn, on top of a 2-step summary embedded in each user message. Measured with the Qwen3-1.7B tokenizer on real ScienceWorld rooms, the prompt crosses `max_prompt_tokens: 10240` somewhere around turn 11 (object-rich room) to turn 17 (sparse room) of a 30-turn episode. Past that point the vLLM engine is *not called at all*: the framework returns a placeholder experience with an all-zero action mask, so the turn contributes no gradient, the parsed action is empty, and the agent stops acting for the rest of the episode. Grep a run log for `Prompt was truncated to` to count how often it happens. Upstream hit a more severe version of this and fixed half of it: TCOD commit `17a8af2` ("use compact action representation to avoid token overflow") replaced a ~1162-item full action×object list with a ~43-item templates+objects pair, because *single-step* prompts were already exceeding the limit. That shrank the per-turn prompt; it did not change the accumulation. **2. The `` instruction is inert.** Both the system prompt and every user message require reasoning "enclosed within ` ` tags", but the configs run Qwen3 with `enable_thinking: false`, and that chat template pre-emits *and closes* an empty `\n\n\n\n` before the model writes a token. Across 11,264 recorded generation turns from three models in earlier measurements, `` opened in 0.0% of them. The instruction costs tokens every turn and `parse_action` only ever reads ``. **3. The full action×object list is hidden from the model but used to judge it.** FTB's `_is_valid_action` checks a proposed action against `get_valid_action_object_combinations()` — the same ~1000-item list that commit `17a8af2` removed from the prompt. The student must compose a string that lands exactly in that set from two separate short lists, and a miss discards the whole bridge candidate. **4. FTB is B2F plus bridging.** `FutureBridgeScienceWorldWorkflow` subclasses `TCOD_b2f_scienceworld_workflow` and overrides only `_finalize_turn_responses`. If you want to know whether FTB's gain comes from the bridge or from the B2F curriculum underneath it, `tcod_b2f` is the controlled comparison — not `opd`. ## Troubleshooting | symptom | cause / fix | | --- | --- | | `Failed to look up actor 'synchronizer'` during weight sync | an engine was set to TP1; keep `tensor_parallel_size: 2` for both rollout and teacher | | teacher engine OOM at startup | cards smaller than 80 GB; there is no supported fallback at this parallelism | | wandb hangs then kills the explorer | only `configs/ftb.yaml` uses wandb (it is the upstream file). `wandb login`, or switch its `monitor_type` to tensorboard like the four baselines. `WANDB_MODE=offline` does not reach the Ray actor. | | `ModuleNotFoundError: scienceworld`, or JVM errors mid-run | `pip install scienceworld==1.2.2`; confirm `java -version` works | | `flash-attn` build failure | it compiles against the installed torch; install torch first, then `pip install flash-attn==2.8.1 --no-build-isolation`. Training can proceed without it. | | stale Ray state after a crash | `ray stop`, and delete the run's buffer db under `outputs/buffers/` if restarting from scratch | | two runs at once | not supported: one Ray cluster, all 8 GPUs per run | ## Credits Method and implementation are the work of the upstream authors: - **TCOD** — *Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents*, https://github.com/kokolerk/TCOD - **FutureBridge-OPD** — *Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillation*, https://github.com/ChenChiShui/FutureBridge-OPD This bundle only packages them for reproducible execution and adds the missing baseline configurations. Please cite the upstream work.