| --- |
| license: apache-2.0 |
| tags: |
| - reinforcement-learning |
| - on-policy-distillation |
| - llm-agents |
| - scienceworld |
| language: |
| - en |
| --- |
| |
| # ftb-sciworld-repro |
|
|
| A self-contained, runnable bundle for training **five multi-turn on-policy |
| distillation methods on ScienceWorld** with a Qwen3-1.7B student and a Qwen3-32B |
| teacher: |
|
|
| | method | what it does | |
| | --- | --- | |
| | **OPD** | vanilla on-policy distillation: the student runs whole episodes alone, the teacher scores its tokens afterwards | |
| | **Guided-OPD** | per-turn coin flip picks teacher or student; teacher turns get SFT cross-entropy, student turns get the OPD reverse-KL | |
| | **TCOD-B2F** | replays a gold-action prefix, student takes over near the end; the takeover point walks backwards over training | |
| | **TCOD-F2B** | student always starts at turn 1, but episodes are truncated short and grow over training | |
| | **FTB** | FutureBridge: inserts a validated teacher "bridge" at the highest-disagreement student turn | |
|
|
| Everything needed to run is in here except the model weights, the task data and |
| the Python dependencies, which `scripts/setup.sh` fetches. |
|
|
| ## Requirements |
|
|
| - **8 GPUs with ≥ 80 GB each, on one node.** The teacher is dense Qwen3-32B at |
| TP2 — roughly 32 GB of weights per device before any KV cache — so 40 GB cards |
| cannot hold it at this parallelism. The split is 4 student rollout + 2 teacher |
| + 2 FSDP trainer. |
| - Linux with CUDA and NCCL |
| - Python 3.10 |
| - **Java 17+ on PATH** — ScienceWorld starts a JVM per episode |
| - ~80 GB free disk for models, plus ~4 GB per saved checkpoint (4 per run) |
|
|
| ## Quick start |
|
|
| ```bash |
| hf download SeanWang0027/ftb-sciworld-repro --local-dir ftb-sciworld-repro |
| cd ftb-sciworld-repro |
| |
| # If the node already has a working Trinity/TCOD environment: |
| bash scripts/setup.sh |
| |
| # If it does not, also install the pinned dependency set: |
| bash scripts/setup.sh --with-deps |
| |
| # Reuse models already on the node instead of downloading ~68 GB: |
| bash scripts/setup.sh --models /path/to/models # expects Qwen3-1.7B, Qwen3-32B |
| |
| # Then, one method at a time (each wants all 8 GPUs): |
| bash scripts/run.sh ftb |
| bash scripts/run.sh opd |
| bash scripts/run.sh guided_opd |
| bash scripts/run.sh tcod_f2b |
| bash scripts/run.sh tcod_b2f |
| ``` |
|
|
| Behind a scheduler: `sbatch -A <acct> -p <partition> --export=ALL,METHOD=ftb scripts/run_slurm.sbatch`. |
|
|
| (The Hub does not preserve the executable bit, so invoke the scripts as |
| `bash scripts/...` — as above — rather than `./scripts/...`.) |
|
|
| `setup.sh` is idempotent — re-run it after a partial failure. It pip-installs |
| the vendored `tcod/` (with `--no-deps` unless you pass `--with-deps`), copies the |
| FutureBridge overlay over it, **verifies all five workflow classes import**, |
| puts the models under `./models`, downloads the task split to |
| `./data/scienceworld`, and creates `./outputs/`. |
|
|
| A run is 200 training steps, batch 16 tasks × up to 30 environment turns, with a |
| checkpoint every 50 steps under |
| `outputs/checkpoints/FutureBridge-OPD/<run_name>/global_step_*/`. Expect hours to |
| a day per method; B2F and FTB are slowest (they boot two ScienceWorld JVMs per |
| training task for gold-path replay). |
|
|
| ## What is upstream and what is not |
|
|
| This matters if you are reproducing published numbers. |
|
|
| **Upstream, redistributed unmodified** (both Apache-2.0, see `NOTICE`): |
|
|
| - `tcod/` — [kokolerk/TCOD](https://github.com/kokolerk/TCOD) at commit |
| `17a8af2`, exported with `git archive` |
| - `overlay/` — [ChenChiShui/FutureBridge-OPD](https://github.com/ChenChiShui/FutureBridge-OPD) |
| `source/` at commit `bfcdf91` |
| - `configs/ftb.yaml` — verbatim from that release |
|
|
| **Added here:** `configs/{opd,guided_opd,tcod_b2f,tcod_f2b}.yaml`, |
| `configs/bench.yaml.tmpl`, everything in `scripts/`, this README, `NOTICE`. |
|
|
| The FutureBridge release ships ScienceWorld configs for FutureBridge and its |
| ablations only. For the four baselines it ships the *workflow implementations* |
| but no configuration — instead its `configs/README.md` gives a "Baseline workflow |
| mappings" table saying to swap `default_workflow_type` and keep the paper |
| settings. The four baseline YAMLs here are a reconstruction from that table: |
| every shared setting is identical to `ftb.yaml`, and the deltas are the workflow |
| class, the method's own `workflow_args`, run/buffer names, the monitor backend |
| (tensorboard rather than wandb), and — for Guided-OPD only — the `mix` |
| policy-loss block. |
|
|
| **Judgement calls the upstream table does not determine**, in case you want to |
| change them: |
|
|
| | setting | value here | basis | |
| | --- | --- | --- | |
| | B2F / F2B `checkpoint_steps` | 5 / 6 | upstream TCOD's own `TCOD_examples/scienceworld/tcod_{b2f,f2b}.yaml` | |
| | Guided-OPD `beta_start/end`, `curriculum_ratio` | 1.0 / 0.0 / 0.8 | defaults in `guided_opd_workflow.py`'s own schedule function | |
| | Guided-OPD **`mu`** | **0.5** | weighs the forward-KL (teacher-turn SFT) and reverse-KL (student-turn OPD) terms equally. `MIXPolicyLossFn`'s own default is **0.1**. Nothing in the release fixes this value — treat it as a hyperparameter, not a reproduction. | |
| | `mix` batch-size args | derived from the 8-GPU split | not specified upstream | |
|
|
| ## Reporting results |
|
|
| **The upstream protocol does not evaluate.** Every released config sets |
| `eval_interval: 9999` and `eval_on_startup: false`, so the declared eval task set |
| never fires, and no evaluation entry point is shipped. Its |
| `configs/README.md` says to "run the configured evaluation task set once after |
| training" but provides no mechanism. |
|
|
| What the release's own analysis code reads is the **training** rollout metric — |
| `analysis/plot_training_dynamics.py` maps `rollout/env_done/mean` to the panel |
| labelled "Completion Rate". So the faithful reproduction is: run the five |
| methods and plot `rollout/env_done/mean` against step. |
|
|
| One caveat if you compare those curves across methods: the training rollout is |
| not produced by the same policy in each arm. OPD's trajectories are 100% student; |
| B2F's begin with a gold-action prefix; Guided-OPD's contain teacher turns with |
| probability β (≈1 early in training); FTB's contain both. Early-training |
| completion is therefore partly the teacher's or the gold path's work, not the |
| student's. |
|
|
| `scripts/eval.sh` is provided as an **addition, not a reproduction**: it scores a |
| run's checkpoints on the held-out 1,308-task test split through a single shared |
| workflow, so the arms are measured under identical conditions with no teacher |
| intervention. See the header of that script for why it forces one workflow for |
| all five methods. |
|
|
| Either way the metric is `env_done`. **There is no score metric anywhere**, here |
| or upstream: ScienceWorld's 0–100 partial credit is computed and stored on |
| `Experience.reward`, which never reaches the metrics dictionary that the monitor |
| aggregates. And `env_done` counts any terminating episode — ScienceWorld also |
| terminates on *failure*, by driving the score negative — so it is not a clean |
| success rate. |
|
|
| ## Known properties worth knowing before you interpret runs |
|
|
| These are properties of the upstream implementation, not bugs introduced here. |
| None of them are patched in this bundle. |
|
|
| **1. The prompt context accumulates, and overflow stops the agent.** Four of the |
| five workflows (all but Guided-OPD, which keeps a 10-turn window) send the entire |
| growing chat history every turn, on top of a 2-step summary embedded in each user |
| message. Measured with the Qwen3-1.7B tokenizer on real ScienceWorld rooms, the |
| prompt crosses `max_prompt_tokens: 10240` somewhere around turn 11 (object-rich |
| room) to turn 17 (sparse room) of a 30-turn episode. Past that point the vLLM |
| engine is *not called at all*: the framework returns a placeholder experience |
| with an all-zero action mask, so the turn contributes no gradient, the parsed |
| action is empty, and the agent stops acting for the rest of the episode. Grep a |
| run log for `Prompt was truncated to` to count how often it happens. |
|
|
| Upstream hit a more severe version of this and fixed half of it: TCOD commit |
| `17a8af2` ("use compact action representation to avoid token overflow") replaced |
| a ~1162-item full action×object list with a ~43-item templates+objects pair, |
| because *single-step* prompts were already exceeding the limit. That shrank the |
| per-turn prompt; it did not change the accumulation. |
|
|
| **2. The `<think>` instruction is inert.** Both the system prompt and every user |
| message require reasoning "enclosed within `<think> </think>` tags", but the |
| configs run Qwen3 with `enable_thinking: false`, and that chat template |
| pre-emits *and closes* an empty `<think>\n\n</think>\n\n` before the model writes |
| a token. Across 11,264 recorded generation turns from three models in earlier |
| measurements, `<think>` opened in 0.0% of them. The instruction costs tokens |
| every turn and `parse_action` only ever reads `<action>`. |
|
|
| **3. The full action×object list is hidden from the model but used to judge it.** |
| FTB's `_is_valid_action` checks a proposed action against |
| `get_valid_action_object_combinations()` — the same ~1000-item list that commit |
| `17a8af2` removed from the prompt. The student must compose a string that lands |
| exactly in that set from two separate short lists, and a miss discards the whole |
| bridge candidate. |
|
|
| **4. FTB is B2F plus bridging.** `FutureBridgeScienceWorldWorkflow` subclasses |
| `TCOD_b2f_scienceworld_workflow` and overrides only `_finalize_turn_responses`. |
| If you want to know whether FTB's gain comes from the bridge or from the B2F |
| curriculum underneath it, `tcod_b2f` is the controlled comparison — not `opd`. |
|
|
| ## Troubleshooting |
|
|
| | symptom | cause / fix | |
| | --- | --- | |
| | `Failed to look up actor 'synchronizer'` during weight sync | an engine was set to TP1; keep `tensor_parallel_size: 2` for both rollout and teacher | |
| | teacher engine OOM at startup | cards smaller than 80 GB; there is no supported fallback at this parallelism | |
| | wandb hangs then kills the explorer | only `configs/ftb.yaml` uses wandb (it is the upstream file). `wandb login`, or switch its `monitor_type` to tensorboard like the four baselines. `WANDB_MODE=offline` does not reach the Ray actor. | |
| | `ModuleNotFoundError: scienceworld`, or JVM errors mid-run | `pip install scienceworld==1.2.2`; confirm `java -version` works | |
| | `flash-attn` build failure | it compiles against the installed torch; install torch first, then `pip install flash-attn==2.8.1 --no-build-isolation`. Training can proceed without it. | |
| | stale Ray state after a crash | `ray stop`, and delete the run's buffer db under `outputs/buffers/` if restarting from scratch | |
| | two runs at once | not supported: one Ray cluster, all 8 GPUs per run | |
|
|
| ## Credits |
|
|
| Method and implementation are the work of the upstream authors: |
|
|
| - **TCOD** — *Exploring Temporal Curriculum in On-Policy Distillation for |
| Multi-turn Autonomous Agents*, https://github.com/kokolerk/TCOD |
| - **FutureBridge-OPD** — *Look Ahead Before You Distill: Future Trajectory |
| Validation of Teacher Guidance for Agentic On-Policy Distillation*, |
| https://github.com/ChenChiShui/FutureBridge-OPD |
|
|
| This bundle only packages them for reproducible execution and adds the missing |
| baseline configurations. Please cite the upstream work. |
|
|
