| # Spike 002a — Trace Collection via TRL + OpenEnv |
|
|
| > **Risk:** MEDIUM. Validates whether TRL's `GRPOTrainer` + OpenEnv environment registry produce clean, schema-stable trace JSONL. |
| > **Status:** 📋 planned (depends on 001 verdict) |
| > **Comparison spike:** runs head-to-head with `002b-trace-collection-prime-rl/`. |
|
|
| ## Question (Given / When / Then) |
|
|
| **Given** Qwen3-7B base + TRL `GRPOTrainer` + a SWE-bench-lite OpenEnv, |
| **when** we run 100 rollouts, |
| **then** all rollouts emit complete `(state_t, action_t, reward_t)` tuples to JSONL with no truncation or schema drift, and the JSONL is loadable by spike 003 without preprocessing. |
|
|
| ## Approach (TBD) |
|
|
| 1. Set up a minimal TRL `GRPOTrainer` config pointing at SWE-bench-lite as an OpenEnv environment. |
| 2. Run 100 rollouts, capturing trace tuples to `traces.jsonl`. |
| 3. Verify schema, count truncations, count missing reward signals. |
|
|
| ## Why this risk-tier |
|
|
| If the trace stream is dirty (missing fields, schema drift mid-rollout, truncated states), spike 003 (DPO-pair extraction) gets nothing useful. But this risk is *medium* not *high* because both TRL and OpenEnv are well-tested upstream — the fail mode is integration glue, not feasibility. |
|
|
| ## Files (planned) |
|
|
| - `setup.py` — TRL + verifiers + transformers + accelerate install |
| - `train_config.py` — minimal `GRPOConfig` |
| - `run_rollout.py` — collect 100 rollouts to traces.jsonl |
| - `validate_schema.py` — schema check + completeness stats |
| - `traces.jsonl` (gitignored — large; uploaded to dataset repo) |
| - `verdict.md` — final verdict |
|
|
| ## Hardware |
|
|
| - 1× A100 80GB on Modal (per `modal-llm-training` skill) |
| - Wallclock estimate: ~4–8 hours for 100 rollouts (depends on rollout length) |
|
|
| ## Blocked on |
|
|
| Spike 001 verdict. If 001 fails, this spike is moot. |
|
|
