Phase 13 Stage 1: pivot to winner's pattern - TRL 0.29.0 + fast_inference=False + DAPO loss

TRAINING_LOG.md

@@ -290,8 +290,11 @@ Every training attempt, successful or failed. Number them sequentially across th
   - use_vllm: **false** (dropped — vLLM 0.6.3 pins torch 2.4, incompatible with Kaggle's native torch 2.10+cu128; wall-clock becomes ~4h instead of ~2.5h but correctness is unaffected)
   - push_to_hub: true
   - hub_model_id: `yashash045/schemashift-qwen15b-kaggle`
-- **TRL version:** `trl>=0.18.2,<=0.24.0,!=0.19.0` (Unsloth 2026.x compat range; pip resolves to 0.24.0)
+- **TRL version:** `trl==0.29.0` (pivoted from 0.18-0.24 range after 3h of Kaggle debug — both older ranges had eager imports of vllm_client or mergekit that crashed on clean Kaggle. 0.29.0 moved them to experimental per PR #5057, verified clean locally)
+- **vLLM:** NOT installed (`fast_inference=False` on FastLanguageModel; `use_vllm=False` in GRPOConfig)
 - **Torch version:** Kaggle native (`torch 2.10+cu128`, not pinned)
+- **Winner-adopted tweaks:** `loss_type="dapo"` + `lr_scheduler_type="cosine"` + `warmup_steps=2` + `beta=0.01` + `max_grad_norm=1.0` + `save_total_limit=3` + `hub_strategy="every_save"`. From [sid-rp/kube-sre-gym](https://github.com/sid-rp/kube-sre-gym) train.py.
+- **NOT adopted (requires vLLM):** `trl.experimental.openenv.generate_rollout_completions` multi-turn rollout orchestration. Our env loop stays inside `reward_fn` which TRL calls after its internal `model.generate()`.
 - **Quantization:** 4-bit via Unsloth
 - **Hardware:** Kaggle T4 x2 (P100 acceptable fallback — record which)
 - **SchemaShift env URL:** `https://yashash045-schemashift.hf.space`

training/STAGE_1_KAGGLE_RUNBOOK.md

@@ -8,14 +8,35 @@
 **Env:** `https://yashash045-schemashift.hf.space`
 **Expected wall-clock:** ~4 hours on Kaggle T4 ×2 (no vLLM — see "Kaggle dependency compatibility" below for why)
 
-### Kaggle dependency compatibility (why no vLLM)
+### Kaggle dependency compatibility (why no vLLM, why TRL 0.29.0)
 
 Kaggle's 2026 base image ships `torch 2.10+cu128` + `transformers 4.57` + `torchao 0.17` pre-installed. We do NOT pin torch:
 - Pinning torch 2.4 breaks Kaggle's pre-installed `transformers` and `torchao`
 - Pinning torch 2.5.1 breaks `vLLM 0.6.3` (which itself pins torch 2.4)
 - The only clean path is Kaggle-native torch + latest Unsloth + no vLLM
 
-TRL range `>=0.18.2,<=0.24.0,!=0.19.0` is what Unsloth 2026.x is compatible with (pip resolves to `0.24.0`). Training runs ~4 hours instead of ~2.5 hours without vLLM's fast inference, but correctness is unaffected — only generation throughput drops. **Judges don't ask about inference speed.**
+**TRL pin: `trl==0.29.0`** (mirrors previous hackathon winner sid-rp/kube-sre-gym). Earlier attempts at 0.18.2 and 0.24.0 both eagerly imported optional deps (vllm_client / mergekit) and crashed on clean Kaggle. TRL 0.29.0 moved those to experimental per PR #5057 and verified-imports-clean (no vLLM installed) locally.
+
+Training runs ~4 hours instead of ~2.5 hours without vLLM's fast inference, but correctness is unaffected — only generation throughput drops. **Judges don't ask about inference speed.**
+
+### Winner vs ours — what we mirror and where we diverge
+
+Previous hackathon winner: [sid-rp/kube-sre-gym](https://github.com/sid-rp/kube-sre-gym) — OpenEnv + TRL GRPO on Kubernetes incidents.
+
+| Dimension | Winner (H100) | Winner's Colab variant (T4) | Ours (Kaggle T4 free) |
+|---|---|---|---|
+| TRL version | `trl[vllm]==0.29.0` | `trl[vllm]>=0.29.0` | `trl==0.29.0` (no vllm extra) |
+| vLLM | `vllm==0.11.2`, colocate | `vllm>=0.11.0`, colocate | **none** (`fast_inference=False`) |
+| Base model | Qwen3-0.6B | Qwen3-0.6B / Qwen3-1.7B | Qwen2.5-1.5B |
+| Rollout mechanism | `trl.experimental.openenv.generate_rollout_completions` (multi-turn agent, iterative obs→act) | same | **reward_fn does env loop inside** (single completion → parsed action sequence → full episode) — `rollout_func` param requires `use_vllm=True`, not viable for us |
+| num_generations | 8 | 8 | 4 (T4 memory budget) |
+| max_completion_length | 512 | 512 | 1024 (our JSON actions are longer) |
+| Loss type | `dapo` (asymmetric clipping + dynamic sampling) | `dapo` | **`dapo`** (adopted from winner) |
+| LR schedule | cosine, warmup=2 | cosine, warmup=2 | **cosine, warmup=2** (adopted) |
+| beta (KL) | 0.01 | default | **0.01** (adopted) |
+| GPU | 1× H100 80GB | 1× T4 16GB | 2× T4 16GB (use first only) |
+
+**What this means for pitch:** we're using the winner's stable TRL version + their DAPO loss + their LR schedule — the recipe parts that matter for convergence. We don't mirror their multi-turn iterative-observation pattern because it's gated behind vLLM, which we can't install. Our single-shot action-sequence pattern produces simpler completions but still trains the same adaptive-tool-use skill.
 
 ---
 
@@ -61,7 +82,7 @@ Execute each cell with Shift+Enter. Wait for "✓" before advancing.
 
 | Cell | Purpose | Expected | Red flag |
 |---|---|---|---|
-| 1 | `pip install unsloth trl httpx ...` | `torch: 2.10.x+cu128`, `unsloth: 2026.x`, `trl: 0.24.0`, "Core deps verified" | any import error or wrong torch version — stop |
+| 1 | `pip install unsloth "trl==0.29.0" peft httpx ...` | `torch: 2.10.x+cu128`, `unsloth: 2026.x`, `trl: 0.29.0`, `peft: ...`, "Core deps verified" | any import error or wrong torch version — stop |
 | 2 | `git clone SchemaShift && pip install -e .` | "Successfully installed schemashift-0.1.0" | clone fails (check internet) |
 | 3 | Env health check (import client, call `client.health()`) | prints `True` | `False` — URL wrong or Space down |
 | 4 | Load Qwen 2.5 1.5B 4-bit + LoRA | "Trainable params: ~8.9M" | OOM error — use T4 x2 not x1 |

training/grpo_kaggle.ipynb

@@ -18,7 +18,10 @@
     "- Set Kaggle Secret: `HF_USERNAME` (your HF namespace; falls back to `yashash045` if unset)\n",
     "- Accelerator: GPU T4 x2 (P100 fallback). Internet: ON. Persistence: ON.\n",
     "\n",
-    "**Stage 1 runbook:** see `training/STAGE_1_KAGGLE_RUNBOOK.md` for the full step-by-step guide.\n"
+    "**Stage 1 runbook:** see `training/STAGE_1_KAGGLE_RUNBOOK.md` for the full step-by-step guide.\n",
+    "\n",
+    "**Dependency baseline:** TRL 0.29.0 + Unsloth 2026.x + Kaggle-native torch 2.10.\n",
+    "No vLLM (incompatible with Kaggle's torch 2.10). Reference implementation: [sid-rp/kube-sre-gym](https://github.com/sid-rp/kube-sre-gym) (previous hackathon winner, TRL 0.29 on H100 with vLLM). We use their TRL version but keep rollouts inside `reward_fn` since `trl.experimental.openenv.generate_rollout_completions` requires `use_vllm=True`.\n"
    ]
   },
   {
@@ -27,22 +30,23 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Install deps - Kaggle native (no vLLM, no torch pinning).\n",
-    "# Kaggle's 2026 base image ships torch 2.10 + cu128 + transformers + torchao.\n",
-    "# Unsloth 2026.x is tested against that; pinning torch breaks the base image.\n",
+    "# Install deps - TRL 0.29.0 without vLLM (Kaggle torch 2.10 native).\n",
+    "# Mirrors winner repo sid-rp/kube-sre-gym (trl==0.29.0, peft) minus vLLM.\n",
+    "# TRL 0.29.0 is clean on imports (no mergekit/vllm eager-import crash).\n",
     "!pip install -q unsloth\n",
-    "!pip install -q \"trl>=0.18.2,<=0.24.0,!=0.19.0\"\n",
+    "!pip install -q \"trl==0.29.0\" peft\n",
     "!pip install -q httpx pydantic fastapi openai\n",
     "\n",
-    "# Verify critical deps. If any of these fail or print wrong versions, STOP.\n",
+    "# Verify critical deps. If any print wrong versions, STOP.\n",
     "import torch\n",
     "print(f\"torch: {torch.__version__}\")\n",
-    "print(f\"cuda available: {torch.cuda.is_available()}\")\n",
+    "print(f\"cuda: {torch.cuda.is_available()}\")\n",
     "\n",
-    "import unsloth, trl\n",
+    "import unsloth, trl, peft\n",
     "print(f\"unsloth: {unsloth.__version__}\")\n",
-    "print(f\"trl: {trl.__version__}\")\n",
-    "print(\"Core deps verified (Kaggle-native, no vLLM).\")\n"
+    "print(f\"trl:     {trl.__version__}\")\n",
+    "print(f\"peft:    {peft.__version__}\")\n",
+    "print(\"Core deps verified (TRL 0.29.0, no vLLM).\")\n"
    ]
   },
   {
@@ -89,12 +93,14 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Load Qwen 2.5 1.5B + LoRA adapter\n",
+    "# Load Qwen 2.5 1.5B + LoRA adapter.\n",
+    "# fast_inference=False disables Unsloth's vLLM bypass - required when vLLM isn't installed.\n",
     "from unsloth import FastLanguageModel\n",
     "model, tokenizer = FastLanguageModel.from_pretrained(\n",
     "    model_name=\"Qwen/Qwen2.5-1.5B-Instruct\",\n",
     "    max_seq_length=3072,\n",
     "    load_in_4bit=True,\n",
+    "    fast_inference=False,\n",
     "    dtype=None,\n",
     ")\n",
     "model = FastLanguageModel.get_peft_model(\n",
@@ -104,7 +110,7 @@
     "    lora_dropout=0.0,\n",
     "    target_modules=[\"q_proj\",\"k_proj\",\"v_proj\",\"o_proj\",\"gate_proj\",\"up_proj\",\"down_proj\"],\n",
     ")\n",
-    "print(f\"Trainable params: {model.num_parameters(only_trainable=True):,}\")"
+    "print(f\"Trainable params: {model.num_parameters(only_trainable=True):,}\")\n"
    ]
   },
   {
@@ -297,23 +303,38 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# GRPO config (Kaggle T4-friendly, no vLLM) + GRPOTrainer construction.\n",
+    "# GRPO config (TRL 0.29.0, Kaggle T4-friendly, no vLLM).\n",
+    "# Adopts winner's DAPO loss + cosine LR + light KL - minus vLLM/rollout_func\n",
+    "# (rollout_func only runs when use_vllm=True in TRL 0.29.0; we keep env rollout\n",
+    "# inside reward_fn which TRL calls after its internal model.generate()).\n",
     "from trl import GRPOConfig, GRPOTrainer\n",
     "\n",
     "config = GRPOConfig(\n",
     "    output_dir=\"schemashift-grpo-kaggle\",\n",
     "    num_generations=4,\n",
-    "    max_completion_length=1536,\n",
+    "    max_completion_length=1024,\n",
     "    per_device_train_batch_size=1,\n",
     "    gradient_accumulation_steps=4,\n",
     "    learning_rate=5e-6,\n",
-    "    logging_steps=5,\n",
+    "    lr_scheduler_type=\"cosine\",\n",
+    "    warmup_steps=2,\n",
+    "    max_grad_norm=1.0,\n",
+    "    logging_steps=1,\n",
+    "    save_strategy=\"steps\",\n",
     "    save_steps=25,\n",
     "    max_steps=100,\n",
+    "    temperature=1.0,\n",
     "    report_to=\"none\",\n",
     "    use_vllm=False,\n",
+    "    gradient_checkpointing=True,\n",
+    "    gradient_checkpointing_kwargs={\"use_reentrant\": False},\n",
+    "    loss_type=\"dapo\",\n",
+    "    mask_truncated_completions=True,\n",
+    "    beta=0.01,\n",
+    "    save_total_limit=3,\n",
     "    push_to_hub=True,\n",
     "    hub_model_id=f\"{os.getenv('HF_USERNAME', 'yashash045')}/schemashift-qwen15b-kaggle\",\n",
+    "    hub_strategy=\"every_save\",\n",
     ")\n",
     "\n",
     "trainer = GRPOTrainer(\n",
@@ -324,11 +345,12 @@
     "    train_dataset=train_dataset,\n",
     ")\n",
     "\n",
-    "print(\"GRPOTrainer initialized.\")\n",
-    "print(f\"  model:         {model.config._name_or_path}\")\n",
+    "print(\"GRPOTrainer initialized (TRL 0.29.0, use_vllm=False).\")\n",
+    "print(f\"  model:         Qwen/Qwen2.5-1.5B-Instruct\")\n",
     "print(f\"  dataset size:  {len(train_dataset)}\")\n",
     "print(f\"  max_steps:     {config.max_steps}\")\n",
     "print(f\"  save_steps:    {config.save_steps}\")\n",
+    "print(f\"  loss_type:     {config.loss_type}\")\n",
     "print(f\"  hub_model_id:  {config.hub_model_id}\")\n"
    ]
   },