Upload README.md with huggingface_hub

README.md

@@ -22,26 +22,32 @@ A reasoning-enhanced version of [Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3
 
 ## Results
 
-| Model | GSM8K 8-shot CoT | GSM8K Zero-shot |
-|-------|:-----------------:|:---------------:|
-| Qwen3.5-0.8B (baseline) | 53.5% | 52.1% |
-| **Qwen3.5-0.8B-GRPO-Math** | 50.4% | **58.0% (+5.9pp)** |
+| Model | GSM8K 8-shot CoT | GSM8K Zero-shot | Notes |
+|-------|:-----------------:|:---------------:|-------|
+| Qwen3.5-0.8B (baseline) | 53.5% | 52.1% | Pre-trained, no fine-tuning |
+| + SFT only | not evaluated | not evaluated | Teaches `<think>` format |
+| **+ SFT + GRPO (ours)** | 50.4% (-3.1pp) | **58.0% (+5.9pp)** | Internalized reasoning |
 
-The model was trained to reason using `<think>` tags. **Zero-shot performance improved by +5.9 percentage points** because the model internalized step-by-step reasoning — it no longer needs few-shot examples. The 8-shot drop is expected: few-shot examples conflict with the model's learned reasoning format.
+**Zero-shot improved by +5.9pp** — the model internalized step-by-step reasoning and no longer needs few-shot examples. The 8-shot drop (-3.1pp) is because few-shot examples conflict with the model's learned `<think>` tag format.
+
+> **Honest note:** We did not evaluate the SFT-only checkpoint, so we cannot isolate whether the 8-shot regression came from SFT (format overwrite) or GRPO. See [Lessons Learned](#lessons-learned).
 
 ## Training Pipeline
 
 ### Phase 1: SFT Warmup
-- **Data:** [3,558 reasoning examples](https://huggingface.co/datasets/celestialcreator/Qwen3.5-0.8B-GRPO-Math-Dataset) (Claude-generated math chains + community reasoning data)
-- **Purpose:** Teach the model `<think>` tag format before RL
+- **Data:** [3,558 reasoning examples](https://huggingface.co/datasets/celestialcreator/Qwen3.5-0.8B-GRPO-Math-Dataset) from 3 sources, standardized to `<think>` tags
+  - 1K Claude Sonnet math chains (originally `<thought>` → converted to `<think>`)
+  - ~250 from TeichAI Opus reasoning (already `<think>`)
+  - ~2.3K from Opus 4.6 Reasoning (thinking/solution → combined into `<think>`)
+- **Purpose:** Solve the cold-start problem — teach the 0.8B model `<think>` tag format before RL exploration
 - **Stats:** 1 epoch, loss 0.932, 78% token accuracy
 
 ### Phase 2: GRPO Training
 - **Data:** GSM8K train split (7,473 math word problems)
 - **Rewards:** Math correctness (1.0/0.0) + format reward (0.3 for `<think>` tags, 0.2 for `####` answer)
-- **Config:** 8 generations/prompt, batch size 1 x 8 grad accum, lr 1e-6, beta=0.04
+- **Config:** 8 generations/prompt, batch size 1 × 8 grad accum, lr 1e-6, β=0.04
 - **Hardware:** Single NVIDIA RTX 5090 (32GB VRAM)
-- **Duration:** ~77 hours, 15,900 steps (epoch 2.13)
+- **Duration:** ~77 hours, 15,900 steps (epoch 2.13, rewards had plateaued)
 
 ### What is GRPO?
 
@@ -51,7 +57,24 @@ GRPO eliminates the need for a separate reward model and critic network (unlike
 3. Normalizes rewards within the group (relative advantage)
 4. Updates the policy using a clipped surrogate objective
 
-This means only 2 models in memory (policy + reference) instead of 4, making it feasible on consumer GPUs.
+Only 2 models in memory (policy + reference) instead of 4 — feasible on consumer GPUs.
+
+## Lessons Learned
+
+### What worked
+- **GRPO improved zero-shot reasoning** — +5.9pp on GSM8K, model internalized step-by-step thinking
+- **Format + correctness rewards together** — `<think>` tag bonus helped learn structured reasoning alongside accuracy
+- **Single consumer GPU is viable** — full pipeline on one RTX 5090 with ~10-12 GB of 32 GB used
+
+### What we'd do differently
+- **Eval after SFT** — we skipped this, so we can't isolate SFT's contribution. Critical missing data point.
+- **Try GRPO without SFT** — ablation would show if SFT warmup is necessary or trades few-shot ability for format
+- **Larger model** — 0.8B is near capacity ceiling. Successful open GRPO reproductions start at 1.5B+.
+
+### Technical findings
+- **Qwen3.5 DeltaNet needs FLA** — install `flash-linear-attention` + `causal-conv1d`, otherwise torch fallback is ~10x slower
+- **SDPA > FLA for inference** — 3.6x faster first call. Use `attn_implementation="sdpa"` for eval
+- **Rewards plateau ~epoch 1.2** — math_reward at 0.45-0.53, diminishing returns beyond 2 epochs at this scale
 
 ## Usage
 
@@ -80,14 +103,7 @@ print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_t
 
 ## Training Code
 
-Full training pipeline (Dockerfile, k8s configs, scripts): [github.com/CelestialCreator/gpu-lab/tree/main/projects/05-grpo-reasoning](https://github.com/CelestialCreator/gpu-lab/tree/main/projects/05-grpo-reasoning)
-
-## Key Findings
-
-- **Qwen3.5-0.8B uses DeltaNet** (hybrid Gated DeltaNet + Gated Attention layers). Install `flash-linear-attention` + `causal-conv1d` for fast generation.
-- **SDPA > FLA for inference** — 3.6x faster on first call. Use `attn_implementation="sdpa"`.
-- **Zero-shot is the right eval for RL-trained reasoning models** — few-shot examples conflict with learned reasoning patterns.
-- **0.8B is near the capacity ceiling for GRPO** — the model internalizes reasoning format but has limited room for math accuracy gains. Consider 1.5B+ for stronger results.
+Full pipeline (Dockerfile, k8s configs, scripts): [github.com/CelestialCreator/gpu-lab/tree/main/projects/05-grpo-reasoning](https://github.com/CelestialCreator/gpu-lab/tree/main/projects/05-grpo-reasoning)
 
 ## Acknowledgments