Upload README.md with huggingface_hub

README.md

@@ -22,30 +22,36 @@ A reasoning-enhanced version of [Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3
 
 ## Results
 
-| Model | GSM8K 8-shot CoT | GSM8K Zero-shot | Notes |
-|-------|:-----------------:|:---------------:|-------|
-| Qwen3.5-0.8B (baseline) | 53.5% | 52.1% | Pre-trained, no fine-tuning |
-| + SFT only | not evaluated | not evaluated | Teaches `<think>` format |
-| **+ SFT + GRPO (ours)** | 50.4% (-3.1pp) | **58.0% (+5.9pp)** | Internalized reasoning |
+| Eval Setting | GSM8K Accuracy | Notes |
+|---|:-:|---|
+| Baseline 8-shot CoT | 53.5% | Pre-trained, no fine-tuning |
+| Baseline zero-shot | 52.1% | Pre-trained, no fine-tuning |
+| **GRPO zero-shot** | **58.0% (+5.9pp)** | Best result — model reasons autonomously |
+| GRPO 8-shot (plain format) | 50.4% (-3.1pp) | Few-shot examples conflict with learned policy |
+| GRPO 8-shot (`<think>` aligned) | 34.1% (-19.4pp) | Format-aligned examples hurt even more |
 
-**Zero-shot improved by +5.9pp** — the model internalized step-by-step reasoning and no longer needs few-shot examples. The 8-shot drop (-3.1pp) is because few-shot examples conflict with the model's learned `<think>` tag format.
+### Key Finding: Demonstration to Policy Shift
 
-> **Honest note:** We did not evaluate the SFT-only checkpoint, so we cannot isolate whether the 8-shot regression came from SFT (format overwrite) or GRPO. See [Lessons Learned](#lessons-learned).
+GRPO training shifted the model from **demonstration-based reasoning** to **policy-based reasoning**.
+
+After training, the model:
+- **Performs best in zero-shot** — it reasons autonomously using `<think>` tags
+- **Is hurt by few-shot examples** — any demonstrations conflict with its learned internal reasoning policy
+- **Is hurt even more by format-aligned few-shot** — `<think>` tags in examples caused the model to confuse context with its own generation, dropping to 34.1%
+
+This is a behavioral shift, not a regression. The model no longer needs (or wants) demonstrations. This mirrors what DeepSeek-R1 demonstrated at 670B scale.
 
 ## Training Pipeline
 
 ### Phase 1: SFT Warmup
 - **Data:** [3,558 reasoning examples](https://huggingface.co/datasets/celestialcreator/Qwen3.5-0.8B-GRPO-Math-Dataset) from 3 sources, standardized to `<think>` tags
-  - 1K Claude Sonnet math chains (originally `<thought>` → converted to `<think>`)
-  - ~250 from TeichAI Opus reasoning (already `<think>`)
-  - ~2.3K from Opus 4.6 Reasoning (thinking/solution → combined into `<think>`)
 - **Purpose:** Solve the cold-start problem — teach the 0.8B model `<think>` tag format before RL exploration
 - **Stats:** 1 epoch, loss 0.932, 78% token accuracy
 
 ### Phase 2: GRPO Training
 - **Data:** GSM8K train split (7,473 math word problems)
 - **Rewards:** Math correctness (1.0/0.0) + format reward (0.3 for `<think>` tags, 0.2 for `####` answer)
-- **Config:** 8 generations/prompt, batch size 1 × 8 grad accum, lr 1e-6, β=0.04
+- **Config:** 8 generations/prompt, batch size 1 x 8 grad accum, lr 1e-6, beta=0.04
 - **Hardware:** Single NVIDIA RTX 5090 (32GB VRAM)
 - **Duration:** ~77 hours, 15,900 steps (epoch 2.13, rewards had plateaued)
 
@@ -62,19 +68,21 @@ Only 2 models in memory (policy + reference) instead of 4 — feasible on consum
 ## Lessons Learned
 
 ### What worked
-- **GRPO improved zero-shot reasoning** — +5.9pp on GSM8K, model internalized step-by-step thinking
+- **GRPO improved zero-shot reasoning** — +5.9pp, model internalized step-by-step thinking
+- **Demonstration to policy shift** — the model developed its own reasoning strategy instead of relying on examples
 - **Format + correctness rewards together** — `<think>` tag bonus helped learn structured reasoning alongside accuracy
-- **Single consumer GPU is viable** — full pipeline on one RTX 5090 with ~10-12 GB of 32 GB used
+- **Single consumer GPU is viable** — full pipeline on one RTX 5090
 
 ### What we'd do differently
-- **Eval after SFT** — we skipped this, so we can't isolate SFT's contribution. Critical missing data point.
+- **Eval after SFT** — we skipped this, so we can't isolate SFT's contribution
 - **Try GRPO without SFT** — ablation would show if SFT warmup is necessary or trades few-shot ability for format
-- **Larger model** — 0.8B is near capacity ceiling. Successful open GRPO reproductions start at 1.5B+.
+- **Larger model** — 0.8B is near capacity ceiling. Successful open GRPO reproductions start at 1.5B+
 
 ### Technical findings
 - **Qwen3.5 DeltaNet needs FLA** — install `flash-linear-attention` + `causal-conv1d`, otherwise torch fallback is ~10x slower
-- **SDPA > FLA for inference** — 3.6x faster first call. Use `attn_implementation="sdpa"` for eval
-- **Rewards plateau ~epoch 1.2** — math_reward at 0.45-0.53, diminishing returns beyond 2 epochs at this scale
+- **SDPA > FLA for inference** — 3.6x faster first call. Use `attn_implementation="sdpa"`
+- **Rewards plateau ~epoch 1.2** — diminishing returns beyond 2 epochs at this scale
+- **RL-trained models are few-shot sensitive** — even format-aligned examples hurt (34.1%), suggesting the model confuses example `<think>` tags with its own generation context
 
 ## Usage
 
@@ -90,6 +98,7 @@ model = AutoModelForCausalLM.from_pretrained(
     trust_remote_code=True,
 )
 
+# Best used in zero-shot — the model has its own reasoning policy
 messages = [
     {"role": "system", "content": "You are a helpful assistant that thinks step by step. Show your reasoning inside <think> tags before giving your final answer. End math answers with: #### <number>"},
     {"role": "user", "content": "If a train travels at 60 mph for 2.5 hours, how far does it go?"},
@@ -101,6 +110,8 @@ outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
 print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
 ```
 
+> **Note:** This model performs best in **zero-shot** mode. Do not use few-shot examples — they conflict with the model's learned reasoning policy and reduce accuracy.
+
 ## Training Code
 
 Full pipeline (Dockerfile, k8s configs, scripts): [github.com/CelestialCreator/gpu-lab/tree/main/projects/05-grpo-reasoning](https://github.com/CelestialCreator/gpu-lab/tree/main/projects/05-grpo-reasoning)