Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,107 @@
+---
+language:
+- en
+license: apache-2.0
+base_model: Qwen/Qwen3.5-0.8B
+tags:
+- reasoning
+- math
+- grpo
+- reinforcement-learning
+- rlvr
+- qwen3.5
+datasets:
+- gsm8k
+- celestialcreator/Qwen3.5-0.8B-GRPO-Math-Dataset
+pipeline_tag: text-generation
+---
+
+# Qwen3.5-0.8B-GRPO-Math
+
+A reasoning-enhanced version of [Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B), trained using **GRPO (Group Relative Policy Optimization)** — the RL technique behind DeepSeek-R1 — on a single RTX 5090.
+
+## Results
+
+| Model | GSM8K 8-shot CoT | GSM8K Zero-shot |
+|-------|:-----------------:|:---------------:|
+| Qwen3.5-0.8B (baseline) | 53.5% | 52.1% |
+| **Qwen3.5-0.8B-GRPO-Math** | 50.4% | **58.0% (+5.9pp)** |
+
+The model was trained to reason using `<think>` tags. **Zero-shot performance improved by +5.9 percentage points** because the model internalized step-by-step reasoning — it no longer needs few-shot examples. The 8-shot drop is expected: few-shot examples conflict with the model's learned reasoning format.
+
+## Training Pipeline
+
+### Phase 1: SFT Warmup
+- **Data:** [3,558 reasoning examples](https://huggingface.co/datasets/celestialcreator/Qwen3.5-0.8B-GRPO-Math-Dataset) (Claude-generated math chains + community reasoning data)
+- **Purpose:** Teach the model `<think>` tag format before RL
+- **Stats:** 1 epoch, loss 0.932, 78% token accuracy
+
+### Phase 2: GRPO Training
+- **Data:** GSM8K train split (7,473 math word problems)
+- **Rewards:** Math correctness (1.0/0.0) + format reward (0.3 for `<think>` tags, 0.2 for `####` answer)
+- **Config:** 8 generations/prompt, batch size 1 x 8 grad accum, lr 1e-6, beta=0.04
+- **Hardware:** Single NVIDIA RTX 5090 (32GB VRAM)
+- **Duration:** ~77 hours, 15,900 steps (epoch 2.13)
+
+### What is GRPO?
+
+GRPO eliminates the need for a separate reward model and critic network (unlike PPO). For each prompt, it:
+1. Samples G completions from the policy
+2. Scores each with a verifiable reward (exact math answer checking)
+3. Normalizes rewards within the group (relative advantage)
+4. Updates the policy using a clipped surrogate objective
+
+This means only 2 models in memory (policy + reference) instead of 4, making it feasible on consumer GPUs.
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_name = "celestialcreator/Qwen3.5-0.8B-GRPO-Math"
+tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    torch_dtype="auto",
+    device_map="auto",
+    trust_remote_code=True,
+)
+
+messages = [
+    {"role": "system", "content": "You are a helpful assistant that thinks step by step. Show your reasoning inside <think> tags before giving your final answer. End math answers with: #### <number>"},
+    {"role": "user", "content": "If a train travels at 60 mph for 2.5 hours, how far does it go?"},
+]
+
+text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = tokenizer(text, return_tensors="pt").to(model.device)
+outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
+print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
+```
+
+## Training Code
+
+Full training pipeline (Dockerfile, k8s configs, scripts): [github.com/CelestialCreator/gpu-lab/tree/main/projects/05-grpo-reasoning](https://github.com/CelestialCreator/gpu-lab/tree/main/projects/05-grpo-reasoning)
+
+## Key Findings
+
+- **Qwen3.5-0.8B uses DeltaNet** (hybrid Gated DeltaNet + Gated Attention layers). Install `flash-linear-attention` + `causal-conv1d` for fast generation.
+- **SDPA > FLA for inference** — 3.6x faster on first call. Use `attn_implementation="sdpa"`.
+- **Zero-shot is the right eval for RL-trained reasoning models** — few-shot examples conflict with learned reasoning patterns.
+- **0.8B is near the capacity ceiling for GRPO** — the model internalizes reasoning format but has limited room for math accuracy gains. Consider 1.5B+ for stronger results.
+
+## Acknowledgments
+
+- [TRL](https://github.com/huggingface/trl) for the GRPOTrainer implementation
+- [Qwen Team](https://github.com/QwenLM) for the base model
+- [DeepSeek](https://arxiv.org/abs/2402.03300) for the GRPO algorithm
+
+## Citation
+
+```bibtex
+@misc{qwen35-grpo-math-2026,
+  author = {Akshay Mhaskar},
+  title = {Qwen3.5-0.8B-GRPO-Math: Teaching a Small Model to Reason with RL},
+  year = {2026},
+  url = {https://huggingface.co/celestialcreator/Qwen3.5-0.8B-GRPO-Math},
+}
+```