|
|
--- |
|
|
base_model: |
|
|
- Qwen/Qwen2.5-3B-Instruct |
|
|
tags: |
|
|
- text-generation-inference |
|
|
- transformers |
|
|
- unsloth |
|
|
- qwen2 |
|
|
- trl |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- en |
|
|
datasets: |
|
|
- openai/gsm8k |
|
|
metrics: |
|
|
- accuracy |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
# Uploaded model |
|
|
|
|
|
- **Developed by:** rushigulum |
|
|
- **License:** apache-2.0 |
|
|
- **Finetuned from model :** Qwen/Qwen2.5-3B-Instruct |
|
|
|
|
|
This qwen2 model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library. |
|
|
|
|
|
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth) |
|
|
|
|
|
|
|
|
Nano R1 is a fine-tuned variant of Qwen2.5-3B-Instruct, aligned using Group Relative Preference Optimization (GRPO) |
|
|
for reasoning-intensive tasks such as math problem-solving. |
|
|
The model is trained with Unsloth + TRL + vLLM to ensure efficient fine-tuning, faster inference, and improved contextual accuracy. |
|
|
|
|
|
Key Highlights: |
|
|
|
|
|
- = Base Model: Qwen2.5-3B-Instruct (via HuggingFace) |
|
|
- = Fine-Tuning: GRPO reinforcement learning with custom reward functions |
|
|
- = Optimizations: LoRA adapters, 4-bit quantization, vLLM inference |
|
|
- = Dataset: GSM8K (math reasoning) with structured XML reasoning prompts |
|
|
- = Deployment: Hugging Face Hub integration |
|
|
|
|
|
- Model Loading with LoRA |
|
|
Base: Qwen/Qwen2.5-3B-Instruct |
|
|
Optimizations: 4-bit quantization, LoRA rank=64, gradient checkpointing. |
|
|
|
|
|
- Reward Functions |
|
|
Semantic Correctness → via Sentence-BERT embeddings. |
|
|
Strict XML Compliance → ensures reasoning/answer separation. |
|
|
Numerical Answer Check → enforces valid math outputs. |
|
|
Length & Format Penalty → prevents overly long/unstructured responses. |
|
|
|
|
|
|
|
|
- GRPO Training |
|
|
Optimizer: AdamW (8-bit) |
|
|
Batch size: 1 (accumulated) |
|
|
Learning rate: 5e-6 |
|
|
Steps: 150 (demo run) |
|
|
Inference engine: vLLM for efficiency |
|
|
|
|
|
|
|
|
- Valuation |
|
|
Benchmarked on GSM8K validation set. |
|
|
Metrics: |
|
|
Final Answer Accuracy (semantic similarity > threshold). |
|
|
Format Compliance (% responses following XML structure). |
|
|
Average Reward Score across completions. |
|
|
|
|
|
- Results |
|
|
Improved reasoning structure with consistent <reasoning>/<answer> format. |
|
|
Higher semantic accuracy vs baseline Qwen2.5-3B. |
|
|
Optimized inference speed using vLLM. |
