metadata
license: apache-2.0
tags:
- speculative-decoding
- knowledge-distillation
- mistral
base_model: Lite-Mistral-150M-v2-Instruct
drafter-understanding
Domain-specific draft model for speculative decoding, trained on Understanding tasks.
Model Details
| Parameter | Value |
|---|---|
| Base model | Lite-Mistral-150M-v2-Instruct (156M params) |
| Architecture | MistralForCausalLM |
| Target model | TurboSparse-Mistral-Instruct (7B) |
| Domain | Understanding (21 Flan clusters) |
| Training samples | 395K |
| Epochs | 4.5 (early stop from 25) |
| Training time | 4.3 hours (1x RTX 3090) |
| Loss | Mixed: 0.5 x CE + 0.5 x KL (T=1.0) |
| Final eval_loss | 1.193 |
| Final top1_accuracy | 65.05% |
| Overlap Area (AR proxy) | 0.7558 on own domain |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("MikhailRudenko/drafter-understanding")
tokenizer = AutoTokenizer.from_pretrained("MikhailRudenko/drafter-understanding")
Training Data
MikhailRudenko/domain-aware-sd-synthetic
Citation
Part of the Domain-Aware Speculative Decoding research project: GitHub