metadata
license: apache-2.0
tags:
- speculative-decoding
- knowledge-distillation
- mistral
base_model: Lite-Mistral-150M-v2-Instruct
drafter-mixed-ut
Domain-specific draft model for speculative decoding, trained on Mixed (Understanding + Text Reformulation) tasks.
Model Details
| Parameter | Value |
|---|---|
| Base model | Lite-Mistral-150M-v2-Instruct (156M params) |
| Architecture | MistralForCausalLM |
| Target model | TurboSparse-Mistral-Instruct (7B) |
| Domain | Mixed (Understanding + Text Reformulation) (32 Flan clusters) |
| Training samples | 722K |
| Epochs | 5 |
| Training time | 12.6 hours (1x RTX 3090) |
| Loss | Mixed: 0.5 x CE + 0.5 x KL (T=1.0) |
| Final eval_loss | 1.788 |
| Final top1_accuracy | 56.64% |
| Overlap Area (AR proxy) | 0.7191 on own domain |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("MikhailRudenko/drafter-mixed-ut")
tokenizer = AutoTokenizer.from_pretrained("MikhailRudenko/drafter-mixed-ut")
Training Data
MikhailRudenko/domain-aware-sd-synthetic
Citation
Part of the Domain-Aware Speculative Decoding research project: GitHub