File size: 1,317 Bytes
2ae841b fa0df6c 2ae841b fa0df6c 2ae841b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | ---
license: apache-2.0
tags:
- speculative-decoding
- knowledge-distillation
- mistral
base_model: Lite-Mistral-150M-v2-Instruct
---
# drafter-understanding
Domain-specific draft model for speculative decoding, trained on **Understanding** tasks.
## Model Details
| Parameter | Value |
|-----------|-------|
| Base model | Lite-Mistral-150M-v2-Instruct (156M params) |
| Architecture | MistralForCausalLM |
| Target model | TurboSparse-Mistral-Instruct (7B) |
| Domain | Understanding (21 Flan clusters) |
| Training samples | 395K |
| Epochs | 4.5 (early stop from 25) |
| Training time | 4.3 hours (1x RTX 3090) |
| Loss | Mixed: 0.5 x CE + 0.5 x KL (T=1.0) |
| Final eval_loss | 1.193 |
| Final top1_accuracy | 65.05% |
| Overlap Area (AR proxy) | **0.7558** on own domain |
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("MikhailRudenko/drafter-understanding")
tokenizer = AutoTokenizer.from_pretrained("MikhailRudenko/drafter-understanding")
```
## Training Data
[MikhailRudenko/domain-aware-sd-synthetic](https://huggingface.co/datasets/MikhailRudenko/domain-aware-sd-synthetic)
## Citation
Part of the Domain-Aware Speculative Decoding research project:
[GitHub](https://github.com/MikhailRudenk0/Domain-Aware-SD)
|