---
license: apache-2.0
tags:
  - speculative-decoding
  - knowledge-distillation
  - mistral
base_model: Lite-Mistral-150M-v2-Instruct
---

# drafter-understanding

Domain-specific draft model for speculative decoding, trained on **Understanding** tasks.

## Model Details

| Parameter | Value |
|-----------|-------|
| Base model | Lite-Mistral-150M-v2-Instruct (156M params) |
| Architecture | MistralForCausalLM |
| Target model | TurboSparse-Mistral-Instruct (7B) |
| Domain | Understanding (21 Flan clusters) |
| Training samples | 395K |
| Epochs | 4.5 (early stop from 25) |
| Training time | 4.3 hours (1x RTX 3090) |
| Loss | Mixed: 0.5 x CE + 0.5 x KL (T=1.0) |
| Final eval_loss | 1.193 |
| Final top1_accuracy | 65.05% |
| Overlap Area (AR proxy) | **0.7558** on own domain |

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("MikhailRudenko/drafter-understanding")
tokenizer = AutoTokenizer.from_pretrained("MikhailRudenko/drafter-understanding")
```

## Training Data

[MikhailRudenko/domain-aware-sd-synthetic](https://huggingface.co/datasets/MikhailRudenko/domain-aware-sd-synthetic)

## Citation

Part of the Domain-Aware Speculative Decoding research project:
[GitHub](https://github.com/MikhailRudenk0/Domain-Aware-SD)