---
license: apache-2.0
tags:
  - speculative-decoding
  - knowledge-distillation
  - mistral
base_model: Lite-Mistral-150M-v2-Instruct
---

# drafter-mixed-ut

Domain-specific draft model for speculative decoding, trained on **Mixed (Understanding + Text Reformulation)** tasks.

## Model Details

| Parameter | Value |
|-----------|-------|
| Base model | Lite-Mistral-150M-v2-Instruct (156M params) |
| Architecture | MistralForCausalLM |
| Target model | TurboSparse-Mistral-Instruct (7B) |
| Domain | Mixed (Understanding + Text Reformulation) (32 Flan clusters) |
| Training samples | 722K |
| Epochs | 5 |
| Training time | 12.6 hours (1x RTX 3090) |
| Loss | Mixed: 0.5 x CE + 0.5 x KL (T=1.0) |
| Final eval_loss | 1.788 |
| Final top1_accuracy | 56.64% |
| Overlap Area (AR proxy) | **0.7191** on own domain |

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("MikhailRudenko/drafter-mixed-ut")
tokenizer = AutoTokenizer.from_pretrained("MikhailRudenko/drafter-mixed-ut")
```

## Training Data

[MikhailRudenko/domain-aware-sd-synthetic](https://huggingface.co/datasets/MikhailRudenko/domain-aware-sd-synthetic)

## Citation

Part of the Domain-Aware Speculative Decoding research project:
[GitHub](https://github.com/MikhailRudenk0/Domain-Aware-SD)