| --- |
| license: apache-2.0 |
| tags: |
| - speculative-decoding |
| - knowledge-distillation |
| - mistral |
| base_model: Lite-Mistral-150M-v2-Instruct |
| --- |
| |
| # drafter-understanding |
|
|
| Domain-specific draft model for speculative decoding, trained on **Understanding** tasks. |
|
|
| ## Model Details |
|
|
| | Parameter | Value | |
| |-----------|-------| |
| | Base model | Lite-Mistral-150M-v2-Instruct (156M params) | |
| | Architecture | MistralForCausalLM | |
| | Target model | TurboSparse-Mistral-Instruct (7B) | |
| | Domain | Understanding (21 Flan clusters) | |
| | Training samples | 395K | |
| | Epochs | 4.5 (early stop from 25) | |
| | Training time | 4.3 hours (1x RTX 3090) | |
| | Loss | Mixed: 0.5 x CE + 0.5 x KL (T=1.0) | |
| | Final eval_loss | 1.193 | |
| | Final top1_accuracy | 65.05% | |
| | Overlap Area (AR proxy) | **0.7558** on own domain | |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model = AutoModelForCausalLM.from_pretrained("MikhailRudenko/drafter-understanding") |
| tokenizer = AutoTokenizer.from_pretrained("MikhailRudenko/drafter-understanding") |
| ``` |
|
|
| ## Training Data |
|
|
| [MikhailRudenko/domain-aware-sd-synthetic](https://huggingface.co/datasets/MikhailRudenko/domain-aware-sd-synthetic) |
|
|
| ## Citation |
|
|
| Part of the Domain-Aware Speculative Decoding research project: |
| [GitHub](https://github.com/MikhailRudenk0/Domain-Aware-SD) |
|
|
