File size: 2,886 Bytes
ca3c60a 4abc1d6 ca3c60a ef1d52e ca3c60a 704e411 ca3c60a 856ede3 ca3c60a a517f8e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 |
---
license: mit
language:
- en
base_model:
- unsloth/phi-4
- microsoft/phi-4
pipeline_tag: text-generation
---
# Phi-4 converted for ExLlamaV3
[ExLlamaV3 is an optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs.](https://github.com/turboderp-org/exllamav3)
**This is an early preview release of ExLlamaV3.**
| | Quant type | File Size | Vram*|
| -------- | ---------- | --------- | -------- |
| [phi-4_3bpw](https://huggingface.co/cmh/phi-4_exl3/tree/3bpw) | 3 bits per weight | 6.53 GB | **9.4 GB** |
| [phi-4_4bpw](https://huggingface.co/cmh/phi-4_exl3/tree/4bpw) | 4 bits per weight | 8.24 GB | **11.0 GB** |
| [phi-4_5bpw](https://huggingface.co/cmh/phi-4_exl3/tree/5bpw) | 5 bits per weight | 9.94 GB | **12,6 GB** |
| [phi-4_6bpw](https://huggingface.co/cmh/phi-4_exl3/tree/6bpw) | 6 bits per weight | 11.65 GB | **14,2 GB** |
| [phi-4_7bpw](https://huggingface.co/cmh/phi-4_exl3/tree/7bpw) | 7 bits per weight | 13.35 GB | **15,8 GB** |
| [phi-4_8bpw](https://huggingface.co/cmh/phi-4_exl3/tree/8bpw) | 8 bits per weight | 15.05 GB | **17,3 GB** |
<sub>*approximate value at **16k context**.<sup>
---------------------------------------------
# Phi-4 Model Card
[Phi-4 Technical Report](https://arxiv.org/pdf/2412.08905)
## Model Summary
| | |
|-------------------------|-------------------------------------------------------------------------------|
| **Developers** | Microsoft Research |
| **Description** | `phi-4` is a state-of-the-art open model built upon a blend of synthetic datasets, data from filtered public domain websites, and acquired academic books and Q&A datasets. The goal of this approach was to ensure that small capable models were trained with data focused on high quality and advanced reasoning.<br><br>`phi-4` underwent a rigorous enhancement and alignment process, incorporating both supervised fine-tuning and direct preference optimization to ensure precise instruction adherence and robust safety measures |
| **Architecture** | 14B parameters, dense decoder-only Transformer model |
| **Context length** | 16384 tokens |
## Usage
### Input Formats
Given the nature of the training data, `phi-4` is best suited for prompts using the chat format as follows:
```bash
<|im_start|>system<|im_sep|>
You are a medieval knight and must provide explanations to modern people.<|im_end|>
<|im_start|>user<|im_sep|>
How should I explain the Internet?<|im_end|>
<|im_start|>assistant<|im_sep|>
```
### With exllamav3's chat.py:
python examples\chat.py -m models\phi-4_exl3\4bpw -mode raw |