File size: 4,991 Bytes
c024e48 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 | ---
license: apache-2.0
language:
- en
- es
- fr
- de
- it
- pt
- ru
- ar
- hi
- ko
- zh
library_name: transformers
base_model:
- arcee-ai/Trinity-Mini
base_model_relation: quantized
---
<div align="center">
<picture>
<img
src="https://cdn-uploads.huggingface.co/production/uploads/6435718aaaef013d1aec3b8b/i-v1KyAMOW_mgVGeic9WJ.png"
alt="Arcee Trinity Mini"
style="max-width: 100%; height: auto;"
>
</picture>
</div>
# Trinity Mini FP8-Block
**This repository contains the FP8 block-quantized weights of Trinity-Mini (FP8 weights and activations with per-block scaling).**
Trinity Mini is an Arcee AI 26B MoE model with 3B active parameters. It is the medium-sized model in our new Trinity family, a series of open-weight models for enterprise and tinkerers alike.
This model is tuned for reasoning, but in testing, it uses a similar total token count to competitive instruction-tuned models.
***
Trinity Mini is trained on 10T tokens gathered and curated through a key partnership with [Datology](https://www.datologyai.com/), building upon the excellent dataset we used on [AFM-4.5B](https://huggingface.co/arcee-ai/AFM-4.5B) with additional math and code.
Training was performed on a cluster of 512 H200 GPUs powered by [Prime Intellect](https://www.primeintellect.ai/) using HSDP parallelism.
More details, including key architecture decisions, can be found on our blog [here](https://www.arcee.ai/blog/the-trinity-manifesto)
Try it out now at [chat.arcee.ai](http://chat.arcee.ai/)
***
## Model Details
* **Model Architecture:** AfmoeForCausalLM
* **Parameters:** 26B, 3B active
* **Experts:** 128 total, 8 active, 1 shared
* **Context length:** 128k
* **Training Tokens:** 10T
* **License:** [Apache 2.0](https://huggingface.co/arcee-ai/Trinity-Mini#license)
* **Recommended settings:**
* temperature: 0.15
* top_k: 50
* top_p: 0.75
* min_p: 0.06
***
## Quantization Details
- **Scheme:** `FP8 Block` (FP8 weights and activations, per-block scaling with E8M0 scale format)
- **Format:** `compressed-tensors`
- **Intended use:** High-throughput FP8 deployment of Trinity-Mini with near-lossless quality, optimized for NVIDIA Hopper GPUs
- **Supported backends:** [DeepGEMM](https://github.com/deepseek-ai/DeepGEMM), vLLM CUTLASS, Triton
## Benchmarks
<div align="center">
<picture>
<img src="https://cdn-uploads.huggingface.co/production/uploads/6435718aaaef013d1aec3b8b/sSVjGNHfrJKmQ6w8I18ek.png" style="background-color:ghostwhite;padding:5px;" width="17%" alt="Powered by Datology">
</picture>
</div>
### Running our model
- [VLLM](https://huggingface.co/arcee-ai/Trinity-Mini-FP8-Block#vllm)
- [Transformers](https://huggingface.co/arcee-ai/Trinity-Mini-FP8-Block#transformers)
## VLLM
Supported in VLLM release 0.18.0+ with DeepGEMM FP8 MoE acceleration.
```
# pip
pip install "vllm>=0.18.0"
```
Serving the model with DeepGEMM enabled:
```
VLLM_USE_DEEP_GEMM=1 vllm serve arcee-ai/Trinity-Mini-FP8-Block \
--trust-remote-code \
--max-model-len 4096 \
--enable-auto-tool-choice \
--reasoning-parser deepseek_r1 \
--tool-call-parser hermes
```
Serving without DeepGEMM (falls back to CUTLASS/Triton):
```
vllm serve arcee-ai/Trinity-Mini-FP8-Block \
--trust-remote-code \
--max-model-len 4096 \
--enable-auto-tool-choice \
--reasoning-parser deepseek_r1 \
--tool-call-parser hermes
```
## Transformers
Use the `main` transformers branch
```
git clone https://github.com/huggingface/transformers.git
cd transformers
# pip
pip install '.[torch]'
# uv
uv pip install '.[torch]'
```
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
model_id = "arcee-ai/Trinity-Mini-FP8-Block"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True
)
messages = [
{"role": "user", "content": "Who are you?"},
]
input_ids = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt"
).to(model.device)
outputs = model.generate(
input_ids,
max_new_tokens=256,
do_sample=True,
temperature=0.15,
top_k=50,
top_p=0.75,
min_p=0.06
)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
```
## API
Trinity Mini is available today on openrouter:
https://openrouter.ai/arcee-ai/trinity-mini
```
curl -X POST "https://openrouter.ai/v1/chat/completions" \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "arcee-ai/trinity-mini",
"messages": [
{
"role": "user",
"content": "What are some fun things to do in New York?"
}
]
}'
```
## License
Trinity-Mini-FP8-Block is released under the Apache-2.0 license. |