arcee-ai/Trinity-Nano-Preview-W4A16

Trinity Nano Preview W4A16

Trinity Nano Preview is a preview of Arcee AI's 6B MoE model with 1B active parameters. It is the small-sized model in our new Trinity family, a series of open-weight models for enterprise and tinkerers alike.

This is a chat tuned model, with a delightful personality and charm we think users will love. We note that this model is pushing the limits of sparsity in small language models with only 800M non-embedding parameters active per token, and as such may be unstable in certain use cases, especially in this preview.

This is an experimental release, it's fun to talk to but will not be hosted anywhere, so download it and try it out yourself!

---

Trinity Nano Preview is trained on 10T tokens gathered and curated through a key partnership with Datology, building upon the excellent dataset we used on AFM-4.5B with additional math and code.

Training was performed on a cluster of 512 H200 GPUs powered by Prime Intellect using HSDP parallelism.

More details, including key architecture decisions, can be found on our blog here

---

This repository contains the W4A16 quantized weights of Trinity-Nano-Preview (INT4 weights, 16-bit activations).

Model Details

• Model Architecture: AfmoeForCausalLM
• Parameters: 6B, 1B active
• Experts: 128 total, 8 active, 1 shared
• Context length: 128k
• Training Tokens: 10T
• License: Apache 2.0

---

Quantization Details

• Scheme: W4A16 (INT4 weights, 16-bit activations)
• Intended use: quality-preserving 4-bit deployment of Trinity-Nano-Preview

Running our model

• Transformers
• VLLM
• llama.cpp
• LM Studio

Transformers

Use the main transformers branch

git clone https://github.com/huggingface/transformers.git
cd transformers

# pip
pip install '.[torch]'

# uv
uv pip install '.[torch]'

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "arcee-ai/Trinity-Nano-Preview"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

messages = [
    {"role": "user", "content": "Who are you?"},
]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    input_ids,
    max_new_tokens=256,
    do_sample=True,
    temperature=0.5,
    top_k=50,
    top_p=0.95
)

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

If using a released transformers, simply pass "trust_remote_code=True":

model_id = "arcee-ai/Trinity-Nano-Preview"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

VLLM

Supported in VLLM release 0.11.1

# pip
pip install "vllm>=0.11.1"

Serving the model with suggested settings:

vllm serve arcee-train/Trinity-Nano-Preview \
  --dtype bfloat16 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_r1 \
  --tool-call-parser hermes

llama.cpp

Supported in llama.cpp release b7061

Download the latest llama.cpp release

llama-server -hf arcee-ai/Trinity-Nano-Preview-GGUF:q4_k_m

LM Studio

Supported in latest LM Studio runtime

Update to latest available, then verify your runtime by:

1. Click "Power User" at the bottom left
2. Click the green "Developer" icon at the top left
3. Select "LM Runtimes" at the top
4. Refresh the list of runtimes and verify that the latest is installed

Then, go to Model Search and search for arcee-ai/Trinity-Nano-Preview-GGUF, download your prefered size, and load it up in the chat

License

Trinity-Nano-Preview-W4A16 is released under the Apache-2.0 license.