README.md

---
license: apache-2.0
base_model: tencent/Hy3-FP8
base_model_relation: quantized
language:
  - en
  - zh
tags:
  - hy_v3
  - hunyuan
  - hy3
  - int4
  - cpu
  - moe
  - expert-streaming
  - colibri
  - text-generation
library_name: colibri
---

# Hy3 — colibrì int4 container (~142 GB)

Pre-converted weights for **[colibrì](https://github.com/JustVugg/colibri)** — a pure-C engine that runs huge MoE models on consumer hardware by keeping the dense stack resident in RAM and **streaming routed experts from disk**.

This repo is the **first public Colibri-format checkpoint for [Tencent Hy3](https://huggingface.co/tencent/Hy3)**. It lets you run Hy3's **295B-parameter MoE** (21B active per token) on a machine with modest RAM and a fast local NVMe drive, instead of an 8×GPU datacenter setup.

**⚠️ This is NOT a GGUF / AWQ / GPTQ / MLX / vLLM model.** It uses colibrì's own safetensors container: for each quantized weight, `name` (packed int4/int8) + `name.qs` (F32 per-row scales), with packing math bit-identical to the engine's C kernels. **It only works with the colibrì `hy3` engine** (see the Hy3 port in this conversion fork).

## What we contribute

The upstream [tencent/Hy3](https://huggingface.co/tencent/Hy3) weights ship for **BF16 / FP8 inference stacks** (vLLM, SGLang, Transformers). Community quants (GGUF, AWQ, etc.) target **other runtimes** and cannot be loaded by colibrì.

This release adds the missing **disk-streaming Colibri container** plus the runtime pieces needed to use it:

| Contribution | Description |
|---|---|
| **Colibri int4 container** | 107 `out-*.safetensors` shards (~142 GB) converted from [tencent/Hy3-FP8](https://huggingface.co/tencent/Hy3-FP8) with `convert_hy3.py` — shard-at-a-time, resumable, never needs the full FP8 tree on disk at once |
| **Hy3-aware quantization** | Mixed precision tuned for streaming: **int4** routed experts (disk), **int8** attention / shared expert / embeddings, **F32** norms + sigmoid router + expert bias; Hy3-FP8 **per-tensor `weight_scale`** dequant path (not GLM's block scales) |
| **MTP head shards** | 6 `out-mtp-*.safetensors` shards (layer 80, int8) — weights included for future speculative decode; MTP runtime in `hy3.c` is not wired yet |
| **`hy3.c` engine** | New pure-C forward for `hy_v3`: **GQA** attention (64Q / 8KV, Q/K RMSNorm, rotate-half RoPE), **sigmoid MoE router** with bias + route normalization + `router_scaling_factor`, batch-union expert matmul, LRU/pin expert cache |
| **Converter + validation** | `convert_hy3.py`, `make_hy3_oracle.py`, `audit_hy3_tensors.py`; oracle teacher-forcing validated on a tiny Hy3 fixture before scaling to full weights |
| **Chat / serve integration** | Hy3 Hunyuan chat template (`｜` delimiters), `coli chat`, OpenAI-compatible `coli serve` |

**Bottom line:** same model family as [tencent/Hy3](https://huggingface.co/tencent/Hy3), but repackaged so a **single NVMe + ~16–64 GB RAM** machine can run it via expert streaming — the same idea as [jlnsrk/GLM-5.2-colibri-int4](https://huggingface.co/jlnsrk/GLM-5.2-colibri-int4), applied to Hy3.

## About Hy3 (from the base model)

Hy3 is Tencent's **295B-parameter Mixture-of-Experts** instruct model with **21B activated parameters** per token, developed by the Tencent Hy Team. See the [official model card](https://huggingface.co/tencent/Hy3) for benchmarks, agent capabilities, and deployment recipes for GPU stacks.

| Property | Value |
|---|---|
| Architecture | Mixture-of-Experts (MoE), `hy_v3` |
| Total parameters | 295B |
| Activated parameters | 21B |
| Layers (excl. MTP) | 80 (layer 0 dense, 79 sparse MoE) |
| Attention | GQA — 64 heads, 8 KV heads, head dim 128 |
| Hidden size | 4096 |
| Experts | 192 routed + 1 shared, **top-8** per token |
| Context length | 256K |
| Vocabulary | 120832 |

## Usage

### 1. Get the engine (Hy3 port)

```bash
git clone https://github.com/JustVugg/colibri.git   # or your Hy3 port fork with c/hy3.c
cd colibri/c
./setup.sh    # builds hy3 + self-test on hy3_tiny fixture
```

### 2. Download this repo

Store on a **fast local disk** (NVMe, native ext4). Avoid WSL `/mnt/c` (9p) or network mounts — random expert reads are disk-bound.

```bash
hf download <your-username>/Hy3-colibri-int4 --local-dir /path/to/hy3_i4
```

### 3. Chat

```bash
COLI_MODEL=/path/to/hy3_i4 ./coli chat --ram 56 --topp 0.7
```

Recommended sampling for int4 streaming (differs from the official GPU defaults of `temperature=0.9, top_p=1.0` on [tencent/Hy3](https://huggingface.co/tencent/Hy3)):

- `--temp 0.7 --topp 0.7` — colibrì defaults tuned to reduce quantization tail noise

### 4. OpenAI-compatible API

```bash
COLI_MODEL=/path/to/hy3_i4 ./coli serve --ram 56 --host 127.0.0.1 --port 8000
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="hy3-colibri",
    messages=[{"role": "user", "content": "Who are you?"}],
    temperature=0.7,
    top_p=0.7,
)
print(response.choices[0].message.content)
```

### Requirements

- Linux or WSL2 with **native ext4** model path
- gcc + OpenMP, **AVX2**
- **≥16 GB RAM** (32–64 GB recommended for a comfortable expert cache)
- **~150 GB free** on a fast NVMe SSD
- Optional: CUDA build for future GPU-pinned expert tier (`make hy3 CUDA=1`)

## What's inside

| File | Contents |
|---|---|
| `out-*.safetensors` (107 shards) | Dense weights (GQA attention, layer-0 FFN, shared expert, embeddings) + **15,168 routed experts** (79 MoE layers × 192), int4 with per-row `.qs` scales; router / norms / bias kept F32 |
| `out-mtp-*.safetensors` (6 shards) | MTP layer 80 weights (int8) — for future speculative decoding |
| `config.json`, `tokenizer.json`, `tokenizer_config.json`, `generation_config.json` | Copied from the Hy3 base repo |

**On-disk size:** ~142 GB (int4 container). **Resident RAM (dense only):** ~4.1 GB at load time; peak RSS depends on context length, expert cache cap, and `--ram` budget.

## Convert it yourself

If you prefer to reproduce the container locally instead of downloading:

```bash
cd colibri/c
pip install torch safetensors huggingface_hub numpy
./coli convert --repo tencent/Hy3-FP8 --model /path/to/hy3_i4
```

The converter downloads one FP8 shard at a time, dequants, requantizes to the Colibri layout, and deletes the source shard — the full FP8 checkpoint never has to exist on disk at once.

## Why not use existing Hy3 quants?

| Format | Works with colibrì? |
|---|---|
| [tencent/Hy3-FP8](https://huggingface.co/tencent/Hy3-FP8) | **Input only** — must be converted to `out-*.safetensors` |
| [tencent/Hy3](https://huggingface.co/tencent/Hy3) BF16 | Convertible, but larger download |
| GGUF / llama.cpp / Ollama quants | **No** — different tensor layout and metadata |
| AWQ / GPTQ / HF quant repos | **No** — different scale packing and loaders |
| vLLM / SGLang | **No** — GPU serving stacks, not Colibri |

## Expected performance (honest)

Hy3 is **smaller per expert** than GLM-5.2 (~9 MB vs ~19 MB int4), with **192 experts/layer** and **79 MoE layers** — cold decode still moves **~5–6 GB of expert data per token** from disk. Throughput is **disk-bound** on a cold cache (often **0.1–0.5 tok/s** class on consumer NVMe, highly setup-dependent). Warm LRU / pinned hot experts improve this substantially.

This is not a replacement for [vLLM / SGLang deployment](https://huggingface.co/tencent/Hy3#deployment) on 8× datacenter GPUs. It is a way to **run the same model class on hardware you already own**.

## Status & limitations (early port)

- **Beta:** the Hy3 `hy3.c` engine is a new port (validated on oracle fixtures; full-model chat is still being stabilized).
- **MTP:** weights are included; native speculative decode in `hy3.c` is **not implemented yet**.
- **CUDA expert tier:** planned (`make hy3 CUDA=1`); not production-ready for Hy3 yet.
- **KV disk persistence:** not yet ported from GLM's MLA path (GQA layout differs).
- **Not loadable** in `transformers.AutoModelForCausalLM` — use colibrì only.

## Provenance & license

- **Base weights:** [tencent/Hy3-FP8](https://huggingface.co/tencent/Hy3-FP8) and [tencent/Hy3](https://huggingface.co/tencent/Hy3) (Apache 2.0).
- **Architecture & benchmarks:** see [tencent/Hy3](https://huggingface.co/tencent/Hy3).
- **Engine & converter:** [colibrì](https://github.com/JustVugg/colibri) (MIT) + Hy3 port (`hy3.c`, `convert_hy3.py`).
- This derivative checkpoint remains under the **Apache 2.0** terms of the Hy3 base model.

## Citation

If you use the base Hy3 model in academic work, cite Tencent's Hy3 release. If you use this Colibri container, please also credit the [colibrì project](https://github.com/JustVugg/colibri).