rina-llama.cpp / README.md
XenonFear128's picture
README: all-core 9950X numbers (dual-channel DDR5-6400)
119db25 verified
|
Raw History Blame Contribute Delete
6.65 kB
---
license: mit
tags:
- llama.cpp
- gguf
- ternary
- cpu
- x86
- avx2
- avx512
---
# rina-llama.cpp
A llama.cpp build for ternary GGUF models on x86 CPUs, maintained by Rinstitute. It is the
[PrismML llama.cpp fork](https://github.com/PrismML-Eng/llama.cpp) (branch `prism`, which tracks upstream
llama.cpp) plus **tq16**, a set of CPU kernels for the three ternary formats used by
[Rina Juvenile Exp](https://huggingface.co/Rinstitute/Rina-Juvenile-Exp-2B-GGUF): PQ2_0, PTQ1_0 and Q2_0.
- 中文:面向 x86 CPU 的 llama.cpp 构建,为三值 GGUF(PQ2_0 / PTQ1_0 / Q2_0)加了 AVX2 / AVX-VNNI / AVX-512 内核。
- 日本語:三値 GGUF(PQ2_0 / PTQ1_0 / Q2_0)向けに AVX2 / AVX-VNNI / AVX-512 カーネルを追加した、x86 CPU 用の llama.cpp ビルドです。
## Release 2026.11.14
| File | Contents |
|---|---|
| `rina-llama.cpp-2026.11.14-linux-x64.tar.gz` | Linux x86-64, statically linked (any distribution): `llama-cli`, `llama-completion`, `llama-server`, `llama-bench` |
| `rina-llama.cpp-2026.11.14-src.tar.gz` | Full source of this release |
| `source/0001-*.patch`, `source/0002-*.patch` | The tq16 changes as patches on top of PrismML `prism` 6447906 |
| `SHA256SUMS` | Checksums |
Base: PrismML `prism` at `6447906bf96c1a023980d3453fb59daa6af3bb23`; tq16 at `de0d48e42aaaaae3bad1cc2d44e1bcf8a80a3867`.
Windows builds are not included yet.
## Quick start (Linux)
```bash
tar xzf rina-llama.cpp-2026.11.14-linux-x64.tar.gz
cd rina-llama.cpp-2026.11.14-linux-x64
./llama-cli -m Rina-Juvenile-Exp-2B-PQ2_0.gguf
./llama-server -m Rina-Juvenile-Exp-2B-PQ2_0.gguf --host 127.0.0.1 --port 8080
```
The top-level `llama-*` files are small launchers. They read `/proc/cpuinfo` and run one of three builds in `bin/`:
| Level | Requires | Typical CPUs |
|---|---|---|
| `avx512` | AVX-512 F/CD/BW/DQ/VL + VNNI | Intel Cascade Lake, Ice Lake, Sapphire Rapids and later Xeons; AMD Zen 4, Zen 5 |
| `avxvnni` | AVX2, FMA, F16C, BMI2, AVX-VNNI | Intel Alder Lake and later Core |
| `avx2` | AVX2, FMA, F16C, BMI2 | Intel Haswell and later; AMD Zen 1–3 |
`RINA_CPU_LEVEL=avx2` (or `avxvnni`, `avx512`) forces a level. CPUs without AVX2 are not supported by this
package; use an upstream llama.cpp build there.
## What tq16 changes
- At load time the weights of all three formats are rearranged into one layout of 16-column tiles. Each 64-byte
line holds 2-bit codes such that `(line >> 2s) & 3` is a 16-column × 4-input operand for `vpdpbusd` (AVX-512 VNNI,
AVX-VNNI) or `pmaddubsw` (plain AVX2). Every output column accumulates in its own SIMD lane.
- PTQ1_0 trits are expanded to 2-bit codes when the model loads.
- Activations are quantized to int8 with one scale per 32 values.
- The output of a token does not depend on how many tokens are in the batch, so speculative decoding reproduces
plain decoding bit for bit.
Environment variables:
| Variable | Effect |
|---|---|
| `GGML_TQ16=0` | Use the original PrismML kernels |
| `GGML_TQ16_ACT=128` | One activation scale per weight group instead of per 32 values: about 10% faster prompt processing on VNNI CPUs, slightly lower precision (KLD 1.46e-4 instead of 1.02e-4) |
## Speed
Rina Juvenile Exp 2B, `llama-bench -p 512 -n 128`, tokens per second. “PrismML” is the same source with
`GGML_TQ16=0`. The `avxvnni` and `avx2` rows are the same CPU running builds without AVX-512.
**2-vCPU virtual machine** (host Ryzen 9 9950X, about 20 GB/s memory bandwidth), 2 threads:
| Format | Level | PrismML pp512 | tq16 pp512 | PrismML tg128 | tq16 tg128 |
|---|---|---:|---:|---:|---:|
| PQ2_0 | avx512 | 195 | 377 | 49.3 | 54.0 |
| PQ2_0 | avxvnni | 136 | 258 | 35.1 | 53.2 |
| PQ2_0 | avx2 | 92 | 163 | 30.2 | 55.9 |
| PTQ1_0 | avx512 | 58 | 387 | 28.6 | 53.2 |
| PTQ1_0 | avx2 | 40 | 177 | 22.8 | 55.0 |
| Q2_0 | avx512 | 56 | 371 | 27.0 | 49.9 |
| Q2_0 | avx2 | 53 | 167 | 25.5 | 49.4 |
On this machine, generation at 2 threads is close to the memory-bandwidth limit.
**Ryzen 9 9950X, all cores** (16 cores / 32 threads, dual-channel DDR5-6400, Linux under WSL 2, this release
package, `avx512` level):
| Format | PrismML pp512, 16 threads | tq16 pp512, 16 threads | PrismML tg128 | tq16 tg128 |
|---|---:|---:|---:|---:|
| PQ2_0 | 827 | 1575 | 94.9 | 98.5 |
| PTQ1_0 | 325 | 1659 | 102.3 | 95.5 |
| Q2_0 | 311 | 1759 | 91.0 | 95.7 |
- tg128 is the median of 10 samples at the best thread count, which was 8 to 12 threads.
- Generation is limited by memory bandwidth here (about 51 GB/s), so thread counts from 6 to 16 give similar
results.
- Using all 32 threads (SMT) is slower than 16 for both prompt processing and generation.
- With a 4K context (`-d 4096`, PQ2_0, 16 threads), tq16 gives pp512 1447 and tg128 82.4.
Single core of the same CPU:
| Format | Level | PrismML pp512 | tq16 pp512 | PrismML tg128 | tq16 tg128 |
|---|---|---:|---:|---:|---:|
| PQ2_0 | avx512 | 111 | 229 | 37.7 | 64.9 |
| PQ2_0 | avx2 | 59 | 99 | 24.8 | 55.4 |
| PTQ1_0 | avx512 | 33 | 232 | 22.2 | 65.7 |
| Q2_0 | avx512 | 30 | 225 | 20.5 | 66.2 |
## Accuracy
KL divergence against an F16 copy of the same weights (exact for ternary weights × fp16 scales), 40 chunks of 512
tokens of Chinese / Japanese / English chat text:
| Kernel | KLD | Same top-1 token |
|---|---:|---:|
| PrismML, AVX-512 | 1.00e-4 | 99.45% |
| PrismML, AVX2, PQ2_0 | 1.66e-4 | 99.37% |
| tq16, all levels | 1.02e-4 – 1.03e-4 | 99.43% – 99.53% |
A unit test (`tests/test-tq16.cpp` in the source) checks matrix products against a double-precision reference:
300 shapes per format, each with two activation distributions.
## Known limitations
- Only x86-64 Linux binaries so far. tq16 does nothing on ARM; there, PrismML's own kernels are used.
- PTQ1_0 uses as much memory as PQ2_0 after loading, because its trits are expanded to 2-bit codes. On machines
with many cores and limited memory bandwidth, PTQ1_0 generation can be a few percent slower than with
`GGML_TQ16=0`: about 5% on the 9950X at 8 or more threads.
- `llama-server` is built without TLS. Put a reverse proxy in front of it if it must be reachable from other
machines.
## Building from source
```bash
tar xzf rina-llama.cpp-2026.11.14-src.tar.gz && cd rina-llama.cpp-2026.11.14-src
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON
cmake --build build -j --target llama-cli llama-server
```
To apply the patches to a newer PrismML `prism` checkout, run `git am source/*.patch`.
## License
MIT, the same as llama.cpp. Copyright (c) 2023-2026 The ggml authors; the tq16 changes are by Rinstitute. The
licenses of bundled third-party code are in `licenses/` inside each archive.