|
Download README.md from Rinstitute/rina-llama.cpp: direct link, hf CLI and curl.
- Browser
- Download file 6.65 kB
-
https://huggingface.co/Rinstitute/rina-llama.cpp/resolve/main/README.md
- Command line
-
hf download hf://Rinstitute/rina-llama.cpp/README.md
-
curl -L -o README.md https://huggingface.co/Rinstitute/rina-llama.cpp/resolve/main/README.md
6.65 kB
| license: mit | |
| tags: | |
| - llama.cpp | |
| - gguf | |
| - ternary | |
| - cpu | |
| - x86 | |
| - avx2 | |
| - avx512 | |
| # rina-llama.cpp | |
| A llama.cpp build for ternary GGUF models on x86 CPUs, maintained by Rinstitute. It is the | |
| [PrismML llama.cpp fork](https://github.com/PrismML-Eng/llama.cpp) (branch `prism`, which tracks upstream | |
| llama.cpp) plus **tq16**, a set of CPU kernels for the three ternary formats used by | |
| [Rina Juvenile Exp](https://huggingface.co/Rinstitute/Rina-Juvenile-Exp-2B-GGUF): PQ2_0, PTQ1_0 and Q2_0. | |
| - 中文:面向 x86 CPU 的 llama.cpp 构建,为三值 GGUF(PQ2_0 / PTQ1_0 / Q2_0)加了 AVX2 / AVX-VNNI / AVX-512 内核。 | |
| - 日本語:三値 GGUF(PQ2_0 / PTQ1_0 / Q2_0)向けに AVX2 / AVX-VNNI / AVX-512 カーネルを追加した、x86 CPU 用の llama.cpp ビルドです。 | |
| ## Release 2026.11.14 | |
| | File | Contents | | |
| |---|---| | |
| | `rina-llama.cpp-2026.11.14-linux-x64.tar.gz` | Linux x86-64, statically linked (any distribution): `llama-cli`, `llama-completion`, `llama-server`, `llama-bench` | | |
| | `rina-llama.cpp-2026.11.14-src.tar.gz` | Full source of this release | | |
| | `source/0001-*.patch`, `source/0002-*.patch` | The tq16 changes as patches on top of PrismML `prism` 6447906 | | |
| | `SHA256SUMS` | Checksums | | |
| Base: PrismML `prism` at `6447906bf96c1a023980d3453fb59daa6af3bb23`; tq16 at `de0d48e42aaaaae3bad1cc2d44e1bcf8a80a3867`. | |
| Windows builds are not included yet. | |
| ## Quick start (Linux) | |
| ```bash | |
| tar xzf rina-llama.cpp-2026.11.14-linux-x64.tar.gz | |
| cd rina-llama.cpp-2026.11.14-linux-x64 | |
| ./llama-cli -m Rina-Juvenile-Exp-2B-PQ2_0.gguf | |
| ./llama-server -m Rina-Juvenile-Exp-2B-PQ2_0.gguf --host 127.0.0.1 --port 8080 | |
| ``` | |
| The top-level `llama-*` files are small launchers. They read `/proc/cpuinfo` and run one of three builds in `bin/`: | |
| | Level | Requires | Typical CPUs | | |
| |---|---|---| | |
| | `avx512` | AVX-512 F/CD/BW/DQ/VL + VNNI | Intel Cascade Lake, Ice Lake, Sapphire Rapids and later Xeons; AMD Zen 4, Zen 5 | | |
| | `avxvnni` | AVX2, FMA, F16C, BMI2, AVX-VNNI | Intel Alder Lake and later Core | | |
| | `avx2` | AVX2, FMA, F16C, BMI2 | Intel Haswell and later; AMD Zen 1–3 | | |
| `RINA_CPU_LEVEL=avx2` (or `avxvnni`, `avx512`) forces a level. CPUs without AVX2 are not supported by this | |
| package; use an upstream llama.cpp build there. | |
| ## What tq16 changes | |
| - At load time the weights of all three formats are rearranged into one layout of 16-column tiles. Each 64-byte | |
| line holds 2-bit codes such that `(line >> 2s) & 3` is a 16-column × 4-input operand for `vpdpbusd` (AVX-512 VNNI, | |
| AVX-VNNI) or `pmaddubsw` (plain AVX2). Every output column accumulates in its own SIMD lane. | |
| - PTQ1_0 trits are expanded to 2-bit codes when the model loads. | |
| - Activations are quantized to int8 with one scale per 32 values. | |
| - The output of a token does not depend on how many tokens are in the batch, so speculative decoding reproduces | |
| plain decoding bit for bit. | |
| Environment variables: | |
| | Variable | Effect | | |
| |---|---| | |
| | `GGML_TQ16=0` | Use the original PrismML kernels | | |
| | `GGML_TQ16_ACT=128` | One activation scale per weight group instead of per 32 values: about 10% faster prompt processing on VNNI CPUs, slightly lower precision (KLD 1.46e-4 instead of 1.02e-4) | | |
| ## Speed | |
| Rina Juvenile Exp 2B, `llama-bench -p 512 -n 128`, tokens per second. “PrismML” is the same source with | |
| `GGML_TQ16=0`. The `avxvnni` and `avx2` rows are the same CPU running builds without AVX-512. | |
| **2-vCPU virtual machine** (host Ryzen 9 9950X, about 20 GB/s memory bandwidth), 2 threads: | |
| | Format | Level | PrismML pp512 | tq16 pp512 | PrismML tg128 | tq16 tg128 | | |
| |---|---|---:|---:|---:|---:| | |
| | PQ2_0 | avx512 | 195 | 377 | 49.3 | 54.0 | | |
| | PQ2_0 | avxvnni | 136 | 258 | 35.1 | 53.2 | | |
| | PQ2_0 | avx2 | 92 | 163 | 30.2 | 55.9 | | |
| | PTQ1_0 | avx512 | 58 | 387 | 28.6 | 53.2 | | |
| | PTQ1_0 | avx2 | 40 | 177 | 22.8 | 55.0 | | |
| | Q2_0 | avx512 | 56 | 371 | 27.0 | 49.9 | | |
| | Q2_0 | avx2 | 53 | 167 | 25.5 | 49.4 | | |
| On this machine, generation at 2 threads is close to the memory-bandwidth limit. | |
| **Ryzen 9 9950X, all cores** (16 cores / 32 threads, dual-channel DDR5-6400, Linux under WSL 2, this release | |
| package, `avx512` level): | |
| | Format | PrismML pp512, 16 threads | tq16 pp512, 16 threads | PrismML tg128 | tq16 tg128 | | |
| |---|---:|---:|---:|---:| | |
| | PQ2_0 | 827 | 1575 | 94.9 | 98.5 | | |
| | PTQ1_0 | 325 | 1659 | 102.3 | 95.5 | | |
| | Q2_0 | 311 | 1759 | 91.0 | 95.7 | | |
| - tg128 is the median of 10 samples at the best thread count, which was 8 to 12 threads. | |
| - Generation is limited by memory bandwidth here (about 51 GB/s), so thread counts from 6 to 16 give similar | |
| results. | |
| - Using all 32 threads (SMT) is slower than 16 for both prompt processing and generation. | |
| - With a 4K context (`-d 4096`, PQ2_0, 16 threads), tq16 gives pp512 1447 and tg128 82.4. | |
| Single core of the same CPU: | |
| | Format | Level | PrismML pp512 | tq16 pp512 | PrismML tg128 | tq16 tg128 | | |
| |---|---|---:|---:|---:|---:| | |
| | PQ2_0 | avx512 | 111 | 229 | 37.7 | 64.9 | | |
| | PQ2_0 | avx2 | 59 | 99 | 24.8 | 55.4 | | |
| | PTQ1_0 | avx512 | 33 | 232 | 22.2 | 65.7 | | |
| | Q2_0 | avx512 | 30 | 225 | 20.5 | 66.2 | | |
| ## Accuracy | |
| KL divergence against an F16 copy of the same weights (exact for ternary weights × fp16 scales), 40 chunks of 512 | |
| tokens of Chinese / Japanese / English chat text: | |
| | Kernel | KLD | Same top-1 token | | |
| |---|---:|---:| | |
| | PrismML, AVX-512 | 1.00e-4 | 99.45% | | |
| | PrismML, AVX2, PQ2_0 | 1.66e-4 | 99.37% | | |
| | tq16, all levels | 1.02e-4 – 1.03e-4 | 99.43% – 99.53% | | |
| A unit test (`tests/test-tq16.cpp` in the source) checks matrix products against a double-precision reference: | |
| 300 shapes per format, each with two activation distributions. | |
| ## Known limitations | |
| - Only x86-64 Linux binaries so far. tq16 does nothing on ARM; there, PrismML's own kernels are used. | |
| - PTQ1_0 uses as much memory as PQ2_0 after loading, because its trits are expanded to 2-bit codes. On machines | |
| with many cores and limited memory bandwidth, PTQ1_0 generation can be a few percent slower than with | |
| `GGML_TQ16=0`: about 5% on the 9950X at 8 or more threads. | |
| - `llama-server` is built without TLS. Put a reverse proxy in front of it if it must be reachable from other | |
| machines. | |
| ## Building from source | |
| ```bash | |
| tar xzf rina-llama.cpp-2026.11.14-src.tar.gz && cd rina-llama.cpp-2026.11.14-src | |
| cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON | |
| cmake --build build -j --target llama-cli llama-server | |
| ``` | |
| To apply the patches to a newer PrismML `prism` checkout, run `git am source/*.patch`. | |
| ## License | |
| MIT, the same as llama.cpp. Copyright (c) 2023-2026 The ggml authors; the tq16 changes are by Rinstitute. The | |
| licenses of bundled third-party code are in `licenses/` inside each archive. | |