gguf-gemv

GEMV directly on packed GGUF blocks, aarch64 first, loadable through kernels. Weight rows stream as raw block bytes; each block decodes to f32 in a stack buffer and accumulates against the activations, so no dequantized weight tensor is ever materialized. The reference baseline is the ggml / gguf-py dequantization, reproduced bit for bit. Companions: bitnet-cpu, quant-matmul (linears), cpu-attn, decode-ops.

The GGUF catalogue is the largest library of quantized models in existence, and the Python stack cannot use it directly: loading a 4-bit checkpoint into transformers expands it to float first, so the file that fit on the board no longer does. This kernel runs the matrix-vector product against the packed bytes as they sit in the page cache, so a GGUF layer costs its file size in memory rather than four times its unpacked size, and BitNet's ternary GGUF formats become loadable from Python at all.

A 4096 x 4096 layer measured on the boards: Q4_K at 2.95 ms on a Pi 5's Cortex-A76 and 34.5 ms on a Pi 4's A72, Q8_0 at 3.32 and 20.8, Q6_K at 8.12 and 62.5, with the packed weights at 9 to 17 MB against 64 MB in fp32 and decode matching the reference dequantization to 2e-6.

Usage

import torch
from kernels import get_kernel

gg = get_kernel("phanerozoic/gguf-gemv", version=1, trust_remote_code=True)

tensors = gg.load_gguf("model.gguf")       # quantized 2D -> GGUFLinear
layer = tensors["blk.0.ffn_up.weight"]     # GGUFLinear
y = layer(x)                               # [..., N]

load_gguf reads tensor data through the gguf package's copy-on-write mmap, so weight bytes stay file-backed and the kernel reads them out of the page cache. version selects the release branch; trust_remote_code is required by kernels for publishers without the trusted-publisher mark.

API

Method

Supported ggml block types: Q4_0, Q8_0, Q4_K, Q5_K, Q6_K (ids 2, 8, 12, 13, 14), the mainline ternary TQ2_0 (35), and the BitNet.cpp ternary I2_S (36). The first six cover the common K-quant files (Q4_K_M, Q5_K_M) and reproduce the ggml / gguf-py reference dequantization bit for bit (max abs difference 0 on aarch64 and x86-64). I2_S follows the BitNet.cpp quantize_i2_s / vec_dot_i2_i8_s layout (128-element blocks, one f32 tensor-wide scale, codes {0,1,2} -> {-1,0,+1}), verified against that spec and against the packed tensors of microsoft/bitnet-embedding-0.6b, which makes the BitNet ternary GGUFs loadable through the Python stack.

Measured

Raspberry Pi 5 (4x Cortex-A76 2.4 GHz) and Pi 4 Model B (4x Cortex-A72 1.8 GHz), torch 2.13 CPU, a 4096 x 4096 layer at M = 1:

Further shapes on the Pi 5 and Pi 4: Q4_K 1, 11008, 4096 runs 7.66 and 24.85 ms; Q4_K 8, 4096, 4096 runs 9.53 and 37.19 ms. Decode is memory-bound at the block-byte stream; Q6_K is slower for its heavier per-block bit assembly.

Correctness

The GEMV agrees with the reference dequantization followed by a float product to 2e-6 relative across all four measured formats on both boards, and dequant itself matches ggml / gguf-py bit for bit.

Requirements and limits

• K a multiple of the block size (32 for Q4_0/Q8_0, 256 for the K-quants and TQ2_0, 128 for I2_S).
• f32 or bf16 activations; output follows the input dtype.
• load_gguf reads every type from the raw file, including I2_S, which gguf-py's own reader rejects, and needs no external package.
• The block decode is scalar on both aarch64 and x86-64 (the f32 accumulate vectorizes): correct and bit-exact, not micro-tuned.
• A dedicated llama.cpp or BitNet.cpp runtime remains faster; this is for models composed in Python.

References

The ggml GGUF block formats and their reference dequantization; Microsoft BitNet.cpp's quantize_i2_s / vec_dot_i2_i8_s layout.

License

Apache-2.0.