VelociQ8

Get VelociQ8 Embedding models Documentation

VelociQ8 is a build of llama.cpp that runs 8-bit models faster on CPUs. It adds Q8A64, an 8-bit weight format with its own CPU kernels for Arm (dotprod) and x86-64 (AVX2 + FMA). The tools are the usual llama-server, llama-cli, llama-embedding, llama-bench and llama-quantize, with the same options and the same HTTP API, including the OpenAI-compatible endpoints. The main target is embeddings on CPU servers. It also runs dense and mixture-of-experts LLMs.

The models in this organization are ready-to-run Q8A64 files, converted from the original weights.

Results

Measured in October 2026 against llama.cpp v0.7.0 with Q8_0, on the machines named under each result. The embedding runs used VelociQ8 1.0 (1.1 gives the same output on those models) and the mixture-of-experts runs used 1.1.

Where it does not win: text generation runs at about the speed of llama.cpp. On Arm, llama.cpp built with KleidiAI is faster than VelociQ8 on dense LLMs (16 % on Llama-3.2-1B prompt processing), but its output drifts 10 to 18 times further from full precision. Every table and how it was measured: Benchmarks and Quality.

Embeddings on Arm

EmbeddingGemma 300M on Oracle Cloud A1 (Ampere Altra), 4 threads, against three other llama.cpp builds, Ampere's own among them.

Embeddings on x86-64

EmbeddingGemma 300M on an AMD Ryzen 9 5950X (Windows 11), with 1 and 16 threads.

Mixture-of-experts LLMs

LLM files will be added later. VelociQ8 already runs them: prompt processing is faster, text generation is level, and on Arm the peak memory with OLMoE-1B-7B is 6.9 GiB against 13.6 GiB.

Get started

1. Get the runtime. Download the archive for your machine from the table, or from the latest release, and check it against SHA256SUMS from the same release.

The Ollama archives are a portable Ollama for the CPU with VelociQ8 inside, and do not touch an installed Ollama (see Ollama). The Linux archives need glibc 2.28 or newer. There is also a Docker image for Linux on Arm and x86-64, ghcr.io/dhaern/velociq8.

2. Download a model and run it. Each model is a repository VelociQ8/<model>-Q8A64-GGUF with one Q8A64 file. VelociQ8 runs on the CPU only and its tools cannot download models, so fetch the file first:

tar xf velociq8-1.2-linux-arm64.tar.gz && cd velociq8-1.2-linux-arm64
REPO=VelociQ8/embeddinggemma-2-Q8A64-GGUF
FILE=embeddinggemma-2-Q8A64.gguf
curl -L -O https://huggingface.co/$REPO/resolve/main/$FILE
./llama-server -m $FILE --embedding -t 4 -b 2048 -ub 2048 --port 8080

curl -s http://127.0.0.1:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"input": "task: search result | query: Which planet is known as the Red Planet?"}'

Q8A64 files load only in VelociQ8 1.1 or later, including its Ollama archives. Other GGUF tools cannot load them, and the Use this model snippets that Hugging Face shows for them do not work. You can also point VelociQ8 at a BF16, F16 or F32 GGUF: the weights are converted to Q8A64 while the model loads. See Usage.

Models

Ready-to-use Q8A64 files, grouped in collections. Each model keeps the license of its original weights, shown on its model card.

• Embeddings: embedding models
• LLM: coming later, in their own collection.

Links

• GitHub and releases
• Docker image: ghcr.io/dhaern/velociq8
• Support the project: Ko-fi
• Docs: Usage, Compatibility, Benchmarks, Quality, Changelog

License

VelociQ8 is proprietary software, free to use under the VelociQ8 Software License Agreement. You may run it for personal or commercial work. You may not redistribute, modify, decompile or reverse engineer it, except where the law expressly allows it. The agreement covers the runtime only.

The Q8A64 model files keep the license of the original model, stated on each model card. The release archives also contain llama.cpp, which is MIT licensed.

_{Arm is a registered trademark of Arm Limited (or its subsidiaries) in the US and/or elsewhere. Other product and hardware names belong to their respective owners and are used only to say what VelociQ8 is compared against or measured on.}