Qwen2.5-3B-Instruct-numallama
Qwen/Qwen2.5-3B-Instruct, converted for numallama: a C inference engine for CPU servers, with NUMA-aware execution across all sockets of multi-socket machines.
These files are not GGUF. They run only with numallama (they do not work with llama.cpp, Ollama, LM Studio or transformers).
Files
| File | Weights | Size | Decode speed* |
|---|---|---|---|
qwen2.5-3b-instruct-q4.bin |
Q4 | 2.1 GB | 107 tok/s |
qwen2.5-3b-instruct-q8.bin |
Q8 (SmoothQuant) | 3.3 GB | 79 tok/s |
tokenizer.bin |
tokenizer | 2.2 MB |
Q4 is asymmetric 4-bit (groups of 32, fp16 scale and min), quantized with an importance matrix collected by numallama on English, Russian, Chinese and code text. Q8 is int8 with per-group scales. Q4 is faster and smaller; Q8 is closer to the original.
* tokens/s for a single chat, measured on 2 ร Ampere Altra Max M128-30 (256 Neoverse N1 cores), both sockets. Other machines will differ.
Run
Linux. Optimized for ARM64 with NEON dot product (Ampere Altra, AWS Graviton, other Neoverse cores); other CPUs, including x86-64, run portable C code that is not optimized.
git clone https://github.com/miloserdev/numallama && cd numallama && make
pip install huggingface_hub
hf download miloserdev/Qwen2.5-3B-Instruct-numallama --local-dir models/Qwen2.5-3B-Instruct-numallama
# chat in the terminal
./runq models/Qwen2.5-3B-Instruct-numallama/qwen2.5-3b-instruct-q4.bin -z models/Qwen2.5-3B-Instruct-numallama/tokenizer.bin -m chat
# or an OpenAI-compatible server on http://127.0.0.1:8080/v1
./runq models/Qwen2.5-3B-Instruct-numallama/qwen2.5-3b-instruct-q4.bin -z models/Qwen2.5-3B-Instruct-numallama/tokenizer.bin -m server -P 8080
See the numallama README for all options (context length, sampling, tools, speculative decoding, API details).
License
Licensed under the Qwen Research License, like the original model: non-commercial use only.
Original model: Qwen/Qwen2.5-3B-Instruct. Conversion: numallama.