Qwen2.5-3B-Instruct-numallama

Qwen/Qwen2.5-3B-Instruct, converted for numallama: a C inference engine for CPU servers, with NUMA-aware execution across all sockets of multi-socket machines.

These files are not GGUF. They run only with numallama (they do not work with llama.cpp, Ollama, LM Studio or transformers).

Files

File Weights Size Decode speed*
qwen2.5-3b-instruct-q4.bin Q4 2.1 GB 107 tok/s
qwen2.5-3b-instruct-q8.bin Q8 (SmoothQuant) 3.3 GB 79 tok/s
tokenizer.bin tokenizer 2.2 MB

Q4 is asymmetric 4-bit (groups of 32, fp16 scale and min), quantized with an importance matrix collected by numallama on English, Russian, Chinese and code text. Q8 is int8 with per-group scales. Q4 is faster and smaller; Q8 is closer to the original.

* tokens/s for a single chat, measured on 2 ร— Ampere Altra Max M128-30 (256 Neoverse N1 cores), both sockets. Other machines will differ.

Run

Linux. Optimized for ARM64 with NEON dot product (Ampere Altra, AWS Graviton, other Neoverse cores); other CPUs, including x86-64, run portable C code that is not optimized.

git clone https://github.com/miloserdev/numallama && cd numallama && make
pip install huggingface_hub
hf download miloserdev/Qwen2.5-3B-Instruct-numallama --local-dir models/Qwen2.5-3B-Instruct-numallama

# chat in the terminal
./runq models/Qwen2.5-3B-Instruct-numallama/qwen2.5-3b-instruct-q4.bin -z models/Qwen2.5-3B-Instruct-numallama/tokenizer.bin -m chat

# or an OpenAI-compatible server on http://127.0.0.1:8080/v1
./runq models/Qwen2.5-3B-Instruct-numallama/qwen2.5-3b-instruct-q4.bin -z models/Qwen2.5-3B-Instruct-numallama/tokenizer.bin -m server -P 8080

See the numallama README for all options (context length, sampling, tools, speculative decoding, API details).

License

Licensed under the Qwen Research License, like the original model: non-commercial use only.

Original model: Qwen/Qwen2.5-3B-Instruct. Conversion: numallama.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for miloserdev/Qwen2.5-3B-Instruct-numallama

Base model

Qwen/Qwen2.5-3B
Quantized
(347)
this model