Llama-3.2-3B-Instruct-numallama
Built with Llama
meta-llama/Llama-3.2-3B-Instruct, converted for numallama: a C inference engine for CPU servers, with NUMA-aware execution across all sockets of multi-socket machines.
These files are not GGUF. They run only with numallama (they do not work with llama.cpp, Ollama, LM Studio or transformers).
Files
| File | Weights | Size | Decode speed* |
|---|---|---|---|
llama-3.2-3b-instruct-q4.bin |
Q4 | 2.2 GB | 104 tok/s |
llama-3.2-3b-instruct-q8.bin |
Q8 | 3.4 GB | 77 tok/s |
tokenizer.bin |
tokenizer | 1.9 MB |
Q4 is asymmetric 4-bit (groups of 32, fp16 scale and min), quantized with an importance matrix collected by numallama on English, Russian, Chinese and code text. Q8 is int8 with per-group scales. Q4 is faster and smaller; Q8 is closer to the original.
* tokens/s for a single chat, measured on 2 ร Ampere Altra Max M128-30 (256 Neoverse N1 cores), both sockets. Other machines will differ.
Run
Linux. Optimized for ARM64 with NEON dot product (Ampere Altra, AWS Graviton, other Neoverse cores); other CPUs, including x86-64, run portable C code that is not optimized.
git clone https://github.com/miloserdev/numallama && cd numallama && make
pip install huggingface_hub
hf download miloserdev/Llama-3.2-3B-Instruct-numallama --local-dir models/Llama-3.2-3B-Instruct-numallama
# chat in the terminal
./runq models/Llama-3.2-3B-Instruct-numallama/llama-3.2-3b-instruct-q4.bin -z models/Llama-3.2-3B-Instruct-numallama/tokenizer.bin -m chat
# or an OpenAI-compatible server on http://127.0.0.1:8080/v1
./runq models/Llama-3.2-3B-Instruct-numallama/llama-3.2-3b-instruct-q4.bin -z models/Llama-3.2-3B-Instruct-numallama/tokenizer.bin -m server -P 8080
See the numallama README for all options (context length, sampling, tools, speculative decoding, API details).
License
Llama 3.2 is licensed under the Llama 3.2 Community License, Copyright ยฉ Meta Platforms, Inc. All Rights Reserved. Use of this model must comply with the Llama 3.2 Community License and the Acceptable Use Policy.
Original model: meta-llama/Llama-3.2-3B-Instruct. Conversion: numallama.
Model tree for miloserdev/Llama-3.2-3B-Instruct-numallama
Base model
meta-llama/Llama-3.2-3B-Instruct