Llama-3.2-1B-Instruct-numallama

Built with Llama

meta-llama/Llama-3.2-1B-Instruct, converted for numallama: a C inference engine for CPU servers, with NUMA-aware execution across all sockets of multi-socket machines.

These files are not GGUF. They run only with numallama (they do not work with llama.cpp, Ollama, LM Studio or transformers).

Files

File Weights Size Decode speed*
llama-3.2-1b-instruct-q8.bin Q8 1.3 GB 184 tok/s
tokenizer.bin tokenizer 1.9 MB

Q4 is asymmetric 4-bit (groups of 32, fp16 scale and min), quantized with an importance matrix collected by numallama on English, Russian, Chinese and code text. Q8 is int8 with per-group scales. Q4 is faster and smaller; Q8 is closer to the original.

* tokens/s for a single chat, measured on 2 ร— Ampere Altra Max M128-30 (256 Neoverse N1 cores), both sockets. Other machines will differ.

Run

Linux. Optimized for ARM64 with NEON dot product (Ampere Altra, AWS Graviton, other Neoverse cores); other CPUs, including x86-64, run portable C code that is not optimized.

git clone https://github.com/miloserdev/numallama && cd numallama && make
pip install huggingface_hub
hf download miloserdev/Llama-3.2-1B-Instruct-numallama --local-dir models/Llama-3.2-1B-Instruct-numallama

# chat in the terminal
./runq models/Llama-3.2-1B-Instruct-numallama/llama-3.2-1b-instruct-q8.bin -z models/Llama-3.2-1B-Instruct-numallama/tokenizer.bin -m chat

# or an OpenAI-compatible server on http://127.0.0.1:8080/v1
./runq models/Llama-3.2-1B-Instruct-numallama/llama-3.2-1b-instruct-q8.bin -z models/Llama-3.2-1B-Instruct-numallama/tokenizer.bin -m server -P 8080

See the numallama README for all options (context length, sampling, tools, speculative decoding, API details).

License

Llama 3.2 is licensed under the Llama 3.2 Community License, Copyright ยฉ Meta Platforms, Inc. All Rights Reserved. Use of this model must comply with the Llama 3.2 Community License and the Acceptable Use Policy.

Original model: meta-llama/Llama-3.2-1B-Instruct. Conversion: numallama.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for miloserdev/Llama-3.2-1B-Instruct-numallama

Quantized
(439)
this model