| --- |
| base_model: HuggingFaceTB/SmolLM3-3B |
| language: |
| - en |
| library_name: gguf |
| tags: |
| - quantization |
| - llama.cpp |
| - gguf |
| - smollm |
| - nim-kernels |
| - 4-bit |
| - 8-bit |
| - consumer-hardware |
| --- |
| |
| # SmolLM3-3B — 4-bit GGUF with Custom Nim Kernels |
|
|
| Quantized versions of [SmolLM3-3B](https://huggingface.co/HuggingFaceTB/SmolLM3-3B) |
| running on consumer hardware under 6 GB VRAM, with custom matrix kernels written in Nim |
| for hot-path matrix-multiply operations. |
|
|
| Full write-up: [Run LLMs on a 6 GB Laptop GPU & CPU — Medium](https://medium.com/@devbyankit/run-llms-on-a-6-gb-laptop-gpu-cpu-smollm-3-quantization-nim-kernels-6c9cbb233930) |
|
|
| --- |
|
|
| ## What's in this repo |
|
|
| | File | Format | Size approx. | Use case | |
| |------|--------|--------------|----------| |
| | smolLM3-q4_k_m.gguf | Q4_K_M | ~1.9 GB | Best speed, GPU + CPU | |
| | smolLM3-q8_0.gguf | Q8_0 | ~3.3 GB | Near FP16 quality, fits 6 GB | |
|
|
| Memory footprint is reduced ~75% from the FP16 baseline. |
|
|
| --- |
|
|
| ## Performance (512-token prompt, averaged over 3 runs) |
|
|
| | Configuration | Prompt Processing | Text Generation | VRAM / RAM | |
| |--------------|-------------------|-----------------|------------| |
| | 4-bit GPU | 11.25 tok/s | **14.60 tok/s** | ~5.5 GB VRAM | |
| | 8-bit GPU | 2.57 tok/s | 12.65 tok/s | ~5.8 GB VRAM | |
| | 4-bit CPU | 2.38 tok/s | 15.37 tok/s | ~3–4 GB RAM | |
| | 8-bit CPU | 7.79 tok/s | 11.83 tok/s | ~3–4 GB RAM | |
|
|
| Hardware used: NVIDIA GPU with 6 GB VRAM, Windows, CUDA 12.4. |
|
|
| --- |
|
|
| ## How to run |
|
|
| ```bash |
| # 4-bit GPU |
| llama-cli.exe -m smolLM3-q4_k_m.gguf -ngl 36 -n 256 --temp 0.7 \ |
| --repeat-penalty 1.1 --color -sys "you are a helpful assistant" |
| |
| # 8-bit CPU |
| llama-cli.exe -m smolLM3-q8_0.gguf -ngl 0 -n 256 --temp 0.7 |
| ``` |
|
|
| Requires [llama.cpp](https://github.com/ggerganov/llama.cpp) 0.3.14+ built with CUDA |
| support for GPU runs. |
|
|
| --- |
|
|
| ## Custom Nim Kernels |
|
|
| The repo ships two companion DLLs — `libsmolkernels_q4.dll` and `libsmolkernels_q8.dll` |
| — written in Nim, compiled with `-O3` and ORC memory management. They implement a |
| custom dequantization + matrix-multiply path (`mulQ4Mat` / `mulQ8Mat`) that is |
| dynamically loaded by a patched `main.cpp` at runtime based on the detected model type. |
| llama-cli.exe detects "q4_k_m" in filename → loads libsmolkernels_q4.dll → mulQ4Mat() |
| llama-cli.exe detects "q8_0" in filename → loads libsmolkernels_q8.dll → mulQ8Mat() |
| Full build instructions, kernel source code, and the main.cpp patch are in the |
| Medium article linked above. |
| |
| --- |
| |
| ## System requirements |
| |
| - NVIDIA GPU 4 GB+ VRAM (for GPU runs), or 8 GB+ RAM (for CPU runs) |
| - CUDA Toolkit 12.0+ |
| - llama.cpp 0.3.x+ |
| - Windows (DLL-based kernel loader); Linux port straightforward with .so |
| |
| --- |
| |
| ## Related project |
| |
| This work is the research foundation for |
| [QBench CLI](https://github.com/AnkitTsj/qbench) — a C++ command-line tool that |
| automates LLM quantization workflows, model selection, and hardware compatibility |
| checks using the same quantization and kernel techniques developed here. |
