Instructions to use bartowski/SmolLM2-135M-Instruct-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use bartowski/SmolLM2-135M-Instruct-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf bartowski/SmolLM2-135M-Instruct-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf bartowski/SmolLM2-135M-Instruct-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf bartowski/SmolLM2-135M-Instruct-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf bartowski/SmolLM2-135M-Instruct-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf bartowski/SmolLM2-135M-Instruct-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf bartowski/SmolLM2-135M-Instruct-GGUF:Q4_K_M
Use Docker
docker model run hf.co/bartowski/SmolLM2-135M-Instruct-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use bartowski/SmolLM2-135M-Instruct-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "bartowski/SmolLM2-135M-Instruct-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "bartowski/SmolLM2-135M-Instruct-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/bartowski/SmolLM2-135M-Instruct-GGUF:Q4_K_M
- Ollama
How to use bartowski/SmolLM2-135M-Instruct-GGUF with Ollama:
ollama run hf.co/bartowski/SmolLM2-135M-Instruct-GGUF:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use bartowski/SmolLM2-135M-Instruct-GGUF with Docker Model Runner:
docker model run hf.co/bartowski/SmolLM2-135M-Instruct-GGUF:Q4_K_M
- Lemonade
How to use bartowski/SmolLM2-135M-Instruct-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull bartowski/SmolLM2-135M-Instruct-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.SmolLM2-135M-Instruct-GGUF-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Three quants, one prompt, three runs: speed is flat, the answers are not
I ran the same prompt through Q2_K, Q4_K_M and Q8_0 of this repository three times each, in a browser, with llama.cpp compiled to WebAssembly. Same page, same settings, temperature 0.3, the prompt:
List three prime numbers below 20 and say in one sentence why 1 is not prime.
Speed, in tokens per second, from llama.cpp's own counters:
| Quant | File | Run 1 | Run 2 | Run 3 |
|---|---|---|---|---|
| Q2_K | 88 MB | 22.4 | 32.5 | 29.6 |
| Q4_K_M | 106 MB | 28.1 | 30.7 | 29.1 |
| Q8_0 | 145 MB | 30.9 | 34.9 | 36.3 |
Two things stand out.
The spread inside one quant is as large as the spread between quants. Q2_K moved 10 tokens per second between runs while the three files differ by 65 percent in size. On a 135M model in WebAssembly the dequantization cost is small enough that the file size is not the lever people assume it is. Q8_0 was the fastest in every run, which is the opposite of the usual expectation.
The answers were wrong in every quant, and each one failed differently. Q2_K said 1 is composite and, in another run, included 1 in the list of primes. Q4_K_M ignored the bound and listed primes up to 227. Q8_0 said 1 is divisible by 2, 3 and 5. The one clean answer in nine runs came from Q4_K_M. Run to run the output changed more than it changed between quants.
None of that is a criticism of the model. It is a 135M model answering a two-part instruction, and the point is narrower: at this size, choosing a quant for quality is the wrong lever. Choose by download size if bandwidth matters, then spend the effort on the prompt or pick a larger model.
Measured on an M1 Pro, Chrome, single threaded because a static page sends no COOP or COEP headers. The page is https://huggingface.co/spaces/abderrahmane11/gguf-runtime if you want to repeat it with your own prompt.