Hugging Face's logo

AssistantsLab
/
SmolLM2-360M-humanized_GGUF

GGUF
conversational
Model card Files
xet
 Community
How to use from
llama.cpp
Install from brew 
brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf AssistantsLab/SmolLM2-360M-humanized_GGUF:
# Run inference directly in the terminal:
llama-cli -hf AssistantsLab/SmolLM2-360M-humanized_GGUF:
Install from WinGet (Windows) 
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf AssistantsLab/SmolLM2-360M-humanized_GGUF:
# Run inference directly in the terminal:
llama-cli -hf AssistantsLab/SmolLM2-360M-humanized_GGUF:
Use pre-built binary 
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf AssistantsLab/SmolLM2-360M-humanized_GGUF:
# Run inference directly in the terminal:
./llama-cli -hf AssistantsLab/SmolLM2-360M-humanized_GGUF:
Build from source code 
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf AssistantsLab/SmolLM2-360M-humanized_GGUF:
# Run inference directly in the terminal:
./build/bin/llama-cli -hf AssistantsLab/SmolLM2-360M-humanized_GGUF:
Use Docker 
docker model run hf.co/AssistantsLab/SmolLM2-360M-humanized_GGUF:
Quick Links

Introduction

This repo contains the humanized 360M SmolLM2 model in the GGUF Format

  • Quantization: q2_K, q3_K_S, q3_K_M, q3_K_L, q4_0, q4_K_S, q4_K_M, q5_0, q5_K_S, q5_K_M, q6_K, q8_0

More about this model

  • We released a 135M, 360M and 1.7B parameter version of this model. For more information, view our report.

Quickstart

We advise you to clone llama.cpp and install it following the official guide. We follow the latest version of llama.cpp. In the following demonstration, we assume that you are running commands under the repository llama.cpp.

Since cloning the entire repo may be inefficient, you can manually download the GGUF file that you need or use huggingface-cli:

  1. Install
    pip install -U huggingface_hub
  2. Download:
    huggingface-cli download AssistantsLab/SmolLM2-360M-humanized_GGUF smollm2-360M-humanized-q4_k_m.gguf --local-dir . --local-dir-use-symlinks False

Quants

Filename Quant type File Size
smollm2-360m-humanized-q2_k.gguf Q2_K 219MB
smollm2-360m-humanized-q3_k_s.gguf Q3_K_S 219MB
smollm2-360m-humanized-q3_k_m.gguf Q3_K_M 235MB
smollm2-360m-humanized-q3_k_l.gguf Q3_K_L 246MB
smollm2-360m-humanized-q4_0.gguf Q4_0 229MB
smollm2-360m-humanized-q4_k_s.gguf Q4_K_S 260MB
smollm2-360m-humanized-q4_k_m.gguf Q4_K_M 271B
smollm2-360m-humanized-q5_0.gguf Q5_0 268MB
smollm2-360m-humanized-q5_k_s.gguf Q5_K_S 283MB
smollm2-360m-humanized-q5_k_m.gguf Q5_K_M 290MB
smollm2-360m-humanized-q6_k.gguf Q6_K 367MB
smollm2-360m-humanized-q8_0.gguf Q8_0 386MB

More information

For more information about this model, please visit the original model here.

License

Apache 2.0

Citation

SmolLM2:

@misc{allal2024SmolLM2,
      title={SmolLM2 - with great data, comes great performance}, 
      author={Loubna Ben Allal and Anton Lozhkov and Elie Bakouch and Gabriel Martín Blázquez and Lewis Tunstall and Agustín Piqueres and Andres Marafioti and Cyril Zakka and Leandro von Werra and Thomas Wolf},
      year={2024},
}

Human-Like-DPO-Dataset:

@misc{çalık2025enhancinghumanlikeresponseslarge,
      title={Enhancing Human-Like Responses in Large Language Models}, 
      author={Ethem Yağız Çalık and Talha Rüzgar Akkuş},
      year={2025},
      eprint={2501.05032},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2501.05032}, 
}
Downloads last month
103
GGUF
Model size
0.4B params
Architecture
llama
Hardware compatibility
Log In to add your hardware

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collection including AssistantsLab/SmolLM2-360M-humanized_GGUF

Paper for AssistantsLab/SmolLM2-360M-humanized_GGUF