Instructions to use Tiiny/SmallThinker-4BA0.6B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Tiiny/SmallThinker-4BA0.6B-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Tiiny/SmallThinker-4BA0.6B-Instruct", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("Tiiny/SmallThinker-4BA0.6B-Instruct", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use Tiiny/SmallThinker-4BA0.6B-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Tiiny/SmallThinker-4BA0.6B-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Tiiny/SmallThinker-4BA0.6B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Tiiny/SmallThinker-4BA0.6B-Instruct

SGLang

How to use Tiiny/SmallThinker-4BA0.6B-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Tiiny/SmallThinker-4BA0.6B-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Tiiny/SmallThinker-4BA0.6B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Tiiny/SmallThinker-4BA0.6B-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Tiiny/SmallThinker-4BA0.6B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Tiiny/SmallThinker-4BA0.6B-Instruct with Docker Model Runner:
```
docker model run hf.co/Tiiny/SmallThinker-4BA0.6B-Instruct
```

nielsr HF Staff commited on Jul 29, 2025

Commit

6526c24

verified ·

1 Parent(s): 512a7f0

Improve model card: Add `library_name`, explicit paper and code links

Browse files

This PR enhances the model card by:
- Adding `library_name: transformers` to the metadata, which enables the "Use in Transformers" snippet on the Hub and improves model categorization.
- Adding the paper title as the main heading.
- Adding explicit "Paper" and "Code" sections at the top for better discoverability and direct access to the research and source code.

Files changed (1) hide show

README.md +10 -4

README.md CHANGED Viewed

@@ -1,9 +1,16 @@
 ---
-license: apache-2.0
 language:
 - en
 pipeline_tag: text-generation
 ---
 ## Introduction
 <p align="center">
@@ -42,7 +49,7 @@ All models are evaluated in non-thinking mode.
 | Qwen3 0.6B                                    | 0.6                 | 148.56   | 94.91      | 45.93        | 15.29  | 27.44          | 13.32  | 9.76   |
 | Qwen3 1.7B                                    | 1.3                 | 62.24    | 41.00      | 20.29        | 6.09   | 11.08          | 6.35   | 4.15   |
 | Qwen3 1.7B+limited memory                     | limit 1G            | 2.66     | 1.09       | 1.00         | 0.47   | -              | -      | 0.11   |
-| Gemma3n E2B                                   | 1G, theoretically   | 36.88    | 27.06      | 12.50        | 3.80   | 6.66           | 3.46   | 2.45   |
 Note: i9 14900, 1+13 8ge4 use 4 threads, others use the number of threads that can achieve the maximum speed. All models here have been quantized to q4_0.
@@ -115,5 +122,4 @@ from modelscope import AutoModelForCausalLM, AutoTokenizer
 ## Statement
 - Due to the constraints of its model size and the limitations of its training data, its responses may contain factual inaccuracies, biases, or outdated information.
 - Users bear full responsibility for independently evaluating and verifying the accuracy and appropriateness of all generated content.
-- SmallThinker does not possess genuine comprehension or consciousness and cannot express personal opinions or value judgments.

 ---
 language:
 - en
+license: apache-2.0
 pipeline_tag: text-generation
+library_name: transformers
 ---
+# SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment
+**Paper**: [SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment](https://huggingface.co/papers/2507.20984)
+**Code**: [https://github.com/SJTU-IPADS/SmallThinker](https://github.com/SJTU-IPADS/SmallThinker)
 ## Introduction
 <p align="center">
 | Qwen3 0.6B                                    | 0.6                 | 148.56   | 94.91      | 45.93        | 15.29  | 27.44          | 13.32  | 9.76   |
 | Qwen3 1.7B                                    | 1.3                 | 62.24    | 41.00      | 20.29        | 6.09   | 11.08          | 6.35   | 4.15   |
 | Qwen3 1.7B+limited memory                     | limit 1G            | 2.66     | 1.09       | 1.00         | 0.47   | -              | -      | 0.11   |
+| Gemma3n E2B                                   | 1G, theoretically   | 36.88    | 27.06      | 12.50        | 3.80   | 6.66           | 3.80   | 2.45   |
 Note: i9 14900, 1+13 8ge4 use 4 threads, others use the number of threads that can achieve the maximum speed. All models here have been quantized to q4_0.
 ## Statement
 - Due to the constraints of its model size and the limitations of its training data, its responses may contain factual inaccuracies, biases, or outdated information.
 - Users bear full responsibility for independently evaluating and verifying the accuracy and appropriateness of all generated content.
+- SmallThinker does not possess genuine comprehension or consciousness and cannot express personal opinions or value judgments.