Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf cortexso/small-thinker:# Run inference directly in the terminal:
llama-cli -hf cortexso/small-thinker:Use pre-built binary
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf cortexso/small-thinker:# Run inference directly in the terminal:
./llama-cli -hf cortexso/small-thinker:Build from source code
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf cortexso/small-thinker:# Run inference directly in the terminal:
./build/bin/llama-cli -hf cortexso/small-thinker:Use Docker
docker model run hf.co/cortexso/small-thinker:Overview
PowerInfer developed and released the SmallThinker-3B-preview, a fine-tuned version of the Qwen2.5-3B-Instruct model. SmallThinker is optimized for efficient deployment on resource-constrained devices while maintaining high performance in reasoning, coding, and general text generation tasks. It outperforms its base model on key benchmarks, including AIME24, AMC23, and GAOKAO2024, making it a robust tool for both edge deployment and as a draft model for larger systems like QwQ-32B-Preview.
SmallThinker was fine-tuned in two phases using high-quality datasets, including PowerInfer/QWQ-LONGCOT-500K and PowerInfer/LONGCOT-Refine-500K. Its small size allows for up to 70% faster inference speeds compared to larger models, making it ideal for applications requiring quick responses and efficient computation.
Variants
| No | Variant | Cortex CLI command |
|---|---|---|
| 1 | Small-thinker-3b | cortex run small-thinker:3b |
Use it with Jan (UI)
- Install Jan using Quickstart
- Use in Jan model Hub:
cortexso/small-thinker
Use it with Cortex (CLI)
- Install Cortex using Quickstart
- Run the model with command:
cortex run small-thinker
Credits
- Downloads last month
- 41
2-bit
3-bit
4-bit
5-bit
6-bit
8-bit
Install from brew
# Start a local OpenAI-compatible server with a web UI: llama-server -hf cortexso/small-thinker:# Run inference directly in the terminal: llama-cli -hf cortexso/small-thinker: