Instructions to use jun-ai/Rusa-1-Small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use jun-ai/Rusa-1-Small with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf jun-ai/Rusa-1-Small:TQ1_0 # Run inference directly in the terminal: llama cli -hf jun-ai/Rusa-1-Small:TQ1_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf jun-ai/Rusa-1-Small:TQ1_0 # Run inference directly in the terminal: llama cli -hf jun-ai/Rusa-1-Small:TQ1_0
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf jun-ai/Rusa-1-Small:TQ1_0 # Run inference directly in the terminal: ./build/bin/llama cli -hf jun-ai/Rusa-1-Small:TQ1_0
Use Docker
docker model run hf.co/jun-ai/Rusa-1-Small:TQ1_0
- Llama new
- LM Studio
- Jan
- vLLM
How to use jun-ai/Rusa-1-Small with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "jun-ai/Rusa-1-Small" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jun-ai/Rusa-1-Small", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/jun-ai/Rusa-1-Small:TQ1_0
- Ollama
How to use jun-ai/Rusa-1-Small with Ollama:
ollama run hf.co/jun-ai/Rusa-1-Small:TQ1_0
- Unsloth Desktop
- Docker Model Runner
How to use jun-ai/Rusa-1-Small with Docker Model Runner:
docker model run hf.co/jun-ai/Rusa-1-Small:TQ1_0
- Lemonade
How to use jun-ai/Rusa-1-Small with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull jun-ai/Rusa-1-Small:TQ1_0
Run and chat with the model
lemonade run user.Rusa-1-Small-TQ1_0
List all available models
lemonade list
- Atomic Chat
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf jun-ai/Rusa-1-Small:TQ1_0# Run inference directly in the terminal:
llama cli -hf jun-ai/Rusa-1-Small:TQ1_0Build from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf jun-ai/Rusa-1-Small:TQ1_0# Run inference directly in the terminal:
./build/bin/llama cli -hf jun-ai/Rusa-1-Small:TQ1_0Use Docker
docker model run hf.co/jun-ai/Rusa-1-Small:TQ1_0Rusa-1-Small
Experimental CPU GGUFs for rewriting with a 350M-parameter model under 100 MB. These variants use ternary weights in most projections and Q2_K in selected tensors. The standard model is Rusa-1.
| Build | Size | Download |
|---|---|---|
| Q6_K-distilled mixed ternary with learned block scales | 99.03 MB | TQ1_0-QAT |
| Original fitted ternary | 94.12 MB | TQ1_0-Fit-Down |
Evaluation
The 99.03 MB build was selected after two consecutive rounds failed to improve the combined automated development score by two percentage points.
On eight additional drafts, a blinded manual review gave the selected build credit for 10 of 24 factual relationships, compared with 19 of 24 for Q6_K. The selected build produced zero fully correct rewrites on those drafts. The automated checks missed some factual changes. For reliable rewriting, use the standard Rusa-1 GGUFs.
Inference
Use a CPU llama.cpp build with LFM2 and TQ1_0 support. Rusa uses a plain-text prompt. Put the draft after the rewrite instructions and end the prompt with two newlines, ### Rewritten:, and two more newlines. See the Rusa-1 prompt format for the full instructions.
llama-cli -m gguf/Rusa-1-Small-TQ1_0-QAT.gguf -ngl 0 --temp 0 -f prompt.txt
- Downloads last month
- 105
1-bit


Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf jun-ai/Rusa-1-Small:TQ1_0# Run inference directly in the terminal: llama cli -hf jun-ai/Rusa-1-Small:TQ1_0