Instructions to use Genrius-AI/Qwen3-0.6B-Pingala8-2bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Genrius-AI/Qwen3-0.6B-Pingala8-2bit with Trellis:

# No code snippets available yet for this library.

# To use this model, check the repository files and the library's documentation.

# Want to help? PRs adding snippets are welcome at:
# https://github.com/huggingface/huggingface.js

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Genrius-AI/Qwen3-0.6B-Pingala8-2bit with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Genrius-AI/Qwen3-0.6B-Pingala8-2bit
# Run inference directly in the terminal:
llama cli -hf Genrius-AI/Qwen3-0.6B-Pingala8-2bit

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Genrius-AI/Qwen3-0.6B-Pingala8-2bit
# Run inference directly in the terminal:
llama cli -hf Genrius-AI/Qwen3-0.6B-Pingala8-2bit

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Genrius-AI/Qwen3-0.6B-Pingala8-2bit
# Run inference directly in the terminal:
./llama-cli -hf Genrius-AI/Qwen3-0.6B-Pingala8-2bit

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Genrius-AI/Qwen3-0.6B-Pingala8-2bit
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Genrius-AI/Qwen3-0.6B-Pingala8-2bit

Use Docker

docker model run hf.co/Genrius-AI/Qwen3-0.6B-Pingala8-2bit

Llama new
LM Studio
Jan

vLLM

How to use Genrius-AI/Qwen3-0.6B-Pingala8-2bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Genrius-AI/Qwen3-0.6B-Pingala8-2bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Genrius-AI/Qwen3-0.6B-Pingala8-2bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Genrius-AI/Qwen3-0.6B-Pingala8-2bit

Ollama
How to use Genrius-AI/Qwen3-0.6B-Pingala8-2bit with Ollama:
```
ollama run hf.co/Genrius-AI/Qwen3-0.6B-Pingala8-2bit
```
Unsloth Desktop

How to use Genrius-AI/Qwen3-0.6B-Pingala8-2bit with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Genrius-AI/Qwen3-0.6B-Pingala8-2bit

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Genrius-AI/Qwen3-0.6B-Pingala8-2bit"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Genrius-AI/Qwen3-0.6B-Pingala8-2bit with Docker Model Runner:
```
docker model run hf.co/Genrius-AI/Qwen3-0.6B-Pingala8-2bit
```

Lemonade

How to use Genrius-AI/Qwen3-0.6B-Pingala8-2bit with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Genrius-AI/Qwen3-0.6B-Pingala8-2bit

Run and chat with the model

lemonade run user.Qwen3-0.6B-Pingala8-2bit-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use Genrius-AI/Qwen3-0.6B-Pingala8-2bit with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Genrius-AI/Qwen3-0.6B-Pingala8-2bit

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Genrius-AI/Qwen3-0.6B-Pingala8-2bit

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Genrius-AI/Qwen3-0.6B-Pingala8-2bit with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Genrius-AI/Qwen3-0.6B-Pingala8-2bit

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Genrius-AI/Qwen3-0.6B-Pingala8-2bit" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3-0.6B-Pingala8-2bit

Qwen3-0.6B in 0.28 GB, 2-bit: WikiText-2 perplexity 41.34 vs 10791 for llama.cpp IQ2_XXS, in a file 0.09 MB smaller (lower is better).

Honest caveats: a 0.6B model at 2 bits is weak (perplexity 2.16x the 8-bit file; zero-shot scores near chance on ARC-Challenge), and at this size this file generates tokens SLOWER than IQ2_XXS (0.70x). It is mainly useful where 0.28 GB is the hard limit.

Needs the patched llama.cpp runtime (CUDA, NVIDIA RTX 30/40/50): the runtime of the v2.5 one-click release or llama.cpp 7ac59a6 + the patches in the code repository. Stock llama.cpp, Ollama and LM Studio cannot load it. The one-click build has no launcher for this model yet (planned for v2.6); until then start its llama-server with this file (see "How to run").

File	Quant	Size	WikiText-2 PPL
`Qwen3-0.6B-Pingala8-2bit.gguf`	Pingala-8, 2-bit	0.28 GB	41.34
for comparison: IQ2_XXS (llama.cpp, same imatrix)	2-bit	0.28 GB	10791
for comparison: IQ2_KT (ik_llama.cpp, default recipe)	2-bit	0.29 GB	48.97
for comparison: Q8_0 (llama.cpp)	8-bit	0.80 GB	19.11

Author: Harshit Goyal, Genrius AI (harshit@genrius.in). Base model: Qwen/Qwen3-0.6B. Base revision c1899de2. Method paper: Small-State Coset Trellis Coding at IQ2_XXS Size: An Equal-Bytes Study in llama.cpp (preprint, v2.4), DOI 10.5281/zenodo.23481757. The numbers on this page are new measurements and are not in the paper. Code and runtime patch: https://github.com/Genrius-AI/pingala8-llm

What this is

A 2-bit-class quantization of Qwen3-0.6B (28 decoder layers (grouped-query attention and SwiGLU FFN)) at no more bytes than llama.cpp's IQ2_XXS file of the same model. Every linear weight (196 tensors: attn_q, attn_k, attn_v, attn_output, ffn_gate, ffn_up, ffn_down in every layer) is stored with Ungerboeck's 8-state set-partitioned trellis code over 4-weight symbols, behind a random Hadamard rotation and QuIP's LDLQ rounding, one rate class per tensor and incoherence-bound column scales (the B4 recipe of the Qwen3-4B release, unchanged). Layer Hessians come from 256 sequences of 2048 tokens of C4 text. The other tensors (token embedding and tied output, norms) are copied from the stock IQ2_XXS file.

This was a single build with the recipe and the release rule written down before any 2-bit number was measured. To meet the IQ2_XXS byte count, ten tensors (blk.17 to blk.25 attn_v and blk.2 ffn_down) were coded one rate class lower than the recipe's nearest class, chosen by a fixed rule written down before any perplexity was measured (the first build was 698,336 bytes over).

Files

file	bytes	sha256
`Qwen3-0.6B-Pingala8-2bit.gguf`	280,408,160	`0e534cb999adb5473c4d9f5edac81f4a47acd0e4f57cd76c4f5bd9189129be2d`

IQ2_XXS of the same model: 280,496,640 bytes.

Which file should I pick?

If you...	Use
have an NVIDIA RTX 30/40/50 GPU and want Qwen3-0.6B in 0.28 GB	this file with the v2.5 one-click runtime or a patched build
use stock llama.cpp, Ollama, LM Studio, a Mac, an AMD GPU or only a CPU	a standard GGUF from the base model's Quantizations list; this file will not load there
serve many requests at once or process long prompts	a standard quant: this file processes prompts at 0.861x the speed of IQ2_XXS

Prompt format

The Qwen3-0.6B chat template is stored in the GGUF (ChatML, with Qwen3 thinking switch).

Results

WikiText-2 test, llama-perplexity -c 2049 --ppl-stride 4095 (windows of 4096 tokens; this tokenizer gives 73 windows on the text). Paired difference: this file minus the row, nats per token, 95% interval, and the windows on which this file is better.

format	file bytes	PPL	ratio to Q8_0	paired, Pingala-8 - row
Q8_0 (llama.cpp)	0.80 GB	19.1116	1.000	+0.772 +/- 0.023 (0/73)
IQ2_XXS (llama.cpp, imatrix)	280,496,640	10791.2170	564.6	-5.565 +/- 0.173 (73/73)
IQ2_KT (ik_llama.cpp, default recipe, imatrix)	289,226,752	48.9733	2.563	-0.169 +/- 0.020 (72/73)
Pingala-8 (this file)	280,408,160	41.3423	2.163	0

The IQ2_KT file is 3.1% larger than this one, and this file is still better on 72 of 73 windows. At this size llama.cpp's IQ2_XXS breaks down (perplexity in the thousands), so that comparison mainly shows that IQ2_XXS is unusable here. All three 2-bit files use the same importance matrix (C4 text, context 512, 200 chunks).

Zero-shot accuracy, % (0-shot continuation scores from llama-perplexity; HellaSwag 1,000 items, MMLU all 14,042 test items):

format	HellaSwag	WinoGrande	ARC-Easy	ARC-Challenge	MMLU	average
Pingala-8 (this file)	35.5	52.49	39.65	25.08	28.04	36.15
IQ2_XXS	26.8	49.49	28.6	25.42	26.67	31.4

Speed, same-round ratio to IQ2_XXS on one RTX 5070 Ti Laptop GPU, batch 1 (llama-bench -ngl 99 -fa 1, median of 3 rounds of -r 5): token generation (tg128) 0.700x, prompt processing (pp512) 0.861x. Per round: tg128 0.819, 0.700, 0.582 (this file 361.3, 379.1, 317.4 tokens/s; IQ2_XXS 441.1, 541.5, 545.1); pp512 1.232, 0.861, 0.825. At 0.6B the decode step is so short that fixed per-matrix costs of the trellis kernel dominate.

How to run

With the v2.5 one-click release (Linux; on Windows use the same command with bin\llama-server.exe and set for the variables):

hf download Genrius-AI/Qwen3-0.6B-Pingala8-2bit Qwen3-0.6B-Pingala8-2bit.gguf --local-dir models
export GGML_F44=4 GGML_F44_BATCH=2 GGML_VR_NATIVE=1 GGML_VRC_V=5 GGML_VRC_NWP=8 GGML_VR_RL=1 GGML_VR_PRMG=1 GGML_VR_PPC=2
./bin/llama-server -m models/Qwen3-0.6B-Pingala8-2bit.gguf -ngl 99 --port 8080

or build llama.cpp 7ac59a6e3ad851cd41af00f678effab0598ba9a8 with patches/llama_cuda_vs_7ac59a6.patch and patches/q35_loader_count_delta.patch from the code repository (-DGGML_CUDA=ON). The measured perplexity command is llama-perplexity -m Qwen3-0.6B-Pingala8-2bit.gguf -f wikitext2_test_hfjoin.raw -c 2049 --ppl-stride 4095 -ngl 99 (expected final value [73]41.3423).

Limitations

CUDA only, measured on one laptop GPU (RTX 5070 Ti Laptop, WSL2); no Metal, Vulkan or CPU path for this file.
No one-click launcher yet (planned for v2.6).
Prompt processing is slower than IQ2_XXS (0.861x). Batched decoding was not measured on this model; on Qwen3-4B it was much slower than IQ2_XXS (0.18-0.34x for 2 to 8 sequences).
At 2 bits the model is clearly worse than the 8-bit model (perplexity 2.163x Q8_0); small models lose more.
Perplexity was measured at context 4096 only.
With the same recipe the method lost to IQ2_KT on Qwen3-14B (paper, Section 4.3); results do not transfer to every model.
Refusal, bias and hallucination behaviour after quantization were not evaluated. Evaluate the model for your own use.

Citation

@misc{goyal2026pingala8,
  title        = {Small-State Coset Trellis Coding at {IQ2\_XXS} Size: An Equal-Bytes Study in {llama.cpp}},
  author       = {Goyal, Harshit},
  year         = {2026},
  month        = oct,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.23481757},
  url          = {https://doi.org/10.5281/zenodo.23481757},
  note         = {Preprint, version 2.4. Code: https://github.com/Genrius-AI/pingala8-llm}
}

License and notice

Apache-2.0, inherited from Qwen3-0.6B (Qwen team, Alibaba Cloud); see LICENSE and NOTICE. The weights are a quantized derivative of Qwen3-0.6B made by Genrius AI. "Pingala-8" and "Genrius AI" are names of Genrius AI; the license grants no trademark rights.

Downloads last month: -

GGUF

Model size

0.4B params

Architecture

qwen3

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Model tree for Genrius-AI/Qwen3-0.6B-Pingala8-2bit

Base model

Qwen/Qwen3-0.6B-Base

Finetuned

Qwen/Qwen3-0.6B

Quantized

(510)

this model

Evaluation results

Perplexity (lower is better) on WikiText-2 (test, context 4096)
test set Genrius AI measurement (logs in the code repository)

41.342
Accuracy (%) on HellaSwag (0-shot, llama-perplexity)
Genrius AI measurement (logs in the code repository)

35.500
Accuracy (%) on WinoGrande (0-shot, llama-perplexity)
Genrius AI measurement (logs in the code repository)

52.490
Accuracy (%) on ARC-Easy (0-shot, llama-perplexity)
Genrius AI measurement (logs in the code repository)

39.650
Accuracy (%) on ARC-Challenge (0-shot, llama-perplexity)
Genrius AI measurement (logs in the code repository)

25.080
Accuracy (%) on MMLU (0-shot, llama-perplexity)
Genrius AI measurement (logs in the code repository)

28.040