Instructions to use Genrius-AI/Qwen3-1.7B-Pingala8-2bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Genrius-AI/Qwen3-1.7B-Pingala8-2bit with Trellis:

# No code snippets available yet for this library.

# To use this model, check the repository files and the library's documentation.

# Want to help? PRs adding snippets are welcome at:
# https://github.com/huggingface/huggingface.js

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Genrius-AI/Qwen3-1.7B-Pingala8-2bit with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Genrius-AI/Qwen3-1.7B-Pingala8-2bit
# Run inference directly in the terminal:
llama cli -hf Genrius-AI/Qwen3-1.7B-Pingala8-2bit

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Genrius-AI/Qwen3-1.7B-Pingala8-2bit
# Run inference directly in the terminal:
llama cli -hf Genrius-AI/Qwen3-1.7B-Pingala8-2bit

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Genrius-AI/Qwen3-1.7B-Pingala8-2bit
# Run inference directly in the terminal:
./llama-cli -hf Genrius-AI/Qwen3-1.7B-Pingala8-2bit

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Genrius-AI/Qwen3-1.7B-Pingala8-2bit
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Genrius-AI/Qwen3-1.7B-Pingala8-2bit

Use Docker

docker model run hf.co/Genrius-AI/Qwen3-1.7B-Pingala8-2bit

Llama new
LM Studio
Jan

vLLM

How to use Genrius-AI/Qwen3-1.7B-Pingala8-2bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Genrius-AI/Qwen3-1.7B-Pingala8-2bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Genrius-AI/Qwen3-1.7B-Pingala8-2bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Genrius-AI/Qwen3-1.7B-Pingala8-2bit

Ollama
How to use Genrius-AI/Qwen3-1.7B-Pingala8-2bit with Ollama:
```
ollama run hf.co/Genrius-AI/Qwen3-1.7B-Pingala8-2bit
```
Unsloth Desktop

How to use Genrius-AI/Qwen3-1.7B-Pingala8-2bit with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Genrius-AI/Qwen3-1.7B-Pingala8-2bit

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Genrius-AI/Qwen3-1.7B-Pingala8-2bit"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Genrius-AI/Qwen3-1.7B-Pingala8-2bit with Docker Model Runner:
```
docker model run hf.co/Genrius-AI/Qwen3-1.7B-Pingala8-2bit
```

Lemonade

How to use Genrius-AI/Qwen3-1.7B-Pingala8-2bit with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Genrius-AI/Qwen3-1.7B-Pingala8-2bit

Run and chat with the model

lemonade run user.Qwen3-1.7B-Pingala8-2bit-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use Genrius-AI/Qwen3-1.7B-Pingala8-2bit with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Genrius-AI/Qwen3-1.7B-Pingala8-2bit

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Genrius-AI/Qwen3-1.7B-Pingala8-2bit

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Genrius-AI/Qwen3-1.7B-Pingala8-2bit with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Genrius-AI/Qwen3-1.7B-Pingala8-2bit

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Genrius-AI/Qwen3-1.7B-Pingala8-2bit" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3-1.7B-Pingala8-2bit

Qwen3-1.7B in 0.69 GB, 2-bit: WikiText-2 perplexity 28.23 vs 47.96 for llama.cpp IQ2_XXS, in a file 0.45 MB smaller (lower is better).

Honest caveat: on this model ik_llama.cpp's IQ2_KT (a different runtime) is better: perplexity 20.89 against 28.23, at a 3.3% larger file. This file is the better choice only against llama.cpp's own 2-bit IQ2_XXS.

Needs the patched llama.cpp runtime (CUDA, NVIDIA RTX 30/40/50): the runtime of the v2.5 one-click release or llama.cpp 7ac59a6 + the patches in the code repository. Stock llama.cpp, Ollama and LM Studio cannot load it. The one-click build has no launcher for this model yet (planned for v2.6); until then start its llama-server with this file (see "How to run").

File	Quant	Size	WikiText-2 PPL
`Qwen3-1.7B-Pingala8-2bit.gguf`	Pingala-8, 2-bit	0.69 GB	28.23
for comparison: IQ2_XXS (llama.cpp, same imatrix)	2-bit	0.69 GB	47.96
for comparison: IQ2_KT (ik_llama.cpp, default recipe)	2-bit	0.71 GB	20.89
for comparison: Q8_0 (llama.cpp)	8-bit	2.2 GB	15.65

Author: Harshit Goyal, Genrius AI (harshit@genrius.in). Base model: Qwen/Qwen3-1.7B. Base revision 70d244cc. Method paper: Small-State Coset Trellis Coding at IQ2_XXS Size: An Equal-Bytes Study in llama.cpp (preprint, v2.4), DOI 10.5281/zenodo.23481757. The numbers on this page are new measurements and are not in the paper. Code and runtime patch: https://github.com/Genrius-AI/pingala8-llm

What this is

A 2-bit-class quantization of Qwen3-1.7B (28 decoder layers (grouped-query attention and SwiGLU FFN)) at no more bytes than llama.cpp's IQ2_XXS file of the same model. Every linear weight (196 tensors: attn_q, attn_k, attn_v, attn_output, ffn_gate, ffn_up, ffn_down in every layer) is stored with Ungerboeck's 8-state set-partitioned trellis code over 4-weight symbols, behind a random Hadamard rotation and QuIP's LDLQ rounding, one rate class per tensor and incoherence-bound column scales (the B4 recipe of the Qwen3-4B release, unchanged). Layer Hessians come from 256 sequences of 2048 tokens of C4 text. The other tensors (token embedding and tied output, norms) are copied from the stock IQ2_XXS file.

This was a single build with the recipe and the release rule written down before any 2-bit number was measured. To meet the IQ2_XXS byte count, four tensors (blk.0, blk.19 and blk.20 attn_v and blk.1 ffn_down) were coded one rate class lower than the recipe's nearest class, chosen by a fixed rule written down before any perplexity was measured (the first build was 727,040 bytes over).

Files

file	bytes	sha256
`Qwen3-1.7B-Pingala8-2bit.gguf`	692,136,672	`69c9f95952503911ad4e4ea277b657acd668b611166a211facf49f376a42556c`

IQ2_XXS of the same model: 692,589,120 bytes.

Which file should I pick?

If you...	Use
have an NVIDIA RTX 30/40/50 GPU and want Qwen3-1.7B in 0.69 GB	this file with the v2.5 one-click runtime or a patched build
use stock llama.cpp, Ollama, LM Studio, a Mac, an AMD GPU or only a CPU	a standard GGUF from the base model's Quantizations list; this file will not load there
serve many requests at once or process long prompts	a standard quant: this file processes prompts at 0.795x the speed of IQ2_XXS

Prompt format

The Qwen3-1.7B chat template is stored in the GGUF (ChatML, with Qwen3 thinking switch).

Results

WikiText-2 test, llama-perplexity -c 2049 --ppl-stride 4095 (windows of 4096 tokens; this tokenizer gives 73 windows on the text). Paired difference: this file minus the row, nats per token, 95% interval, and the windows on which this file is better.

format	file bytes	PPL	ratio to Q8_0	paired, Pingala-8 - row
Q8_0 (llama.cpp)	2.2 GB	15.6484	1.000	+0.590 +/- 0.020 (0/73)
IQ2_XXS (llama.cpp, imatrix)	692,589,120	47.9569	3.065	-0.530 +/- 0.024 (73/73)
IQ2_KT (ik_llama.cpp, default recipe, imatrix)	714,995,808	20.8926	1.335	+0.301 +/- 0.014 (0/73)
Pingala-8 (this file)	692,136,672	28.2299	1.804	0

IQ2_KT is better than this file on this model: perplexity 20.89 against 28.23, better on all 73 windows, with a file 3.3% (22.9 MB) larger. This file beats llama.cpp's IQ2_XXS, not ik_llama.cpp's IQ2_KT; if you can run ik_llama.cpp, its IQ2_KT is the better 2-bit choice for Qwen3-1.7B. All three 2-bit files use the same importance matrix (C4 text, context 512, 200 chunks).

Zero-shot accuracy, % (0-shot continuation scores from llama-perplexity; HellaSwag 1,000 items, MMLU all 14,042 test items):

format	HellaSwag	WinoGrande	ARC-Easy	ARC-Challenge	MMLU	average
Pingala-8 (this file)	46.1	57.85	45.09	30.1	30.48	41.92
IQ2_XXS	33.7	52.33	38.6	26.09	28.93	35.93

Speed, same-round ratio to IQ2_XXS on one RTX 5070 Ti Laptop GPU, batch 1 (llama-bench -ngl 99 -fa 1, median of 3 rounds of -r 5): token generation (tg128) 1.019x, prompt processing (pp512) 0.795x. Per round: tg128 1.020, 1.019, 0.985; pp512 0.791, 0.798, 0.795.

How to run

With the v2.5 one-click release (Linux; on Windows use the same command with bin\llama-server.exe and set for the variables):

hf download Genrius-AI/Qwen3-1.7B-Pingala8-2bit Qwen3-1.7B-Pingala8-2bit.gguf --local-dir models
export GGML_F44=4 GGML_F44_BATCH=2 GGML_VR_NATIVE=1 GGML_VRC_V=5 GGML_VRC_NWP=8 GGML_VR_RL=1 GGML_VR_PRMG=1 GGML_VR_PPC=2
./bin/llama-server -m models/Qwen3-1.7B-Pingala8-2bit.gguf -ngl 99 --port 8080

or build llama.cpp 7ac59a6e3ad851cd41af00f678effab0598ba9a8 with patches/llama_cuda_vs_7ac59a6.patch and patches/q35_loader_count_delta.patch from the code repository (-DGGML_CUDA=ON). The measured perplexity command is llama-perplexity -m Qwen3-1.7B-Pingala8-2bit.gguf -f wikitext2_test_hfjoin.raw -c 2049 --ppl-stride 4095 -ngl 99 (expected final value [73]28.2299).

Limitations

CUDA only, measured on one laptop GPU (RTX 5070 Ti Laptop, WSL2); no Metal, Vulkan or CPU path for this file.
No one-click launcher yet (planned for v2.6).
Prompt processing is slower than IQ2_XXS (0.795x). Batched decoding was not measured on this model; on Qwen3-4B it was much slower than IQ2_XXS (0.18-0.34x for 2 to 8 sequences).
At 2 bits the model is clearly worse than the 8-bit model (perplexity 1.804x Q8_0); small models lose more.
Perplexity was measured at context 4096 only.
With the same recipe the method lost to IQ2_KT on Qwen3-14B (paper, Section 4.3); results do not transfer to every model.
Refusal, bias and hallucination behaviour after quantization were not evaluated. Evaluate the model for your own use.

Citation

@misc{goyal2026pingala8,
  title        = {Small-State Coset Trellis Coding at {IQ2\_XXS} Size: An Equal-Bytes Study in {llama.cpp}},
  author       = {Goyal, Harshit},
  year         = {2026},
  month        = oct,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.23481757},
  url          = {https://doi.org/10.5281/zenodo.23481757},
  note         = {Preprint, version 2.4. Code: https://github.com/Genrius-AI/pingala8-llm}
}

License and notice

Apache-2.0, inherited from Qwen3-1.7B (Qwen team, Alibaba Cloud); see LICENSE and NOTICE. The weights are a quantized derivative of Qwen3-1.7B made by Genrius AI. "Pingala-8" and "Genrius AI" are names of Genrius AI; the license grants no trademark rights.

Downloads last month: -

GGUF

Model size

1.0B params

Architecture

qwen3

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Model tree for Genrius-AI/Qwen3-1.7B-Pingala8-2bit

Base model

Qwen/Qwen3-1.7B-Base

Finetuned

Qwen/Qwen3-1.7B

Quantized

(439)

this model

Evaluation results

Perplexity (lower is better) on WikiText-2 (test, context 4096)
test set Genrius AI measurement (logs in the code repository)

28.230
Accuracy (%) on HellaSwag (0-shot, llama-perplexity)
Genrius AI measurement (logs in the code repository)

46.100
Accuracy (%) on WinoGrande (0-shot, llama-perplexity)
Genrius AI measurement (logs in the code repository)

57.850
Accuracy (%) on ARC-Easy (0-shot, llama-perplexity)
Genrius AI measurement (logs in the code repository)

45.090
Accuracy (%) on ARC-Challenge (0-shot, llama-perplexity)
Genrius AI measurement (logs in the code repository)

30.100
Accuracy (%) on MMLU (0-shot, llama-perplexity)
Genrius AI measurement (logs in the code repository)

30.480