README.md

---
license: apache-2.0
library_name: llama.cpp
base_model: badtheorylabs/BTL-4
tags:
  - gguf
  - llama.cpp
  - agentic
  - tool-use
  - moe
  - quantized
pipeline_tag: text-generation
---

# BTL-4 Compact

**The whole 35B model in a single 9.96 GB file.** 2.30 bits per weight, and it
retains 94.1% of the full-precision model's measured behaviour.

BTL-4 is a mixture of experts with roughly 2.1B active parameters per token, so
it costs a large model's memory and a small model's compute. Compact is the
edition that runs on hardware you already own — one file, one command, a
running agent. No base download, no reconstruction.

Loads in [llama.cpp](https://github.com/ggml-org/llama.cpp), Ollama and
LM Studio.

Full-precision weights: [`badtheorylabs/BTL-4`](https://huggingface.co/badtheorylabs/BTL-4)

| build | size | bits/weight | behavioural retention |
|---|---|---|---|
| `BTL-4-IQ2_XXS.gguf` | 9.96 GB | 2.30 | **94.1%** |

Retention is measured, not estimated: 118 items on which the full-precision
bf16 model is correct, replayed against this build. It reproduces 111 of them.
Per category: 95.0% short-form factual, 100% grounded extraction, 87.2%
false-premise rejection. The gate resolves to about ±3.4 points, so treat
differences smaller than that as noise.

## Run it

```bash
llama-cli -m BTL-4-IQ2_XXS.gguf --jinja -c 8192 \
  -p "Refactor this function to be pure."
```

```bash
llama-server -m BTL-4-IQ2_XXS.gguf --port 8080 \
  --jinja \
  --reasoning-format deepseek \
  -c 32768 -fa on \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --temp 1.0 --top-p 0.95 --top-k 20
```

Requires a llama.cpp with `qwen3_5_moe` support (`src/models/qwen35moe.cpp`).

### Flags that are not optional

**`--jinja`.** Without it llama.cpp ignores the template embedded in the GGUF
and falls back to a built-in one. BTL-4 emits tool calls as
`<tool_call><function=name><parameter=arg>`, not stock Qwen's JSON form, so
without this flag tool calls do not parse and multi-turn tool use fails.

**`--reasoning-format deepseek`.** Without it, reasoning is left in `content`
instead of being separated into `reasoning_content`. It then accumulates on
every turn, the template cannot strip it from older turns, and the model
repeats turns until it runs out of budget. If your agent loops on an otherwise
sane task, check this flag first.

**Do not pass `--chat-template`.** The GGUF ships the correct one. Overriding it
with a generic Qwen template produces the same repeat-forever failure.

**Prefer `--cache-type-k/v q8_0` over `q4_0`.** At 2.30 bpw the weights are
already heavily compressed; a 4-bit KV cache on top of that degrades long-horizon
state tracking, which shows up as the model redoing work it already completed.
Only 10 of 40 layers keep a growing cache (~20 KB/token), so q8_0 is affordable
even at long context.

## Architecture

| | |
|---|---|
| total parameters | 35.1B (34.7B excluding the vision tower) |
| active per token | ~2.1B |
| layers | 40 — 30 linear-attention, 10 full-attention |
| experts | 256 per layer, 8 routed per token |
| context | 262,144 native |
| KV cache | ~20 KB/token |

Only 10 of 40 layers keep a growing KV cache, and those use 2 KV heads. The
whole 262K window costs about 5.2 GB of cache, so long-context work fits on
consumer hardware.

## Notes on this build

**The MTP layer is disabled.** The source model declares
`mtp_num_hidden_layers: 1` and the converter writes `block_count = 41` while
emitting tensors for only 40 blocks, so a stock loader fails on
`blk.40.attn_norm.weight`. This build sets `block_count = 40` and
`nextn_predict_layers = 0`. The multi-token-prediction head is a speculative
decoding accessory; the model runs without it.

**The vision tower is not included.** This is a text-only build.

## Quantisation

The 120 expert tensors are `IQ2_XXS` (2.0625 bpw); everything else follows the
`Q4_K_M` mixture. An importance matrix was computed over 120 chunks of a 3 MB
corpus of source code, technical documentation and question prompts — a
deliberate match for what this model is for, rather than generic web text.

The router (`ffn_gate_inp`) and every normalisation tensor stay at f32. Routing
decides *which* experts a token reaches, so error there changes which knowledge
gets used rather than degrading it smoothly, and at ~21M parameters it is free
to protect.

Where the 2.30 bpw goes: the experts are 93% of all parameters and contribute
1.92 bpw; the remaining 0.38 comes from the 4-bit and 6-bit non-expert matrices
plus the f32 router and norms.

Two findings from simulation work on this model shaped the recipe. **Range
selection dominates everything else at low bit widths** — replacing min/max
group ranging with a per-group MSE clip search moved retention from 77.1% to
95.8% at an identical byte budget. And **protecting the output head, the usual
recommendation, is worth nothing**: head and embedding at 4-bit retained 118 of
118. `IQ2_XXS` with an imatrix performs its own importance-weighted range
search, which is why it is the build shipped here.

## Licence

Apache-2.0, inherited from the base model.

© 2026 Bad Theory Labs