README.md

---
license: other
license_name: fastino-community-license
license_link: https://huggingface.co/BoldingBuilds/Wildcat-Mini/blob/main/LICENSES/FASTINO-LICENSE.txt
base_model: prism-ml/Ternary-Bonsai-2-27B-gguf
pipeline_tag: zero-shot-classification
library_name: transformers
tags:
- decision-model
- system-one
- classification
- ternary
- 2-bit
- lora
---

# Wildcat Mini

**Wildcat Mini is Ternary Bonsai 2 27B taught to answer decision questions.** Ask it to pick one option, answer
yes / no, or rate something on a scale, and it returns a probability for every answer, in the same format as
[d3-ultra](https://huggingface.co/vllm-sr/d3-ultra) and [Wildcat](https://huggingface.co/BoldingBuilds/Wildcat).
Its language weights stay in Bonsai's own ternary form, about 2 bits per weight, so the model loads in **7.4 GiB of
GPU memory**.

| | |
| --- | --- |
| **Language model** | [Ternary Bonsai 2 27B](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf) (PrismML, Apache-2.0; Qwen3.8-27B family), PQ2_0 weights kept packed |
| **Decision head** | [d3](https://huggingface.co/vllm-sr/d3)'s readout, tokenizer and runtime (Apache-2.0) |
| **Training** | One LoRA (rank 16) taught by d3, then d3-ultra and Fastino GLiDE |
| **GPU memory** | 7.4 GiB loaded |
| **Download** | about 7.5 GiB |
| **License** | Fastino Community License (see below). Built with Fastino GLiDE. |

## How it was made

1. **Swap the language model.** d3's package (code, tokenizer, the 255-code answer readout) with every
   language-model weight replaced by Ternary Bonsai 2 27B's. Both are Qwen3.8-27B shapes. On its own this scores
   41.9 on our sample: the readout does not fit Bonsai's weights yet.
2. **Teach it.** One LoRA is trained in rounds to match a teacher's answer probabilities: first
   [vllm-sr/d3](https://huggingface.co/vllm-sr/d3) on about 33,000 questions and then on the 36,559-question mix
   below, and last on 12,000 of those with d3-ultra's and [Fastino GLiDE](https://huggingface.co/fastino/GLiDE)'s
   answers combined. The questions come only from public training splits, checked against the Decision Index for
   leaks. No Decision Index data was used for training.
3. **Serve it ternary.** At run time the 2-bit codes stay packed on the GPU; a small Triton kernel unpacks each layer
   just before its matmul. It scores the same as with the full 49 GB bf16 weights (60.23 vs 60.25 on our sample).

## Results

Wildcat Mini has **not** been run on the full Decision Index or submitted to its board. All numbers below are from
**our own 2,000-row sample of the Decision Index public set**, the same rows for every model. This sample reads about
2.7 points below the board's public score for d3-ultra, so compare rows with each other, not with the board.

| Model (same 2,000 public rows) | GPU memory | Index |
| --- | --- | --- |
| d3-ultra (bf16) | about 54 GB | 68.84 |
| Wildcat (3-bit d3-ultra) | 13.2 GiB | 69.39 |
| **Wildcat Mini (this model)** | **7.4 GiB** | **60.23** |
| Wildcat Mini, bf16 weights | about 49 GB | 60.25 |
| Ternary Bonsai 2 27B + d3 readout, not trained | 7.4 GiB | 41.91 |

**In short: about 9 points below d3-ultra on our sample, at a seventh of its memory.** If you have the memory for
Wildcat, use Wildcat.

**Speed.** Mini is slower than d3-ultra, because every layer is unpacked from ternary on the fly. On 750 sampled
public requests, one at a time:

| ms per request | median | p80 | mean |
| --- | --- | --- | --- |
| d3-ultra, RTX PRO 6000 | 168 | 1,067 | 932 |
| Wildcat Mini, RTX PRO 6000 | 173 | 1,878 | 1,393 |
| Wildcat Mini, RTX 3090 (2,000 sample rows) | 1,097 | 9,650 | 7,042 |

## What was not tested

- **Cards smaller than 24 GB.** The weights take 7.4 GiB, but long requests need more memory on top. It was run on an
  RTX 3090 (24 GB) and an RTX PRO 6000 (96 GB) only.
- **Images and video, Box and Track questions.** d3's runtime supports them; Mini was only measured on choice, yes / no
  and score questions.
- AMD GPUs. The unpack kernel is Triton and was only run on NVIDIA (CUDA).

## Run it

Needs an NVIDIA GPU, the packages in `requirements.txt`, plus `triton` and `accelerate`.

```bash
hf download BoldingBuilds/Wildcat-Mini --local-dir wildcat-mini
```

With the Decision Index kit:

```bash
PYTHONPATH=wildcat-mini python -m decision_index run --engine d3_tern:D3TernEngine --option model=wildcat-mini \
  --rows <rows.jsonl.gz> --out <run dir>
```

The packed weights and the LoRA in `adapter/` load on their own. Nothing else needs passing.

## Files

| File | What it is |
| --- | --- |
| `mini-base.safetensors` | the packed ternary language weights (2-bit codes + one fp16 scale per 128 weights) plus the other backbone tensors |
| `adapter/` | the LoRA (`adapter.safetensors`, `adapter.json`) |
| `readout.safetensors`, tokenizer, configs, `d3_*.py` | from d3, unchanged |
| `d3_tern.py` | the ternary engine and loader |
| `d3_q3.py` | shared helpers from Wildcat's engine (memory limits for long requests) |

## Training data

The same public-train-split mix as Wildcat: intent and topic (CLINC150, BANKING77, DBpedia-14, LEDGAR); reading,
inference and paraphrase (SQuAD v2, BoolQ, SNLI, WANLI, VitaminC, PAWS, STS-B, PubMedQA, claim_stance +
Perspectrum); commonsense and science (HellaSwag, WinoGrande, PIQA, Social IQa, CommonsenseQA, Cosmos QA, ARC,
OpenBookQA, QASC, RuleTaker); maths and knowledge (GSM8K, AQuA-RAT, MATH, MMLU train, MedMCQA); ratings and taste
(Feedback Collection, HelpSteer2, WMT DA, GoEmotions, Civil Comments, New Yorker Caption Contest, iSarcasmEval);
tools and search (When2Call, Amazon ESCI); and questions generated here (MIT). Several are train splits of
benchmarks the Decision Index tests, so expect Mini to do relatively better there. Each source keeps its own
licence; no non-commercial data.

## License

Wildcat Mini learned from Fastino GLiDE's answers, so it is a Derivative Work under the
[Fastino Community License](LICENSES/FASTINO-LICENSE.txt) and ships under it. In short:

- Free to use, change and share. Keep this license with any copy.
- **Commercial use:** show "Built with Fastino GLiDE" in the product's docs, about page or UI.
- **Companies with more than $10M yearly revenue** using it in production need a separate agreement with Fastino
  (hello@fastino.ai). Evaluation, testing, benchmarking and research are not production.

Ternary Bonsai 2's weights, and d3's readout and runtime, are Apache-2.0 (`LICENSE`, `NOTICE`).

**Built with Fastino GLiDE.**

## Credits

- **Ternary Bonsai 2 27B**: PrismML ([prism-ml/Ternary-Bonsai-2-27B-gguf](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf)). Mini runs on their ternary weights.
- **d3** and **d3-ultra**: the vLLM Semantic Router team ([vllm-sr/d3](https://huggingface.co/vllm-sr/d3), [vllm-sr/d3-ultra](https://huggingface.co/vllm-sr/d3-ultra)). d3 supplies the readout, tokenizer and runtime; both were teachers.
- **Fastino GLiDE**: teacher ([fastino/GLiDE](https://huggingface.co/fastino/GLiDE)).
- **Qwen** (Alibaba Cloud): the base model family.
- The **Decision Index** (multimodalart) for the benchmark and kit.
- The authors of every public dataset in the training mix (above).