| --- |
| license: other |
| license_name: fastino-community-license |
| license_link: https://huggingface.co/BoldingBuilds/Wildcat-Mini/blob/main/LICENSES/FASTINO-LICENSE.txt |
| base_model: prism-ml/Ternary-Bonsai-2-27B-gguf |
| pipeline_tag: zero-shot-classification |
| library_name: transformers |
| tags: |
| - decision-model |
| - system-one |
| - classification |
| - ternary |
| - 2-bit |
| - lora |
| --- |
| |
| # Wildcat Mini |
|
|
| **Wildcat Mini is Ternary Bonsai 2 27B taught to answer decision questions.** Ask it to pick one option, answer |
| yes / no, or rate something on a scale, and it returns a probability for every answer, in the same format as |
| [d3-ultra](https://huggingface.co/vllm-sr/d3-ultra) and [Wildcat](https://huggingface.co/BoldingBuilds/Wildcat). |
| Its language weights stay in Bonsai's own ternary form, about 2 bits per weight, so the model loads in **7.4 GiB of |
| GPU memory**. |
|
|
| | | | |
| | --- | --- | |
| | **Language model** | [Ternary Bonsai 2 27B](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf) (PrismML, Apache-2.0; Qwen3.8-27B family), PQ2_0 weights kept packed | |
| | **Decision head** | [d3](https://huggingface.co/vllm-sr/d3)'s readout, tokenizer and runtime (Apache-2.0) | |
| | **Training** | One LoRA (rank 16) taught by d3, then d3-ultra and Fastino GLiDE | |
| | **GPU memory** | 7.4 GiB loaded | |
| | **Download** | about 7.5 GiB | |
| | **License** | Fastino Community License (see below). Built with Fastino GLiDE. | |
| |
| ## How it was made |
| |
| 1. **Swap the language model.** d3's package (code, tokenizer, the 255-code answer readout) with every |
| language-model weight replaced by Ternary Bonsai 2 27B's. Both are Qwen3.8-27B shapes. On its own this scores |
| 41.9 on our sample: the readout does not fit Bonsai's weights yet. |
| 2. **Teach it.** One LoRA is trained in rounds to match a teacher's answer probabilities: first |
| [vllm-sr/d3](https://huggingface.co/vllm-sr/d3) on about 33,000 questions and then on the 36,559-question mix |
| below, and last on 12,000 of those with d3-ultra's and [Fastino GLiDE](https://huggingface.co/fastino/GLiDE)'s |
| answers combined. The questions come only from public training splits, checked against the Decision Index for |
| leaks. No Decision Index data was used for training. |
| 3. **Serve it ternary.** At run time the 2-bit codes stay packed on the GPU; a small Triton kernel unpacks each layer |
| just before its matmul. It scores the same as with the full 49 GB bf16 weights (60.23 vs 60.25 on our sample). |
| |
| ## Results |
| |
| Wildcat Mini has **not** been run on the full Decision Index or submitted to its board. All numbers below are from |
| **our own 2,000-row sample of the Decision Index public set**, the same rows for every model. This sample reads about |
| 2.7 points below the board's public score for d3-ultra, so compare rows with each other, not with the board. |
| |
| | Model (same 2,000 public rows) | GPU memory | Index | |
| | --- | --- | --- | |
| | d3-ultra (bf16) | about 54 GB | 68.84 | |
| | Wildcat (3-bit d3-ultra) | 13.2 GiB | 69.39 | |
| | **Wildcat Mini (this model)** | **7.4 GiB** | **60.23** | |
| | Wildcat Mini, bf16 weights | about 49 GB | 60.25 | |
| | Ternary Bonsai 2 27B + d3 readout, not trained | 7.4 GiB | 41.91 | |
| |
| **In short: about 9 points below d3-ultra on our sample, at a seventh of its memory.** If you have the memory for |
| Wildcat, use Wildcat. |
| |
| **Speed.** Mini is slower than d3-ultra, because every layer is unpacked from ternary on the fly. On 750 sampled |
| public requests, one at a time: |
| |
| | ms per request | median | p80 | mean | |
| | --- | --- | --- | --- | |
| | d3-ultra, RTX PRO 6000 | 168 | 1,067 | 932 | |
| | Wildcat Mini, RTX PRO 6000 | 173 | 1,878 | 1,393 | |
| | Wildcat Mini, RTX 3090 (2,000 sample rows) | 1,097 | 9,650 | 7,042 | |
| |
| ## What was not tested |
| |
| - **Cards smaller than 24 GB.** The weights take 7.4 GiB, but long requests need more memory on top. It was run on an |
| RTX 3090 (24 GB) and an RTX PRO 6000 (96 GB) only. |
| - **Images and video, Box and Track questions.** d3's runtime supports them; Mini was only measured on choice, yes / no |
| and score questions. |
| - AMD GPUs. The unpack kernel is Triton and was only run on NVIDIA (CUDA). |
| |
| ## Run it |
| |
| Needs an NVIDIA GPU, the packages in `requirements.txt`, plus `triton` and `accelerate`. |
| |
| ```bash |
| hf download BoldingBuilds/Wildcat-Mini --local-dir wildcat-mini |
| ``` |
| |
| With the Decision Index kit: |
| |
| ```bash |
| PYTHONPATH=wildcat-mini python -m decision_index run --engine d3_tern:D3TernEngine --option model=wildcat-mini \ |
| --rows <rows.jsonl.gz> --out <run dir> |
| ``` |
| |
| The packed weights and the LoRA in `adapter/` load on their own. Nothing else needs passing. |
| |
| ## Files |
| |
| | File | What it is | |
| | --- | --- | |
| | `mini-base.safetensors` | the packed ternary language weights (2-bit codes + one fp16 scale per 128 weights) plus the other backbone tensors | |
| | `adapter/` | the LoRA (`adapter.safetensors`, `adapter.json`) | |
| | `readout.safetensors`, tokenizer, configs, `d3_*.py` | from d3, unchanged | |
| | `d3_tern.py` | the ternary engine and loader | |
| | `d3_q3.py` | shared helpers from Wildcat's engine (memory limits for long requests) | |
| |
| ## Training data |
| |
| The same public-train-split mix as Wildcat: intent and topic (CLINC150, BANKING77, DBpedia-14, LEDGAR); reading, |
| inference and paraphrase (SQuAD v2, BoolQ, SNLI, WANLI, VitaminC, PAWS, STS-B, PubMedQA, claim_stance + |
| Perspectrum); commonsense and science (HellaSwag, WinoGrande, PIQA, Social IQa, CommonsenseQA, Cosmos QA, ARC, |
| OpenBookQA, QASC, RuleTaker); maths and knowledge (GSM8K, AQuA-RAT, MATH, MMLU train, MedMCQA); ratings and taste |
| (Feedback Collection, HelpSteer2, WMT DA, GoEmotions, Civil Comments, New Yorker Caption Contest, iSarcasmEval); |
| tools and search (When2Call, Amazon ESCI); and questions generated here (MIT). Several are train splits of |
| benchmarks the Decision Index tests, so expect Mini to do relatively better there. Each source keeps its own |
| licence; no non-commercial data. |
| |
| ## License |
| |
| Wildcat Mini learned from Fastino GLiDE's answers, so it is a Derivative Work under the |
| [Fastino Community License](LICENSES/FASTINO-LICENSE.txt) and ships under it. In short: |
| |
| - Free to use, change and share. Keep this license with any copy. |
| - **Commercial use:** show "Built with Fastino GLiDE" in the product's docs, about page or UI. |
| - **Companies with more than $10M yearly revenue** using it in production need a separate agreement with Fastino |
| (hello@fastino.ai). Evaluation, testing, benchmarking and research are not production. |
| |
| Ternary Bonsai 2's weights, and d3's readout and runtime, are Apache-2.0 (`LICENSE`, `NOTICE`). |
| |
| **Built with Fastino GLiDE.** |
| |
| ## Credits |
| |
| - **Ternary Bonsai 2 27B**: PrismML ([prism-ml/Ternary-Bonsai-2-27B-gguf](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf)). Mini runs on their ternary weights. |
| - **d3** and **d3-ultra**: the vLLM Semantic Router team ([vllm-sr/d3](https://huggingface.co/vllm-sr/d3), [vllm-sr/d3-ultra](https://huggingface.co/vllm-sr/d3-ultra)). d3 supplies the readout, tokenizer and runtime; both were teachers. |
| - **Fastino GLiDE**: teacher ([fastino/GLiDE](https://huggingface.co/fastino/GLiDE)). |
| - **Qwen** (Alibaba Cloud): the base model family. |
| - The **Decision Index** (multimodalart) for the benchmark and kit. |
| - The authors of every public dataset in the training mix (above). |
| |
