--- license: other license_name: fastino-community-license license_link: https://huggingface.co/BoldingBuilds/Wildcat-Mini/blob/main/LICENSES/FASTINO-LICENSE.txt base_model: prism-ml/Ternary-Bonsai-2-27B-gguf pipeline_tag: zero-shot-classification library_name: transformers tags: - decision-model - system-one - classification - ternary - 2-bit - lora --- # Wildcat Mini **Wildcat Mini is Ternary Bonsai 2 27B taught to answer decision questions.** Ask it to pick one option, answer yes / no, or rate something on a scale, and it returns a probability for every answer, in the same format as [d3-ultra](https://huggingface.co/vllm-sr/d3-ultra) and [Wildcat](https://huggingface.co/BoldingBuilds/Wildcat). Its language weights stay in Bonsai's own ternary form, about 2 bits per weight, so the model loads in **7.4 GiB of GPU memory**. | | | | --- | --- | | **Language model** | [Ternary Bonsai 2 27B](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf) (PrismML, Apache-2.0; Qwen3.8-27B family), PQ2_0 weights kept packed | | **Decision head** | [d3](https://huggingface.co/vllm-sr/d3)'s readout, tokenizer and runtime (Apache-2.0) | | **Training** | One LoRA (rank 16) taught by d3, then d3-ultra and Fastino GLiDE | | **GPU memory** | 7.4 GiB loaded | | **Download** | about 7.5 GiB | | **License** | Fastino Community License (see below). Built with Fastino GLiDE. | ## How it was made 1. **Swap the language model.** d3's package (code, tokenizer, the 255-code answer readout) with every language-model weight replaced by Ternary Bonsai 2 27B's. Both are Qwen3.8-27B shapes. On its own this scores 41.9 on our sample: the readout does not fit Bonsai's weights yet. 2. **Teach it.** One LoRA is trained in rounds to match a teacher's answer probabilities: first [vllm-sr/d3](https://huggingface.co/vllm-sr/d3) on about 33,000 questions and then on the 36,559-question mix below, and last on 12,000 of those with d3-ultra's and [Fastino GLiDE](https://huggingface.co/fastino/GLiDE)'s answers combined. The questions come only from public training splits, checked against the Decision Index for leaks. No Decision Index data was used for training. 3. **Serve it ternary.** At run time the 2-bit codes stay packed on the GPU; a small Triton kernel unpacks each layer just before its matmul. It scores the same as with the full 49 GB bf16 weights (60.23 vs 60.25 on our sample). ## Results Wildcat Mini has **not** been run on the full Decision Index or submitted to its board. All numbers below are from **our own 2,000-row sample of the Decision Index public set**, the same rows for every model. This sample reads about 2.7 points below the board's public score for d3-ultra, so compare rows with each other, not with the board. | Model (same 2,000 public rows) | GPU memory | Index | | --- | --- | --- | | d3-ultra (bf16) | about 54 GB | 68.84 | | Wildcat (3-bit d3-ultra) | 13.2 GiB | 69.39 | | **Wildcat Mini (this model)** | **7.4 GiB** | **60.23** | | Wildcat Mini, bf16 weights | about 49 GB | 60.25 | | Ternary Bonsai 2 27B + d3 readout, not trained | 7.4 GiB | 41.91 | **In short: about 9 points below d3-ultra on our sample, at a seventh of its memory.** If you have the memory for Wildcat, use Wildcat. **Speed.** Mini is slower than d3-ultra, because every layer is unpacked from ternary on the fly. On 750 sampled public requests, one at a time: | ms per request | median | p80 | mean | | --- | --- | --- | --- | | d3-ultra, RTX PRO 6000 | 168 | 1,067 | 932 | | Wildcat Mini, RTX PRO 6000 | 173 | 1,878 | 1,393 | | Wildcat Mini, RTX 3090 (2,000 sample rows) | 1,097 | 9,650 | 7,042 | ## What was not tested - **Cards smaller than 24 GB.** The weights take 7.4 GiB, but long requests need more memory on top. It was run on an RTX 3090 (24 GB) and an RTX PRO 6000 (96 GB) only. - **Images and video, Box and Track questions.** d3's runtime supports them; Mini was only measured on choice, yes / no and score questions. - AMD GPUs. The unpack kernel is Triton and was only run on NVIDIA (CUDA). ## Run it Needs an NVIDIA GPU, the packages in `requirements.txt`, plus `triton` and `accelerate`. ```bash hf download BoldingBuilds/Wildcat-Mini --local-dir wildcat-mini ``` With the Decision Index kit: ```bash PYTHONPATH=wildcat-mini python -m decision_index run --engine d3_tern:D3TernEngine --option model=wildcat-mini \ --rows --out ``` The packed weights and the LoRA in `adapter/` load on their own. Nothing else needs passing. ## Files | File | What it is | | --- | --- | | `mini-base.safetensors` | the packed ternary language weights (2-bit codes + one fp16 scale per 128 weights) plus the other backbone tensors | | `adapter/` | the LoRA (`adapter.safetensors`, `adapter.json`) | | `readout.safetensors`, tokenizer, configs, `d3_*.py` | from d3, unchanged | | `d3_tern.py` | the ternary engine and loader | | `d3_q3.py` | shared helpers from Wildcat's engine (memory limits for long requests) | ## Training data The same public-train-split mix as Wildcat: intent and topic (CLINC150, BANKING77, DBpedia-14, LEDGAR); reading, inference and paraphrase (SQuAD v2, BoolQ, SNLI, WANLI, VitaminC, PAWS, STS-B, PubMedQA, claim_stance + Perspectrum); commonsense and science (HellaSwag, WinoGrande, PIQA, Social IQa, CommonsenseQA, Cosmos QA, ARC, OpenBookQA, QASC, RuleTaker); maths and knowledge (GSM8K, AQuA-RAT, MATH, MMLU train, MedMCQA); ratings and taste (Feedback Collection, HelpSteer2, WMT DA, GoEmotions, Civil Comments, New Yorker Caption Contest, iSarcasmEval); tools and search (When2Call, Amazon ESCI); and questions generated here (MIT). Several are train splits of benchmarks the Decision Index tests, so expect Mini to do relatively better there. Each source keeps its own licence; no non-commercial data. ## License Wildcat Mini learned from Fastino GLiDE's answers, so it is a Derivative Work under the [Fastino Community License](LICENSES/FASTINO-LICENSE.txt) and ships under it. In short: - Free to use, change and share. Keep this license with any copy. - **Commercial use:** show "Built with Fastino GLiDE" in the product's docs, about page or UI. - **Companies with more than $10M yearly revenue** using it in production need a separate agreement with Fastino (hello@fastino.ai). Evaluation, testing, benchmarking and research are not production. Ternary Bonsai 2's weights, and d3's readout and runtime, are Apache-2.0 (`LICENSE`, `NOTICE`). **Built with Fastino GLiDE.** ## Credits - **Ternary Bonsai 2 27B**: PrismML ([prism-ml/Ternary-Bonsai-2-27B-gguf](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf)). Mini runs on their ternary weights. - **d3** and **d3-ultra**: the vLLM Semantic Router team ([vllm-sr/d3](https://huggingface.co/vllm-sr/d3), [vllm-sr/d3-ultra](https://huggingface.co/vllm-sr/d3-ultra)). d3 supplies the readout, tokenizer and runtime; both were teachers. - **Fastino GLiDE**: teacher ([fastino/GLiDE](https://huggingface.co/fastino/GLiDE)). - **Qwen** (Alibaba Cloud): the base model family. - The **Decision Index** (multimodalart) for the benchmark and kit. - The authors of every public dataset in the training mix (above).