Model card

README.md

@@ -0,0 +1,140 @@
+---
+license: other
+license_name: fastino-community-license
+license_link: https://huggingface.co/BoldingBuilds/Wildcat-Mini/blob/main/LICENSES/FASTINO-LICENSE.txt
+base_model: prism-ml/Ternary-Bonsai-2-27B-gguf
+pipeline_tag: zero-shot-classification
+library_name: transformers
+tags:
+- decision-model
+- system-one
+- classification
+- ternary
+- 2-bit
+- lora
+---
+
+# Wildcat Mini
+
+**Wildcat Mini is Ternary Bonsai 2 27B taught to answer decision questions.** Ask it to pick one option, answer
+yes / no, or rate something on a scale, and it returns a probability for every answer, in the same format as
+[d3-ultra](https://huggingface.co/vllm-sr/d3-ultra) and [Wildcat](https://huggingface.co/BoldingBuilds/Wildcat).
+Its language weights stay in Bonsai's own ternary form, about 2 bits per weight, so the model loads in **7.4 GiB of
+GPU memory**.
+
+| | |
+| --- | --- |
+| **Language model** | [Ternary Bonsai 2 27B](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf) (PrismML, Apache-2.0; Qwen3.8-27B family), PQ2_0 weights kept packed |
+| **Decision head** | [d3](https://huggingface.co/vllm-sr/d3)'s readout, tokenizer and runtime (Apache-2.0) |
+| **Training** | One LoRA (rank 16) taught by d3, then d3-ultra and Fastino GLiDE |
+| **GPU memory** | 7.4 GiB loaded |
+| **Download** | about 7.5 GiB |
+| **License** | Fastino Community License (see below). Built with Fastino GLiDE. |
+
+## How it was made
+
+1. **Swap the language model.** d3's package (code, tokenizer, the 255-code answer readout) with every
+   language-model weight replaced by Ternary Bonsai 2 27B's. Both are Qwen3.8-27B shapes. On its own this scores
+   41.9 on our sample: the readout does not fit Bonsai's weights yet.
+2. **Teach it.** One LoRA is trained in rounds to match a teacher's answer probabilities: first
+   [vllm-sr/d3](https://huggingface.co/vllm-sr/d3) on about 33,000 questions and then on the 36,559-question mix
+   below, and last on 12,000 of those with d3-ultra's and [Fastino GLiDE](https://huggingface.co/fastino/GLiDE)'s
+   answers combined. The questions come only from public training splits, checked against the Decision Index for
+   leaks. No Decision Index data was used for training.
+3. **Serve it ternary.** At run time the 2-bit codes stay packed on the GPU; a small Triton kernel unpacks each layer
+   just before its matmul. It scores the same as with the full 49 GB bf16 weights (60.23 vs 60.25 on our sample).
+
+## Results
+
+Wildcat Mini has **not** been run on the full Decision Index or submitted to its board. All numbers below are from
+**our own 2,000-row sample of the Decision Index public set**, the same rows for every model. This sample reads about
+2.7 points below the board's public score for d3-ultra, so compare rows with each other, not with the board.
+
+| Model (same 2,000 public rows) | GPU memory | Index |
+| --- | --- | --- |
+| d3-ultra (bf16) | about 54 GB | 68.84 |
+| Wildcat (3-bit d3-ultra) | 13.2 GiB | 69.39 |
+| **Wildcat Mini (this model)** | **7.4 GiB** | **60.23** |
+| Wildcat Mini, bf16 weights | about 49 GB | 60.25 |
+| Ternary Bonsai 2 27B + d3 readout, not trained | 7.4 GiB | 41.91 |
+
+**In short: about 9 points below d3-ultra on our sample, at a seventh of its memory.** If you have the memory for
+Wildcat, use Wildcat.
+
+**Speed.** Mini is slower than d3-ultra, because every layer is unpacked from ternary on the fly. On 750 sampled
+public requests, one at a time:
+
+| ms per request | median | p80 | mean |
+| --- | --- | --- | --- |
+| d3-ultra, RTX PRO 6000 | 168 | 1,067 | 932 |
+| Wildcat Mini, RTX PRO 6000 | 173 | 1,878 | 1,393 |
+| Wildcat Mini, RTX 3090 (2,000 sample rows) | 1,097 | 9,650 | 7,042 |
+
+## What was not tested
+
+- **Cards smaller than 24 GB.** The weights take 7.4 GiB, but long requests need more memory on top. It was run on an
+  RTX 3090 (24 GB) and an RTX PRO 6000 (96 GB) only.
+- **Images and video, Box and Track questions.** d3's runtime supports them; Mini was only measured on choice, yes / no
+  and score questions.
+- AMD GPUs. The unpack kernel is Triton and was only run on NVIDIA (CUDA).
+
+## Run it
+
+Needs an NVIDIA GPU, the packages in `requirements.txt`, plus `triton` and `accelerate`.
+
+```bash
+hf download BoldingBuilds/Wildcat-Mini --local-dir wildcat-mini
+```
+
+With the Decision Index kit:
+
+```bash
+PYTHONPATH=wildcat-mini python -m decision_index run --engine d3_tern:D3TernEngine --option model=wildcat-mini \
+  --rows <rows.jsonl.gz> --out <run dir>
+```
+
+The packed weights and the LoRA in `adapter/` load on their own. Nothing else needs passing.
+
+## Files
+
+| File | What it is |
+| --- | --- |
+| `mini-base.safetensors` | the packed ternary language weights (2-bit codes + one fp16 scale per 128 weights) plus the other backbone tensors |
+| `adapter/` | the LoRA (`adapter.safetensors`, `adapter.json`) |
+| `readout.safetensors`, tokenizer, configs, `d3_*.py` | from d3, unchanged |
+| `d3_tern.py` | the ternary engine and loader |
+| `d3_q3.py` | shared helpers from Wildcat's engine (memory limits for long requests) |
+
+## Training data
+
+The same public-train-split mix as Wildcat: intent and topic (CLINC150, BANKING77, DBpedia-14, LEDGAR); reading,
+inference and paraphrase (SQuAD v2, BoolQ, SNLI, WANLI, VitaminC, PAWS, STS-B, PubMedQA, claim_stance +
+Perspectrum); commonsense and science (HellaSwag, WinoGrande, PIQA, Social IQa, CommonsenseQA, Cosmos QA, ARC,
+OpenBookQA, QASC, RuleTaker); maths and knowledge (GSM8K, AQuA-RAT, MATH, MMLU train, MedMCQA); ratings and taste
+(Feedback Collection, HelpSteer2, WMT DA, GoEmotions, Civil Comments, New Yorker Caption Contest, iSarcasmEval);
+tools and search (When2Call, Amazon ESCI); and questions generated here (MIT). Several are train splits of
+benchmarks the Decision Index tests, so expect Mini to do relatively better there. Each source keeps its own
+licence; no non-commercial data.
+
+## License
+
+Wildcat Mini learned from Fastino GLiDE's answers, so it is a Derivative Work under the
+[Fastino Community License](LICENSES/FASTINO-LICENSE.txt) and ships under it. In short:
+
+- Free to use, change and share. Keep this license with any copy.
+- **Commercial use:** show "Built with Fastino GLiDE" in the product's docs, about page or UI.
+- **Companies with more than $10M yearly revenue** using it in production need a separate agreement with Fastino
+  (hello@fastino.ai). Evaluation, testing, benchmarking and research are not production.
+
+Ternary Bonsai 2's weights, and d3's readout and runtime, are Apache-2.0 (`LICENSE`, `NOTICE`).
+
+**Built with Fastino GLiDE.**
+
+## Credits
+
+- **Ternary Bonsai 2 27B**: PrismML ([prism-ml/Ternary-Bonsai-2-27B-gguf](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf)). Mini runs on their ternary weights.
+- **d3** and **d3-ultra**: the vLLM Semantic Router team ([vllm-sr/d3](https://huggingface.co/vllm-sr/d3), [vllm-sr/d3-ultra](https://huggingface.co/vllm-sr/d3-ultra)). d3 supplies the readout, tokenizer and runtime; both were teachers.
+- **Fastino GLiDE**: teacher ([fastino/GLiDE](https://huggingface.co/fastino/GLiDE)).
+- **Qwen** (Alibaba Cloud): the base model family.
+- The **Decision Index** (multimodalart) for the benchmark and kit.
+- The authors of every public dataset in the training mix (above).