BoldingBuilds
/

Wildcat

+---
+license: other
+license_name: fastino-community-license
+license_link: https://huggingface.co/BoldingBuilds/Wildcat/blob/main/LICENSES/FASTINO-LICENSE.txt
+base_model: vllm-sr/d3-ultra
+pipeline_tag: zero-shot-classification
+library_name: transformers
+tags:
+- decision-model
+- system-one
+- classification
+- 3-bit
+- hqq
+- lora
+---
+<!-- DRAFT for Josh's review. Not uploaded. -->
+# Wildcat
+**Wildcat is d3-ultra squeezed to 3 bits, then repaired.** It answers the same decision questions as
+[vllm-sr/d3-ultra](https://huggingface.co/vllm-sr/d3-ultra) (pick one option, yes / no, rate on a scale) and
+returns a probability for every answer. It loads in **about 13 GB of GPU memory** and runs on a single 24 GB card
+(tested on an RTX 3090). d3-ultra in bf16 needs about 54 GB.
+| | |
+| --- | --- |
+| **Base** | d3-ultra (27B, Qwen3.5 family), Apache-2.0 |
+| **Weights** | 400 text Linears in 3-bit HQQ (group 64), packed; everything else bf16 as in d3-ultra |
+| **Repair** | Two LoRA adapters (rank 32, two training seeds) averaged, applied on top of the 3-bit weights |
+| **GPU memory** | 13.19 GiB loaded; runs on 24 GB cards |
+| **Download** | 13.6 GiB |
+| **License** | Fastino Community License (see below). Built with Fastino GLiDE. |
+## How it was made
+1. **Squeeze.** d3-ultra's text-layer weights are quantized to 3 bits with HQQ (group size 64). The vision tower,
+   embeddings, norms and the small `in_proj_a` / `in_proj_b` projections stay in bf16. The 3-bit codes are stored
+   packed (a 2-bit plane plus a 1-bit plane); a small Triton kernel unpacks each layer just before its matmul, bit
+   for bit equal to HQQ's own dequantize.
+2. **Repair.** A LoRA is trained on top of the frozen 3-bit model to match the answer probabilities of three
+   teachers: d3-ultra, [Jeffan](https://huggingface.co/tattrongvu) and [Fastino GLiDE](https://huggingface.co/fastino/GLiDE).
+   The questions come only from public training splits of permissively licensed datasets, checked against the
+   Decision Index for leaks. No Decision Index data was used for training.
+3. **Blend.** Two runs with different seeds are averaged (each adapter at half strength).
+## Results
+**Decision Index 0.3 public index: 70.66** (complete run, all 140,620 requests, one RTX PRO 6000). d3-ultra's public
+index on the board is 71.57, so Wildcat is **0.9 points below the model it was made from**, at a quarter of the memory.
+The board's own ranking uses its private tests, which we have not seen.
+| Area (public index, skill) | Wildcat | d3-ultra (board) |
+| --- | --- | --- |
+| Knowledge & reasoning | 58.9 | 62.3 |
+| Language understanding | 73.7 | 72.4 |
+| Retrieval & classification | 71.9 | 71.8 |
+| Tools & automation | 87.2 | 87.8 |
+| Arts & human taste | 60.7 | 63.5 |
+| **Index** | **70.66** | **71.57** |
+Where the 3-bit squeeze shows: multi-step reasoning and hard knowledge (MuSR, GPQA Diamond, CRUXEval, MMLU-Pro,
+GSM8K are 3 to 10 points lower). The repair LoRA gains on language tasks (iSarcasmEval +8.7, FinEntity +2.7).
+Speed over the full public run on one RTX PRO 6000 (in-process, one request at a time): median 157 ms, p80 351 ms
+per request. The mean is 1,258 ms because about 1,000 very long ToolRet requests take minutes each; without ToolRet
+the mean is 438 ms. On 750 sampled rows timed the same way, Wildcat and d3-ultra were level (mean 818 vs 932 ms).
+On cards under 40 GB the engine answers one question at a time (batch size 1) so that requests with many long
+questions fit; on cards over 72 GB it unpacks to bf16 and merges the LoRA for speed. Very long requests can fall
+back to a slower path that still answers them.
+## What was not tested
+- **Box and Track** (image and video locations). d3-ultra supports them; Wildcat was only measured on choice,
+  yes / no and score questions. Expect them to be weaker than d3-ultra's.
+- AMD GPUs. The packed kernel is Triton and was only run on NVIDIA (CUDA).
+## Run it
+Needs an NVIDIA GPU with 24 GB or more, and the packages in `requirements.txt` plus `triton` and `accelerate`.
+```bash
+hf download BoldingBuilds/Wildcat --local-dir wildcat
+```
+With the Decision Index kit (this is how the board runs it):
+```bash
+PYTHONPATH=wildcat python -m decision_index run --engine d3_q3:D3Q3Engine --option model=wildcat \
+  --rows <rows.jsonl.gz> --out <run dir>
+```
+The LoRA in `lora.safetensors` attaches on load. Nothing else needs passing.
+<!-- TODO: a plain Python example (D3.from_pretrained with d3_q3.install()) once checked. -->
+## Files
+| File | What it is |
+| --- | --- |
+| `q3.safetensors` | the packed 3-bit Linears (`<module>.lo / .hi / .scale / .zero`) |
+| `model-rest.safetensors` | every other backbone tensor, bf16 |
+| `lora.safetensors` | the repair LoRA, both seeds stacked and scaled (`<module>.A / .B`) |
+| `readout.safetensors`, tokenizer, configs, `d3_*.py` | from d3-ultra, unchanged |
+| `d3_q3.py` | the 3-bit engine and loader |
+| `MODEL_MANIFEST.json` | SHA-256 of every file; checked on load |
+## License
+Wildcat learned from Fastino GLiDE's answers, so it is a Derivative Work under the
+[Fastino Community License](LICENSES/FASTINO-LICENSE.txt) and ships under it. In short:
+- Free to use, change and share. Keep this license with any copy.
+- **Commercial use:** show "Built with Fastino GLiDE" in the product's docs, about page or UI.
+- **Companies with more than $10M yearly revenue** using it in production need a separate agreement with Fastino
+  (hello@fastino.ai). Evaluation, testing, benchmarking and research are not production.
+d3-ultra's weights, readout and runtime are Apache-2.0 (`LICENSE`, `NOTICE`). Qwen's Apache-2.0 license is in
+`LICENSES/Qwen3.8-27B-LICENSE`.
+**Built with Fastino GLiDE.**
+## Credits
+- **d3-ultra** and its runtime: the vLLM Semantic Router team ([vllm-sr/d3-ultra](https://huggingface.co/vllm-sr/d3-ultra)). Wildcat is their model, made smaller.
+- **Fastino GLiDE**: teacher ([fastino/GLiDE](https://huggingface.co/fastino/GLiDE)).
+- **Jeffan** (tattrongvu): teacher.
+- **Qwen** (Alibaba Cloud): the base model family.
+- **HQQ** (Mobius Labs): the 3-bit quantizer.
+- The **Decision Index** (multimodalart) for the benchmark and kit.
+- The authors of every public dataset in the repair mix (below).
+## Repair data
+36,559 questions, all from **train splits** of public datasets (or generated here), each checked against the
+Decision Index public rows for overlapping text; overlapping rows were dropped. Several sources are the train
+splits of benchmarks the Decision Index tests (for example BANKING77, CLINC150, GSM8K, WinoGrande, HellaSwag,
+When2Call, Amazon ESCI, iSarcasmEval, New Yorker captions), so expect Wildcat to do relatively better there.
+- Intent and topic: CLINC150, BANKING77, DBpedia-14, LEDGAR
+- Reading, inference and paraphrase: SQuAD v2, BoolQ, SNLI, WANLI, VitaminC, PAWS, STS-B, PubMedQA, claim_stance + Perspectrum
+- Commonsense and science: HellaSwag, WinoGrande, PIQA, Social IQa, CommonsenseQA, Cosmos QA, ARC, OpenBookQA, QASC, RuleTaker
+- Maths and knowledge: GSM8K, AQuA-RAT, MATH (hendrycks), MMLU (train), MedMCQA
+- Ratings and taste: Feedback Collection, HelpSteer2, WMT DA, GoEmotions, Civil Comments, New Yorker Caption Contest, iSarcasmEval
+- Tools and search: When2Call, Amazon ESCI
+- Generated here (MIT): chord, CRUX-style code tracing, BBH-style, CLadder-style and SQuAD-based hallucination questions
+Each source keeps its own licence (CC-BY, CC-BY-SA, Apache-2.0, MIT, CC0, AFL-3.0); no non-commercial data.