| --- |
| license: other |
| license_name: fastino-community-license |
| license_link: https://huggingface.co/BoldingBuilds/Wildcat/blob/main/LICENSES/FASTINO-LICENSE.txt |
| base_model: vllm-sr/d3-ultra |
| pipeline_tag: zero-shot-classification |
| library_name: transformers |
| tags: |
| - decision-model |
| - system-one |
| - classification |
| - 3-bit |
| - hqq |
| - lora |
| --- |
| |
| # Wildcat |
|
|
| **Wildcat is d3-ultra squeezed to 3 bits, then repaired.** It answers the same decision questions as |
| [vllm-sr/d3-ultra](https://huggingface.co/vllm-sr/d3-ultra) (pick one option, yes / no, rate on a scale) and |
| returns a probability for every answer. It loads in **about 13 GB of GPU memory** and runs on a single 24 GB card |
| (tested on an RTX 3090). d3-ultra in bf16 needs about 54 GB. |
|
|
| | | | |
| | --- | --- | |
| | **Base** | d3-ultra (27B, Qwen3.5 family), Apache-2.0 | |
| | **Weights** | 400 text Linears in 3-bit HQQ (group 64), packed; everything else bf16 as in d3-ultra | |
| | **Repair** | Two LoRA adapters (rank 32, two training seeds) averaged, applied on top of the 3-bit weights | |
| | **GPU memory** | 13.19 GiB loaded; runs on 24 GB cards | |
| | **Download** | 13.6 GiB | |
| | **License** | Fastino Community License (see below). Built with Fastino GLiDE. | |
|
|
| ## How it was made |
|
|
| 1. **Squeeze.** d3-ultra's text-layer weights are quantized to 3 bits with HQQ (group size 64). The vision tower, |
| embeddings, norms and the small `in_proj_a` / `in_proj_b` projections stay in bf16. The 3-bit codes are stored |
| packed (a 2-bit plane plus a 1-bit plane); a small Triton kernel unpacks each layer just before its matmul, bit |
| for bit equal to HQQ's own dequantize. |
| 2. **Repair.** A LoRA is trained on top of the frozen 3-bit model to match the answer probabilities of three |
| teachers: d3-ultra, [Jeffan](https://huggingface.co/tattrongvu) and [Fastino GLiDE](https://huggingface.co/fastino/GLiDE). |
| The questions come only from public training splits of permissively licensed datasets, checked against the |
| Decision Index for leaks. No Decision Index data was used for training. |
| 3. **Blend.** Two runs with different seeds are averaged (each adapter at half strength). |
|
|
| ## Results |
|
|
| **Decision Index 0.3 public index: 70.66** (complete run, all 140,620 requests, one RTX PRO 6000). d3-ultra's public |
| index on the board is 71.57, so Wildcat is **0.9 points below the model it was made from**, at a quarter of the memory. |
| The board's own ranking uses its private tests, which we have not seen. |
|
|
| | Area (public index, skill) | Wildcat | d3-ultra (board) | |
| | --- | --- | --- | |
| | Knowledge & reasoning | 58.9 | 62.3 | |
| | Language understanding | 73.7 | 72.4 | |
| | Retrieval & classification | 71.9 | 71.8 | |
| | Tools & automation | 87.2 | 87.8 | |
| | Arts & human taste | 60.7 | 63.5 | |
| | **Index** | **70.66** | **71.57** | |
|
|
| Where the 3-bit squeeze shows: multi-step reasoning and hard knowledge (MuSR, GPQA Diamond, CRUXEval, MMLU-Pro, |
| GSM8K are 3 to 10 points lower). The repair LoRA gains on language tasks (iSarcasmEval +8.7, FinEntity +2.7). |
|
|
| Speed over the full public run on one RTX PRO 6000 (in-process, one request at a time): median 157 ms, p80 351 ms |
| per request. The mean is 1,258 ms because about 1,000 very long ToolRet requests take minutes each; without ToolRet |
| the mean is 438 ms. On 750 sampled rows timed the same way, Wildcat and d3-ultra were level (mean 818 vs 932 ms). |
|
|
| On cards under 40 GB the engine answers one question at a time (batch size 1) so that requests with many long |
| questions fit; on cards over 72 GB it unpacks to bf16 and merges the LoRA for speed. Very long requests can fall |
| back to a slower path that still answers them. |
|
|
| ## What was not tested |
|
|
| - **Box and Track** (image and video locations). d3-ultra supports them; Wildcat was only measured on choice, |
| yes / no and score questions. Expect them to be weaker than d3-ultra's. |
| - AMD GPUs. The packed kernel is Triton and was only run on NVIDIA (CUDA). |
|
|
| ## Run it |
|
|
| Needs an NVIDIA GPU with 24 GB or more, and the packages in `requirements.txt` plus `triton` and `accelerate`. |
|
|
| ```bash |
| hf download BoldingBuilds/Wildcat --local-dir wildcat |
| ``` |
|
|
| With the Decision Index kit (this is how the board runs it): |
|
|
| ```bash |
| PYTHONPATH=wildcat python -m decision_index run --engine d3_q3:D3Q3Engine --option model=wildcat \ |
| --rows <rows.jsonl.gz> --out <run dir> |
| ``` |
|
|
| The LoRA in `lora.safetensors` attaches on load. Nothing else needs passing. |
|
|
| ## Files |
|
|
| | File | What it is | |
| | --- | --- | |
| | `q3.safetensors` | the packed 3-bit Linears (`<module>.lo / .hi / .scale / .zero`) | |
| | `model-rest.safetensors` | every other backbone tensor, bf16 | |
| | `lora.safetensors` | the repair LoRA, both seeds stacked and scaled (`<module>.A / .B`) | |
| | `readout.safetensors`, tokenizer, configs, `d3_*.py` | from d3-ultra, unchanged | |
| | `d3_q3.py` | the 3-bit engine and loader | |
| | `MODEL_MANIFEST.json` | SHA-256 of every file; checked on load | |
|
|
| ## License |
|
|
| Wildcat learned from Fastino GLiDE's answers, so it is a Derivative Work under the |
| [Fastino Community License](LICENSES/FASTINO-LICENSE.txt) and ships under it. In short: |
|
|
| - Free to use, change and share. Keep this license with any copy. |
| - **Commercial use:** show "Built with Fastino GLiDE" in the product's docs, about page or UI. |
| - **Companies with more than $10M yearly revenue** using it in production need a separate agreement with Fastino |
| (hello@fastino.ai). Evaluation, testing, benchmarking and research are not production. |
|
|
| d3-ultra's weights, readout and runtime are Apache-2.0 (`LICENSE`, `NOTICE`). Qwen's Apache-2.0 license is in |
| `LICENSES/Qwen3.8-27B-LICENSE`. |
|
|
| **Built with Fastino GLiDE.** |
|
|
| ## Credits |
|
|
| - **d3-ultra** and its runtime: the vLLM Semantic Router team ([vllm-sr/d3-ultra](https://huggingface.co/vllm-sr/d3-ultra)). Wildcat is their model, made smaller. |
| - **Fastino GLiDE**: teacher ([fastino/GLiDE](https://huggingface.co/fastino/GLiDE)). |
| - **Jeffan** (tattrongvu): teacher. |
| - **Qwen** (Alibaba Cloud): the base model family. |
| - **HQQ** (Mobius Labs): the 3-bit quantizer. |
| - The **Decision Index** (multimodalart) for the benchmark and kit. |
| - The authors of every public dataset in the repair mix (below). |
|
|
| ## Repair data |
|
|
| 36,559 questions, all from **train splits** of public datasets (or generated here), each checked against the |
| Decision Index public rows for overlapping text; overlapping rows were dropped. Several sources are the train |
| splits of benchmarks the Decision Index tests (for example BANKING77, CLINC150, GSM8K, WinoGrande, HellaSwag, |
| When2Call, Amazon ESCI, iSarcasmEval, New Yorker captions), so expect Wildcat to do relatively better there. |
|
|
| - Intent and topic: CLINC150, BANKING77, DBpedia-14, LEDGAR |
| - Reading, inference and paraphrase: SQuAD v2, BoolQ, SNLI, WANLI, VitaminC, PAWS, STS-B, PubMedQA, claim_stance + Perspectrum |
| - Commonsense and science: HellaSwag, WinoGrande, PIQA, Social IQa, CommonsenseQA, Cosmos QA, ARC, OpenBookQA, QASC, RuleTaker |
| - Maths and knowledge: GSM8K, AQuA-RAT, MATH (hendrycks), MMLU (train), MedMCQA |
| - Ratings and taste: Feedback Collection, HelpSteer2, WMT DA, GoEmotions, Civil Comments, New Yorker Caption Contest, iSarcasmEval |
| - Tools and search: When2Call, Amazon ESCI |
| - Generated here (MIT): chord, CRUX-style code tracing, BBH-style, CLadder-style and SQuAD-based hallucination questions |
| |
| Each source keeps its own licence (CC-BY, CC-BY-SA, Apache-2.0, MIT, CC0, AFL-3.0); no non-commercial data. |
| |
