Inkling-2-Small, EXL3 quants
These are EXL3 quantizations of thinkingmachines/Inkling-2-Small,
revision 8361e7a31ae7e9966a49a91d3994fde618936c2b. We made them with exllamav3.
Each bitrate is a branch. main holds this card, the license and the exllamav3 patch that loads the model.
The quants are text only. The vision tower and the MTP head are copied into each branch unchanged (bf16).
Branches
| Branch | Total bpw | Expert bpw | Trunk / head K | Size (GB) |
|---|---|---|---|---|
| source (reference) | β | NVFP4 / MXFP8 experts | bf16 | 393.5 |
2.99bpw |
2.987 | 2.958 | 6 | 228.4 |
1.99bpw |
1.987 | 1.958 | 5 | 154.3 |
1.75bpw |
1.746 | 1.724 | 4 | 136.4 |
1.02bpw |
1.019 | 1.000 | 3 | 82.5 |
- Total bpw counts every quantized tensor: routed experts, trunk and head. Expert bpw counts the routed experts only.
- Size is the whole branch, including the copied vision tower and MTP head (4.1 GB). 1 GB is 10^9 bytes.
Evaluation
exllamav3's own eval/model_diff.py at its defaults: wikitext-2 test, 100 rows of 2048 tokens. Model A is the source checkpoint (its NVFP4/MXFP8 experts dequantized), model B the quant; both run on exllamav3 with our inkling2 architecture code. -gp prepends the chat template's generation prompt to every row. The values are as model_diff printed them.
python eval/model_diff.py -ma <Inkling-2-Small> -mb <quant>
python eval/model_diff.py -ma <Inkling-2-Small> -mb <quant> -gp
python eval/ppl.py -m <quant>
| Branch | B perplexity | Top-1 agreement | KL (A, B) | KL (B, A) | B perplexity -gp |
Top-1 agreement -gp |
KL (A, B) -gp |
|---|---|---|---|---|---|---|---|
| source (model A) | 4.06381460 | β | β | β | 8.85000714 | β | β |
2.99bpw |
4.75948204 | 0.8672 | 0.25917166 | 0.16717809 | 10.98381872 | 0.8367 | 0.26225683 |
1.99bpw |
5.60385430 | 0.7978 | 0.54950179 | 0.33236999 | 13.01754256 | 0.7750 | 0.48791120 |
1.75bpw |
4.53966664 | 0.8685 | 0.15623883 | 0.12469767 | 10.09789441 | 0.8442 | 0.20987597 |
1.02bpw |
10.75714223 | 0.6312 | 1.79623947 | 0.99631458 | 19.89612224 | 0.6355 | 1.46618018 |
eval/ppl.py at its defaults loads the whole model onto one GPU. On our 96 GB GPU only 1.02bpw fits (perplexity 10.757142, the same as model_diff's B perplexity); the others and the source stop with "Insufficient VRAM in split for model and cache". The generation evals (humaneval.py, mmlu.py, ifbench.py, bbeh_mini.py, longctx.py) need exllamav3's Generator and paged cache, which the inkling2 code does not have, so they were not run.
Loader status
Stock exllamav3 cannot load these quants. It has no inkling2 architecture. We wrote one, and it is in
exllamav3-inkling2/ on main, as a patch series and a
git bundle. Apply it to exllamav3 dev at commit db26d5d80ee0058da4e615696b2e329dc82dc364.
The patch is enough for conversion, measurement and simple greedy generation. It runs the routed experts on exllamav3's dense per-expert path. Its attention is a plain implementation of the relative-logit bias and sinks. It has no paged-cache or Generator support, so it does not work in TabbyAPI or other exllamav3 servers. It is not part of upstream exllamav3.
The full model is larger than one 96 GB GPU. Our tests streamed the layers, or kept the trunk on the GPU and paged the routed experts in on demand.
How we made them
We used exllamav3's self-calibration pipeline:
sc_trace.pysampled a calibration trace from the original model on OpenRouter (thinkingmachines/inkling-2-small), pinned to the provider Together. The trace has 374 responses and 697,822 response tokens, packed into 250 rows of up to 2048 tokens.sc_measure.py --streaming --shapedmeasured how sensitive each layer's experts are. It ran once, on the source, and serves all branches.sc_optimize.py -mink 1 -maxk 4picked one K for each layer's routed experts, for the target bitrate.- The trunk (attention projections) and the head get K = the highest expert K + 2.
convert.py -rcp <recipe> -cd <trace> -cb mcg -mae 0.01quantized the model.-mae 0.01gives each expert its own Hessian. It cut the layer error by 27β34% in a test on the first layers.
Not quantized: token embedding, router, attention sinks, short convolutions, norms and attention-residual queries.
Bit allocation
Each layer's 384 routed experts share one K. The per-layer K maps are:
2.99bpw: 0β22 K4, 23β31 K3, 32β36 K2, 37 K1, 38 K2, 39β41 K1, 42 K2, 43β47 K1.1.99bpw: 0β12 K3, 13β14 K2, 15β17 K3, 18β29 K2, 30β47 K1.1.02bpw: 0β47 K1.1.75bpw: K per expert, not per layer. In some layers the least-routed experts (calibration routing counts) are K1 and the rest K2: 0: least 25% K1; 1β5: least 50%; 6β11: all K1; 12: K2; 13β14: least 25%; 15β17: K2; 18β29: least 25%; 30β36: K2; 37: least 25%; 38: K2; 39β41: least 25%; 42β47: K2. Its card says more.
Files
Each branch holds the EXL3 shards, config.json, the tokenizer files, the chat template, recipe.yaml (the
per-tensor K), vision.safetensors and mtp.safetensors (unchanged), a copy of the patch and its own card.
Licenses
The model weights are Apache-2.0 (Thinking Machines). The EXL3 format and exllamav3 are MIT.
Credits
Thanks to Thinking Machines for Inkling-2-Small and to turboderp for exllamav3 and EXL3.
Model tree for OpensourceWTF/Inkling-2-Small-exl3
Base model
thinkingmachines/Inkling-2-Small