Inkling-2-Small, EXL3 quants

These are EXL3 quantizations of thinkingmachines/Inkling-2-Small, revision 8361e7a31ae7e9966a49a91d3994fde618936c2b. We made them with exllamav3. Each bitrate is a branch. main holds this card, the license and the exllamav3 patch that loads the model.

The quants are text only. The vision tower and the MTP head are copied into each branch unchanged (bf16).

Branches

Branch Total bpw Expert bpw Trunk / head K Size (GB)
source (reference) β€” NVFP4 / MXFP8 experts bf16 393.5
2.99bpw 2.987 2.958 6 228.4
1.99bpw 1.987 1.958 5 154.3
1.75bpw 1.746 1.724 4 136.4
1.02bpw 1.019 1.000 3 82.5
  • Total bpw counts every quantized tensor: routed experts, trunk and head. Expert bpw counts the routed experts only.
  • Size is the whole branch, including the copied vision tower and MTP head (4.1 GB). 1 GB is 10^9 bytes.

Evaluation

exllamav3's own eval/model_diff.py at its defaults: wikitext-2 test, 100 rows of 2048 tokens. Model A is the source checkpoint (its NVFP4/MXFP8 experts dequantized), model B the quant; both run on exllamav3 with our inkling2 architecture code. -gp prepends the chat template's generation prompt to every row. The values are as model_diff printed them.

python eval/model_diff.py -ma <Inkling-2-Small> -mb <quant>
python eval/model_diff.py -ma <Inkling-2-Small> -mb <quant> -gp
python eval/ppl.py -m <quant>
Branch B perplexity Top-1 agreement KL (A, B) KL (B, A) B perplexity -gp Top-1 agreement -gp KL (A, B) -gp
source (model A) 4.06381460 β€” β€” β€” 8.85000714 β€” β€”
2.99bpw 4.75948204 0.8672 0.25917166 0.16717809 10.98381872 0.8367 0.26225683
1.99bpw 5.60385430 0.7978 0.54950179 0.33236999 13.01754256 0.7750 0.48791120
1.75bpw 4.53966664 0.8685 0.15623883 0.12469767 10.09789441 0.8442 0.20987597
1.02bpw 10.75714223 0.6312 1.79623947 0.99631458 19.89612224 0.6355 1.46618018

eval/ppl.py at its defaults loads the whole model onto one GPU. On our 96 GB GPU only 1.02bpw fits (perplexity 10.757142, the same as model_diff's B perplexity); the others and the source stop with "Insufficient VRAM in split for model and cache". The generation evals (humaneval.py, mmlu.py, ifbench.py, bbeh_mini.py, longctx.py) need exllamav3's Generator and paged cache, which the inkling2 code does not have, so they were not run.

Loader status

Stock exllamav3 cannot load these quants. It has no inkling2 architecture. We wrote one, and it is in exllamav3-inkling2/ on main, as a patch series and a git bundle. Apply it to exllamav3 dev at commit db26d5d80ee0058da4e615696b2e329dc82dc364.

The patch is enough for conversion, measurement and simple greedy generation. It runs the routed experts on exllamav3's dense per-expert path. Its attention is a plain implementation of the relative-logit bias and sinks. It has no paged-cache or Generator support, so it does not work in TabbyAPI or other exllamav3 servers. It is not part of upstream exllamav3.

The full model is larger than one 96 GB GPU. Our tests streamed the layers, or kept the trunk on the GPU and paged the routed experts in on demand.

How we made them

We used exllamav3's self-calibration pipeline:

  1. sc_trace.py sampled a calibration trace from the original model on OpenRouter (thinkingmachines/inkling-2-small), pinned to the provider Together. The trace has 374 responses and 697,822 response tokens, packed into 250 rows of up to 2048 tokens.
  2. sc_measure.py --streaming --shaped measured how sensitive each layer's experts are. It ran once, on the source, and serves all branches.
  3. sc_optimize.py -mink 1 -maxk 4 picked one K for each layer's routed experts, for the target bitrate.
  4. The trunk (attention projections) and the head get K = the highest expert K + 2.
  5. convert.py -rcp <recipe> -cd <trace> -cb mcg -mae 0.01 quantized the model. -mae 0.01 gives each expert its own Hessian. It cut the layer error by 27–34% in a test on the first layers.

Not quantized: token embedding, router, attention sinks, short convolutions, norms and attention-residual queries.

Bit allocation

Each layer's 384 routed experts share one K. The per-layer K maps are:

  • 2.99bpw: 0–22 K4, 23–31 K3, 32–36 K2, 37 K1, 38 K2, 39–41 K1, 42 K2, 43–47 K1.
  • 1.99bpw: 0–12 K3, 13–14 K2, 15–17 K3, 18–29 K2, 30–47 K1.
  • 1.02bpw: 0–47 K1.
  • 1.75bpw: K per expert, not per layer. In some layers the least-routed experts (calibration routing counts) are K1 and the rest K2: 0: least 25% K1; 1–5: least 50%; 6–11: all K1; 12: K2; 13–14: least 25%; 15–17: K2; 18–29: least 25%; 30–36: K2; 37: least 25%; 38: K2; 39–41: least 25%; 42–47: K2. Its card says more.

Files

Each branch holds the EXL3 shards, config.json, the tokenizer files, the chat template, recipe.yaml (the per-tensor K), vision.safetensors and mtp.safetensors (unchanged), a copy of the patch and its own card.

Licenses

The model weights are Apache-2.0 (Thinking Machines). The EXL3 format and exllamav3 are MIT.

Credits

Thanks to Thinking Machines for Inkling-2-Small and to turboderp for exllamav3 and EXL3.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for OpensourceWTF/Inkling-2-Small-exl3

Quantized
(21)
this model