sabrewing-engine/Inkling-Small-colibri-int4

Inkling-Small — colibri int4

Thinking Machines Inkling-Small converted to the colibri/sabrewing snapshot format: int4 routed experts with bf16 residents. 131 GiB, down from 495 GiB in bf16.

This is not a transformers checkpoint. The tensors are packed for sabrewing, a dependency-free C engine for MoE inference. AutoModel.from_pretrained will not load it. If you want the original weights, use the base model.

Why this exists

Inkling-Small is ~266B total parameters. In bf16 that is 495 GiB — more than fits in a workstation. At int4 the routed experts (which are ~95% of the weights) drop to 131 GiB total, which fits in RAM on a 187 GB machine, so decode never touches the NVMe expert-streaming path that the 975B Inkling depends on.

Quickstart

hf download sabrewing-engine/Inkling-Small-colibri-int4 --local-dir ~/Models/inkling_small_i4

git clone git@github.com:Schneewolf-Labs/sabrewing.git && cd sabrewing/c
make inkling ARCH=native                  # CPU; add CUDA=1 for the GPU resident tier

SNAP=~/Models/inkling_small_i4 ./inkling -p "The capital of France is" -n 64

Put the snapshot on NVMe, not spinning disk. The engine reads expert weights from the snapshot on cache misses; on an HDD that dominates everything else.

What is in the container

Tensor class	dtype	Notes
Routed experts (256/layer, top-6)	int4 + f32 row scales	~95% of parameters
Attention projections, shared experts, dense MLP, embed, lm_head	bf16	converted on load
Norms, short-conv weights, rel-bias banks, router (+bias/scales)	f32
model.audio.*, model.visual.*, model.mtp.*	—	skipped: text-only

Quantization is per-row symmetric, low nibble = even column, np.rint matching the C engine's lrintf. Identical math to the other colibri int4 containers.

The base model is multimodal (image-text-to-text); this conversion is text-only. The vision and audio towers are not included, and neither is the MTP draft head.

Architecture

Inkling-Small is a straight scale-down of Inkling — same MoE structure, smaller dimensions. No engine changes were needed to run it.

	Inkling-Small	Inkling
hidden size	4096	6144
layers	42	66
attention heads	32	64
expert intermediate	2048	3072
routed experts	256, top-6	256, top-6
shared experts	2	2
vocab	201,024	201,024
total params	~266B	~975B

Measured performance

Ryzen 9 7900 (12c/24t), 187 GB DDR5, snapshot on NVMe, CPU build, ARCH=native. 24-token greedy decode, single runs.

Condition	Cache hit	Prefill	Decode
Warm, pins trained on the prompt	100%	1.1 s	3.0 tok/s
Warm, novel prompt	82%	15.5 s	0.86 tok/s

At 100% cache hit there is zero disk I/O (fill 0.0s), so the warm number is pure compute. The novel-prompt number is dominated by loading the ~18% of experts the pins missed.

If you get ~0.3 tok/s, check your thread count

OpenMP defaults its team size to nproc. On an SMT machine that runs two threads per physical core, both contending for the same vector units, and the int4 kernels fall off a cliff — measured ~10x on this model and ~28x on Laguna:

OMP_NUM_THREADS	Decode
24 (logical CPUs)	0.26–0.29 tok/s
12 (physical cores)	2.74–3.02 tok/s

sabrewing sets one thread per physical core automatically as of PR #10. On older builds, set OMP_NUM_THREADS to your physical core count by hand.

Cache warming

The engine counts expert selections per (layer, expert) and writes .coli_usage into the snapshot directory, pinning the top experts on the next start. Counts accumulate across runs, so warm it on a diverse prompt set — pins trained on one prompt overfit badly. That file is deliberately not shipped here; it is generated locally from your own workload.

SNAP=~/Models/inkling_small_i4 ./inkling -f warmup_prompts.txt -n 32

Reproducing the conversion

hf download thinkingmachines/Inkling-Small --local-dir <bf16-dir>
python3 c/tools/convert_inkling_int4.py --indir <bf16-dir> --outdir <out-dir>

--watch converts shards while the download is still running. That is a win when the source is on NVMe; when source and output share one spinning disk it is a ~3x pessimization (measured 197 MB/s serialized vs 60 MB/s overlapped) because the head thrashes between the download's writes and the converter's reads. Serialize on HDD.

Caveats

• Not validated token-exact. sabrewing's usual bar is bit-for-bit agreement with transformers via a tiny-model oracle. The oracle builder needs transformers >= 5.7, which was not available on the conversion machine, so this container has not been checked against the reference implementation. Kernel correctness was verified separately (make kernel-check passes: int4/int8 kernels against a double-precision dequant reference), and generation is coherent — but that is weaker than the usual guarantee.
• CPU only, so far. The CUDA resident tier was not exercised; the test machine's GPU was occupied. Expect it to help — at 100% cache hit this model is compute-bound, which is what a GPU fixes.
• Text-only; no vision, audio, or MTP.

License

Apache 2.0, inherited from the base model.