---
library_name: transformers
license: apache-2.0
base_model: thinkingmachines/Inkling
tags:
- moe
- testing
- minimized
- not-for-production
---

# Inkling-mini-bf16

> **This is a broken test fixture, not a usable model.**
>
> It is a deliberately crippled cut-down of
> [`thinkingmachines/Inkling`](https://huggingface.co/thinkingmachines/Inkling), built only to exercise
> inference-engine plumbing. Most of the network has been deleted:
> 12 of 66 layers and 16 of 256
> experts remain. **It produces gibberish. Do not use it for generation, do
> not evaluate it, and do not treat any output as representative of Inkling.**
> If you want the real model, use
> [`thinkingmachines/Inkling`](https://huggingface.co/thinkingmachines/Inkling).

## Attribution

All weights here are the work of **Thinking Machines Lab**, sliced without
modification from [`thinkingmachines/Inkling`](https://huggingface.co/thinkingmachines/Inkling)
(revision `main`) and redistributed under that model's Apache-2.0 licence.

To be precise about what "minimized" means: the retained tensors are the
**real trained values, bit-for-bit** — nothing is randomly initialized and no
weight is altered. The reduction is purely deletion, keeping a prefix of the
layers and the first 16 experts of each MoE layer. That
is what makes the fixture useful for shape and dataflow testing, and it is
also why upstream attribution and licensing apply in full.

No claim is made about this artifact's quality, and no endorsement by Thinking
Machines Lab is implied.

## What was reduced

| | upstream | here |
|---|---|---|
| text layers | 66 | 12 |
| global attention layers | 11 | 2 (at 5, 11) |
| dense MLP layers | 0-1 | 0-1 |
| routed experts | 256 | 16 |
| experts per token | 6 | 6 |
| shared expert sinks | 2 | 2 |
| MTP layers | 8 | 2 |
| hidden size / heads / head_dim | 6144 / 64 / 128 | unchanged |
| vocab | 201024 (200058 unpadded) | unchanged |

30.79 GiB, 298 tensors, 9 shards.

The retained layers are a **prefix** of the upstream schedule, so layer
indices, `local_layer_ids` and `dense_mlp_idx` carry over unchanged and the
real 5:1 sliding-window / full-attention alternation is preserved rather than
hand-written. Hidden size, head counts and `head_dim` are untouched so kernel
shapes match production. The vision and audio towers are complete.

## Why the output is gibberish

This is expected and by design, not a bug to report. At
12 of 66 layers and 16 of 256
experts the output distribution is broad and near-uniform. Greedy decoding
collapses onto a single special token; temperature 1.0 returns unrelated
vocabulary tokens. Logprobs are finite and greedy decoding is deterministic,
which is all the fixture needs to be.

This fixture validates *plumbing* — tensor shapes, the four short-convolution
sites per layer, per-layer-type KV head counts and relative-bias widths,
gate-plus-sink routing, embedding norm, muP logit scaling, the sampler — and it
is a suitable input for a logit-level comparison against a reference
implementation. It does not validate correctness on its own, and it should not
be evaluated for quality.

## Serving

Verified with vLLM 0.26.0 on a single B200:

```bash
export VLLM_USE_V2_MODEL_RUNNER=1
export MAX_JOBS=16                # FlashInfer's ninja JIT otherwise fans out
                                  # to the host CPU count and can OOM the box
vllm serve Inkling-mini-bf16 \
  --tokenizer-mode inkling \
  --tensor-parallel-size 1 \
  --kernel-config.enable_flashinfer_autotune=False \
  --max-model-len 4096 \
  --max-num-seqs 16
```

`--max-num-seqs` matters: vLLM allocates `max_num_seqs * max_model_len * 4 B`
of pinned host memory, and the default can stall `cudaHostAlloc` for minutes.
`scipy` is an undeclared runtime dependency of the vision tower. At TP=1 the
fused Lamport collective logs a `ValueError: TP world size must be 2, 4, or 8`
traceback at ERROR level; it is a benign fallback to NCCL, not a failure.

## Licence

Apache-2.0, inherited from [`thinkingmachines/Inkling`](https://huggingface.co/thinkingmachines/Inkling).
The weights are Thinking Machines Lab's; this repository only removes layers
and experts from them.