Instructions to use tbmod/Inkling-mini-bf16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tbmod/Inkling-mini-bf16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="tbmod/Inkling-mini-bf16")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("tbmod/Inkling-mini-bf16")
model = AutoModelForMultimodalLM.from_pretrained("tbmod/Inkling-mini-bf16", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tbmod/Inkling-mini-bf16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tbmod/Inkling-mini-bf16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tbmod/Inkling-mini-bf16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/tbmod/Inkling-mini-bf16

SGLang

How to use tbmod/Inkling-mini-bf16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tbmod/Inkling-mini-bf16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tbmod/Inkling-mini-bf16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tbmod/Inkling-mini-bf16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tbmod/Inkling-mini-bf16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use tbmod/Inkling-mini-bf16 with Docker Model Runner:
```
docker model run hf.co/tbmod/Inkling-mini-bf16
```

Inkling-mini-bf16

File size: 4,180 Bytes

903c060

---
library_name: transformers
license: apache-2.0
base_model: thinkingmachines/Inkling
tags:
- moe
- testing
- minimized
- not-for-production
---

# Inkling-mini-bf16

> **This is a broken test fixture, not a usable model.**
>
> It is a deliberately crippled cut-down of
> [`thinkingmachines/Inkling`](https://huggingface.co/thinkingmachines/Inkling), built only to exercise
> inference-engine plumbing. Most of the network has been deleted:
> 12 of 66 layers and 16 of 256
> experts remain. **It produces gibberish. Do not use it for generation, do
> not evaluate it, and do not treat any output as representative of Inkling.**
> If you want the real model, use
> [`thinkingmachines/Inkling`](https://huggingface.co/thinkingmachines/Inkling).

## Attribution

All weights here are the work of **Thinking Machines Lab**, sliced without
modification from [`thinkingmachines/Inkling`](https://huggingface.co/thinkingmachines/Inkling)
(revision `main`) and redistributed under that model's Apache-2.0 licence.

To be precise about what "minimized" means: the retained tensors are the
**real trained values, bit-for-bit** — nothing is randomly initialized and no
weight is altered. The reduction is purely deletion, keeping a prefix of the
layers and the first 16 experts of each MoE layer. That
is what makes the fixture useful for shape and dataflow testing, and it is
also why upstream attribution and licensing apply in full.

No claim is made about this artifact's quality, and no endorsement by Thinking
Machines Lab is implied.

## What was reduced

| | upstream | here |
|---|---|---|
| text layers | 66 | 12 |
| global attention layers | 11 | 2 (at 5, 11) |
| dense MLP layers | 0-1 | 0-1 |
| routed experts | 256 | 16 |
| experts per token | 6 | 6 |
| shared expert sinks | 2 | 2 |
| MTP layers | 8 | 2 |
| hidden size / heads / head_dim | 6144 / 64 / 128 | unchanged |
| vocab | 201024 (200058 unpadded) | unchanged |

30.79 GiB, 298 tensors, 9 shards.

The retained layers are a **prefix** of the upstream schedule, so layer
indices, `local_layer_ids` and `dense_mlp_idx` carry over unchanged and the
real 5:1 sliding-window / full-attention alternation is preserved rather than
hand-written. Hidden size, head counts and `head_dim` are untouched so kernel
shapes match production. The vision and audio towers are complete.

## Why the output is gibberish

This is expected and by design, not a bug to report. At
12 of 66 layers and 16 of 256
experts the output distribution is broad and near-uniform. Greedy decoding
collapses onto a single special token; temperature 1.0 returns unrelated
vocabulary tokens. Logprobs are finite and greedy decoding is deterministic,
which is all the fixture needs to be.

This fixture validates *plumbing* — tensor shapes, the four short-convolution
sites per layer, per-layer-type KV head counts and relative-bias widths,
gate-plus-sink routing, embedding norm, muP logit scaling, the sampler — and it
is a suitable input for a logit-level comparison against a reference
implementation. It does not validate correctness on its own, and it should not
be evaluated for quality.

## Serving

Verified with vLLM 0.26.0 on a single B200:

```bash
export VLLM_USE_V2_MODEL_RUNNER=1
export MAX_JOBS=16                # FlashInfer's ninja JIT otherwise fans out
                                  # to the host CPU count and can OOM the box
vllm serve Inkling-mini-bf16 \
  --tokenizer-mode inkling \
  --tensor-parallel-size 1 \
  --kernel-config.enable_flashinfer_autotune=False \
  --max-model-len 4096 \
  --max-num-seqs 16
```

`--max-num-seqs` matters: vLLM allocates `max_num_seqs * max_model_len * 4 B`
of pinned host memory, and the default can stall `cudaHostAlloc` for minutes.
`scipy` is an undeclared runtime dependency of the vision tower. At TP=1 the
fused Lamport collective logs a `ValueError: TP world size must be 2, 4, or 8`
traceback at ERROR level; it is a benign fallback to NCCL, not a failure.

## Licence

Apache-2.0, inherited from [`thinkingmachines/Inkling`](https://huggingface.co/thinkingmachines/Inkling).
The weights are Thinking Machines Lab's; this repository only removes layers
and experts from them.