Instructions to use tbmod/Inkling-mini-bf16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tbmod/Inkling-mini-bf16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="tbmod/Inkling-mini-bf16")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("tbmod/Inkling-mini-bf16")
model = AutoModelForMultimodalLM.from_pretrained("tbmod/Inkling-mini-bf16", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tbmod/Inkling-mini-bf16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tbmod/Inkling-mini-bf16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tbmod/Inkling-mini-bf16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/tbmod/Inkling-mini-bf16

SGLang

How to use tbmod/Inkling-mini-bf16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tbmod/Inkling-mini-bf16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tbmod/Inkling-mini-bf16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tbmod/Inkling-mini-bf16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tbmod/Inkling-mini-bf16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use tbmod/Inkling-mini-bf16 with Docker Model Runner:
```
docker model run hf.co/tbmod/Inkling-mini-bf16
```

Inkling-mini-bf16 / README.md

tbmod

Add minimized Inkling bf16 fixture (12 layers, 16 experts)

903c060 verified 2 days ago

preview code

Raw

History Blame Contribute Delete

4.18 kB

	---
	library_name: transformers
	license: apache-2.0
	base_model: thinkingmachines/Inkling
	tags:
	- moe
	- testing
	- minimized
	- not-for-production
	---

	# Inkling-mini-bf16

	> This is a broken test fixture, not a usable model.
	>
	> It is a deliberately crippled cut-down of
	> [`thinkingmachines/Inkling`](https://huggingface.co/thinkingmachines/Inkling), built only to exercise
	> inference-engine plumbing. Most of the network has been deleted:
	> 12 of 66 layers and 16 of 256
	> experts remain. **It produces gibberish. Do not use it for generation, do
	> not evaluate it, and do not treat any output as representative of Inkling.**
	> If you want the real model, use
	> [`thinkingmachines/Inkling`](https://huggingface.co/thinkingmachines/Inkling).

	## Attribution

	All weights here are the work of Thinking Machines Lab, sliced without
	modification from [`thinkingmachines/Inkling`](https://huggingface.co/thinkingmachines/Inkling)
	(revision `main`) and redistributed under that model's Apache-2.0 licence.

	To be precise about what "minimized" means: the retained tensors are the
	real trained values, bit-for-bit — nothing is randomly initialized and no
	weight is altered. The reduction is purely deletion, keeping a prefix of the
	layers and the first 16 experts of each MoE layer. That
	is what makes the fixture useful for shape and dataflow testing, and it is
	also why upstream attribution and licensing apply in full.

	No claim is made about this artifact's quality, and no endorsement by Thinking
	Machines Lab is implied.

	## What was reduced

	\| \| upstream \| here \|
	\|---\|---\|---\|
	\| text layers \| 66 \| 12 \|
	\| global attention layers \| 11 \| 2 (at 5, 11) \|
	\| dense MLP layers \| 0-1 \| 0-1 \|
	\| routed experts \| 256 \| 16 \|
	\| experts per token \| 6 \| 6 \|
	\| shared expert sinks \| 2 \| 2 \|
	\| MTP layers \| 8 \| 2 \|
	\| hidden size / heads / head_dim \| 6144 / 64 / 128 \| unchanged \|
	\| vocab \| 201024 (200058 unpadded) \| unchanged \|

	30.79 GiB, 298 tensors, 9 shards.

	The retained layers are a prefix of the upstream schedule, so layer
	indices, `local_layer_ids` and `dense_mlp_idx` carry over unchanged and the
	real 5:1 sliding-window / full-attention alternation is preserved rather than
	hand-written. Hidden size, head counts and `head_dim` are untouched so kernel
	shapes match production. The vision and audio towers are complete.

	## Why the output is gibberish

	This is expected and by design, not a bug to report. At
	12 of 66 layers and 16 of 256
	experts the output distribution is broad and near-uniform. Greedy decoding
	collapses onto a single special token; temperature 1.0 returns unrelated
	vocabulary tokens. Logprobs are finite and greedy decoding is deterministic,
	which is all the fixture needs to be.

	This fixture validates plumbing — tensor shapes, the four short-convolution
	sites per layer, per-layer-type KV head counts and relative-bias widths,
	gate-plus-sink routing, embedding norm, muP logit scaling, the sampler — and it
	is a suitable input for a logit-level comparison against a reference
	implementation. It does not validate correctness on its own, and it should not
	be evaluated for quality.

	## Serving

	Verified with vLLM 0.26.0 on a single B200:

	```bash
	export VLLM_USE_V2_MODEL_RUNNER=1
	export MAX_JOBS=16 # FlashInfer's ninja JIT otherwise fans out
	# to the host CPU count and can OOM the box
	vllm serve Inkling-mini-bf16 \
	--tokenizer-mode inkling \
	--tensor-parallel-size 1 \
	--kernel-config.enable_flashinfer_autotune=False \
	--max-model-len 4096 \
	--max-num-seqs 16
	```

	`--max-num-seqs` matters: vLLM allocates `max_num_seqs * max_model_len * 4 B`
	of pinned host memory, and the default can stall `cudaHostAlloc` for minutes.
	`scipy` is an undeclared runtime dependency of the vision tower. At TP=1 the
	fused Lamport collective logs a `ValueError: TP world size must be 2, 4, or 8`
	traceback at ERROR level; it is a benign fallback to NCCL, not a failure.

	## Licence

	Apache-2.0, inherited from [`thinkingmachines/Inkling`](https://huggingface.co/thinkingmachines/Inkling).
	The weights are Thinking Machines Lab's; this repository only removes layers
	and experts from them.