Instructions to use nm-testing/Inkling-Dense-54M-MTP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nm-testing/Inkling-Dense-54M-MTP with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="nm-testing/Inkling-Dense-54M-MTP")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("nm-testing/Inkling-Dense-54M-MTP")
model = AutoModelForMultimodalLM.from_pretrained("nm-testing/Inkling-Dense-54M-MTP", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use nm-testing/Inkling-Dense-54M-MTP with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nm-testing/Inkling-Dense-54M-MTP"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nm-testing/Inkling-Dense-54M-MTP",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/nm-testing/Inkling-Dense-54M-MTP

SGLang

How to use nm-testing/Inkling-Dense-54M-MTP with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nm-testing/Inkling-Dense-54M-MTP" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nm-testing/Inkling-Dense-54M-MTP",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nm-testing/Inkling-Dense-54M-MTP" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nm-testing/Inkling-Dense-54M-MTP",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use nm-testing/Inkling-Dense-54M-MTP with Docker Model Runner:
```
docker model run hf.co/nm-testing/Inkling-Dense-54M-MTP
```

Inkling-Dense-54M-MTP / README.md

soyrsoyr

Duplicate from soyrsoyr/Inkling-Dense-54M-MTP

1e837c7 21 days ago

preview code

Raw History Blame Contribute Delete

5.32 kB

---
license: apache-2.0
base_model:
- thinkingmachines/Inkling
library_name: transformers
tags:
- testing
- mtp
---
# Inkling-Dense-54M-MTP

A dense Inkling fixture for testing the `model.mtp.*` checkpoint layout in llm-compressor PR #3225. Created with the repository's `create-tiny-model` skill. The backbone is trained on its eight toy texts; the MTP layer is randomly initialized and untrained. This fixture is useful for checkpoint-preservation, quantization/export, and speculative-serving regression tests. It is not a general-purpose language model or a draft-acceptance benchmark.

## Model details

- Architecture: `InklingForConditionalGeneration`.
- Parameters: 54,136,067, including one MTP layer and reduced audio/vision components.
- BF16 weights: 108,279,142 bytes, approximately 103.3 MiB.
- Two dense backbone layers, one sliding and one full attention layer; no routed-expert tensors.
- One sliding-attention MTP layer, saved as 20 `model.mtp.layers.0.*` tensors.
- Original tokenizer and 201,024-entry padded vocabulary retained.

| Setting | Original Inkling | This fixture |
| --- | ---: | ---: |
| Text hidden size | 6,144 | 128 |
| Backbone layers | 66 | 2 |
| MTP layers | 8 | 1 |
| Attention heads | 64 | 4 |
| KV heads | 8 | 2 |
| Dense intermediate size | 24,576 | 512 |
| Head dimension | 128 | 128 |
| Context limit | 1,048,576 | 512 |

Audio and vision projection dimensions match the reduced text hidden size. Text-only serving is tested; multimodal inference is not. Both Transformers' dense-layer fields and vLLM's legacy `dense_mlp_idx` / `dense_intermediate_size` fields are retained in the config.

## Creation and provenance

1. Start from the corrected tiny Inkling config and compare against [original Inkling](https://huggingface.co/thinkingmachines/Inkling/tree/828496eeae4c243ff1a22f7f28ff83694f2f7bc9). No original model weights are copied.
2. Initialize the reduced dense backbone with seed 0. Use the tokenizer and processor from [Inkling-0.92B-MTP](https://huggingface.co/inference-optimization/Inkling-0.92B-MTP/tree/22b31818016a1560cfeb47caaa3296cafa2f43df).
3. Train using the skill's `COPYPASTAS` dataset, learning rate 0.001, sequence length 128, and seed 0. Early stopping finishes at 200 steps. Evaluation perplexity is 1.0279 on the same eight training texts, not a held-out benchmark. The skill's separate sample check gives 1.0051.
4. Initialize one MTP layer with seed 0 and use Transformers' reversed weight conversions to serialize it. All 62 final checkpoint keys occur in the original Inkling checkpoint index. All tensors are finite.

Build provenance and scoped validation results are included in `fixture_provenance.json`. The model was created for [llm-compressor PR #3225](https://github.com/vllm-project/llm-compressor/pull/3225).

## Usage

```python
import torch
from transformers import AutoTokenizer, InklingForConditionalGeneration
path = "soyrsoyr/Inkling-Dense-54M-MTP"
model = InklingForConditionalGeneration.from_pretrained(
    path, dtype=torch.bfloat16, device_map="cuda", attn_implementation="eager"
)
tokenizer = AutoTokenizer.from_pretrained(path)
inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")
output = model.generate(**inputs, max_new_tokens=16)
print(tokenizer.decode(output[0]))
```

## Serving and tested scope

Prepublication validation with `load_context(InklingForConditionalGeneration, load_mtp=True)` detected and attached MTP, quantized its layers with FP8_DYNAMIC, and exported seven MTP weight-scale tensors under the native `model.mtp.*` checkpoint names. A separate control loaded MTP through the same path, kept it in BF16, and successfully served the FP8 backbone with vLLM MTP drafting.
BF16 MTP serving also passed for the source fixture. An additional unloaded-copy check used `load_mtp=False` and `oneshot(..., output_dir=...)`, verified exact equality of all 20 copied MTP tensors, and passed vLLM generation with a positive MTP draft-token counter. These are separate local validation runs, not a claim that fully quantized Inkling MTP serving works.
Quantizing and exporting the MTP layers also succeeds, but serving those quantized MTP layers fails in the tested vLLM implementation: Inkling's MTP decoder is constructed with `quant_config=None` and rejects `weight_scale` tensors. Keep MTP in BF16 for this serving test.
```python
from vllm import LLM, SamplingParams
llm = LLM(
    model="soyrsoyr/Inkling-Dense-54M-MTP",
    dtype="bfloat16",
    max_model_len=256,
    max_num_seqs=2,
    gpu_memory_utilization=0.25,
    enforce_eager=True,
    enable_prefix_caching=False,
    disable_log_stats=False,
    limit_mm_per_prompt={"image": 0, "audio": 0, "video": 0},
    speculative_config={"method": "mtp", "num_speculative_tokens": 1},
)
outputs = llm.generate(["The capital of France is"], SamplingParams(temperature=0, max_tokens=16))
print(outputs[0].outputs[0].text)
```
Tested on one H100 with Transformers 5.17.0 and torch 2.13.0+cu130. The WDC vLLM installation reports distribution metadata 0.30.0 but runtime version 0.26.0; this is validation of that installed build, not a clean-wheel compatibility claim. Multimodal inference and multi-GPU serving are not covered. The dense design intentionally avoids the separate sparse Inkling expert-export issue.

	---
	license: apache-2.0
	base_model:
	- thinkingmachines/Inkling
	library_name: transformers
	tags:
	- testing
	- mtp
	---

	# Inkling-Dense-54M-MTP

	A dense Inkling fixture for testing the `model.mtp.*` checkpoint layout in llm-compressor PR #3225. Created with the repository's `create-tiny-model` skill. The backbone is trained on its eight toy texts; the MTP layer is randomly initialized and untrained. This fixture is useful for checkpoint-preservation, quantization/export, and speculative-serving regression tests. It is not a general-purpose language model or a draft-acceptance benchmark.

	## Model details

	- Architecture: `InklingForConditionalGeneration`.
	- Parameters: 54,136,067, including one MTP layer and reduced audio/vision components.
	- BF16 weights: 108,279,142 bytes, approximately 103.3 MiB.
	- Two dense backbone layers, one sliding and one full attention layer; no routed-expert tensors.
	- One sliding-attention MTP layer, saved as 20 `model.mtp.layers.0.*` tensors.
	- Original tokenizer and 201,024-entry padded vocabulary retained.

	\| Setting \| Original Inkling \| This fixture \|
	\| --- \| ---: \| ---: \|
	\| Text hidden size \| 6,144 \| 128 \|
	\| Backbone layers \| 66 \| 2 \|
	\| MTP layers \| 8 \| 1 \|
	\| Attention heads \| 64 \| 4 \|
	\| KV heads \| 8 \| 2 \|
	\| Dense intermediate size \| 24,576 \| 512 \|
	\| Head dimension \| 128 \| 128 \|
	\| Context limit \| 1,048,576 \| 512 \|

	Audio and vision projection dimensions match the reduced text hidden size. Text-only serving is tested; multimodal inference is not. Both Transformers' dense-layer fields and vLLM's legacy `dense_mlp_idx` / `dense_intermediate_size` fields are retained in the config.

	## Creation and provenance

	1. Start from the corrected tiny Inkling config and compare against [original Inkling](https://huggingface.co/thinkingmachines/Inkling/tree/828496eeae4c243ff1a22f7f28ff83694f2f7bc9). No original model weights are copied.
	2. Initialize the reduced dense backbone with seed 0. Use the tokenizer and processor from [Inkling-0.92B-MTP](https://huggingface.co/inference-optimization/Inkling-0.92B-MTP/tree/22b31818016a1560cfeb47caaa3296cafa2f43df).
	3. Train using the skill's `COPYPASTAS` dataset, learning rate 0.001, sequence length 128, and seed 0. Early stopping finishes at 200 steps. Evaluation perplexity is 1.0279 on the same eight training texts, not a held-out benchmark. The skill's separate sample check gives 1.0051.
	4. Initialize one MTP layer with seed 0 and use Transformers' reversed weight conversions to serialize it. All 62 final checkpoint keys occur in the original Inkling checkpoint index. All tensors are finite.

	Build provenance and scoped validation results are included in `fixture_provenance.json`. The model was created for [llm-compressor PR #3225](https://github.com/vllm-project/llm-compressor/pull/3225).

	## Usage

	```python
	import torch
	from transformers import AutoTokenizer, InklingForConditionalGeneration

	path = "soyrsoyr/Inkling-Dense-54M-MTP"
	model = InklingForConditionalGeneration.from_pretrained(
	path, dtype=torch.bfloat16, device_map="cuda", attn_implementation="eager"
	)
	tokenizer = AutoTokenizer.from_pretrained(path)
	inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")
	output = model.generate(**inputs, max_new_tokens=16)
	print(tokenizer.decode(output[0]))
	```

	## Serving and tested scope

	Prepublication validation with `load_context(InklingForConditionalGeneration, load_mtp=True)` detected and attached MTP, quantized its layers with FP8_DYNAMIC, and exported seven MTP weight-scale tensors under the native `model.mtp.*` checkpoint names. A separate control loaded MTP through the same path, kept it in BF16, and successfully served the FP8 backbone with vLLM MTP drafting.

	BF16 MTP serving also passed for the source fixture. An additional unloaded-copy check used `load_mtp=False` and `oneshot(..., output_dir=...)`, verified exact equality of all 20 copied MTP tensors, and passed vLLM generation with a positive MTP draft-token counter. These are separate local validation runs, not a claim that fully quantized Inkling MTP serving works.

	Quantizing and exporting the MTP layers also succeeds, but serving those quantized MTP layers fails in the tested vLLM implementation: Inkling's MTP decoder is constructed with `quant_config=None` and rejects `weight_scale` tensors. Keep MTP in BF16 for this serving test.

	```python
	from vllm import LLM, SamplingParams

	llm = LLM(
	model="soyrsoyr/Inkling-Dense-54M-MTP",
	dtype="bfloat16",
	max_model_len=256,
	max_num_seqs=2,
	gpu_memory_utilization=0.25,
	enforce_eager=True,
	enable_prefix_caching=False,
	disable_log_stats=False,
	limit_mm_per_prompt={"image": 0, "audio": 0, "video": 0},
	speculative_config={"method": "mtp", "num_speculative_tokens": 1},
	)
	outputs = llm.generate(["The capital of France is"], SamplingParams(temperature=0, max_tokens=16))
	print(outputs[0].outputs[0].text)
	```

	Tested on one H100 with Transformers 5.17.0 and torch 2.13.0+cu130. The WDC vLLM installation reports distribution metadata 0.30.0 but runtime version 0.26.0; this is validation of that installed build, not a clean-wheel compatibility claim. Multimodal inference and multi-GPU serving are not covered. The dense design intentionally avoids the separate sparse Inkling expert-export issue.