---
license: apache-2.0
base_model:
- thinkingmachines/Inkling
library_name: transformers
tags:
- testing
- mtp
---

# Inkling-Dense-54M-MTP

A dense Inkling fixture for testing the `model.mtp.*` checkpoint layout in llm-compressor PR #3225. Created with the repository's `create-tiny-model` skill. The backbone is trained on its eight toy texts; the MTP layer is randomly initialized and untrained. This fixture is useful for checkpoint-preservation, quantization/export, and speculative-serving regression tests. It is not a general-purpose language model or a draft-acceptance benchmark.

## Model details

- Architecture: `InklingForConditionalGeneration`.
- Parameters: 54,136,067, including one MTP layer and reduced audio/vision components.
- BF16 weights: 108,279,142 bytes, approximately 103.3 MiB.
- Two dense backbone layers, one sliding and one full attention layer; no routed-expert tensors.
- One sliding-attention MTP layer, saved as 20 `model.mtp.layers.0.*` tensors.
- Original tokenizer and 201,024-entry padded vocabulary retained.

| Setting | Original Inkling | This fixture |
| --- | ---: | ---: |
| Text hidden size | 6,144 | 128 |
| Backbone layers | 66 | 2 |
| MTP layers | 8 | 1 |
| Attention heads | 64 | 4 |
| KV heads | 8 | 2 |
| Dense intermediate size | 24,576 | 512 |
| Head dimension | 128 | 128 |
| Context limit | 1,048,576 | 512 |

Audio and vision projection dimensions match the reduced text hidden size. Text-only serving is tested; multimodal inference is not. Both Transformers' dense-layer fields and vLLM's legacy `dense_mlp_idx` / `dense_intermediate_size` fields are retained in the config.

## Creation and provenance

1. Start from the corrected tiny Inkling config and compare against [original Inkling](https://huggingface.co/thinkingmachines/Inkling/tree/828496eeae4c243ff1a22f7f28ff83694f2f7bc9). No original model weights are copied.
2. Initialize the reduced dense backbone with seed 0. Use the tokenizer and processor from [Inkling-0.92B-MTP](https://huggingface.co/inference-optimization/Inkling-0.92B-MTP/tree/22b31818016a1560cfeb47caaa3296cafa2f43df).
3. Train using the skill's `COPYPASTAS` dataset, learning rate 0.001, sequence length 128, and seed 0. Early stopping finishes at 200 steps. Evaluation perplexity is 1.0279 on the same eight training texts, not a held-out benchmark. The skill's separate sample check gives 1.0051.
4. Initialize one MTP layer with seed 0 and use Transformers' reversed weight conversions to serialize it. All 62 final checkpoint keys occur in the original Inkling checkpoint index. All tensors are finite.

Build provenance and scoped validation results are included in `fixture_provenance.json`. The model was created for [llm-compressor PR #3225](https://github.com/vllm-project/llm-compressor/pull/3225).

## Usage

```python
import torch
from transformers import AutoTokenizer, InklingForConditionalGeneration

path = "soyrsoyr/Inkling-Dense-54M-MTP"
model = InklingForConditionalGeneration.from_pretrained(
    path, dtype=torch.bfloat16, device_map="cuda", attn_implementation="eager"
)
tokenizer = AutoTokenizer.from_pretrained(path)
inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")
output = model.generate(**inputs, max_new_tokens=16)
print(tokenizer.decode(output[0]))
```

## Serving and tested scope

Prepublication validation with `load_context(InklingForConditionalGeneration, load_mtp=True)` detected and attached MTP, quantized its layers with FP8_DYNAMIC, and exported seven MTP weight-scale tensors under the native `model.mtp.*` checkpoint names. A separate control loaded MTP through the same path, kept it in BF16, and successfully served the FP8 backbone with vLLM MTP drafting.

BF16 MTP serving also passed for the source fixture. An additional unloaded-copy check used `load_mtp=False` and `oneshot(..., output_dir=...)`, verified exact equality of all 20 copied MTP tensors, and passed vLLM generation with a positive MTP draft-token counter. These are separate local validation runs, not a claim that fully quantized Inkling MTP serving works.

Quantizing and exporting the MTP layers also succeeds, but serving those quantized MTP layers fails in the tested vLLM implementation: Inkling's MTP decoder is constructed with `quant_config=None` and rejects `weight_scale` tensors. Keep MTP in BF16 for this serving test.

```python
from vllm import LLM, SamplingParams

llm = LLM(
    model="soyrsoyr/Inkling-Dense-54M-MTP",
    dtype="bfloat16",
    max_model_len=256,
    max_num_seqs=2,
    gpu_memory_utilization=0.25,
    enforce_eager=True,
    enable_prefix_caching=False,
    disable_log_stats=False,
    limit_mm_per_prompt={"image": 0, "audio": 0, "video": 0},
    speculative_config={"method": "mtp", "num_speculative_tokens": 1},
)
outputs = llm.generate(["The capital of France is"], SamplingParams(temperature=0, max_tokens=16))
print(outputs[0].outputs[0].text)
```

Tested on one H100 with Transformers 5.17.0 and torch 2.13.0+cu130. The WDC vLLM installation reports distribution metadata 0.30.0 but runtime version 0.26.0; this is validation of that installed build, not a clean-wheel compatibility claim. Multimodal inference and multi-GPU serving are not covered. The dense design intentionally avoids the separate sparse Inkling expert-export issue.