Instructions to use inference-optimization/GLM-5-0.88B-MTP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use inference-optimization/GLM-5-0.88B-MTP with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="inference-optimization/GLM-5-0.88B-MTP")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("inference-optimization/GLM-5-0.88B-MTP")
model = AutoModelForCausalLM.from_pretrained("inference-optimization/GLM-5-0.88B-MTP", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use inference-optimization/GLM-5-0.88B-MTP with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "inference-optimization/GLM-5-0.88B-MTP"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/GLM-5-0.88B-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/inference-optimization/GLM-5-0.88B-MTP

SGLang

How to use inference-optimization/GLM-5-0.88B-MTP with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "inference-optimization/GLM-5-0.88B-MTP" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/GLM-5-0.88B-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "inference-optimization/GLM-5-0.88B-MTP" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/GLM-5-0.88B-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use inference-optimization/GLM-5-0.88B-MTP with Docker Model Runner:
```
docker model run hf.co/inference-optimization/GLM-5-0.88B-MTP
```

soyrsoyr commited on 19 days ago

Commit

0dab512

verified ·

1 Parent(s): 10d9a07

simplify model card to recipe and usage

Browse files

Files changed (1) hide show

README.md +22 -63

README.md CHANGED Viewed

@@ -9,73 +9,32 @@ tags:
 - test-fixture
 ---
-# GLM-5-0.88B-MTP-PR3225
-Toy-trained BF16 source for [LLM Compressor PR #3225](https://github.com/vllm-project/llm-compressor/pull/3225). This is a small architecture and checkpoint test fixture, not a production language model. Its backbone was initialized randomly and trained on the repository tiny-model skill's toy text corpus; no pretrained base-model weights were used.
-## Model and provenance
-- Original architecture/tokenizer: [zai-org/GLM-5](https://huggingface.co/zai-org/GLM-5), revision `c183ef8c61faee82855eca1ed9bb3a9a7ce3b0b2`.
-- Transformers class: `GlmMoeDsaForCausalLM`.
-- Total source parameters including synthetic MTP: **881,036,032** (0.881B).
-- Backbone parameters: **765,051,392**; MTP parameters: **115,984,640**.
-- Estimated active backbone parameters: **708,428,288**. This routing estimate includes embeddings/dense components, excludes MTP, and is not a FLOP measurement.
-- Precision: BF16.
-- Checkpoint: 5 indexed safetensors shards; 192 indexed tensors. Tokenizer assets are included.
-## Configuration
-| Field | Original | Tiny |
-| --- | --- | --- |
-| `num_hidden_layers` | 78 | 4 |
-| `hidden_size` | 6144 | 1536 |
-| `intermediate_size` | 12288 | 6144 |
-| `moe_intermediate_size` | 2048 | 1024 |
-| `n_routed_experts` | 256 | 16 |
-| `num_experts_per_tok` | 8 | 4 |
-| `num_attention_heads` | 64 | 16 |
-| `num_key_value_heads` | 64 | 16 |
-| `q_lora_rank` | 2048 | 2048 |
-| `kv_lora_rank` | 512 | 512 |
-| `num_mtp_layers` | not explicitly set | 1 |
-The saved `config.json` is authoritative. Depth is reduced; the limitations below apply.
-## Validation and scope
-The BF16 backbone reload passed. Model-based MTP loading fails for this reduced-depth fixture: the upstream registered checkpoint pattern expects layer 78, while this fixture's MTP tensors begin at layer 4. No loader or architecture override was applied. MFPTQ successfully quantized 136 modules, including 56 MTP modules. Its Transformers backbone reload had toy perplexity 1.440128 (BF16: 1.439453). MTP execution remains unverified; the vLLM attempt encountered a DeepGEMM extension ABI mismatch.
-The reloaded **BF16 backbone** toy perplexity was **1.439453** on the same small corpus used for training. This demonstrates learning/reload integrity, not generalization or benchmark quality.
-MTP projections are synthetic initializations and decoder weights were copied from trained backbone blocks. **MTP was not separately trained**, so these artifacts do not establish draft acceptance quality or inference acceleration.
-The tested model-based environment used **Transformers 5.17.0**, Torch 2.14.0+cu130, LLM Compressor [`2d52420`](https://github.com/vllm-project/llm-compressor/commit/2d5242056a8a00028686a303771ffc8e2fe27d07), and compressed-tensors [`e69c8dc`](https://github.com/vllm-project/compressed-tensors/commit/e69c8dc58aa152e5f5e36e85800ec1d2e7de5271). Serving smoke checks used vLLM 0.30.0. GLM/DeepSeek model-based MTP loading failed on Transformers **5.15.0**; **5.16 was not tested**. Calibration-dependent MTP quantization remains follow-up work.
-The model class/loader was not patched to make tests pass. Construction changes were configuration reductions and explicit synthetic checkpoint fixture creation. Structured results and configuration provenance are in [validation.json](validation.json); [artifact-manifest.json](artifact-manifest.json) records checkpoint file hashes.
-## Backbone loading
-With the tested Transformers environment (and compressed-tensors for FP8), this loads the backbone. MTP is opt-in; ordinary Transformers backbone generation does not execute it.
 ```python
-import torch
-from transformers import AutoModelForCausalLM, AutoTokenizer
-model_id = "inference-optimization/GLM-5-0.88B-MTP-PR3225"
-tokenizer = AutoTokenizer.from_pretrained(model_id)
-model = AutoModelForCausalLM.from_pretrained(
-    model_id, dtype=torch.bfloat16, attn_implementation="eager",
 )
-inputs = tokenizer("The capital of France is", return_tensors="pt")
-output = model.generate(**inputs, max_new_tokens=16, do_sample=False)
-print(tokenizer.decode(output[0], skip_special_tokens=True))
 ```
-## Training
-Random seed 3225; AdamW with learning rate 0.0004 and weight decay 0.01; batch size 2; text truncated to 160 tokens. Training stopped after three consecutive checks at toy perplexity <=3. The corpus follows the [repository tiny-model workflow](https://github.com/vllm-project/llm-compressor/tree/2d5242056a8a00028686a303771ffc8e2fe27d07/.agents/skills/create-tiny-model), with one additional short saying. These are the trained/reloaded artifacts used in testing.
-## License and attribution
-Upstream architecture/tokenizer provenance and license: [zai-org/GLM-5](https://huggingface.co/zai-org/GLM-5/tree/c183ef8c61faee82855eca1ed9bb3a9a7ce3b0b2).

 - test-fixture
 ---
+# GLM-5-0.88B-MTP
+Small BF16 test model using [zai-org/GLM-5](https://huggingface.co/zai-org/GLM-5), with a toy-trained backbone and synthetic MTP weights.
+## Quantization recipe
+Uses [LLM Compressor](https://github.com/vllm-project/llm-compressor/pull/3225) model-free quantization. MTP inference has not been verified for this fixture.
 ```python
+from llmcompressor import model_free_ptq
+MODEL_ID = "inference-optimization/GLM-5-0.88B-MTP"
+SAVE_DIR = "GLM-5-0.88B-MTP-FP8-Dynamic-MFPTQ"
+model_free_ptq(
+    model_stub=MODEL_ID,
+    save_directory=SAVE_DIR,
+    scheme="FP8_DYNAMIC",
+    ignore=[
+        "lm_head",
+        "model.embed_tokens",
+        r"re:.*\.mlp\.gate$",
+        r"re:.*\.eh_proj$",
+        r"re:.*\.indexer\..*",
+    ],
+    max_workers=2,
+    device="cuda",
 )
 ```