Instructions to use inference-optimization/GLM-5-2.9B-A2.8B-small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use inference-optimization/GLM-5-2.9B-A2.8B-small with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="inference-optimization/GLM-5-2.9B-A2.8B-small")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("inference-optimization/GLM-5-2.9B-A2.8B-small")
model = AutoModelForCausalLM.from_pretrained("inference-optimization/GLM-5-2.9B-A2.8B-small", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use inference-optimization/GLM-5-2.9B-A2.8B-small with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "inference-optimization/GLM-5-2.9B-A2.8B-small"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/GLM-5-2.9B-A2.8B-small",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/inference-optimization/GLM-5-2.9B-A2.8B-small

SGLang

How to use inference-optimization/GLM-5-2.9B-A2.8B-small with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "inference-optimization/GLM-5-2.9B-A2.8B-small" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/GLM-5-2.9B-A2.8B-small",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "inference-optimization/GLM-5-2.9B-A2.8B-small" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/GLM-5-2.9B-A2.8B-small",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use inference-optimization/GLM-5-2.9B-A2.8B-small with Docker Model Runner:
```
docker model run hf.co/inference-optimization/GLM-5-2.9B-A2.8B-small
```

GLM-5-2.9B-A2.8B-small

This is a small version of zai-org/GLM-5 created for testing and development. It keeps every hyperparameter of the base model identical (hidden size, MLA/DSA dimensions, indexer parameters, MoE topology, vocabulary, untied embeddings, rope settings) and only reduces the number of decoder layers, the per-layer type lists, and the number of routed experts per MoE layer.

Model Details

Base Model: zai-org/GLM-5
Architecture: GlmMoeDsaForCausalLM (glm_moe_dsa)
Total Parameters: 2.89B
Activated Parameters: 2.82B (topk 8 of 10 routed experts per sparse layer)

Configuration Changes

Only the allowed change classes were applied; everything else is identical to the base:

Parameter	Base	Small
`num_hidden_layers`	78	2
`layer_types`	78 x `deepseek_sparse_attention`	2 x `deepseek_sparse_attention`
`mlp_layer_types`	`[dense x3, sparse x75]`	`[dense, sparse]`
`first_k_dense_replace`	3	1
`indexer_types`	78 x `full` (all base layers run their own indexer)	`[full, full]`
`n_routed_experts`	256	10

The 2 layers are the minimal covering set of the distinct (attention, MLP, indexer) combinations that occur in the base:

layer 0: DSA with full indexer + dense MLP (base layers 0-2)
layer 1: DSA with full indexer + sparse MoE (base layers 3-77)

Unlike GLM-5.2, every one of the 78 base layers carries a full indexer (verified against the base checkpoint: every layer, including the MTP layer 78, has self_attn.indexer.* tensors), so there is no shared-indexer layer type to cover. 10 experts = topk (8) + 2 so that routing is actually exercised; num_experts_per_tok (topk 8) is unchanged. All other values are kept from the base, including hidden_size 6144, intermediate_size 12288, moe_intermediate_size 2048, n_shared_experts 1, MLA dimensions (q_lora_rank 2048, kv_lora_rank 512, qk_nope_head_dim 192, qk_rope_head_dim 64, v_head_dim 256, 64/64 heads), indexer parameters (index_topk 2048, index_n_heads 32, index_head_dim 128, indexer_rope_interleave true), sigmoid noaux_tc routing with routed_scaling_factor 2.5, rope_theta 1e6, vocab_size 154880, untied embeddings.

Checkpoint Structure

Single-file bf16 model.safetensors (69 tensors) using the original base checkpoint layout: per-expert tensors model.layers.N.mlp.experts.{i}.{gate,up,down}_proj.weight, per-layer mlp.gate.weight + mlp.gate.e_score_correction_bias + shared-expert mlp.shared_experts.{gate,up,down}_proj.weight, MLA names (self_attn.{q_a_proj,q_a_layernorm,q_b_proj,kv_a_proj_with_mqa,kv_a_layernorm,kv_b_proj,o_proj}), DSA indexer tensors on every layer (self_attn.indexer.{wq_b,wk,k_norm,weights_proj}), and untied lm_head.weight. Tensor-name patterns match the base model.safetensors.index.json exactly, modulo layer and expert counts (validated programmatically: 28/28 normalized patterns, 0 missing, 0 extra; per-layer key sets identical to the analogous base layers 0 / 3).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("inference-optimization/GLM-5-2.9B-A2.8B-small", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/GLM-5-2.9B-A2.8B-small")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model skill:

Loaded the base AutoConfig and applied only the allowed changes (2 layers with the per-layer lists above, 10 experts).
Built the model with skip_weights_download, re-initialized weights (init_weights() + NaN fixup), cast fully to bf16, and saved with the base's weight conversions recorded (so the save emits the base per-expert layout directly).
Verified the saved tensor structure against the base model.safetensors.index.json immediately after the first save.
Fine-tuned on the skill's copypasta dataset in bf16 (lr 1e-4, batch 2, early-stopped after ~180 steps / 21s) to target perplexity.
Re-validated the fine-tuned checkpoint structure and model quality.

Validation

Perplexity on the skill's validation text: 1.00002 (target <= 10)
Greedy generation from "According to all known laws": produces the trained copypasta continuation correctly.

Notes

MTP: num_nextn_predict_layers is kept at 1 from the base; as with transformers saves of this family, the checkpoint is backbone-only (the base stores its MTP layer as model.layers.78.*, which is intentionally absent here, matching the convention of the other GLM small models). This entry replaces a previous MTP fixture with a backbone-only checkpoint.
Size: 2.89B total is the forced minimum for a layer set covering all distinct (attention, MLP, indexer) combos at hidden size 6144 — the untied 154880 x 6144 embeddings alone are 1.9B and a 10-expert MoE layer adds ~~0.45B. Strictly far below the base (~~757B). The briefing's ~3-3.5B estimate assumed a third distinct (shared-indexer) combo as in GLM-5.2; GLM-5 has only the two combos above, so the minimal covering set is 2 layers.
No quantization_config in the base (unquantized bf16 checkpoint, ~1.5TB); nothing stripped. bf16 throughout (full bf16 cast works around _keep_in_fp32_modules on indexer.weights_proj / e_score_correction_bias).
training_args.bin and optimizer files are not part of the upload.

Downloads last month: 36

Safetensors

Model size

3B params

Tensor type

BF16

Model tree for inference-optimization/GLM-5-2.9B-A2.8B-small

Base model

zai-org/GLM-5

Finetuned

(38)

this model

Collection including inference-optimization/GLM-5-2.9B-A2.8B-small

Small Models

Collection

Small versions of known models: same vLLM loading characteristics as the base, fewer layers/experts, identical hidden dims and hyperparameters • 25 items • Updated 6 days ago