Instructions to use inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small")
model = AutoModelForMultimodalLM.from_pretrained("inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small

SGLang

How to use inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small with Docker Model Runner:
```
docker model run hf.co/inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small
```

GLM-5.3-Flash-2.6B-A2.3B-small

This is a small version of zai-org/GLM-5.3-Flash created for testing and development. It keeps all hidden dimensions and hyperparameters of the base model identical, reducing only the number of layers, the layer-type layout, and the number of MoE experts, so it exercises the same architecture (hybrid KDA linear attention + DeepSeek-style sparse attention with indexer, MoE with sigmoid noaux-tc routing, multi-hyper-connection, vision tower) at a fraction of the size.

Model Details

Base Model: zai-org/GLM-5.3-Flash
Architecture: glm5_next (Glm5NextForConditionalGeneration, multimodal VLM: glm5_next_text + glm5_next_vision)
Total Parameters: 2.57B (2,570,456,610)
Activated Parameters: 2.28B (2,275,605,794; text-only per-token estimate, vision tower excluded, top-8-of-10 experts + shared expert per MoE layer)

Configuration Changes

Only the three allowed categories of changes were made (layer counts, layer-type layout, expert count); every other hyperparameter is identical to the base:

Parameter	Base	Small
text `num_hidden_layers`	45	3
text `layer_types`	34x `linear_attention` + 11x `deepseek_sparse_attention` (every 4th layer DSA)	`linear_attention`, `linear_attention`, `deepseek_sparse_attention`
text `mlp_layer_types`	3x `dense` + 42x `sparse`	`dense`, `sparse`, `sparse`
text `first_k_dense_replace`	3	1
text `indexer_types`	45x `full`	3x `full`
text `linear_attn_config.kda_layers` / `full_attn_layers`	[0,1,2,4,...,44] / [3,7,...,43]	[0,1] / [2]
text `n_routed_experts`	288	10
vision `depth`	24	2

The 3 text layers are the minimal set covering every distinct attention layer type (linear_attention KDA, deepseek_sparse_attention MLA+indexer) and every distinct MLP type (dense, sparse MoE), i.e. all three (attention, MLP) combinations that occur in the base. 10 experts >= num_experts_per_tok 8 (topk+2 for non-degenerate routing). Everything else — hidden size 4096, vocab 154880, MLA dims (kv_lora_rank 512, q_lora_rank 1536, nope-only heads), indexer params (index_topk 2048, kpool 4), MoE routing (sigmoid, noaux_tc, norm_topk_prob, routed_scaling_factor 2.5, 1 shared expert), multi-hyper-connection (mhc, hc_mult 4), KDA linear attention dims, rope/max_position_embeddings 1048576, tie_word_embeddings: false, num_nextn_predict_layers: 1 — is unchanged from the base.

Checkpoint Structure

Single-file model.safetensors (bf16, ~5.1GB), unsharded. Tensor naming follows the base checkpoint exactly (validated: all 75 tensor-name patterns match the base model.safetensors.index.json modulo layer/expert/vision-block counts): model.language_model.* text tower (per-expert ungrouped mlp.experts.<i>.{gate,up,down}_proj.weight, self_attn.forget_gate-style original names f_a_proj/f_b_proj/dt_bias/A_log, hc_attn_*/hc_ffn_* hyper-connection names, split q/k/v_conv1d), model.visual.* vision tower, and untied lm_head.weight.

Usage

from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained("inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small", device_map="auto")
processor = AutoProcessor.from_pretrained("inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small")

input_ids = processor.tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(processor.tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model skill.

The config was derived from the base AutoConfig applying only the changes in the table above, the model was built with random init (skip_weights_download + init_weights()), and then fine-tuned on the skill's toy copypasta dataset (8 texts, ~160 steps, lr 1e-4, bf16) until the perplexity stopping criterion was reached.

Validation

Perplexity on the toy dataset text: 1.0000129 (target <= 10)
Greedy generation reproduces the training text ("According to all known laws of aviation, there is no way a bee should be able to fly.")
Multimodal forward smoke test: image through processor + 2-block vision tower produces finite logits
Checkpoint tensor-name structure matches the base index.json (75/75 patterns)

Notes

The base model ships with an fp8 (e4m3, dynamic activation scheme) quantization_config; it was stripped and this small model is unquantized bf16, matching the convention for the other small models in this series.
num_nextn_predict_layers: 1 is kept from the base config; transformers does not materialize MTP weights, so the checkpoint is backbone-only (the base stores its MTP layer as model.layers.45.*, which is intentionally absent here, matching e.g. inference-optimization/GLM-5.2-0.8B-A0.8B).
Training loss plateaus around 0.30 (train ppl ~1.35) due to the GLM tokenizer's left padding interacting with the causal shift (one unpredictable first-token term per text); unpadded validation perplexity is ~1.0.

Downloads last month: 35

Safetensors

Model size

3B params

Tensor type

BF16

Model tree for inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small

Base model

zai-org/GLM-5.3-Flash

Finetuned

(32)

this model

Collection including inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small

Small Models

Collection

Small versions of known models: same vLLM loading characteristics as the base, fewer layers/experts, identical hidden dims and hyperparameters • 25 items • Updated 6 days ago