GLM-4.7-Flash-1.0B-A0.9B-small

This is a small version of zai-org/GLM-4.7-Flash created for testing and development. Unlike the previous "tiny" models, all hidden dimensions and hyperparameters are kept identical to the base model; only the layer count, layer types per position, and expert count are reduced.

Model Details

• Base Model: zai-org/GLM-4.7-Flash
• Architecture: glm4_moe_lite (Glm4MoeLiteForCausalLM, native in transformers 5.17)
• Total Parameters: 1.039B
• Activated Parameters: ~0.926B (4 of 8 routed experts per token + 1 shared expert, topk unchanged from base)

Configuration Changes

The following parameters were reduced from the original model:

Parameter	Original	Small
num_hidden_layers	47	4
mlp_layer_types	47 entries ([dense x1, sparse x46])	[dense, sparse, sparse, sparse]
n_routed_experts	64	8

All other hyperparameters are identical to the base: hidden_size 2048, vocab_size 154880, intermediate_size 10240, moe_intermediate_size 1536, num_experts_per_tok 4 (topk unchanged), n_shared_experts 1, kv_lora_rank 512, q_lora_rank 768, qk_nope_head_dim 192, qk_rope_head_dim 64, qk_head_dim 256, v_head_dim 256, num_attention_heads 20, rope_theta 1e6 (rope_interleave), routed_scaling_factor 1.8, noaux_tc routing with n_group 1, tie_word_embeddings: false, and num_nextn_predict_layers: 1.

The base has a single attention type (MLA) and two MLP types (dense, sparse); the small model keeps 1 dense layer followed by 3 sparse layers, mirroring the base's layout (first_k_dense_replace: 1 unchanged). 8 experts (>= topk 4) still exercises routing.

Note: the parameter count (~1.04B, of which the two untied 154880x2048 embedding matrices are ~0.63B) is expected with fixed base dims; the model is still far smaller than the base.

Checkpoint Structure

Single-file bf16 checkpoint (model.safetensors, 133 tensors) using the same naming convention as the base checkpoint: model.layers.N.self_attn.q_a_proj.weight, model.layers.N.mlp.experts.E.gate_proj/up_proj/down_proj.weight, model.embed_tokens.weight, model.norm.weight, lm_head.weight. Tensor-name structure matches the base model.safetensors.index.json exactly modulo layer count (23/23 patterns; the base's model.layers.47.* MTP layer is intentionally absent — see Notes).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "inference-optimization/GLM-4.7-Flash-1.0B-A0.9B-small",
    device_map="auto", dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/GLM-4.7-Flash-1.0B-A0.9B-small")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model skill.

1. Inspected the GLM-4.7-Flash config: 47 layers (1 dense + 46 sparse MoE), 64 routed experts with topk 4, MLA attention (single attention type), num_nextn_predict_layers: 1, no quantization_config.
2. Built the small model keeping all dims identical: 4 layers [dense, sparse, sparse, sparse], 8 routed experts.
3. Fine-tuned on the copypasta toy dataset (bf16, batch size 2, lr 1e-4 → 5e-5) until perplexity < 3.0.
4. Validated the tensor-name structure against the base model.safetensors.index.json (pattern sets match exactly modulo layer count; no converter needed — transformers saves this architecture directly in the base naming).
5. Validated perplexity and generation (output below).

Notes

• Backbone-only checkpoint: num_nextn_predict_layers=1 is kept from the base config, and no MTP weights are materialized — matching the base checkpoint layout, where the MTP layer is stored as an extra model.layers.47.* block that is intentionally absent here.
• No quantization_config in the base (bf16); none stripped.
• Training-loss note: the GLM tokenizer pads on the left, so with the standard causal shift the first real token of each training text is predicted from the final padding position; this contributes a fixed irreducible term to the training loss (~0.29 average plateau). The model itself converged fully: every real-position prediction reaches ~0 loss, validation perplexity is 1.0000067, and greedy generation reproduces the training text.
• Validation output:

  Success: 1.0000066757202148 <= 10.0
  Generating sample text:
  According to all known laws of aviation, there is no way a bee should be able to fly.