---
library_name: transformers
base_model:
- tencent/Hy4-preview-FP8
---
This tiny model is intended for debugging. It is randomly initialized using the configuration adapted from [tencent/Hy4-preview-FP8](https://huggingface.co/tencent/Hy4-preview-FP8).
Structural comparison with `tencent/Hy4-preview-FP8`:
| Feature | Source | Tiny |
|---|---|---|
| Architecture / input | HYV4; text | Same |
| Vocabulary / tied embeddings | 120,832 / false | Same |
| Backbone layers / hidden size | 78 / 6,144 | 10 / 64 |
| MLP layout | 1 dense layer, then MoE | Same |
| Routed experts / top-k / shared experts | 256 / 8 / 1 | 32 / 8 / 1 |
| Dense / expert intermediate size | 18,432 / 2,048 | 128 / 512 |
| Attention Q heads; non-RoPE / RoPE / V dimensions | 64; 192 / 64 / 256 | 8; 32 / 64 / 32 |
| Q / KV LoRA rank | 2,048 / 512 | 64 / 512 |
| Indexer heads / dimension / selected tokens | 32 / 128 / 2,048 | Same |
| Full indexer layers | 0, 1, 5, 9, ... | 0, 1, 5, 9; remaining layers reuse the preceding indexer |
| Independent Hyper-Connections | 4 streams; gated pre/post branches | Same |
| MLA gating / learned attention sinks / SwiGLU clamp | Elementwise / enabled / 10 | Same |
| MTP | 1 plain DSA/MoE decoder; shared embedding/head | Same |
| Checkpoint precision | MXFP8 E4M3 weights + uint8 E8M0 scales, 1x32 groups; BF16 embedding/head/gates; FP32 iHC/sinks | Same |
Tokenizer, chat template and generation assets are copied unchanged from the pinned source.
The registered Transformers `indexed_attention` config tag represents the source DSA layers.
The H20 Marlin backend requires width 64. The two untied BF16 token matrices alone
occupy 30,932,992 bytes, exceeding the 30 MB target; the checkpoint is 70,661,788 bytes. TP=4 Marlin also requires a 128-wide
expert intermediate partition, hence a 512-wide full expert intermediate size.
Checkpoint precision matches the source ModelOpt MXFP8 config. Marlin uses BF16
activations at runtime (W8A16).
Transformers on one GPU and vLLM at TP=4 were validated on NVIDIA H20 with English, Chinese and
long DSA prompts. The vLLM run enabled three-token MTP and CUDA Graph. Its separate
test environment used vLLM 0.32.0, Torch 2.13.0+cu130, CUDA 13.0, Transformers 5.19.0
and FlashInfer 0.7.0.post1; the Transformers/SGLang environment is listed below.
| File path | Size |
|------|------|
| model.safetensors | 70.7MB |
### Example usage:
- vLLM (✅ verified)
```bash
vllm serve tiny-random/hy4-preview-fp8 \
--tensor-parallel-size 4 \
--kernel-config '{"linear_backend":"marlin","moe_backend":"marlin"}' \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
--max-model-len 4096 \
--max-num-batched-tokens 512 \
--max-num-seqs 4 \
--gpu-memory-utilization 0.3 \
--compilation-config '{"cudagraph_capture_sizes":[1,2,4,8,16]}'
```
- SGLang (unverified)
- Unverified: SGLang's native Hy4 MXFP8 backend requires SM100+; the available H20 GPUs are SM90. Use the verified vLLM Marlin example on H20.
- This command targets four one-GPU nodes with NODE_RANK=0..3, a shared DIST_INIT_ADDR and an available local NCCL_PORT. TP=4, NEXTN/MTP and CUDA Graph are enabled.
```bash
model_id=tiny-random/hy4-preview-fp8
node_rank=${NODE_RANK:-0}
dist_init_addr=${DIST_INIT_ADDR}
nccl_port=${NCCL_PORT}
export SGLANG_DG_CACHE_DIR=/tmp/hy4-deepgemm-$node_rank
sglang serve \
--model-path $model_id \
--tp-size 4 \
--nnodes 4 \
--node-rank $node_rank \
--dist-init-addr $dist_init_addr \
--nccl-port $nccl_port \
--fp8-gemm-backend deep_gemm \
--moe-runner-backend deep_gemm \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--context-length 4096 \
--max-total-tokens 4096 \
--chunked-prefill-size 512 \
--mem-fraction-static 0.3 \
--cuda-graph-config '{"decode": {"bs": [1, 2, 4]}}' \
--disable-custom-all-reduce
```
- Transformers (✅ verified)
```python
import torch
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer, FineGrainedFP8Config
model_id = "tiny-random/hy4-preview-fp8"
config = AutoConfig.from_pretrained(model_id)
precision = config.quantization_config["quantization"]
# Adapt the loader descriptor in memory; the checkpoint keeps the source ModelOpt config.
config.quantization_config = FineGrainedFP8Config(
quant_method="mxfp8", weight_block_size=(1, 32), scale_fmt="ue8m0",
modules_to_not_convert=[name for name in precision["exclude_modules"] if name != "model.layers.0.mlp.gate"],
).to_dict()
# Transformers 5.19.0 implements Hy4's DSA and learned sinks only with eager attention.
model = AutoModelForCausalLM.from_pretrained(
model_id, config=config, dtype=torch.bfloat16, device_map="cuda", attn_implementation="eager", experts_implementation="eager",
key_mapping={r"\.weight_scale$": ".weight_scale_inv", r"\.(gate_up_proj|down_proj)_scale$": r".\1_scale_inv"},
)
# Source MXFP8 stores E8M0 scale bits in uint8 containers.
for module in model.modules():
for name, param in list(module.named_parameters(recurse=False)):
if name.endswith("_scale_inv") and param.dtype == torch.uint8:
setattr(module, name, torch.nn.Parameter(param.detach().view(torch.float8_e8m0fnu), requires_grad=False))
tokenizer = AutoTokenizer.from_pretrained(model_id)
inputs = tokenizer.apply_chat_template(
[{"role": "user", "content": "Hello"}], tokenize=True,
add_generation_prompt=True, return_dict=True, return_tensors="pt",
).to(model.device)
output = model.generate(**inputs, do_sample=False, max_new_tokens=10)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```
### Codes to create this repo (✅ verified):
Click to expand
```python
import json
import re
from copy import deepcopy
from pathlib import Path
from shutil import copyfile
import torch
from huggingface_hub import hf_hub_download, list_repo_files
from safetensors.torch import save_file
from transformers import AutoConfig, AutoModelForCausalLM, set_seed
source_model_id = "tencent/Hy4-preview-FP8"
save_folder = "/tmp/tiny-random/hy4-preview-fp8"
source_revision = '4215ec29de873a998e849cee902654490c7ff4d1'
set_seed(42)
folder = Path(save_folder)
folder.mkdir(parents=True, exist_ok=True)
raw = json.loads(Path(hf_hub_download(source_model_id, "config.json", revision=source_revision)).read_text())
original_quantization = raw.pop("quantization_config")
raw.update(
hidden_size=64, num_hidden_layers=10, num_attention_heads=8,
intermediate_size=128, moe_intermediate_size=512, n_routed_experts=32,
q_lora_rank=64, qk_nope_head_dim=32, v_head_dim=32, qk_head_dim=96,
layer_types=["indexed_attention"] * 10,
)
raw["indexer_types"] = raw["indexer_types"][:10]
raw["mlp_layer_types"] = raw["mlp_layer_types"][:10]
(folder / "config.json").write_text(json.dumps(raw, indent=2))
config = AutoConfig.from_pretrained(folder)
# Transformers 5.19.0 implements Hy4's DSA and learned sinks only with eager attention.
model = AutoModelForCausalLM.from_config(config, dtype=torch.bfloat16, attn_implementation="eager")
backbone = model.model
last = backbone.layers[-1]
model.model.mtp_layers = torch.nn.ModuleList([torch.nn.ModuleDict({
"enorm": deepcopy(backbone.norm), "hnorm": deepcopy(backbone.norm),
"eh_proj": torch.nn.Linear(2 * config.hidden_size, config.hidden_size, bias=False, dtype=torch.bfloat16),
"input_layernorm": deepcopy(last.input_layernorm), "self_attn": deepcopy(last.self_attn),
"post_attention_layernorm": deepcopy(last.post_attention_layernorm), "mlp": deepcopy(last.mlp),
"final_layernorm": deepcopy(backbone.norm),
})])
model.model.mtp_layers.apply(model._init_weights)
def public_name(name):
for native, source in (("attn_hc", "hc_attn_layer"), ("ffn_hc", "hc_mlp_layer")):
name = re.sub(r"\." + native + r"\.(fn|base|scale)$", r"." + source + r".hc_pre.hc_\1", name)
name = name.replace(".self_attn.gate_proj", ".self_attn.linear_gate")
name = name.replace(".self_attn.sinks", ".self_attn.learnable_sink_param")
for suffix in ("fn", "base", "scale"):
name = name.replace(".hc_head.hc_" + suffix, ".hc_head.hc_head_" + suffix)
return name
state = {}
for name, value in model.state_dict().items():
name = public_name(name)
full_precision = name.endswith((".hc_fn", ".hc_base", ".hc_scale", ".hc_head_fn", ".hc_head_base", ".hc_head_scale"))
full_precision |= name.endswith((".learnable_sink_param", ".e_score_correction_bias")) and not name.startswith("model.mtp_layers.")
state[name] = value.detach().to(torch.float32 if full_precision else torch.bfloat16).cpu().contiguous()
# Keep the source exclusions, pruning module names absent from the reduced model.
source_names = {public_name(name): name for name, _ in model.named_modules()}
source_names.update({public_name(name): name for name, _ in model.named_parameters()})
exclusions = [name for name in original_quantization["quantization"]["exclude_modules"] if name in source_names]
for name in list(state):
excluded = any(name == prefix or name.startswith(prefix + ".") for prefix in exclusions)
expert = name.endswith((".experts.gate_up_proj", ".experts.down_proj"))
linear = name.endswith(".weight") and state[name].ndim == 2
if excluded or not (expert or linear):
continue
weight = state[name].float()
blocks = weight.reshape(*weight.shape[:-1], weight.shape[-1] // 32, 32)
exponent = torch.ceil(torch.log2((blocks.abs().amax(-1) / 448).clamp_min(2.0 ** -127))).clamp(-127, 127)
scale = torch.exp2(exponent)
state[name] = (blocks / scale.unsqueeze(-1)).clamp(-448, 448).reshape(weight.shape).to(torch.float8_e4m3fn).contiguous()
scale_name = name.removesuffix(".weight") + ".weight_scale" if linear else name + "_scale"
state[scale_name] = (exponent + 127).to(torch.uint8).contiguous()
save_file(state, folder / "model.safetensors", metadata={"format": "pt"})
raw["quantization_config"] = original_quantization
(folder / "config.json").write_text(json.dumps(raw, indent=2))
assets = {"tokenizer.json", "tokenizer_config.json", "chat_template.jinja", "generation_config.json", "LICENSE"}
for name in sorted(set(list_repo_files(source_model_id, revision=source_revision)) & assets):
copyfile(hf_hub_download(source_model_id, name, revision=source_revision), folder / name)
```
Click to expand
```text
HYV4ForCausalLM(
(model): HYV4Model(
(embed_tokens): Embedding(120832, 64, padding_idx=120002)
(layers): ModuleList(
(0): HYV4DecoderLayer(
(self_attn): HYV4Attention(
(q_a_proj): Linear(in_features=64, out_features=64, bias=False)
(q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
(q_b_proj): Linear(in_features=64, out_features=768, bias=False)
(kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
(kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
(kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
(o_proj): Linear(in_features=256, out_features=64, bias=False)
(indexer): HYV4Indexer(
(wq_b): Linear(in_features=64, out_features=4096, bias=False)
(wk): Linear(in_features=64, out_features=128, bias=False)
(k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
(weights_proj): Linear(in_features=64, out_features=32, bias=False)
)
(gate_proj): Linear(in_features=64, out_features=256, bias=False)
)
(mlp): HYV4MLP(
(gate_proj): Linear(in_features=64, out_features=128, bias=False)
(up_proj): Linear(in_features=64, out_features=128, bias=False)
(down_proj): Linear(in_features=128, out_features=64, bias=False)
(act_fn): SiLUActivation()
)
(input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
(post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
(attn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
(ffn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
)
(1): HYV4DecoderLayer(
(self_attn): HYV4Attention(
(q_a_proj): Linear(in_features=64, out_features=64, bias=False)
(q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
(q_b_proj): Linear(in_features=64, out_features=768, bias=False)
(kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
(kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
(kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
(o_proj): Linear(in_features=256, out_features=64, bias=False)
(indexer): HYV4Indexer(
(wq_b): Linear(in_features=64, out_features=4096, bias=False)
(wk): Linear(in_features=64, out_features=128, bias=False)
(k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
(weights_proj): Linear(in_features=64, out_features=32, bias=False)
)
(gate_proj): Linear(in_features=64, out_features=256, bias=False)
)
(mlp): HYV4MoE(
(experts): HYV4Experts()
(gate): HYV4TopkRouter()
(shared_experts): HYV4MLP(
(gate_proj): Linear(in_features=64, out_features=512, bias=False)
(up_proj): Linear(in_features=64, out_features=512, bias=False)
(down_proj): Linear(in_features=512, out_features=64, bias=False)
(act_fn): SiLUActivation()
)
)
(input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
(post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
(attn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
(ffn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
)
(2-4): 3 x HYV4DecoderLayer(
(self_attn): HYV4Attention(
(q_a_proj): Linear(in_features=64, out_features=64, bias=False)
(q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
(q_b_proj): Linear(in_features=64, out_features=768, bias=False)
(kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
(kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
(kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
(o_proj): Linear(in_features=256, out_features=64, bias=False)
(gate_proj): Linear(in_features=64, out_features=256, bias=False)
)
(mlp): HYV4MoE(
(experts): HYV4Experts()
(gate): HYV4TopkRouter()
(shared_experts): HYV4MLP(
(gate_proj): Linear(in_features=64, out_features=512, bias=False)
(up_proj): Linear(in_features=64, out_features=512, bias=False)
(down_proj): Linear(in_features=512, out_features=64, bias=False)
(act_fn): SiLUActivation()
)
)
(input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
(post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
(attn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
(ffn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
)
(5): HYV4DecoderLayer(
(self_attn): HYV4Attention(
(q_a_proj): Linear(in_features=64, out_features=64, bias=False)
(q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
(q_b_proj): Linear(in_features=64, out_features=768, bias=False)
(kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
(kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
(kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
(o_proj): Linear(in_features=256, out_features=64, bias=False)
(indexer): HYV4Indexer(
(wq_b): Linear(in_features=64, out_features=4096, bias=False)
(wk): Linear(in_features=64, out_features=128, bias=False)
(k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
(weights_proj): Linear(in_features=64, out_features=32, bias=False)
)
(gate_proj): Linear(in_features=64, out_features=256, bias=False)
)
(mlp): HYV4MoE(
(experts): HYV4Experts()
(gate): HYV4TopkRouter()
(shared_experts): HYV4MLP(
(gate_proj): Linear(in_features=64, out_features=512, bias=False)
(up_proj): Linear(in_features=64, out_features=512, bias=False)
(down_proj): Linear(in_features=512, out_features=64, bias=False)
(act_fn): SiLUActivation()
)
)
(input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
(post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
(attn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
(ffn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
)
(6-8): 3 x HYV4DecoderLayer(
(self_attn): HYV4Attention(
(q_a_proj): Linear(in_features=64, out_features=64, bias=False)
(q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
(q_b_proj): Linear(in_features=64, out_features=768, bias=False)
(kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
(kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
(kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
(o_proj): Linear(in_features=256, out_features=64, bias=False)
(gate_proj): Linear(in_features=64, out_features=256, bias=False)
)
(mlp): HYV4MoE(
(experts): HYV4Experts()
(gate): HYV4TopkRouter()
(shared_experts): HYV4MLP(
(gate_proj): Linear(in_features=64, out_features=512, bias=False)
(up_proj): Linear(in_features=64, out_features=512, bias=False)
(down_proj): Linear(in_features=512, out_features=64, bias=False)
(act_fn): SiLUActivation()
)
)
(input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
(post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
(attn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
(ffn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
)
(9): HYV4DecoderLayer(
(self_attn): HYV4Attention(
(q_a_proj): Linear(in_features=64, out_features=64, bias=False)
(q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
(q_b_proj): Linear(in_features=64, out_features=768, bias=False)
(kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
(kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
(kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
(o_proj): Linear(in_features=256, out_features=64, bias=False)
(indexer): HYV4Indexer(
(wq_b): Linear(in_features=64, out_features=4096, bias=False)
(wk): Linear(in_features=64, out_features=128, bias=False)
(k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
(weights_proj): Linear(in_features=64, out_features=32, bias=False)
)
(gate_proj): Linear(in_features=64, out_features=256, bias=False)
)
(mlp): HYV4MoE(
(experts): HYV4Experts()
(gate): HYV4TopkRouter()
(shared_experts): HYV4MLP(
(gate_proj): Linear(in_features=64, out_features=512, bias=False)
(up_proj): Linear(in_features=64, out_features=512, bias=False)
(down_proj): Linear(in_features=512, out_features=64, bias=False)
(act_fn): SiLUActivation()
)
)
(input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
(post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
(attn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
(ffn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
)
)
(norm): HYV4RMSNorm((64,), eps=1e-05)
(rotary_emb): HYV4RotaryEmbedding()
(hc_head): HYV4HyperHead(
(input_norm): HYV4UnweightedRMSNorm()
)
(mtp_layers): ModuleList(
(0): ModuleDict(
(enorm): HYV4RMSNorm((64,), eps=1e-05)
(hnorm): HYV4RMSNorm((64,), eps=1e-05)
(eh_proj): Linear(in_features=128, out_features=64, bias=False)
(input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
(self_attn): HYV4Attention(
(q_a_proj): Linear(in_features=64, out_features=64, bias=False)
(q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
(q_b_proj): Linear(in_features=64, out_features=768, bias=False)
(kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
(kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
(kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
(o_proj): Linear(in_features=256, out_features=64, bias=False)
(indexer): HYV4Indexer(
(wq_b): Linear(in_features=64, out_features=4096, bias=False)
(wk): Linear(in_features=64, out_features=128, bias=False)
(k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
(weights_proj): Linear(in_features=64, out_features=32, bias=False)
)
(gate_proj): Linear(in_features=64, out_features=256, bias=False)
)
(post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
(mlp): HYV4MoE(
(experts): HYV4Experts()
(gate): HYV4TopkRouter()
(shared_experts): HYV4MLP(
(gate_proj): Linear(in_features=64, out_features=512, bias=False)
(up_proj): Linear(in_features=64, out_features=512, bias=False)
(down_proj): Linear(in_features=512, out_features=64, bias=False)
(act_fn): SiLUActivation()
)
)
(final_layernorm): HYV4RMSNorm((64,), eps=1e-05)
)
)
)
(lm_head): Linear(in_features=64, out_features=120832, bias=False)
)
```