Upload folder using huggingface_hub

.gitattributes

@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 qwen3/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+Hy-MT2/tokenizer.json filter=lfs diff=lfs merge=lfs -text

Hy-MT2/analysis.md

@@ -0,0 +1,164 @@
+# Hy-MT2-7B 在 cccc 上的可行性分析
+
+## 模型概述
+
+| 属性 | 值 |
+|------|-----|
+| 来源 | https://huggingface.co/tencent/Hy-MT2-7B |
+| 架构 | `hunyuan_v1_dense` (HunYuanDenseV1ForCausalLM) |
+| 用途 | 多语言翻译模型（36种语言，支持指令翻译） |
+| 大小 | 7B 参数，BF16 约 16 GB，FP16 同等 |
+| 权重文件 | 4 × safetensors（共 ~16 GB），另有 FP8/GGUF 版本 |
+
+---
+
+## 架构参数对比
+
+| 参数 | Qwen3-8B（已支持） | Hy-MT2-7B（待支持） |
+|------|-------------------|---------------------|
+| `num_hidden_layers` | 36 | **32** |
+| `hidden_size` (D) | 4096 | 4096 |
+| `intermediate_size` (I) | 12288 | **14336** |
+| `num_attention_heads` (H) | 32 | 32 |
+| `num_key_value_heads` (Hkv) | 8 | 8 |
+| `head_dim` (hd) | 128 | 128 |
+| `vocab_size` (V) | 151936 | **128167** |
+| `rope_theta` | 1,000,000 | **10,000** |
+| `rope_scaling` | 无 | dynamic YaRN (factor=1.0) |
+| `hidden_act` | silu | silu |
+| `norm_type` | rms | rms |
+| QK 归一化 (query/key_layernorm) | ✅ | ✅ |
+| `tie_word_embeddings` | ❌ | **✅ (lm_head = embed)** |
+
+---
+
+## cccc 算子支持情况
+
+| 功能 | cccc 算子 | 支持状态 |
+|------|-----------|---------|
+| Token Embedding | `embed` | ✅ |
+| RMSNorm | `rmsNorm` | ✅ |
+| QK Norm | `rmsNorm`（作用于 Q/K） | ✅ |
+| Q/K/V/O 线性投影 | `batchedMul` | ✅ |
+| GQA（K/V 头复制） | `tile` + `reshapeBatch` | ✅ |
+| RoPE 位置编码 | `rope`, `ropeCosTbl`, `ropeSinTbl` | ✅ |
+| KV 缓存 | `kvcache` | ✅ |
+| Scaled Dot-Product Attention | `attention` | ✅ |
+| SwiGLU FFN | `silu` + `elementMul` + `batchedMul` | ✅ |
+| Tied Embeddings (lm_head=embed) | 同一 `MatrixWithName` 复用 | ✅（见 ini） |
+
+**结论：架构层面 cccc 完全兼容 Hy-MT2-7B，无需新增算子。**
+
+---
+
+## 需要适配的部分
+
+### 1. 新建 ini 配置（简单，已提供模板）
+- 修改层数 32、FFN 维度 14336、词表 128167、rope_theta 10000.0
+- W_lm_head 复用 W_emb（tied embeddings）
+
+### 2. 权重转换（Python 脚本，已提供 `convert_weights.py`）
+- 从 HuggingFace safetensors 下载权重
+- 转置并重命名矩阵 → cccc 的 `named_weights` 格式
+- 保存为 `INIReaderBin` 格式的 `.bin` 文件
+
+**权重名称映射（HF → cccc）：**
+
+| HuggingFace 名称 | cccc 名称 |
+|-----------------|----------|
+| `model.embed_tokens.weight` [V,D] → 转置 | `W_emb` [D,V] |
+| `lm_head.weight` [V,D] → 转置（与 embed 相同） | `W_lm_head` [D,V] |
+| `model.norm.weight` | `W_rms_final` |
+| `model.layers.{i}.input_layernorm.weight` | `W_rms_attn_{i}` |
+| `model.layers.{i}.self_attn.q_proj.weight` [Dq,D] → 转置 | `W_q_{i}` [D,Dq] |
+| `model.layers.{i}.self_attn.k_proj.weight` [Dkv,D] → 转置 | `W_k_{i}` [D,Dkv] |
+| `model.layers.{i}.self_attn.v_proj.weight` [Dkv,D] → 转置 | `W_v_{i}` [D,Dkv] |
+| `model.layers.{i}.self_attn.o_proj.weight` [D,D] → 转置 | `W_o_{i}` [D,D] |
+| `model.layers.{i}.self_attn.query_layernorm.weight` | `W_qnorm_{i}` |
+| `model.layers.{i}.self_attn.key_layernorm.weight` | `W_knorm_{i}` |
+| `model.layers.{i}.post_attention_layernorm.weight` | `W_rms_ffn_{i}` |
+| `model.layers.{i}.mlp.gate_proj.weight` [I,D] → 转置 | `W_gate_{i}` [D,I] |
+| `model.layers.{i}.mlp.up_proj.weight` [I,D] → 转置 | `W_up_{i}` [D,I] |
+| `model.layers.{i}.mlp.down_proj.weight` [D,I] → 转置 | `W_down_{i}` [I,D] |
+
+### 3. cccc-llm.cpp 适配（需修改 C++ 代码）
+
+这是**唯一需要改 C++ 的地方**，涉及两处硬编码：
+
+#### 3a. EOS token 硬编码（`run_generate` 函数）
+```cpp
+// 当前（Qwen3 硬编码）：
+const int EOS_IM_END = 151645;
+const int EOS_EOT    = 151643;
+
+// Hy-MT2 需要：
+const int EOS = 127960;  // <|eos|>
+```
+
+#### 3b. 对话格式（`make_turn` + `llm_init`）
+
+**Hy-MT2 Chat Template（来自 chat_template.jinja）：**
+```
+System:    <|startoftext|>{content}<|extra_4|>
+User(1st): {content}<|extra_0|>       ← 首条 user（system 后）
+User(nth): <|startoftext|>{content}<|extra_0|>
+Assistant: {content}<|eos|>           ← 无 start marker，直接输出
+```
+
+**当前 cccc 格式（ChatML）：**
+```
+<|im_start|>system\n{content}<|im_end|>\n
+<|im_start|>user\n{content}<|im_end|>\n
+<|im_start|>assistant\n
+```
+
+需修改 `make_turn`、`asst_header`、`im_end_nl` 的构造逻辑。
+
+> 建议：在 `[llm]` section 里增加 `chat_format=` 参数支持不同格式，避免每次改 C++ 代码。
+
+---
+
+## RoPE Scaling 说明
+
+Hy-MT2-7B 的 `rope_scaling.type = "dynamic"`, `factor = 1.0`。
+
+- **factor=1.0 时**：YaRN/dynamic NTK 缩放因子为 1，与普通 RoPE 等效
+- **对于 ≤1024 token 的上下文**：用标准 RoPE（theta=10000）完全没有精度损失
+- **如需超长上下文（>32k）**：才需实现 dynamic/YaRN 频率插值
+
+当前 cccc 的 `ropeCosTbl/ropeSinTbl` 使用标准 RoPE，对翻译任务（通常几百 token）完全够用。
+
+---
+
+## 实施路线图
+
+```
+步骤 1：下载权重
+  huggingface-cli download tencent/Hy-MT2-7B --local-dir ./Hy-MT2-weights
+  （或直接下载 GGUF 版 Hy-MT2-7B.gguf 用 llama.cpp 验证先）
+
+步骤 2：运行转换脚本
+  python convert_weights.py \
+      --hf_dir ./Hy-MT2-weights \
+      --output hy_mt2_7b_fp16_cccc.bin
+
+步骤 3：复制 tokenizer.json
+  cp ./Hy-MT2-weights/tokenizer.json .
+
+步骤 4：修改 cccc-llm.cpp（EOS tokens + chat format）
+  → 详见 analysis.md §3
+
+步骤 5：重编 cccc-llm.dll，加载 hy_mt2_7b_fp16.ini 测试
+```
+
+---
+
+## 预估难度
+
+| 工作项 | 难度 | 工时估算 |
+|--------|------|----------|
+| 写 ini 配置 | 低（复制 qwen3 改参数）| 已完成 |
+| 权重转换脚本 | 低（已提供） | 已完成 |
+| cccc-llm.cpp EOS 修改 | 低（改 2 个常量）| 0.5h |
+| cccc-llm.cpp 格式修改 | 中（改对话拼接逻辑）| 1-2h |
+| 下载权重 + 验证运行 | 低 | 依网速 |

Hy-MT2/config.json

@@ -0,0 +1,60 @@
+{
+  "add_classification_head": false,
+  "architectures": [
+    "HunYuanDenseV1ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.1,
+  "attention_head_dim": 128,
+  "bos_token_id": 127958,
+  "cla_share_factor": 2,
+  "class_num": 0,
+  "dense_list": [
+    4096,
+    0
+  ],
+  "torch_dtype": "bfloat16",
+  "eos_token_id": 127960,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "im_end_id": 5,
+  "im_newline_id": 11,
+  "im_start_id": 4,
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "mask_init_id": 12,
+  "max_position_embeddings": 262144,
+  "mlp_bias": false,
+  "model_type": "hunyuan_v1_dense",
+  "norm_type": "rms",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "org_vocab_size": 128167,
+  "pad_id": 127961,
+  "pad_token_id": 127961,
+  "pool_type": "last",
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": {
+    "alpha": 1000.0,
+    "beta_fast": 32,
+    "beta_slow": 1,
+    "factor": 1.0,
+    "mscale": 1.0,
+    "mscale_all_dim": 1.0,
+    "type": "dynamic"
+  },
+  "rope_theta": 10000.0,
+  "sep_token_id": 127962,
+  "text_end_id": 7,
+  "text_start_id": 6,
+  "tie_word_embeddings": true,
+  "transformers_version": "4.57.6",
+  "use_cache": true,
+  "use_cla": false,
+  "use_qk_norm": true,
+  "use_rotary_pos_emb": true,
+  "vocab_size": 128167
+}

Hy-MT2/convert_weights.py

@@ -0,0 +1,165 @@
+"""
+convert_weights.py — 将 Hy-MT2-7B HuggingFace safetensors 转换为 cccc INIReaderBin 格式
+
+用法：
+  python convert_weights.py --hf_dir <HF权重目录> --output hy_mt2_7b_fp16_cccc.bin [--dtype half]
+
+依赖：
+  pip install safetensors numpy
+
+INIReaderBin 格式说明：
+  [0..31]       头部 "CFG_BIN INI"（32字节）
+  [32..39]      INI 索引文本长度（uint64 LE）
+  [40..40+N-1]  INI 索引文本（每个键对应 "offset,length" 偏移和字节数）
+  [40+N..]      二进制数据（各权重依次拼接）
+"""
+
+import argparse
+import os
+import struct
+import sys
+import numpy as np
+
+try:
+    import torch
+except ImportError:
+    print("请先安装: pip install torch")
+    sys.exit(1)
+
+try:
+    from safetensors import safe_open
+except ImportError:
+    print("请先安装: pip install safetensors")
+    sys.exit(1)
+
+
+# ── 权重名称映射：HuggingFace → cccc ─────────────────────────────────────────
+# 格式: (hf_name, cccc_name, needs_transpose)
+# needs_transpose=False：cccc 使用列主序 BLAS（第一维最快），HF [out,in] C-order
+# 在 cccc (in, out, 1, 1) 矩阵中 element(in_i, out_j) 在 in_i + out_j*in_features，
+# 恰好与 HF [out,in] C-order 的 out_j*in + in_i 相同（加法交换律），因此不需要转置。
+def build_weight_map(num_layers=32):
+    mapping = []
+    # 全局权重
+    mapping.append(("model.embed_tokens.weight", "W_emb", False))  # [V,D] 直接存，embed op 按 id*D 取行
+    mapping.append(("lm_head.weight",            "W_lm_head", False))# tied with embed, 保持一致
+    mapping.append(("model.norm.weight",          "W_rms_final", False))
+
+    for i in range(num_layers):
+        pfx = f"model.layers.{i}"
+        mapping.extend([
+            (f"{pfx}.input_layernorm.weight",          f"W_rms_attn_{i}", False),
+            (f"{pfx}.self_attn.q_proj.weight",         f"W_q_{i}",        False),
+            (f"{pfx}.self_attn.k_proj.weight",         f"W_k_{i}",        False),
+            (f"{pfx}.self_attn.v_proj.weight",         f"W_v_{i}",        False),
+            (f"{pfx}.self_attn.o_proj.weight",         f"W_o_{i}",        False),
+            (f"{pfx}.self_attn.query_layernorm.weight",f"W_qnorm_{i}",    False),
+            (f"{pfx}.self_attn.key_layernorm.weight",  f"W_knorm_{i}",    False),
+            (f"{pfx}.post_attention_layernorm.weight", f"W_rms_ffn_{i}",  False),
+            (f"{pfx}.mlp.gate_proj.weight",            f"W_gate_{i}",     False),
+            (f"{pfx}.mlp.up_proj.weight",              f"W_up_{i}",       False),
+            (f"{pfx}.mlp.down_proj.weight",            f"W_down_{i}",     False),
+        ])
+    return mapping
+
+
+# ── INIReaderBin 序列化 ───────────────────────────────────────────────────────
+def build_inireaderbin(named_blobs: dict) -> bytes:
+    """
+    named_blobs: {key: bytes}
+    返回完整的 INIReaderBin 二进制内容。
+    """
+    # 1. 拼接所有二进制 blob，记录 (offset, length)
+    blob_parts = []
+    index_lines = []
+    offset = 0
+    for key, data in named_blobs.items():
+        blob_parts.append(data)
+        index_lines.append(f"{key} = {offset},{len(data)}\n")
+        offset += len(data)
+
+    ini_text = "".join(index_lines)
+    ini_bytes = ini_text.encode("utf-8")
+    binary_content = b"".join(blob_parts)
+
+    # 2. 构建文件
+    header = b"CFG_BIN INI" + b"\x00" * (32 - len("CFG_BIN INI"))
+    size_ini = struct.pack("<Q", len(ini_bytes))
+    return header + size_ini + ini_bytes + binary_content
+
+
+def convert(hf_dir: str, output_path: str, dtype_str: str = "bfloat16"):
+    mapping = build_weight_map(num_layers=32)
+    hf_to_cccc = {hf: (cccc, tp) for hf, cccc, tp in mapping}
+
+    # 扫描所有 safetensors 文件
+    shard_files = sorted(
+        f for f in os.listdir(hf_dir) if f.endswith(".safetensors")
+    )
+    if not shard_files:
+        print(f"在 {hf_dir} 中未找到 .safetensors 文件")
+        sys.exit(1)
+    print(f"找到 {len(shard_files)} 个 safetensors 分片：{shard_files}")
+
+    # 打开所有分片（lazy load）
+    handles = {}
+    for sf in shard_files:
+        path = os.path.join(hf_dir, sf)
+        h = safe_open(path, framework="pt", device="cpu")
+        for key in h.keys():
+            handles[key] = h
+
+    named_blobs = {}
+    # 元信息（data_type 与 cccc ini 中的 data_type 对应）
+    named_blobs["data_type"] = dtype_str.encode()
+    named_blobs["named_weights"] = b"1"
+
+    missing = []
+    for hf_name, cccc_name, needs_transpose in mapping:
+        if hf_name not in handles:
+            missing.append(hf_name)
+            continue
+
+        tensor = handles[hf_name].get_tensor(hf_name)  # torch.Tensor, original dtype (may be bfloat16)
+
+        if needs_transpose and tensor.ndim == 2:
+            tensor = tensor.T.contiguous()
+
+        # 转换到目标精度
+        if dtype_str == "half":
+            tensor = tensor.to(torch.float16)
+            blob = tensor.numpy().tobytes()
+        elif dtype_str == "bfloat16":
+            tensor = tensor.to(torch.bfloat16)
+            # numpy 不支持 bfloat16；用 view(int16) 直接取原始字节
+            blob = tensor.view(torch.int16).numpy().tobytes()
+        elif dtype_str == "float":
+            tensor = tensor.to(torch.float32)
+            blob = tensor.numpy().tobytes()
+        else:
+            raise ValueError(f"不支持的 dtype: {dtype_str}")
+        named_blobs[f"weight_{cccc_name}"] = blob
+        print(f"  {hf_name:60s} → {cccc_name:30s}  shape={tensor.shape}  {len(blob)//1024}KB")
+
+    if missing:
+        print(f"\n警告：以下权重未找到：{missing}")
+
+    print(f"\n正在写入 {output_path} ...")
+    content = build_inireaderbin(named_blobs)
+    with open(output_path, "wb") as f:
+        f.write(content)
+    print(f"完成！文件大小：{os.path.getsize(output_path) / 1024**3:.2f} GB")
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Hy-MT2-7B → cccc bin 转换工具")
+    parser.add_argument("--hf_dir",  required=True, help="HuggingFace 权重目录（含 *.safetensors）")
+    parser.add_argument("--output",  default="hy_mt2_7b_bf16_cccc.bin", help="输出 .bin 文件路径")
+    parser.add_argument("--dtype",   default="bfloat16", choices=["half", "bfloat16", "float"], help="存储精度（默认 bfloat16，无损）")
+    args = parser.parse_args()
+
+    convert(args.hf_dir, args.output, args.dtype)
+
+
+if __name__ == "__main__":
+    main()

Hy-MT2/hy_mt2_7b_bf16_cccc.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:59f8cebc4f048be063bee5969dfad174b9f1921e28dc41c6a069947645b02b37
+size 16059093959

Hy-MT2/hy_mt2_7b_fp16.ini

@@ -0,0 +1,244 @@
+[train]
+load_file = hy_mt2_7b_bf16_cccc.bin
+load_net = 1
+WorkType = 0
+train_epochs = 0
+Batch = 1
+output_net = 1
+gpu = 1
+mp = 1
+mp_device = 0
+named_weights = 1
+data_type = bfloat16
+#disable_activation_memory_reuse=1
+
+[llm]
+tokenizer = tokenizer.json
+
+# EOS token: <|eos|> = 127960
+eos_tokens = 127960
+
+# No thinking support
+think_open_id = -1
+think_close_id = -1
+no_think_str =
+
+# Chat format for Hy-MT2-7B
+# System:  <|startoftext|>{content}<|extra_4|>
+# User:    <|startoftext|>{content}<|extra_0|>
+# Asst:    {content}<|eos|>
+sys_prefix = <|startoftext|>
+sys_suffix = <|extra_4|>
+user_prefix = <|startoftext|>
+user_suffix = <|extra_0|>
+asst_prefix =
+asst_suffix = <|eos|>
+
+[net]
+net_num = 2
+
+# ── prefill net (T tokens at once) ──────────────────────────────────────────
+structure0='
+    D    = 4096;
+    Dq   = 4096;
+    Dkv  = 1024;
+    H    = 32;
+    Hkv  = 8;
+    hd   = 128;
+    I    = 14336;
+    V    = 128167;
+    T    = 1024;
+    HB   = 32;
+    HkvB = 8;
+
+    W_emb = MatrixWithName("W_emb", D, 1, 1, V);
+    token_ids = MatrixF(T, 1, 1, 1);
+    X = embed(token_ids, W_emb);
+
+    cos_tab = ropeCosTbl(T, hd, 10000.0);
+    sin_tab = ropeSinTbl(T, hd, 10000.0);
+
+    for (i = 0; i < 32; i++)
+    {
+        W_rms_attn = MatrixWithName("W_rms_attn_" + to_string(i), D, 1);
+        X_norm = rmsNorm(X, W_rms_attn);
+
+        W_q = MatrixWithName("W_q_" + to_string(i), D, Dq, 1, 1);
+        Q_dq = batchedMul(W_q, X_norm, 1, 0);
+        Q_hHT = reshape(Q_dq, {hd, H, T, 1});
+        W_qnorm = MatrixWithName("W_qnorm_" + to_string(i), hd, 1);
+        Q_qnorm = rmsNorm(Q_hHT, W_qnorm);
+        Q_hTH = permute(Q_qnorm, {0, 2, 1, 3});
+        Q_hb = reshapeBatch(Q_hTH, {hd, T, 1, HB});
+        Q_r = rope(Q_hb, cos_tab, sin_tab);
+
+        W_k = MatrixWithName("W_k_" + to_string(i), D, Dkv, 1, 1);
+        K_dkv = batchedMul(W_k, X_norm, 1, 0);
+        K_hHkvT = reshape(K_dkv, {hd, Hkv, T, 1});
+        W_knorm = MatrixWithName("W_knorm_" + to_string(i), hd, 1);
+        K_knorm = rmsNorm(K_hHkvT, W_knorm);
+        K_hTHkv = permute(K_knorm, {0, 2, 1, 3});
+        K_hb = reshapeBatch(K_hTHkv, {hd, T, 1, HkvB});
+        K_r = rope(K_hb, cos_tab, sin_tab);
+
+        W_v = MatrixWithName("W_v_" + to_string(i), D, Dkv, 1, 1);
+        V_dkv = batchedMul(W_v, X_norm, 1, 0);
+        V_hHkvT = reshape(V_dkv, {hd, Hkv, T, 1});
+        V_hTHkv = permute(V_hHkvT, {0, 2, 1, 3});
+        V_hb = reshapeBatch(V_hTHkv, {hd, T, 1, HkvB});
+
+        Kcache = MatrixWithName("Kcache_" + to_string(i), hd, T, 1, HkvB);
+        setIsWeight(Kcache, 0);
+        registerMatrix("Kcache_" + to_string(i), Kcache);
+        K_cached = kvcache(K_r, Kcache);
+
+        Vcache = MatrixWithName("Vcache_" + to_string(i), hd, T, 1, HkvB);
+        setIsWeight(Vcache, 0);
+        registerMatrix("Vcache_" + to_string(i), Vcache);
+        V_cached = kvcache(V_hb, Vcache);
+
+        K_r2   = reshapeBatch(K_cached,  {hd, T, HkvB, 1});
+        K_r3   = tile(K_r2, {1, 1, 1, 4});
+        K_r4   = permute(K_r3, {0, 1, 3, 2});
+        K_tiled = reshapeBatch(K_r4, {hd, T, 1, HB});
+
+        V_r2   = reshapeBatch(V_cached, {hd, T, HkvB, 1});
+        V_r3   = tile(V_r2, {1, 1, 1, 4});
+        V_r4   = permute(V_r3, {0, 1, 3, 2});
+        V_tiled = reshapeBatch(V_r4, {hd, T, 1, HB});
+
+        Attn = attention(Q_r, K_tiled, V_tiled, hd, 1);
+
+        Attn_hTH = reshapeBatch(Attn, {hd, T, H, 1});
+        Attn_hHT = permute(Attn_hTH, {0, 2, 1, 3});
+        Attn_flat = reshape(Attn_hHT, {Dq, T, 1, 1});
+        W_o = MatrixWithName("W_o_" + to_string(i), Dq, D, 1, 1);
+        O_out = batchedMul(W_o, Attn_flat, 1, 0);
+
+        R1 = X + O_out;
+
+        W_rms_ffn = MatrixWithName("W_rms_ffn_" + to_string(i), D, 1);
+        R1_norm = rmsNorm(R1, W_rms_ffn);
+
+        W_gate = MatrixWithName("W_gate_" + to_string(i), D, I, 1, 1);
+        W_up   = MatrixWithName("W_up_" + to_string(i), D, I, 1, 1);
+        gate_out = silu(batchedMul(W_gate, R1_norm, 1, 0));
+        up_out   = batchedMul(W_up, R1_norm, 1, 0);
+        gated    = elementMul(gate_out, up_out);
+        W_down = MatrixWithName("W_down_" + to_string(i), I, D, 1, 1);
+        ffn_out = batchedMul(W_down, gated, 1, 0);
+
+        X = R1 + ffn_out;
+    }
+
+    W_rms_final = MatrixWithName("W_rms_final", D, 1);
+    X_final = rmsNorm(X, W_rms_final);
+    // tie_word_embeddings=true: reshape W_emb to (D, V, 1, 1) view for lm_head
+    W_lm_head = reshapeBatch(W_emb, {D, V, 1, 1});
+    logits = batchedMul(W_lm_head, X_final, 1, 0);
+
+    setXY(token_ids, logits);
+'
+
+# ── decode net (single token step) ──────────────────────────────────────────
+structure1='
+    D    = 4096;
+    Dq   = 4096;
+    Dkv  = 1024;
+    H    = 32;
+    Hkv  = 8;
+    hd   = 128;
+    I    = 14336;
+    V    = 128167;
+    T    = 1;
+    T_kv = 1024;
+    HB   = 32;
+    HkvB = 8;
+
+    W_emb = MatrixWithName("W_emb", D, 1, 1, V);
+    token_ids = MatrixF(T, 1, 1, 1);
+    X = embed(token_ids, W_emb);
+
+    cos_tab = ropeCosTbl(T_kv, hd, 10000.0);
+    sin_tab = ropeSinTbl(T_kv, hd, 10000.0);
+
+    for (i = 0; i < 32; i++)
+    {
+        W_rms_attn = MatrixWithName("W_rms_attn_" + to_string(i), D, 1);
+        X_norm = rmsNorm(X, W_rms_attn);
+
+        W_q = MatrixWithName("W_q_" + to_string(i), D, Dq, 1, 1);
+        Q_dq = batchedMul(W_q, X_norm, 1, 0);
+        Q_hHT = reshape(Q_dq, {hd, H, T, 1});
+        W_qnorm = MatrixWithName("W_qnorm_" + to_string(i), hd, 1);
+        Q_qnorm = rmsNorm(Q_hHT, W_qnorm);
+        Q_hTH = permute(Q_qnorm, {0, 2, 1, 3});
+        Q_hb = reshapeBatch(Q_hTH, {hd, T, 1, HB});
+        Q_r = rope(Q_hb, cos_tab, sin_tab);
+
+        W_k = MatrixWithName("W_k_" + to_string(i), D, Dkv, 1, 1);
+        K_dkv = batchedMul(W_k, X_norm, 1, 0);
+        K_hHkvT = reshape(K_dkv, {hd, Hkv, T, 1});
+        W_knorm = MatrixWithName("W_knorm_" + to_string(i), hd, 1);
+        K_knorm = rmsNorm(K_hHkvT, W_knorm);
+        K_hTHkv = permute(K_knorm, {0, 2, 1, 3});
+        K_hb = reshapeBatch(K_hTHkv, {hd, T, 1, HkvB});
+        K_r = rope(K_hb, cos_tab, sin_tab);
+
+        W_v = MatrixWithName("W_v_" + to_string(i), D, Dkv, 1, 1);
+        V_dkv = batchedMul(W_v, X_norm, 1, 0);
+        V_hHkvT = reshape(V_dkv, {hd, Hkv, T, 1});
+        V_hTHkv = permute(V_hHkvT, {0, 2, 1, 3});
+        V_hb = reshapeBatch(V_hTHkv, {hd, T, 1, HkvB});
+
+        // 复用 prefill net 预载的共享 KV cache
+        Kcache = MatrixWithName("Kcache_" + to_string(i), hd, T_kv, 1, HkvB);
+        setIsWeight(Kcache, 0);
+        K_cached = kvcache(K_r, Kcache);
+
+        Vcache = MatrixWithName("Vcache_" + to_string(i), hd, T_kv, 1, HkvB);
+        setIsWeight(Vcache, 0);
+        V_cached = kvcache(V_hb, Vcache);
+
+        K_r2   = reshapeBatch(K_cached, {hd, T_kv, HkvB, 1});
+        K_r3   = tile(K_r2, {1, 1, 1, 4});
+        K_r4   = permute(K_r3, {0, 1, 3, 2});
+        K_tiled = reshapeBatch(K_r4, {hd, T_kv, 1, HB});
+
+        V_r2   = reshapeBatch(V_cached, {hd, T_kv, HkvB, 1});
+        V_r3   = tile(V_r2, {1, 1, 1, 4});
+        V_r4   = permute(V_r3, {0, 1, 3, 2});
+        V_tiled = reshapeBatch(V_r4, {hd, T_kv, 1, HB});
+
+        Attn = attention(Q_r, K_tiled, V_tiled, hd, 1);
+
+        Attn_hTH = reshapeBatch(Attn, {hd, T, H, 1});
+        Attn_hHT = permute(Attn_hTH, {0, 2, 1, 3});
+        Attn_flat = reshape(Attn_hHT, {Dq, T, 1, 1});
+        W_o = MatrixWithName("W_o_" + to_string(i), Dq, D, 1, 1);
+        O_out = batchedMul(W_o, Attn_flat, 1, 0);
+
+        R1 = X + O_out;
+
+        W_rms_ffn = MatrixWithName("W_rms_ffn_" + to_string(i), D, 1);
+        R1_norm = rmsNorm(R1, W_rms_ffn);
+
+        W_gate = MatrixWithName("W_gate_" + to_string(i), D, I, 1, 1);
+        W_up   = MatrixWithName("W_up_" + to_string(i), D, I, 1, 1);
+        gate_out = silu(batchedMul(W_gate, R1_norm, 1, 0));
+        up_out   = batchedMul(W_up, R1_norm, 1, 0);
+        gated    = elementMul(gate_out, up_out);
+        W_down = MatrixWithName("W_down_" + to_string(i), I, D, 1, 1);
+        ffn_out = batchedMul(W_down, gated, 1, 0);
+
+        X = R1 + ffn_out;
+    }
+
+    W_rms_final = MatrixWithName("W_rms_final", D, 1);
+    X_final = rmsNorm(X, W_rms_final);
+    // tie_word_embeddings=true: reshape W_emb to (D, V, 1, 1) view for lm_head
+    W_lm_head = reshapeBatch(W_emb, {D, V, 1, 1});
+    logits = batchedMul(W_lm_head, X_final, 1, 0);
+
+    setXY(token_ids, logits);
+'

Hy-MT2/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4185389a69284fdd3ebf737c6120bbeee36cbed9763151fa5bac5986a39e05d4
+size 16388221

qwen3/qwen3_0.5b_fp16.ini

@@ -14,6 +14,22 @@ data_type = half
 [llm]
 tokenizer = tokenizer.json
 
+# EOS tokens: <|im_end|>=151645, <|endoftext|>=151643
+eos_tokens = 151645,151643
+
+# Thinking tokens (Qwen3)
+think_open_id = 151667
+think_close_id = 151668
+no_think_str = <think>\n</think>\n
+
+# Chat format (ChatML)
+sys_prefix = <|im_start|>system\n
+sys_suffix = <|im_end|>\n
+user_prefix = <|im_start|>user\n
+user_suffix = <|im_end|>\n
+asst_prefix = <|im_start|>assistant\n
+asst_suffix = <|im_end|>\n
+
 [net]
 net_num = 2
 

qwen3/qwen3_8b_fp16.ini

@@ -10,10 +10,27 @@ mp = 1
 mp_device = 0
 named_weights = 1
 data_type = half
+#disable_activation_memory_reuse=1
 
 [llm]
 tokenizer = tokenizer.json
 
+# EOS tokens: <|im_end|>=151645, <|endoftext|>=151643
+eos_tokens = 151645,151643
+
+# Thinking tokens (Qwen3)
+think_open_id = 151667
+think_close_id = 151668
+no_think_str = <think>\n</think>\n
+
+# Chat format (ChatML)
+sys_prefix = <|im_start|>system\n
+sys_suffix = <|im_end|>\n
+user_prefix = <|im_start|>user\n
+user_suffix = <|im_end|>\n
+asst_prefix = <|im_start|>assistant\n
+asst_suffix = <|im_end|>\n
+
 [net]
 net_num = 2