onnx-int4

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+onnx-int4/model.onnx.data filter=lfs diff=lfs merge=lfs -text

onnx-int4/README.md

@@ -0,0 +1,48 @@
+# AgenticASR-Refiner ONNX (INT4)
+
+INT4 weight-only quantization of the Optimum ONNX export of
+`Andrew0425/AgenticASR-Refiner` (Llama-based ASR transcript refiner, 24 layers,
+hidden 1536, GQA 16/2, head_dim 128, vocab 130560).
+
+- `model.onnx` + `model.onnx.data` — INT4 (MatMulNBits, block size 32, symmetric,
+  accuracy level 4), ~1.3 GB. Graph I/O is identical to the fp32 export
+  (`input_ids` / `attention_mask` / `position_ids` / `past_key_values.*`).
+- Quantized with `onnxruntime 1.24.4` `MatMulNBitsQuantizer`
+  (`onnxruntime.quantization.matmul_nbits_quantizer`).
+- Requires ONNX Runtime >= 1.20 (CPU EP supports `MatMulNBits`).
+
+Tokenization uses the original repo tokenizer (`tokenizer.json` at the repo root).
+
+## Verified generation (ONNX Runtime 1.28, CPU)
+
+| Input | Output |
+|---|---|
+| 我今天去了公司然后然后开了个会，明天再去见张总 | 我今天去了公司然后开了个会，明天再去见张总 |
+| 你好你好你好我是那个小李啊 电话是13800138000 | 你好我是小李，电话是13800138000 |
+
+## Usage
+
+```python
+import numpy as np
+import onnxruntime as ort
+from transformers import AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("Andrew0425/AgenticASR-Refiner")
+session = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
+input_names = [i.name for i in session.get_inputs()]
+kv_names = [n for n in input_names if n.startswith("past_key_values")]
+
+prompt = tokenizer.apply_chat_template(
+    [{"role": "system", "content": "你是 ASR 文本纠错助手。保留原意，最小修改。"},
+     {"role": "user", "content": "我今天去了公司然后然后开了个会"}],
+    tokenize=False, add_generation_prompt=True,
+)
+ids = tokenizer(prompt).input_ids
+pasts = [np.zeros((1, 2, 0, 128), dtype=np.float32) for _ in kv_names]
+
+# prefill, then loop decode: feed input_ids/attention_mask/position_ids + pasts
+```
+
+Note: `optimum-onnx 0.1.0` cannot yet run this checkpoint (dummy KV-cache shape
+uses `hidden_size // num_heads` = 96 instead of `head_dim` = 128); use the raw
+ONNX Runtime loop above, or a future fixed version of optimum-onnx.

onnx-int4/config.json

@@ -0,0 +1,34 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dtype": "float32",
+  "eos_token_id": 1,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 1536,
+  "initializer_range": 0.02,
+  "intermediate_size": 4608,
+  "max_position_embeddings": 131072,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "num_key_value_heads": 2,
+  "pad_token_id": 1,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 5000000,
+    "rope_type": "default"
+  },
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.6",
+  "use_cache": true,
+  "vocab_size": 130560
+}

onnx-int4/generation_config.json

@@ -0,0 +1,14 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "do_sample": true,
+  "eos_token_id": [
+    1,
+    1,
+    130073
+  ],
+  "pad_token_id": 1,
+  "temperature": 0.9,
+  "top_p": 0.95,
+  "transformers_version": "4.57.6"
+}

onnx-int4/model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:602b5e2192cf05f9656f419fe9a9cc7a45c23ced3659aa5441fbef8ba8387f85
+size 1144846

onnx-int4/model.onnx.data

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b93ed739800441b1e11563b8cf10bb0643e4db5e7cd82c35b305e1af55a7ed83
+size 1352472576