Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,71 @@
+---
+license: apache-2.0
+tags:
+- executorch
+- xnnpack
+- pte
+- on-device
+- text-generation
+base_model:
+- Qwen/Qwen3-4B
+---
+# Qwen3-4B — ExecuTorch XNNPACK 8da4w + 8-bit embedding
+
+`qwen3_4b_xnnpack_8da4w_e8.pte` (2469.4 MB)
+
+- **Source**: Qwen/Qwen3-4B
+- **License**: Apache-2.0
+- **Quantization**: 8da4w linear + 8-bit embedding (`embedding_quantize: "8,0"`)
+- **Export**: executorch 1.4.0 `export_llm`, static shape (seq_len=1), max_seq_length 2048,
+  XNNPACK extended_ops
+- **Config**: `llm_params/qwen3_4b_xnnpack_8da4w_e8.yaml`
+
+## Verification (Mac arm64, 2026-08-21)
+
+`llm_params/gen_static.py`, token-by-token prefill then greedy decode, a fresh process per
+prompt so no answer is read through the previous one's cache:
+
+| prompt | answer |
+|---|---|
+| capital of France? | opens a `<think>` block and reasons before answering |
+| 17 times 4? | same, working the multiplication out in the block |
+
+Decode **28.9 tok/s**, from one pass over every model on this shelf with nothing else
+running. That matters more than it sounds: the same file measured a quarter of its rate
+while an export was running alongside.
+
+Chat template: ChatML, bos 151643, eos [151645, 151643].
+
+Not measured on a phone.
+
+## Three things checked before exporting
+
+- **`use_sdpa_with_kv_cache` is on.** Upstream's `qwen3_5` config leaves it off with no
+  reason given while the equally hybrid `lfm2` config has it on; measured on Qwen3.5-2B in
+  one run, that is 8.20 tok/s against 16.64.
+- **`dim` and `hidden_dim` both divide by the quantizer's group size.** 8da4w only touches a
+  linear whose in_features divide by it, and skips the rest silently — SmolLM2-135M, which
+  is 576 wide, came out at 475 MB against fp32's 540 with no warning at all.
+- **Every field of the params json is read by the generic path**, via
+  `convert/check_params_used.py`. SmolLM3 sets `no_rope_layer_interval`, which `ModelArgs`
+  declares and only the MLX and Qualcomm backends read, and it exports fine and then repeats
+  a single word forever.
+
+## Running it
+
+```bash
+python llm_params/gen_static.py \
+  --pte qwen3_4b_xnnpack_8da4w_e8.pte \
+  --tokenizer tokenizer.json \
+  --prompt $'<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n' \
+  --eos_ids "[151645, 151643]"
+```
+
+The 8-bit embedding needs `from executorch.kernels import quantized` before the program is
+loaded, and `portable_lib._load_for_executorch` rather than `executorch.runtime`. Without
+that the method will not load at all — `kernel
+'quantized_decomposed::embedding_byte.dtype_out' not found` — which reads like a broken
+export rather than a runtime missing its kernels.
+
+(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models) ·
+iOS sample: [executorch-samples](https://github.com/john-rocky/executorch-samples))