Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -24,7 +24,7 @@ token_embeddings input_ids (1, L) int64 -> rows (1, L, 10
|
|
| 24 |
text_model rows, positions, slots -> logits (1, L, 151936)
|
| 25 |
```
|
| 26 |
|
| 27 |
-
- **File**: `qwen3_asr_0_6b_xnnpack_8da4w.pte` — **697.
|
| 28 |
- **Source**: [Qwen/Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B) — 782.4M parameters
|
| 29 |
- **License**: apache-2.0
|
| 30 |
- **Languages**: 30, auto-detected — the model emits `language <Name>` before the text
|
|
@@ -88,7 +88,9 @@ tipping rather than silence gradually swamping the signal. The errors look like
|
|
| 88 |
|
| 89 |
**Hence the ladder.** Thirteen encoder methods cost **3.8 MB** in total, because
|
| 90 |
methods in one `.pte` share their constants: one window is 3753.4 MB in fp32 and
|
| 91 |
-
thirteen are 3757.2 MB. There is no reason to make a caller over-pad.
|
|
|
|
|
|
|
| 92 |
|
| 93 |
## Verification
|
| 94 |
|
|
@@ -99,7 +101,7 @@ and window. Nine clips, Japanese / Portuguese / Russian, windows 2–4 s:
|
|
| 99 |
| bundle | worst CER | ignoring case and terminal punctuation |
|
| 100 |
|---|---|---|
|
| 101 |
| fp32 (3757.2 MB, not published) | **0.0000** | 0.0000 |
|
| 102 |
-
| 8da4w (697.
|
| 103 |
|
| 104 |
Both arms run at the same window on purpose, so this measures the conversion and not
|
| 105 |
the window choice, which is measured separately above.
|
|
@@ -140,16 +142,28 @@ reproduces the fp32 model, and nothing about word error rate on real speakers.
|
|
| 140 |
Measured standalone at the 30-second window, swapping only the audio half and reading
|
| 141 |
the transcript:
|
| 142 |
|
| 143 |
-
| build | size (MB) | Mac ms* | corr vs eager | worst CER |
|
| 144 |
-
|---|---|---|---|---|
|
| 145 |
-
| XNNPACK fp32 | 746.
|
| 146 |
-
| XNNPACK fp16 | 381.
|
| 147 |
-
| XNNPACK int8 | 200.
|
| 148 |
-
| Core ML | 374.1 | **40.
|
| 149 |
-
|
| 150 |
-
\*Mac arm64, median of
|
| 151 |
-
Torch eager fp32 on the same machine is
|
| 152 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 153 |
|
| 154 |
Two things worth taking from that table. **int8 on the audio tower is free** in the
|
| 155 |
only unit that matters, which is why the shipped bundle uses it there. And **the Core
|
|
|
|
| 24 |
text_model rows, positions, slots -> logits (1, L, 151936)
|
| 25 |
```
|
| 26 |
|
| 27 |
+
- **File**: `qwen3_asr_0_6b_xnnpack_8da4w.pte` — **697.0 MB**, fifteen methods
|
| 28 |
- **Source**: [Qwen/Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B) — 782.4M parameters
|
| 29 |
- **License**: apache-2.0
|
| 30 |
- **Languages**: 30, auto-detected — the model emits `language <Name>` before the text
|
|
|
|
| 88 |
|
| 89 |
**Hence the ladder.** Thirteen encoder methods cost **3.8 MB** in total, because
|
| 90 |
methods in one `.pte` share their constants: one window is 3753.4 MB in fp32 and
|
| 91 |
+
thirteen are 3757.2 MB. There is no reason to make a caller over-pad. (Those two fp32
|
| 92 |
+
figures are from the pre-rewrite graph; the rewrite moved the published 8da4w file by
|
| 93 |
+
0.5 MB, and the +3.8 MB it is making a point about is unaffected.)
|
| 94 |
|
| 95 |
## Verification
|
| 96 |
|
|
|
|
| 101 |
| bundle | worst CER | ignoring case and terminal punctuation |
|
| 102 |
|---|---|---|
|
| 103 |
| fp32 (3757.2 MB, not published) | **0.0000** | 0.0000 |
|
| 104 |
+
| 8da4w (697.0 MB, this file) | 0.1333 | **0.0667** |
|
| 105 |
|
| 106 |
Both arms run at the same window on purpose, so this measures the conversion and not
|
| 107 |
the window choice, which is measured separately above.
|
|
|
|
| 142 |
Measured standalone at the 30-second window, swapping only the audio half and reading
|
| 143 |
the transcript:
|
| 144 |
|
| 145 |
+
| build | size (MB) | Mac ms* | XNNPACK takes | corr vs eager | worst CER† |
|
| 146 |
+
|---|---|---|---|---|---|
|
| 147 |
+
| XNNPACK fp32 | 746.0 | 143.6 | 71.9% | 0.999999 | 0.0000 |
|
| 148 |
+
| XNNPACK fp16 | 381.7 | 237.6 | 64.5% | 0.999995 | 0.0000 |
|
| 149 |
+
| XNNPACK int8 | 200.6 | 123.5 | 78.0% | 0.998266 | **0.0000** |
|
| 150 |
+
| Core ML | 374.1 | **40.6** | 100% | 0.972872 | **0.0690** |
|
| 151 |
+
|
| 152 |
+
\*Mac arm64, median of 10 — a reference point for relative cost, not a device number.
|
| 153 |
+
Torch eager fp32 on the same machine is 600.6 ms, so the XNNPACK build is 4.2x faster
|
| 154 |
+
than eager.
|
| 155 |
+
|
| 156 |
+
†The CER column was measured on the pre-rewrite graph (below). The rewrite agrees with
|
| 157 |
+
it to 1.3e-06 at the tensor level and leaves every transcript in this card unchanged,
|
| 158 |
+
but the transcripts behind that column were not re-read.
|
| 159 |
+
|
| 160 |
+
**Those speeds moved because the windowed attention stopped calling
|
| 161 |
+
`F.scaled_dot_product_attention`.** It decomposes to `_safe_softmax`, whose guard for
|
| 162 |
+
fully-masked rows is six operations XNNPACK cannot take — and the call has no mask and
|
| 163 |
+
`is_causal=False`, so the guard can never fire. There are 18 layers x 4 windows of it at
|
| 164 |
+
30 seconds. Writing the softmax out took the fp32 tower from **55.8% delegated at
|
| 165 |
+
170.0 ms to 71.9% at 143.6 ms**, and int8 from 149.4 ms to 123.5 ms, agreeing with the
|
| 166 |
+
old graph to 7.1e-07.
|
| 167 |
|
| 168 |
Two things worth taking from that table. **int8 on the audio tower is free** in the
|
| 169 |
only unit that matters, which is why the shipped bundle uses it there. And **the Core
|