| # Validation |
|
|
| Validation was run on 2026-08-01 with an Apple M5 Pro, 24 GB unified memory, |
| macOS 26.6, Python 3.12.13, and MLX 0.32.0. |
|
|
| ## Checkpoint conversion |
|
|
| - Source: `AutoArk-AI/ARK-ASR-3B` |
| - Revision: `1e28271b79edc97635783bea65abc89195a09ed3` |
| - Source tensors: 926 |
| - Intentionally dropped tensors: 2 |
| - Retained parameter tensors: 924 |
| - Missing, unexpected, or shape-mismatched tensors: 0 |
| - Output dtype: BF16 |
| - Output size: 7.0 GiB |
| - Output SHA-256: |
| `a5e9431bdd648340a40c092e385c4fe1d445d8ad3311dd94609e72af36b0256d` |
|
|
| Source shard hashes are recorded in `conversion.json`. |
|
|
| ## PyTorch parity |
|
|
| The original remote-code model and native MLX model were run on the pinned |
| `Narsil/asr_dummy/1.flac` LibriSpeech sample. |
|
|
| - Prompt token IDs: identical |
| - BF16 input features: maximum absolute difference `0.0` |
| - Adapted audio feature cosine similarity: `0.998567558665821` |
| - Initial decoder logits cosine similarity: `0.9999223476845636` |
| - Greedy generation token IDs: identical |
| - Final decoded text: identical |
|
|
| Both implementations produced: |
|
|
| > he hoped there would be stew for dinner turnips and carrots and bruised |
| > potatoes and fat mutton pieces to be ladled out in thick peppered flour |
| > fattened sauce |
|
|
| The validation command was: |
|
|
| ```bash |
| python scripts/validate_parity.py /path/to/1.flac \ |
| --model . \ |
| --source AutoArk-AI/ARK-ASR-3B \ |
| --revision 1e28271b79edc97635783bea65abc89195a09ed3 \ |
| --min-adapter-cosine 0.998 |
| ``` |
|
|
| ## MLX benchmark |
|
|
| The same 12.1-second audio sample was measured after the checkpoint was already |
| present in the operating-system file cache: |
|
|
| - Model load: `1.011 s` |
| - Audio preprocessing: `0.499 s` |
| - First token: `0.885 s` |
| - Full 34-token generation: `1.965 s` |
| - Generation throughput: `17.30 tokens/s` |
| - MLX peak memory: `7.820 GB` |
|
|
| These numbers describe this machine and test clip; they are not portable |
| performance guarantees. |
|
|
|
|
