fp16-safe conversion: fused x-logsumexp(x) tail; encoder weights byte-identical to upstream pin
Browse files- CHECKSUMS.sha256 +9 -0
- README.md +60 -0
- base960h_aligner.mlmodelc/analytics/coremldata.bin +3 -0
- base960h_aligner.mlmodelc/coremldata.bin +3 -0
- base960h_aligner.mlmodelc/metadata.json +81 -0
- base960h_aligner.mlmodelc/model.mil +0 -0
- base960h_aligner.mlmodelc/weights/weight.bin +3 -0
- base960h_aligner.mlpackage/Data/com.apple.CoreML/model.mlmodel +3 -0
- base960h_aligner.mlpackage/Data/com.apple.CoreML/weights/weight.bin +3 -0
- base960h_aligner.mlpackage/Manifest.json +18 -0
CHECKSUMS.sha256
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
d07ca6d8c69f2921810ca9bc919f0334f78de40a8f0c50e125358765fc5769fd ./base960h_aligner.mlmodelc/analytics/coremldata.bin
|
| 2 |
+
64b5214b50e003435d6659b8a3e4178e58ba48f3738daca0d6a1101412cb5a55 ./base960h_aligner.mlmodelc/coremldata.bin
|
| 3 |
+
b7e9bf36c9290d843804f4082b5442098fd14d26f2e0b032225ddebe3df729b2 ./base960h_aligner.mlmodelc/metadata.json
|
| 4 |
+
2d55c09bd32ee0810214d3896d0a5e87b2e8643b77c034bfcc908836e10d3809 ./base960h_aligner.mlmodelc/model.mil
|
| 5 |
+
de51193fe73fb3aad085f9c794f08bfde1b939fc12f92e0834edcd4cb712e642 ./base960h_aligner.mlmodelc/weights/weight.bin
|
| 6 |
+
ff4b7c3781d2a07698cb8883050697a45abb9e21325da3d9254df35139553c9c ./base960h_aligner.mlpackage/Data/com.apple.CoreML/model.mlmodel
|
| 7 |
+
de51193fe73fb3aad085f9c794f08bfde1b939fc12f92e0834edcd4cb712e642 ./base960h_aligner.mlpackage/Data/com.apple.CoreML/weights/weight.bin
|
| 8 |
+
bae000742a392d10914bce5d02bcb86ba4bbda83de8c7a141d27c79035fab0d3 ./base960h_aligner.mlpackage/Manifest.json
|
| 9 |
+
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ./CHECKSUMS.sha256
|
README.md
ADDED
|
@@ -0,0 +1,60 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
tags:
|
| 4 |
+
- coreml
|
| 5 |
+
- forced-alignment
|
| 6 |
+
- wav2vec2
|
| 7 |
+
- ctc
|
| 8 |
+
- apple-neural-engine
|
| 9 |
+
library_name: coremlit
|
| 10 |
+
---
|
| 11 |
+
|
| 12 |
+
# alignkit-coreml — fp16-safe CoreML conversion of wav2vec2-base-960h for forced alignment
|
| 13 |
+
|
| 14 |
+
A CoreML conversion of **facebook/wav2vec2-base-960h** (CTC head, 29-class character
|
| 15 |
+
vocabulary) with an **fp16-survivable log-probability tail**, for word-level forced
|
| 16 |
+
alignment. Consumed by the `alignkit` crate of the
|
| 17 |
+
[coremlit](https://github.com/findit-studio/coremlit) workspace.
|
| 18 |
+
|
| 19 |
+
## Why this conversion exists
|
| 20 |
+
|
| 21 |
+
The stock conversion's tail is a decomposed `softmax → log(eps = 2^-149)`. That epsilon is
|
| 22 |
+
below fp16's smallest subnormal (`2^-24`), so on the Apple Neural Engine the guard rounds
|
| 23 |
+
to zero and `log(0)` saturates to **≈ −45440**, corrupting **16.7% of emission cells** and
|
| 24 |
+
shifting word timings by up to **881 ms** — silently, since the values are still finite and
|
| 25 |
+
negative. (A naive `torch.log_softmax` re-export reproduces the defect under coremltools
|
| 26 |
+
9.0.) This conversion uses an explicitly **fused `x − logsumexp(x)`** tail: the emissions
|
| 27 |
+
are exact log-probabilities on CPU and bounded (min > −100) on every placement.
|
| 28 |
+
|
| 29 |
+
## Contents
|
| 30 |
+
|
| 31 |
+
- `base960h_aligner.{mlpackage,mlmodelc}` — encoder weights **byte-identical** to the
|
| 32 |
+
established conversion (`weight.bin` sha256 `de51193f…`); only the tail graph changed.
|
| 33 |
+
- `CHECKSUMS.sha256` — per-file digests.
|
| 34 |
+
|
| 35 |
+
Input: 16 kHz mono f32, up to a 960,000-sample (60 s) window. Output: `[1, T, 29]`
|
| 36 |
+
log-probabilities, 320-sample hop.
|
| 37 |
+
|
| 38 |
+
## Placement guidance (measured)
|
| 39 |
+
|
| 40 |
+
- **CPU: exact** log-probabilities (per-frame `logsumexp ≈ 0`) and the fastest placement
|
| 41 |
+
at this model size — the recommended default.
|
| 42 |
+
- **GPU / ANE: numerically safe with this conversion** (emissions bounded, no saturation),
|
| 43 |
+
but the wav2vec2 encoder itself is fp16-sensitive (group-norm): expect ~98% frame-argmax
|
| 44 |
+
agreement vs fp32, i.e. small word-timing differences. Consumers that require exact
|
| 45 |
+
timings should stay on CPU; the crate's runtime guard rejects any corrupted emission
|
| 46 |
+
matrix regardless of placement.
|
| 47 |
+
|
| 48 |
+
## Verification (summary)
|
| 49 |
+
|
| 50 |
+
MIL-level fp16-guard audit clean (27 guard sites, every effective floor ≥ `2^-24`);
|
| 51 |
+
emissions min > −100 on ANE (−23.81) and GPU (−28.36) with real speech; word-timing parity
|
| 52 |
+
vs an independent ONNX implementation: median **0.0 ms** on unpadded 60 s audio (367/372
|
| 53 |
+
boundaries within one 20 ms frame). Toolchain pinned: coremltools 9.0, torch/torchaudio
|
| 54 |
+
2.5.1, python 3.11.15.
|
| 55 |
+
|
| 56 |
+
## Upstream provenance & licensing
|
| 57 |
+
|
| 58 |
+
Weights: [facebook/wav2vec2-base-960h](https://huggingface.co/facebook/wav2vec2-base-960h)
|
| 59 |
+
(Apache-2.0), via torchaudio's `WAV2VEC2_ASR_BASE_960H` pipeline. This is a derivative
|
| 60 |
+
conversion: identical weights, corrected graph numerics.
|
base960h_aligner.mlmodelc/analytics/coremldata.bin
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d07ca6d8c69f2921810ca9bc919f0334f78de40a8f0c50e125358765fc5769fd
|
| 3 |
+
size 243
|
base960h_aligner.mlmodelc/coremldata.bin
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:64b5214b50e003435d6659b8a3e4178e58ba48f3738daca0d6a1101412cb5a55
|
| 3 |
+
size 534
|
base960h_aligner.mlmodelc/metadata.json
ADDED
|
@@ -0,0 +1,81 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{
|
| 3 |
+
"metadataOutputVersion" : "3.0",
|
| 4 |
+
"shortDescription" : "wav2vec2-base-960h forced aligner, epsilon-safe (reduce_log_sum_exp->sub tail)",
|
| 5 |
+
"outputSchema" : [
|
| 6 |
+
{
|
| 7 |
+
"hasShapeFlexibility" : "0",
|
| 8 |
+
"isOptional" : "0",
|
| 9 |
+
"dataType" : "Float32",
|
| 10 |
+
"formattedType" : "MultiArray (Float32 1 × 2999 × 29)",
|
| 11 |
+
"shortDescription" : "",
|
| 12 |
+
"shape" : "[1, 2999, 29]",
|
| 13 |
+
"name" : "emissions",
|
| 14 |
+
"type" : "MultiArray"
|
| 15 |
+
}
|
| 16 |
+
],
|
| 17 |
+
"storagePrecision" : "Float16",
|
| 18 |
+
"modelParameters" : [
|
| 19 |
+
|
| 20 |
+
],
|
| 21 |
+
"author" : "Phase-A reconversion: wav2vec2-base-960h aligner, stable fused log-softmax",
|
| 22 |
+
"specificationVersion" : 8,
|
| 23 |
+
"mlProgramOperationTypeHistogram" : {
|
| 24 |
+
"Ios17.mul" : 13,
|
| 25 |
+
"Ios17.sqrt" : 1,
|
| 26 |
+
"Ios17.square" : 1,
|
| 27 |
+
"Ios17.transpose" : 51,
|
| 28 |
+
"Ios17.sub" : 2,
|
| 29 |
+
"Ios17.linear" : 74,
|
| 30 |
+
"Ios17.conv" : 8,
|
| 31 |
+
"Ios17.matmul" : 24,
|
| 32 |
+
"Ios16.reduceLogSumExp" : 1,
|
| 33 |
+
"Ios17.add" : 27,
|
| 34 |
+
"Ios17.realDiv" : 1,
|
| 35 |
+
"Ios16.reduceMean" : 2,
|
| 36 |
+
"Ios17.sliceByIndex" : 1,
|
| 37 |
+
"Ios16.softmax" : 12,
|
| 38 |
+
"Ios16.gelu" : 20,
|
| 39 |
+
"Ios17.expandDims" : 1,
|
| 40 |
+
"Ios17.reshape" : 50,
|
| 41 |
+
"Ios17.cast" : 2,
|
| 42 |
+
"Ios17.layerNorm" : 26
|
| 43 |
+
},
|
| 44 |
+
"computePrecision" : "Mixed (Float16, Float32, Int32)",
|
| 45 |
+
"isUpdatable" : "0",
|
| 46 |
+
"stateSchema" : [
|
| 47 |
+
|
| 48 |
+
],
|
| 49 |
+
"availability" : {
|
| 50 |
+
"macOS" : "14.0",
|
| 51 |
+
"tvOS" : "17.0",
|
| 52 |
+
"visionOS" : "1.0",
|
| 53 |
+
"watchOS" : "10.0",
|
| 54 |
+
"iOS" : "17.0",
|
| 55 |
+
"macCatalyst" : "17.0"
|
| 56 |
+
},
|
| 57 |
+
"modelType" : {
|
| 58 |
+
"name" : "MLModelType_mlProgram"
|
| 59 |
+
},
|
| 60 |
+
"inputSchema" : [
|
| 61 |
+
{
|
| 62 |
+
"hasShapeFlexibility" : "0",
|
| 63 |
+
"isOptional" : "0",
|
| 64 |
+
"dataType" : "Float32",
|
| 65 |
+
"formattedType" : "MultiArray (Float32 1 × 960000)",
|
| 66 |
+
"shortDescription" : "",
|
| 67 |
+
"shape" : "[1, 960000]",
|
| 68 |
+
"name" : "waveform",
|
| 69 |
+
"type" : "MultiArray"
|
| 70 |
+
}
|
| 71 |
+
],
|
| 72 |
+
"userDefinedMetadata" : {
|
| 73 |
+
"com.github.apple.coremltools.conversion_date" : "2026-07-16",
|
| 74 |
+
"com.github.apple.coremltools.source" : "torch==2.5.1",
|
| 75 |
+
"com.github.apple.coremltools.version" : "9.0",
|
| 76 |
+
"com.github.apple.coremltools.source_dialect" : "TorchScript"
|
| 77 |
+
},
|
| 78 |
+
"generatedClassName" : "base960h_aligner",
|
| 79 |
+
"method" : "predict"
|
| 80 |
+
}
|
| 81 |
+
]
|
base960h_aligner.mlmodelc/model.mil
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
base960h_aligner.mlmodelc/weights/weight.bin
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:de51193fe73fb3aad085f9c794f08bfde1b939fc12f92e0834edcd4cb712e642
|
| 3 |
+
size 188799802
|
base960h_aligner.mlpackage/Data/com.apple.CoreML/model.mlmodel
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ff4b7c3781d2a07698cb8883050697a45abb9e21325da3d9254df35139553c9c
|
| 3 |
+
size 154521
|
base960h_aligner.mlpackage/Data/com.apple.CoreML/weights/weight.bin
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:de51193fe73fb3aad085f9c794f08bfde1b939fc12f92e0834edcd4cb712e642
|
| 3 |
+
size 188799802
|
base960h_aligner.mlpackage/Manifest.json
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"fileFormatVersion": "1.0.0",
|
| 3 |
+
"itemInfoEntries": {
|
| 4 |
+
"4296390D-EF20-44EA-B09C-2E9B874A1BA0": {
|
| 5 |
+
"author": "com.apple.CoreML",
|
| 6 |
+
"description": "CoreML Model Weights",
|
| 7 |
+
"name": "weights",
|
| 8 |
+
"path": "com.apple.CoreML/weights"
|
| 9 |
+
},
|
| 10 |
+
"EC9FDB7A-F224-4BE0-BE4A-1BD1C62458C4": {
|
| 11 |
+
"author": "com.apple.CoreML",
|
| 12 |
+
"description": "CoreML Model Specification",
|
| 13 |
+
"name": "model.mlmodel",
|
| 14 |
+
"path": "com.apple.CoreML/model.mlmodel"
|
| 15 |
+
}
|
| 16 |
+
},
|
| 17 |
+
"rootModelIdentifier": "EC9FDB7A-F224-4BE0-BE4A-1BD1C62458C4"
|
| 18 |
+
}
|