File size: 3,259 Bytes
8c8c7a3
 
 
 
 
 
 
 
 
 
 
a4d58de
8c8c7a3
 
 
a4d58de
 
 
8c8c7a3
 
 
 
 
 
 
 
a4d58de
 
 
8c8c7a3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
59f2668
 
a4d58de
59f2668
 
 
a4d58de
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8c8c7a3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
---
license: apache-2.0
library_name: pytorch
tags:
- forced-alignment
- word-alignment
- speech
- audio
- gguf
---

# FuseAlign multilingual (noisy student, step 60,000)

Word-level forced aligner: given a recording and its transcript, it predicts each word's start and end on a 1 ms
grid plus a presence probability. A reproduction of FuseAlign ([Vaidya et al., 2026](https://arxiv.org/abs/2609.33650))
by Zyphra: one multilingual noisy student, trained 60,000 steps from the multilingual model's CJK fine-tune on about
25,000 h of audio in 73 languages, mostly labelled on clean audio by a teacher per language and learnt from augmented
audio.

Code, usage, long-form alignment and the C++/ggml CLI:
[github.com/Zyphra/fusealign-multilingual](https://github.com/Zyphra/fusealign-multilingual).

## Files

| File | Size | Use |
| --- | ---: | --- |
| `fusealign-multilingual-ns4-step60000.safetensors` | 1.34 GB | PyTorch package (`torch.hub`, `fusealign_hub`): FuseAlign fp32, Dia text encoder bf16 |
| `fusealign-multilingual-ns4-step60000-f32.gguf` | 1.35 GB | ggml port: FuseAlign F32, Dia BF16, Silero VAD F32 |
| `fusealign-multilingual-ns4-step60000-q8_0.gguf` | 0.51 GB | ggml port: transformer matmuls Q8_0, other Dia tensors BF16, the rest F32 |

## Usage

```python
import soundfile as sf
import torch

aligner = torch.hub.load("Zyphra/fusealign-multilingual", "fusealign", trust_repo=True, device="cuda")
wav, sr = sf.read("speech.wav", dtype="float32")
words = aligner.align(torch.from_numpy(wav), sr, "the transcript of the recording")
# [WordAlignment(word='the', start=..., end=..., presence=..., placed=True), ...]
```

## Evaluation

% of word boundaries off by more than 50 ms (the paper's asymmetric error), against
[Qwen3-ForcedAligner-0.6B](https://huggingface.co/Qwen/Qwen3-ForcedAligner-0.6B) (qwen-asr 0.0.6, bf16, whole clips)
on the same clips, transcripts and references (FuseAlign: the PyTorch package, CUDA, one clip per forward):

| Corpus | FuseAlign | Qwen3-FA |
| --- | ---: | ---: |
| TIMIT | 7.92 | 20.17 |
| Buckeye | 7.23 | 24.60 |
| AMI-silver | 34.22 | 58.22 |
| TIMIT-long | 8.21 | 20.04 |
| BZNSYP Mandarin (gold) | 15.44 | 28.04 |
| Seoul Corpus Korean (gold) | 22.43 | 33.58 |
| FLEURS Mandarin | 17.97 | 11.06 |
| FLEURS Japanese | 13.34 | 8.38 |
| FLEURS Korean | 7.80 | 11.53 |
| FLEURS Spanish | 15.43 | 11.71 |
| FLEURS Portuguese | 9.20 | 10.24 |
| FLEURS French | 13.34 | 12.35 |
| FLEURS German | 13.99 | 13.91 |
| FLEURS Russian | 10.60 | 11.18 |

FLEURS references are Montreal Forced Aligner output, not hand labels; BZNSYP's and the Seoul Corpus's are hand-labelled (Seoul: interviewee turns of 40 speakers' spontaneous interviews, at most 30 s).

## License

Apache-2.0. The checkpoint and GGUFs include the Dia text encoder weights from
[nari-labs/Dia-1.6B-0626](https://huggingface.co/nari-labs/Dia-1.6B-0626) (Apache-2.0); the GGUFs include
[Silero VAD](https://github.com/snakers4/silero-vad) weights (MIT).

## Citation

```bibtex
@article{vaidya2026fusealign,
  title   = {FuseAlign: Forced Alignment in the Wild},
  author  = {Vaidya, Mithilesh and Bailey, Stephen and Badam, Sumukh and Bendel, Matthew and He, Xingzhe},
  journal = {arXiv preprint arXiv:2609.33650},
  year    = {2026}
}
```