danish-char-aligner

A Danish character-level forced aligner. Give it speech and the transcript; it returns a start and end time for every character, on a 20 ms grid, plus word spans and an alignment score.

For a radio clip that starts "Andet må løbe under fabrikken ...":

{"characters": [
  {"text": "A", "index": 0, "start": 0.04, "end": 0.08},
  {"text": "n", "index": 1, "start": 0.08, "end": 0.12},
  {"text": "d", "index": 2, "start": 0.12, "end": 0.16},
  {"text": "e", "index": 3, "start": 0.16, "end": 0.20},
  {"text": "t", "index": 4, "start": 0.20, "end": 0.24},
  ...],
 "words": [{"text": "Andet", "start": 0.04, "end": 0.24},
           {"text": "må", "start": 0.26, "end": 0.36}, ...],
 "score": -1.016}

Character spans are contiguous: each character runs until the next one starts, and the gap between two words is its own " " span. index points back into the transcript you passed in. Letters and digits are timed (a-z, æ, ø, å, é, ü, 0-9, case-insensitive); punctuation is skipped.

Install and use

pip install torch transformers safetensors numpy soundfile huggingface_hub

import sys
from huggingface_hub import snapshot_download

model_dir = snapshot_download("syvai/danish-char-aligner")
sys.path.insert(0, model_dir)
from aligner import Aligner

al = Aligner(model_dir)          # cuda if available, else cpu
out = al.align("clip.wav", "Andet må løbe under fabrikken")
for c in out["characters"]:
    print(c["text"], c["start"], c["end"])

For many clips, use align_batch. On a CUDA GPU it runs a batched fp16 path (fast_inference.py, needs Triton, which ships with PyTorch on Linux) at 2,670x realtime on an RTX 3090; elsewhere it falls back to align per clip. It takes (audio, text) pairs, where audio is a path or a 16 kHz float array, and returns results in input order:

results = al.align_batch([("a.wav", "første tekst"), ("b.wav", "anden tekst")])

Clips are grouped by length into batches of up to batch_seconds=600 seconds of audio; lower it if you run out of GPU memory.

or from the shell, inside the downloaded folder:

python aligner.py . clip.wav "Andet må løbe under fabrikken"

Audio of any sample rate is accepted (resampled to 16 kHz). Align utterances of up to ~30 s at a time; for long recordings, segment first.

score is the mean per-frame log-probability along the alignment path. It is a useful check that the transcript actually matches the audio: compare it against other transcripts for the same clip, not across clips.

How it was built

The model is a 12-layer student of Røst (CoRal-project/roest-v3-wav2vec2-315m, a Danish wav2vec2 CTC model). It keeps every second transformer layer of Røst (layers 1, 3, ..., 23) and Røst's convolutional front end, so it has the same 20 ms frame rate.

It was trained on Danish speech from syvai/danish-asr-unified with three losses:

• KL divergence to Røst's frame posteriors (distillation),
• CTC on the transcript,
• per-frame cross-entropy for a frame-labelling head that predicts which character is being spoken in every 20 ms frame, trained on reference character-level alignments.

At inference the frame-labelling head is aligned to the transcript with a monotonic Viterbi pass ([silence] chars/spaces [silence]). That is what gives contiguous character spans, including the ends of words. A CTC model cannot do this on its own, because it marks each character with a 1-2 frame spike.

164M parameters, 1.9x fewer than Røst (315M).

Results

Agreement with held-out reference character alignments. Røst is aligned with CTC; its character ends are taken as the next character's start inside a word and as spike end + 20 ms at the end of a word.

DR P1 radio (out of domain, 7,889 clips, 675,753 characters):

	start within 20 ms	start within 40 ms	start within 100 ms	end within 20 ms
Røst	79.9%	94.4%	98.9%	79.1% inside words, 60.9% word-final
this model	87.1%	94.7%	97.9%	85.2% (all characters)

Held-out read speech (CoRal read-aloud and Folketinget, 2,910 clips, 205,818 characters):

	start within 20 ms	start within 40 ms	end within 20 ms
Røst	79.1%	93.8%	78.0% inside words, 56.3% word-final
this model	86.1%	93.5%	84.6% (all characters)

Detecting a wrong transcript (1,035 P1 clips; AUC of the alignment score for the true transcript against a corrupted one, higher is better):

	text from another clip	30% of words replaced
Røst	0.991	0.940
this model	0.998	0.924

Speed

	realtime factor
align_batch, RTX 3090, fp16, end to end	2,670x
align, RTX 3090, bf16, forward pass, one clip at a time	910-960x (Røst: 820-860x)
align, CPU 12 threads, fp32, forward pass	23.8x (Røst: 15.2x)
align, CPU 1 thread, fp32, forward pass	5.4x (Røst: 3.4x)

The align_batch figure covers the forward pass, the alignment and building the output, with audio already decoded in memory: 8,445 P1 clips (14.8 h of audio) in about 20 s, 2,620-2,680x across runs. Its timing accuracy matches align in fp32 (86.75% of character starts within 20 ms of the reference for both). CPU: AMD Ryzen 5 7600X.

The batched path gets its speed from length-bucketed fp16 batches, the convolutional front end rewritten as matrix multiplies with a fused LayerNorm+GELU kernel, the positional convolution run in chunks (cuDNN picks a ~20x slower algorithm for long sequences), and a Triton Viterbi kernel.

Limitations

• Danish only. Tested on read speech, parliament and radio, not on noisy or far-field audio.
• Timing resolution is 20 ms.
• The word-final tail is slightly heavier than Røst's (97.9% vs 98.9% of starts within 100 ms).
• Characters outside the vocabulary (punctuation, rare letters) are not timed.

License

CC BY 4.0. Built on Røst by the CoRal project (CoRal-project/roest-v3-wav2vec2-315m).