Very unstable inference

#22
by andypotato - opened

The model has a tendency to completely spiral out of control and only produce garbage output. This often happens at sections where speakers repeat a single word multiple times like "yes yes yes". From this point on inference just keeps repeating this single word.

Another issue is that it will not stop after the file has already been fully transcribed and keeps repeating nonsense characters and end tokens.

This behavior can be reproduced on the gradio demo and also single file demos from the Github repo.

You can refer to this PR

This is still the case

Still happening on the transformers path as of October. A workaround that got 11 of 12 looping runs to finish the whole file:

PR #228 doesn't cover this. It adds a client-side script for the vLLM server, not a change to transformers generate() or the Gradio demo.

My trigger: speech mixed with heavy noise (two 1962 NASA radio clips, 45 s and 100 s). It starts correctly, then repeats one phrase until the token limit. The segment JSON never closes, so post_process_transcription returns [] with no error. Noise alone and clean long recordings (39 and 59 min) did not loop.

Workaround: when a loop is detected, roll back to the last closed segment before it (keep the original token ids, don't re-tokenize the cut) and continue with sampling at temperature 0.3 → 0.6 → 1.0. To continue from a prefix, append it and extend the masks:

extra = torch.tensor([prefix_ids], device="cuda")
inputs["input_ids"] = torch.cat([inputs["input_ids"], extra], dim=1)
inputs["attention_mask"] = torch.cat([inputs["attention_mask"], torch.ones_like(extra)], dim=1)
inputs["acoustic_input_mask"] = torch.cat(
    [inputs["acoustic_input_mask"], torch.zeros_like(extra, dtype=torch.bool)], dim=1)

Loop check, every 50 tokens: the last 400 tokens are periodic (period ≤ 100), or a phrase repeats 4+ times in a row inside a segment's Content. The second check matters because after resampling the model often slips into shorter loops ("we have a," 56×) that end before 400 tokens.

2 clips × hotwords on/off × 3 seeds Greedy + rollback + phrase check
Transcribed to the end of the audio 0/12 12/12 11/12
Phrase repeated 3+ times in a row 12/12 8/12 2/12
Time per clip ~130 s 64–164 s 36–99 s

The one failure looped at every temperature and still returned []. On five saved clean long outputs, neither check ever fired.

Caveats: only two clips, and I chose the 4× threshold after seeing them. No reference transcript; I listened to the two worst spots only. Setup: DGX Spark, bf16, transformers 4.51.3, VibeVoice @ 1541f590.

More on the empty-result behavior: https://github.com/MindrLabs/speech-to-text-vibevoice#repetition-loops-and-silent-empty-results

Sign up or log in to comment