conversion to onnx needs script folder

by TheGreatQuran - opened Jun 23

Jun 23

Alslamo alaikom brother <3
i really love you for doing for the sake of Allah the greatest
ربنا تقبل منا انك انت السميع العليم

May Allah accept from us
I tried to export to onnx using icefall repo export file (edited to fit this model) but it requires 2 files (zipformer_rnnt_ctc_train.py and zipformer_rnnt_ctc_eval.py )
you can use the export script

Export Script

Folder Structure
📁 zipformer_p-quran/ <-- (The Root Directory)
│
├── 📁 icefall/ <-- (Cloned k2-fsa/icefall repository)
├── 📁 scripts/ <-- [MISSING IN HF REPO] Needs zipformer_rnnt_ctc_train.py & eval.py
│
├── 📄 phoneme_units.json
├── 📄 quran_text2phoneme.json
├── 📄 quran_phoneme_zipformer.pt
│
└── 📄 export-onnx-streaming-ctc.py <-- (The script provided below)

Command to run

python export-onnx-streaming-ctc.py
--epoch 99
--avg 1
--exp-dir ./exp
--causal True
--chunk-size 24
--left-context-frames 256
--enable-int8-quantization 1

Muno459

Owner Jun 23

•

edited Jun 23

wa ʿalaykum as-salām wa raḥmatullāh, brother 🤍 ربنا يتقبل منك ومنا.

Done. I just added the two files to the repo under scripts/:

scripts/zipformer_rnnt_ctc_train.py (has build_model + the PhonemeUnitTokenizer)
scripts/zipformer_rnnt_ctc_eval.py (has greedy_ctc_decode)

One note on the checkpoint: quran_phoneme_zipformer.pt is a single file with the weights under the "model" key (not icefall's epoch-N/exp-dir layout). So instead of --epoch 99 --avg 1, build the model and load directly:

from scripts.zipformer_rnnt_ctc_train import build_model, load_tokenizer
tok = load_tokenizer('phoneme_units.json'); blank = tok.get_piece_size(); vocab = blank + 1
model = build_model(vocab, blank, chunk_frames=[24], left_context_frames=256).eval()
import torch; model.load_state_dict(torch.load('quran_phoneme_zipformer.pt', map_location='cpu')['model'])
# now trace model.encoder.forward / model.ctc_head for your streaming ONNX export

Use chunk_size=(24,), left_context_frames=(256,) for the 1000 ms streaming profile (it's causal, so smaller chunks work too). You still need k2-fsa/icefall cloned for the zipformer/scaling/subsampling modules build_model imports, exactly as in your folder layout. Let me know if the export still trips on anything 🤝

Muno459

Owner Jun 23

Update, brother 🤍 you no longer need to export it yourself, I did it up front and pushed the ONNX to the repo:

quran_phoneme_zipformer.onnx , fp32 (encoder + CTC, full-utterance), runs with plain onnxruntime, no PyTorch/icefall needed.
quran_phoneme_zipformer.int8.onnx , dynamic-INT8 (q8), ~75 MB (about 3.5x smaller), and it matches the fp32 output 100% on argmax.
scripts/export_quran_onnx.py , the exact export script if you want to reproduce or re-export.

Inputs: 80-bin kaldi fbank feats (B,T,80) + feat_lens; outputs log_probs (B,T',251) + out_lens (argmax then collapse for phonemes). Use B=1; the graph zero-pads the time axis internally so the icefall downsample traces cleanly (that was the bit your export was tripping on).

import onnxruntime as ort
s = ort.InferenceSession('quran_phoneme_zipformer.int8.onnx', providers=['CPUExecutionProvider'])
log_probs, out_lens = s.run(None, {'feats': feats, 'feat_lens': lens})

جزاك الله خيرًا for the push, it made the repo better for everyone.

TheGreatQuran

Jun 23

Elhamdule Allah and Thanks for fast responsing my brother <3
i did nothing , Elhamdule Allah for Allah's Guide
Thanks for this awesome model

this now is the best model in these days insha'a Allah

TheGreatQuran

Jun 23

May Allah the great accept me and you in the jannat in the life after <3

TheGreatQuran

Jun 23

when i tried this onnx and added metadata to run directly on sherpa onnx it faced an error
when i traced it and analyzed the export script
it was found that the onnx was exported as an offline one .
can you check please ?
also refer to this export as i am trying to export but i am facing k2 module errors in windows and i dont have bandwith to download wsl

Export Streaming Script
python export_streaming.py --exp-dir ./exp --enable-int8-quantization 1

Folder Structure
📁 zipformer_p-quran/ <-- (The Root Directory)
│
├── 📁 icefall/ <-- (Cloned k2-fsa/icefall repository)
├── 📁 scripts/
│
├── 📄 phoneme_units.json
├── 📄 quran_phoneme_zipformer.pt
│
└── 📄 export-streaming.py <-- (The script provided )

I'm still trying to fix k2 module errors to try to export if i didn't , i will download wsl insha'a Allah

sorry for custom or specified requests

Muno459

Owner Jun 23

Fixed, brother 🤍 you were right, the first one was a full-utterance (offline) graph. I re-exported it properly as a cache-aware streaming zipformer2-CTC ONNX and replaced the files in the repo:

quran_phoneme_zipformer.onnx , streaming, chunk-by-chunk.
quran_phoneme_zipformer.int8.onnx , streaming int8 (~73 MB).

It is the standard sherpa-onnx online zipformer2-CTC format: input is one chunk x (1,T,80) + the encoder cache states (cached_key/nonlin_attn/val1/val2/conv1/conv2 per layer, embed_states, processed_lens), outputs log_probs + the new_* states for the next chunk. All the streaming params (model_type=zipformer2, decode_chunk_len=48, T=61, left_context_len, layer dims) are embedded as ONNX metadata, so sherpa-onnx reads them automatically , no manual config. Exported with icefall's own export-onnx-streaming-ctc path, so it should drop straight into the sherpa-onnx online recognizer.

scripts/export_quran_streaming_onnx.py is in the repo if you want to see exactly how. Let me know if sherpa is happy with it now 🤝

TheGreatQuran

Jun 23

Thanks my brother <3
i was making a fight to fix the k2 and didn't fix hahaha
but elhamdule Allah , you were faster than me
iam testing now insha'a Allah

TheGreatQuran

Jun 23

This comment has been hidden

TheGreatQuran

Jun 23

•

edited Jun 23

the pervious problem was a shifting in tokens.txt (maybe because i fixed but didn't test on android yet )
Testing on pc q8 :
الحمد لله رب العالمين

✅ Recording finished. Streaming audio to model in chunks...

[05.9s] Partial: ء َ ل ح َ م د ُ ل ِ ل ل َ ا ا ه ِ ر َ ب ب ِ ل ع َ ا ا ل َ م ِ ۦ ۦ ن

ءَلحَمدُلِللَااهِرَببِلعَاالَمِۦۦن
الحمد لله رب العالمين

Testing on pc fp32 :
✅ Recording finished. Streaming audio to model in chunks...

[05.9s] Partial: ء َ ل ح َ م د ُ ل ِ ل ل َ ا ا ه ِ ر َ ب ب ِ ل ع َ ا ا ل َ م ِ ۦ ۦ ن

ءَلحَمدُلِللَااهِرَببِلعَاالَمِۦۦن

Elhamdule Allah
Thanks my brother <3
Sorry for headache

I'll share the flutter project with you so you can make an ios version and share it on your account (if you want ) insha'a Allah

TheGreatQuran

Jun 23

elhamdle Allah the greatest

you are lucky my brother that god chosen you to do this work

Muno459

Owner Jun 23

•

edited Jun 23

Alhamdulilah, and thank you for your kind words.

Can you tell me if this feels faster/more accurate than fastconformer and if so by how much?

TheGreatQuran

Jun 23

I will make tests on both insha'a Allah
but since i ran the streaming model of fastconformer on onnxruntime (it ofcourse slower than sherpa)

but since the minimum chunk size of the streaming fastconformer it was much slower than this model
this model is almost instant
here the ms is between 5 to 300 max on my old android cpu while in the fastconformer it started from 500ms (there was a delay )
accuracy :
still i am testing this but in the fastconformer streaming , it never got words like : الم
and sometimes hallincated

the streaming had a forgetting problem which when the context increased 10-20 words or maybe 1 minute without clearing the output , it then starts randomizing and losing alot of letters

Muno459

Owner 4 days ago

On the forgetting after 10-20 words or about a minute, akhi: that is the streaming cache, and it is fixable from your side without touching the model. 🤍

The streaming graph carries left_context_frames = 256, which is about 10 seconds of audio, plus a processed_lens counter that keeps growing for as long as you keep feeding the same stream. If you run one continuous session for a minute or more, the left context fills with whatever came before, and eventually the model is attending to a long tail that no longer helps it.

Two things fix this:

Reset the cache states at natural boundaries. End of an ayah, or after a clear silence, start a fresh state set. Do not run one unbroken stream for minutes.
Do not feed silence through the model during a pause. If the reciter stops, stop calling the model and keep the state tensors as they are, then resume when speech returns. If you keep pushing silent chunks, the silence itself pushes the real speech out of the left context, and the first words after the pause come out garbled.

Point 2 is the one people miss. Someone else building on these models reported the same thing (garbled openings after a pause), and it was the same cause.

Barakallah feek for the latency numbers too, 5 to 300 ms on an old Android CPU against 500 ms for the FastConformer streaming, that is useful to know.

TheGreatQuran

4 days ago

On the forgetting after 10-20 words or about a minute, akhi: that is the streaming cache, and it is fixable from your side without touching the model. 🤍

The streaming graph carries left_context_frames = 256, which is about 10 seconds of audio, plus a processed_lens counter that keeps growing for as long as you keep feeding the same stream. If you run one continuous session for a minute or more, the left context fills with whatever came before, and eventually the model is attending to a long tail that no longer helps it.

Two things fix this:

Reset the cache states at natural boundaries. End of an ayah, or after a clear silence, start a fresh state set. Do not run one unbroken stream for minutes.

Do not feed silence through the model during a pause. If the reciter stops, stop calling the model and keep the state tensors as they are, then resume when speech returns. If you keep pushing silent chunks, the silence itself pushes the real speech out of the left context, and the first words after the pause come out garbled.

Point 2 is the one people miss. Someone else building on these models reported the same thing (garbled openings after a pause), and it was the same cause.

Barakallah feek for the latency numbers too, 5 to 300 ms on an old Android CPU against 500 ms for the FastConformer streaming, that is useful to know.

Yes you are right ❤️
keeping cache states of old ayahs causes hallucinates and degrade in model accuracy since the model was trained on separate ayahs

Muno459

Owner 4 days ago

Exactly right akhi, and the reason is worth stating for anyone reading later: the model was trained on
isolated ayah clips, so it never learned that one ayah follows another. Cache carried across an
ayah boundary is out of distribution, and the model does what any model does with out-of-distribution
state, which is invent something plausible. That is the hallucination you saw. 🤍

Now the part I want to warn you about, because your fix walks straight into it:

Flush before you reset, or you will lose the last word of every ayah.

The cache-aware model holds audio in its right-context buffer. If you zero the states the moment you
detect the end of an ayah, whatever is still sitting in that buffer is destroyed before it is ever
emitted. The final word simply disappears, silently, on every single ayah. Nothing errors.

So the order matters:

detect end of ayah
push one more chunk of silence through, carrying the current cache as normal (at least the
right-context length, roughly 1.05 s of zeros)
collect whatever comes out, that is your last word
then zero the states for the next ayah

Two smaller notes:

Do not reset mid-ayah. Within an ayah the left context is helping you, and resetting throws
away exactly the history that makes streaming accurate. Reset on the boundary, not on every pause.
A long silence inside an ayah is not a boundary. A reciter taking a breath mid-ayah should keep
its cache. If you reset on silence alone you will fragment ayat that were recited correctly.

I mention the flush specifically because it cost me hours. My verification harness was scoring 1
segment correct out of 10 and I was convinced my ONNX export was broken. It was not. My loop was
discarding the final partial chunk, exactly as a reset without a flush would.

Barak Allahu feek for working this out yourself.

TheGreatQuran

4 days ago

•

edited 4 days ago

Exactly right akhi, and the reason is worth stating for anyone reading later: the model was trained on
isolated ayah clips, so it never learned that one ayah follows another. Cache carried across an
ayah boundary is out of distribution, and the model does what any model does with out-of-distribution
state, which is invent something plausible. That is the hallucination you saw. 🤍

Now the part I want to warn you about, because your fix walks straight into it:

Flush before you reset, or you will lose the last word of every ayah.

The cache-aware model holds audio in its right-context buffer. If you zero the states the moment you
detect the end of an ayah, whatever is still sitting in that buffer is destroyed before it is ever
emitted. The final word simply disappears, silently, on every single ayah. Nothing errors.

So the order matters:

detect end of ayah

push one more chunk of silence through, carrying the current cache as normal (at least the
right-context length, roughly 1.05 s of zeros)

collect whatever comes out, that is your last word

then zero the states for the next ayah

Two smaller notes:

Do not reset mid-ayah. Within an ayah the left context is helping you, and resetting throws
away exactly the history that makes streaming accurate. Reset on the boundary, not on every pause.

A long silence inside an ayah is not a boundary. A reciter taking a breath mid-ayah should keep
its cache. If you reset on silence alone you will fragment ayat that were recited correctly.

I mention the flush specifically because it cost me hours. My verification harness was scoring 1
segment correct out of 10 and I was convinced my ONNX export was broken. It was not. My loop was
discarding the final partial chunk, exactly as a reset without a flush would.

Barak Allahu feek for working this out yourself.

masha'a Allah

this is really what i was doing in app detecting end of ayah ..but i knew this after testign annd some time to figure out the fix
how did you know this without a real test masha'a Allah
but you made it with a better advanced and easy way ..i will save this for later use insha'a Allah

i prefer using without silerovad because it had problems in my first attempt ...there is another options to find silences fitting only our needs

i made reset cahce on 0.5second silence if at ayah end ..and reset if 2.5silence second in mid ayah (because the cache is already filled with silence)

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment