GGUF + pure-C++ runtime in CrispASR — OmniASR-LLM-300M
#1
by cstr - opened
We've added OmniASR-LLM-300M to CrispASR's omniasr backend. Smaller sibling of LLM-1B — 24-layer encoder (d=1024) + 12-layer LLaMA decoder, same dispatch logic.
The LLM variants give native punctuation/capitalisation (autoregressive) at a slight throughput cost vs the CTC variants. KV-cached decode + flash attention. Dynamic language selection across 1693 FLORES-200 codes.
Pre-quantised GGUFs (Apache-2.0): cstr/omniasr-llm-300m-v2-GGUF
./build/bin/crispasr --backend omniasr -m omniasr-llm-300m-q4_k.gguf -f audio.wav
Larger LLM variant: cstr/omniasr-llm-1b-GGUF. CTC siblings (faster, no native punc): CTC-300M, CTC-1B.