talatapp/parakeet-realtime-eou-120m-onnx-split

parakeet-realtime-eou-120m ONNX (split decoder/joint, 320 ms chunks)

Streaming preview model. Re-export of nvidia/parakeet_realtime_eou_120m-v1 to three separate ONNX files, int8 dynamic-quantized for CPU / DirectML inference. 120M-parameter cache-aware Conformer with end-of-utterance detection at 80–160 ms latency.

File	Inputs	Outputs
streaming_encoder.int8.onnx	mel [1, 128, F] f32, mel_length [1] i32, cache_last_channel [1, 17, 70, 512] f32, cache_last_time [1, 17, 512, 8] f32, cache_last_channel_len [1] i32	encoder [1, 512, T] f32, encoder_length [1] i32, cache_last_channel_next, cache_last_time_next, cache_last_channel_len_next [1] i32
decoder.int8.onnx	targets [1, 1] i32, target_length [1] i32, h_in [1, 1, 640] f32, c_in [1, 1, 640] f32	decoder [1, 640, 1] f32, h_out, c_out
joint_decision.int8.onnx	encoder [1, 512, T] f32, decoder [1, 640, U] f32	token_id [1, T, U] i32, token_prob [1, T, U] f32, duration [1, T, U] i32

The streaming encoder carries cache state (cache_last_channel, cache_last_time, cache_last_channel_len) between chunks. Initial call: feed zero-initialised cache tensors. Subsequent calls: feed the previous call's *_next outputs.

Why split?

NeMo's own asr_model.export() and altunenes/parakeet-rs's ONNX version fuse the decoder and joint network into a single ONNX file. That's fine for inference engines that call the full RNN-T decoder loop in one go, but doesn't fit pipelines that drive the loop themselves and need the sub-graphs callable independently (e.g. the talat Rust streaming inference layer, which mirrors FluidAudio's macOS CoreML 3-file decomposition).

The PyTorch wrappers are adapted from FluidInference/mobius (Apache 2.0).

Chunk size

This export is the 320 ms variant — 5120-sample chunks at 16 kHz, the talat-inference default. The 160 ms and 1280 ms variants exist in FluidInference's CoreML release; re-exports of those will land in this repo's branches when needed.

Quantization

Per-channel int8 weight-only via onnxruntime.quantization.quantize_dynamic. Activations stay fp32 — no calibration dataset needed.

License

Inherits NVIDIA's parakeet_realtime_eou_120m-v1 license (CC-BY-4.0).