[audio.cpp] C++/GGML based Voxtral Mini 4B Realtime impl. 16.39x (0.0610 RTF) realtime offline, 170ms TTFT streaming on RTX 5090
#48
by audio-cpp - opened
Check the new Voxtral Mini 4B Realtime impl released in https://github.com/0xShug0/audio.cpp
On RTX 5090:
Cli:
| GGUF | Measured audio | Measured session.wall_ms | RTF | Speed |
|---|---|---|---|---|
| BF16 | 27.394s | 2312.030 ms | 0.0844 | 11.85x RT |
| Q8_0 | 27.394s | 1671.272 ms | 0.0610 | 16.39x RT |
Server:
Streaming mode + multipart upload:
| GGUF | Server TTFT | Client observed TTFT | Peak VRAM |
|---|---|---|---|
| BF16 | 208.786 ms | 210.831 ms | 10082 MiB |
| Q8_0 | 170.508 ms | 173.336 ms | 6495 MiB |
Longform 30min audio (streaming mode + multipart upload)
| GGUF | Client wall | RTF | Speed | Peak VRAM |
|---|---|---|---|---|
| BF16 | 411.807 s | 0.2288 | 4.37x RT | 12595 MiB |
| Q8_0 | 358.631 s | 0.1992 | 5.02x RT | 8990 MiB |