Tom Schelsen

Thank you Nvidia for this. Could you please make it possible, even if not the default, to call a "locally" / internally deployed LLM (OpenAI compatible endpoint) for the SDG phase ? The whole point of self-hosting a fine-tuned embedding model, appart from accurracy, is that the company's data never leave the premises. Thanks

Yes it is quite recent that VLLM publishes an ARM image, was around version 0.11-0.13 (not sure the exact one)

Why specific Docker image for GH200, what are the differences with stock vllm-openai:vX-aarch64 ? Thanks

If I understood correctly, the two figures in https://huggingface.co/blog/nvidia/nemotron-speech-asr-scaling-voice-agents#results-throughput-accuracy-and-speed-at-scale compare a 1.1B model with a 600M one. This is misleading, as demonstrating the added value of the caching mechanism should be done on models of the same size, otherwise half of the "3x" gain could be attributed to the sole parameter count difference.