Join the community of Machine Learners and AI enthusiasts.
Sign Up
The suite answers how fast for 53 models and how good for 25, and the gap sits exactly where the interesting models are.
Counting rows in your own CSVs: 144 quantized entries across the perf tables, 2 across the accuracy tables (both Cosmos-Reason2). So every W4A16 LLM variant ships a throughput number and no quality number, and those are the ones carrying the win. Qwen3-1.7B goes 49.31 to 189.02 tps on agx_thor with FlashHead-W4A16. Llama-3.2-1B goes 77.30 to 284.76. That is 3.8x and 3.7x.
The headline question feels it hardest. The two smallest LLMs you measured, gemma-3-270m-it and Qwen3-0.6B, have no accuracy row at all. So how small can it go and still run is answered, and still be useful is the half not yet in the data.
Worth saying what the tables do prove, because it reads as under-sold: FlashHead is accuracy-neutral. Llama-3.2-1B-FlashHead matches the base model to two decimals on all seven tasks, and the 3B and Qwen3-1.7B are within 0.01 everywhere. A 1.26x with no quality argument to have is a rare thing to be able to show.
On use cases, mine is agent loops running on-device, where tokens/sec decides how many tool-call retries fit inside a latency budget. At that point a 3.8x is not a faster agent, it is a different agent.
Is the W4A16 accuracy sweep just queued, or is getting lm-eval through a TensorRT engine on device the actual blocker?
Yes, accuracy numbers are a bit thin! For FlashHead you are correct. We have written a paper on FlashHead (see https://arxiv.org/abs/2603.14591) where we show that it is consistently accuracy-preserving: typically within ~99.9% top-1 containment.
For the GenAI models on Jetson: We are actually using the TensorRT vLLM Docker image, so lm-eval has a path and is not blocked by the TRT engine. For example, for Qwen3.6 we have internal accuracy evaluations but not yet pushed.
For the more classical DL models, the benchmark suite mostly compares to quantized variants where we do need to push for more on-device accuracy evaluations.
The notion of picking the largest model that fits the latency (and memory) budget is quite common and that's one focus of our GenAI models . On the classical DL side it is a bit different, edge deployment use-cases are often around making an existing, well performing model faster.
The containment number covers the part that already had accuracy rows.
FlashHead is the one variant in your tables with a quality column, and it is clean: Llama-3.2-1B-FlashHead matches base to two decimals on all seven tasks. The 144 rows with throughput and no quality are the W4A16 ones, and those are the rows carrying the 3.7x and 3.8x.
Worth saying why containment is the right metric for a head and still not a substitute downstream. Top-1 containment is a per-step property and generation is autoregressive, so misses compound: 99.9% per token leaves about 82% of 200-token completions with no miss at all, under independence. Most misses will not change the sampled token and fewer will change the answer. But the failure is trajectory-shaped, and a per-step average is the one summary that hides it. That is an argument for your seven-task table, not against it.
The TRT vLLM image having an lm-eval path is the useful half of your answer. It moves the W4A16 gap from a tooling problem to a queue problem, which is a much better place to be.
If one accuracy row goes up next, which moves your own decision more: Qwen3-1.7B at W4A16, or gemma-3-270m-it where there is no quality number at all?
