Nemotron 3B on RTX 3060: Rust/CUDA prototype results across all three modes
I’m building a Rust/CUDA server for Nemotron-Labs-Diffusion-3B without PyTorch or Transformers in the inference runtime. Here are preliminary measurements of AR, block diffusion, and self-speculation, with Qwythos 9B Q5 in llama.cpp as a practical baseline.
Setup: RTX 3060 12GB, 16,384-token context limit, one request at a time, one backend resident. After a warmup, each configuration ran the same tutorial prompt about merging sorted arrays three times with a 512-token output cap.
Throughput is median API completion tokens / request-to-finish seconds, including prompt processing, HTTP overhead, and thinking tokens.
| Configuration | Tokens/sec | Sampled peak VRAM, MiB |
|---|---|---|
| Qwythos 9B Q5 / llama.cpp | 40.1 | 5,945 |
| Nemotron AR / Rust-CUDA | 33.7 | 10,865 |
| Nemotron block diffusion / Rust-CUDA | 26.5 | 10,865 |
| Nemotron self-speculation / Rust-CUDA | 6.7 | 10,865 |
VRAM is whole-device usage sampled across each configuration’s test suite.
Settings: Native Nemotron used BF16 weights, FP32 KV, temperature 0, block length 32, and threshold 0.9 for both diffusion modes. Qwythos used Q5_K_M weights, q4_0 KV, Flash Attention, low reasoning, a 512-token thinking budget, temperature 0.6, top_p 0.95, top_k 20, and llama.cpp b10603.
Important qualifications:
- Our self-speculation threshold 0.9 differs from NVIDIA’s reference default of 0.0. We also did not load the optional drafter LoRA. The 6.7 figure measures this prototype configuration.
- All tutorial runs hit the cap. Qwythos’s 512 tokens were entirely thinking, with no final answer, so this measures token throughput rather than completed-answer speed.
- Precision, sampling, tokenizers, and implementations differ; these results cannot isolate the decoding algorithm’s contribution.
Workload matters: self-speculation completed a correct 20-prime list in 0.67s, versus 1.43s for AR. Block diffusion returned 21 numbers. In a separate Rust-with-tools-declared check, Qwythos’s code passed compilation/tests; the native prototype returned tool-format errors.
The full report and reproduction instructions include source/runtime pins and further latency checks.
Qwythos remains my practical Pi choice for now, while Nemotron needs reference-matched tuning and integration work. Has anyone measured this checkpoint on an RTX 3060 with threshold 0.0 or the drafter LoRA?
Full report link: https://github.com/InServiceOfX/claw-work/blob/21d39704dc349de0c5ae94ba401856b70cb384f5/Deployments/Scripts/llama-cpp-server/MODE_COMPARISON.md