Qwen3-Coder-480B-A35B-Instruct Model Comparison Analysis

Test Configuration

• Full Precision Model: 4× B300 GPU
• NVFP4 Quantized Model: 2× B300 GPU
• Tested Concurrency Levels: 1, 2, 4, 8, 16, 32
• Engine: TensorRT-LLM

Use it with TensorRT-LLM

docker run \
  --gpus all \
  --rm \
  --ipc=host \
  --ulimit memlock=-1:-1 \
  --ulimit stack=67108864 \
  --shm-size=64G \
  -p 8050:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -w /app/tensorrt_llm \
  nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc5 \
  trtllm-serve \
   OPENZEKA/Qwen3-Coder-480B-A35B-Instruct-NVFP4 \
  --host 0.0.0.0 \
  --port 8000 \
  --backend pytorch \
  --max_batch_size 128 \
  --tp_size 2 \
  --kv_cache_free_gpu_memory_fraction 0.9 

Performance Metrics Comparison

1) Time to First Token (TTFT) — ms

TTFT Insights

• The NVFP4 model shows ~26.5% higher TTFT on average across all concurrency levels.
• The largest TTFT regression is at concurrency 8 with +29.6%.
• The best relative TTFT result is at concurrency 16 with +6.9%.

2) Inter-Token Latency (ITL) — ms

ITL Insights

• At low concurrency (1–2), NVFP4 has slightly higher token-to-token latency.
• At medium-to-high concurrency (8–32), NVFP4 performs better.
• At concurrency 32, NVFP4 achieves 18.5% lower ITL.

3) Tokens Per Second (TPS) — tokens/s

TPS Insights

• At low concurrency (1–2), the full-precision model is better.
• At medium-to-high concurrency (4–32), NVFP4 achieves higher TPS.
• At concurrency 32, NVFP4 delivers 19.3% higher TPS.

4) Total Latency — seconds

Latency Insights

• At low concurrency, the full-precision model is better.
• At medium-to-high concurrency, NVFP4 provides lower total latency.

5) Throughput (RPS) — requests/s

Throughput Insights

• At low concurrency, the full-precision model has higher throughput.
• At medium-to-high concurrency, NVFP4 achieves better throughput overall.