腾云智算（Tenyunw）专注于大模型推理优化和GPU云基础设施。
我们在做什么：
- 推理加速：实现了 Eagle3 for training on Qwen3-8B（20,000+ downloads），显著提升LLM推理速度
- GPU云平台：为AIGC应用提供性价比最优的推理部署方案
- 量化优化：基于QAT+ DPO的训练和推理框架，帮助客户在Blackwell架构硬件上翻倍并发能力
核心团队来自腾讯云、华为云、面壁智能等，在AI基础设施领域有15年以上实战经验。
🎁 限时福利： 如果你在做大模型应用部署，我们提供免费的推理优化咨询（30分钟技术诊断），帮你分析：
- 当前部署方案的性能瓶颈
- 成本优化的具体路径
- 适合你场景的推理加速方案
适合谁：
- 月推理费用 > $5K的团队
- 需要降低推理成本30%+
- 考虑自建推理集群
联系方式：
- 创始人/CTO Rocky（前腾讯云行业架构师团队负责人）
- 微信：[rocket-assassin]
- LinkedIn: [https://www.linkedin.com/in/wangchao0808/]
- 官网：https://www.tenyunw.com/
- Email: rockywang@tenyunw.com
We help AI builders deploy faster and cheaper. Let's talk.
datasets: - abisee/cnn_dailymail - nvidia/Nemotron-Post-Training-Dataset-v2 base_model: - MiniMaxAI/MiniMax-M2.1 base_model_relation: quantized license: mit pipeline_tag: text-generation
MiniMax-M2.1-NVFP4
Format: NVFP4 — optimal partial quantization of weights & activations to NVFP4.
Base model: MiniMax-M2.1-NVFP4
How it was made: AutoQuantized with NVIDIA Model-Optimizer (NVFP4), using the default calibration mix. (cnn_dailymail and nemotron-post-training-dataset-v2)
Check the original model card for information about this model.
sglang Inference Note:
vim /sgl-workspace/sglang/python/sglang/srt/layers/quantization/modelopt_quant.py
change the code in 1517 line like this:
), f"Expected {name}_weight_scale.dim(2) == {expected_blocks[name]}, got {weight_scale.shape[-1]}"
else:
pass
# For other backends, ensure the per-input block dimension is aligned to 16.
#assert (
# weight_scale.shape[assert_dim] % block_size == 0
#), f"Expected {name}_weight_scale.dim({assert_dim}) to be divisible by {block_size}"
deploy command MiniMax-M2.1-NVFP4 on sglang:
python3 -m sglang.launch_server --model-path MiniMax-M2.1-NVFP4/ --quantization modelopt_fp4 --tp 8 --attention-backend flashinfer --trust-remote-code
perf
We performed deployment on 8x 5090, and the stress test performance data is provided below.
Benchmarking summary:
+-----------------------------------+-----------+
| Key | Value |
+===================================+===========+
| Time taken for tests (s) | 10.7085 |
+-----------------------------------+-----------+
| Number of concurrency | 1 |
+-----------------------------------+-----------+
| Request rate (req/s) | -1 |
+-----------------------------------+-----------+
| Total requests | 1 |
+-----------------------------------+-----------+
| Succeed requests | 1 |
+-----------------------------------+-----------+
| Failed requests | 0 |
+-----------------------------------+-----------+
| Output token throughput (tok/s) | 47.8126 |
+-----------------------------------+-----------+
| Total token throughput (tok/s) | 143.438 |
+-----------------------------------+-----------+
| Request throughput (req/s) | 0.0934 |
+-----------------------------------+-----------+
| Average latency (s) | 10.7085 |
+-----------------------------------+-----------+
| Average time to first token (s) | 0.5682 |
+-----------------------------------+-----------+
| Average time per output token (s) | 0.0198 |
+-----------------------------------+-----------+
| Average inter-token latency (s) | 0.0202 |
+-----------------------------------+-----------+
| Average input tokens per request | 1024 |
+-----------------------------------+-----------+
| Average output tokens per request | 512 |
+-----------------------------------+-----------+
2026-01-07 04:00:24 - evalscope - INFO:
Percentile results:
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
| Percentiles | TTFT (s) | ITL (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Output (tok/s) | Total (tok/s) |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
| 10% | 0.5682 | 0.0196 | 0.0198 | 10.7085 | 1024 | 512 | 47.8126 | 143.4379 |
| 25% | 0.5682 | 0.0197 | 0.0198 | 10.7085 | 1024 | 512 | 47.8126 | 143.4379 |
| 50% | 0.5682 | 0.0198 | 0.0198 | 10.7085 | 1024 | 512 | 47.8126 | 143.4379 |
| 66% | 0.5682 | 0.0199 | 0.0198 | 10.7085 | 1024 | 512 | 47.8126 | 143.4379 |
| 75% | 0.5682 | 0.0199 | 0.0198 | 10.7085 | 1024 | 512 | 47.8126 | 143.4379 |
| 80% | 0.5682 | 0.0199 | 0.0198 | 10.7085 | 1024 | 512 | 47.8126 | 143.4379 |
| 90% | 0.5682 | 0.0201 | 0.0198 | 10.7085 | 1024 | 512 | 47.8126 | 143.4379 |
| 95% | 0.5682 | 0.0204 | 0.0198 | 10.7085 | 1024 | 512 | 47.8126 | 143.4379 |
| 98% | 0.5682 | 0.0393 | 0.0198 | 10.7085 | 1024 | 512 | 47.8126 | 143.4379 |
| 99% | 0.5682 | 0.0396 | 0.0198 | 10.7085 | 1024 | 512 | 47.8126 | 143.4379 |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
Benchmarking summary:
+-----------------------------------+-----------+
| Key | Value |
+===================================+===========+
| Time taken for tests (s) | 24.0981 |
+-----------------------------------+-----------+
| Number of concurrency | 16 |
+-----------------------------------+-----------+
| Request rate (req/s) | -1 |
+-----------------------------------+-----------+
| Total requests | 16 |
+-----------------------------------+-----------+
| Succeed requests | 16 |
+-----------------------------------+-----------+
| Failed requests | 0 |
+-----------------------------------+-----------+
| Output token throughput (tok/s) | 339.944 |
+-----------------------------------+-----------+
| Total token throughput (tok/s) | 1019.83 |
+-----------------------------------+-----------+
| Request throughput (req/s) | 0.664 |
+-----------------------------------+-----------+
| Average latency (s) | 24.0845 |
+-----------------------------------+-----------+
| Average time to first token (s) | 5.7343 |
+-----------------------------------+-----------+
| Average time per output token (s) | 0.0359 |
+-----------------------------------+-----------+
| Average inter-token latency (s) | 0.0358 |
+-----------------------------------+-----------+
| Average input tokens per request | 1024 |
+-----------------------------------+-----------+
| Average output tokens per request | 512 |
+-----------------------------------+-----------+
2026-01-07 04:11:34 - evalscope - INFO:
Percentile results:
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
| Percentiles | TTFT (s) | ITL (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Output (tok/s) | Total (tok/s) |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
| 10% | 2.4771 | 0.0275 | 0.0301 | 24.0913 | 1024 | 512 | 21.2486 | 63.7458 |
| 25% | 3.6108 | 0.028 | 0.0313 | 24.0928 | 1024 | 512 | 21.2493 | 63.7479 |
| 50% | 5.8605 | 0.0284 | 0.0357 | 24.0939 | 1024 | 512 | 21.2507 | 63.7521 |
| 66% | 6.985 | 0.0287 | 0.0379 | 24.094 | 1024 | 512 | 21.251 | 63.753 |
| 75% | 8.11 | 0.0289 | 0.0401 | 24.095 | 1024 | 512 | 21.252 | 63.7559 |
| 80% | 8.11 | 0.029 | 0.0401 | 24.095 | 1024 | 512 | 21.252 | 63.7559 |
| 90% | 8.7294 | 0.0295 | 0.0423 | 24.0957 | 1024 | 512 | 21.2525 | 63.7576 |
| 95% | 9.3849 | 0.0298 | 0.0445 | 24.0971 | 1024 | 512 | 21.3819 | 64.1458 |
| 98% | 9.3849 | 0.0308 | 0.0445 | 24.0971 | 1024 | 512 | 21.3819 | 64.1458 |
| 99% | 9.3849 | 0.0328 | 0.0445 | 24.0971 | 1024 | 512 | 21.3819 | 64.1458 |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
Benchmarking summary:
+-----------------------------------+-----------+
| Key | Value |
+===================================+===========+
| Time taken for tests (s) | 35.9271 |
+-----------------------------------+-----------+
| Number of concurrency | 32 |
+-----------------------------------+-----------+
| Request rate (req/s) | -1 |
+-----------------------------------+-----------+
| Total requests | 32 |
+-----------------------------------+-----------+
| Succeed requests | 32 |
+-----------------------------------+-----------+
| Failed requests | 0 |
+-----------------------------------+-----------+
| Output token throughput (tok/s) | 456.034 |
+-----------------------------------+-----------+
| Total token throughput (tok/s) | 1368.1 |
+-----------------------------------+-----------+
| Request throughput (req/s) | 0.8907 |
+-----------------------------------+-----------+
| Average latency (s) | 35.914 |
+-----------------------------------+-----------+
| Average time to first token (s) | 10.2324 |
+-----------------------------------+-----------+
| Average time per output token (s) | 0.0503 |
+-----------------------------------+-----------+
| Average inter-token latency (s) | 0.0501 |
+-----------------------------------+-----------+
| Average input tokens per request | 1024 |
+-----------------------------------+-----------+
| Average output tokens per request | 512 |
+-----------------------------------+-----------+
2026-01-07 04:14:20 - evalscope - INFO:
Percentile results:
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
| Percentiles | TTFT (s) | ITL (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Output (tok/s) | Total (tok/s) |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
| 10% | 3.5765 | 0.0338 | 0.0369 | 35.913 | 1024 | 512 | 14.2526 | 42.7577 |
| 25% | 5.8246 | 0.0351 | 0.0413 | 35.9153 | 1024 | 512 | 14.254 | 42.7621 |
| 50% | 10.3224 | 0.0356 | 0.0501 | 35.9185 | 1024 | 512 | 14.2545 | 42.7636 |
| 66% | 13.6935 | 0.0359 | 0.0564 | 35.9196 | 1024 | 512 | 14.2556 | 42.7667 |
| 75% | 14.8185 | 0.0362 | 0.0589 | 35.9198 | 1024 | 512 | 14.2561 | 42.7682 |
| 80% | 15.9416 | 0.0365 | 0.0611 | 35.9199 | 1024 | 512 | 14.2561 | 42.7683 |
| 90% | 17.0672 | 0.037 | 0.0633 | 35.9233 | 1024 | 512 | 14.2567 | 42.77 |
| 95% | 17.6148 | 0.0373 | 0.0655 | 35.9244 | 1024 | 512 | 14.2573 | 42.7718 |
| 98% | 17.6331 | 0.0379 | 0.0677 | 35.9256 | 1024 | 512 | 14.3066 | 42.9197 |
| 99% | 17.6331 | 0.0394 | 0.0677 | 35.9256 | 1024 | 512 | 14.3066 | 42.9197 |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+