Drop benchmarks not present in chart
Browse files
README.md
CHANGED
|
@@ -317,27 +317,16 @@ After compression and fine-tuning, the BF16 checkpoint was quantized to FP8.
|
|
| 317 |
- **Workload shape:** Sweep across different levels of concurrency. 180s measure windows with ramp-up and cooldown excluded. 8k/16k workload as in the original model's card.
|
| 318 |
- **Estimated Peak Memory Usage:** KV cache utilization is monitored during the phase and we estimate memory usage as follows: `model_ weights_{gb} + kv_ cache_{usage_pct} × (nvml_used_{gb} − model_ weights_{gb})`
|
| 319 |
|
| 320 |
-
### Long Context
|
| 321 |
-

|
| 322 |
-
|
| 323 |
-
Pulsar preserves strong long-context behavior after compression, tracking the Nemotron-3-Nano-30B-A3B baseline closely across retrieval-heavy and full-suite long-context evaluations. Results are reported for LongBench v1, AA-LCR, NIAH, and RULER groupings up to 256k context.
|
| 324 |
-
|
| 325 |
### Evaluation Methodology
|
| 326 |
|
| 327 |
Benchmark scores were obtained with the following setups. Methodology varies by benchmark family.
|
| 328 |
|
| 329 |
-
#### MMLU-Pro, AIME25, GPQA
|
| 330 |
|
| 331 |
- **Framework**: [Nemo-Skills](https://github.com/NVIDIA-NeMo/Skills)
|
| 332 |
- **Inference**: vLLM 0.18.0
|
| 333 |
- **Decode**: temp 1.0, top_p 1.0
|
| 334 |
|
| 335 |
-
#### BFCL v4
|
| 336 |
-
|
| 337 |
-
- **Framework**: [EvalScope](https://github.com/EvalScope/EvalScope) 1.4.1
|
| 338 |
-
- **Inference**: vLLM 0.18.0
|
| 339 |
-
- **Decode**: temp 0.6, top_p 0.95, parallel_tool_calls true
|
| 340 |
-
|
| 341 |
#### Tau2-bench (Telecom)
|
| 342 |
|
| 343 |
- **Framework**: [EvalScope](https://github.com/EvalScope/EvalScope) 1.4.1
|
|
@@ -346,13 +335,6 @@ Benchmark scores were obtained with the following setups. Methodology varies by
|
|
| 346 |
- **Decode (judge / user sim)**: temp 0.7, top_p 0.8, top_k 20, min_p 0
|
| 347 |
- **Repro**: telecom subset; max steps 100; 3 runs; judge parser hermes
|
| 348 |
|
| 349 |
-
#### LongBench v1
|
| 350 |
-
|
| 351 |
-
- **Framework**: CompactifAI evaluation interface with the LongBench harness, benchmark version v1, full suite.
|
| 352 |
-
- **Dataset**: `Xnhyacinth/LongBench`, all LongBench v1 tasks, no subset limit.
|
| 353 |
-
- **Inference**: vLLM 0.18.0
|
| 354 |
-
- **Decode**: max_tokens 1024, temperature 1.0, top_p 0.95, chat template enabled.
|
| 355 |
-
|
| 356 |
#### AA-LCR
|
| 357 |
|
| 358 |
- **Framework**: [EvalScope](https://github.com/EvalScope/EvalScope) 1.4.1
|
|
@@ -361,22 +343,6 @@ Benchmark scores were obtained with the following setups. Methodology varies by
|
|
| 361 |
- **Decode**: temperature 1.0, top_p 1.0
|
| 362 |
- **Decode (judge)**: judge top_p 0.8, top_k 20, min_p 0.0; pattern scoring.
|
| 363 |
|
| 364 |
-
#### NIAH
|
| 365 |
-
|
| 366 |
-
- **Framework**: [Nemo-Skills](https://github.com/NVIDIA-NeMo/Skills)
|
| 367 |
-
- **Dataset / tasks**: RULER NIAH variants: single needle, multi-key, multi-query, and multi-value.
|
| 368 |
-
- **Context lengths**: 4096, 8192, 16384, 32768, 65536, 131072, and 262144 tokens.
|
| 369 |
-
- **Inference**: vLLM 0.18.0
|
| 370 |
-
- **Decode**: max_tokens 128, temperature 1.0, top_p 1.0, top_k 20, min_p 0.0, tokenizer `meta-llama/Llama-3.1-8B`.
|
| 371 |
-
|
| 372 |
-
#### RULER
|
| 373 |
-
|
| 374 |
-
- **Framework**: [Nemo-Skills](https://github.com/NVIDIA-NeMo/Skills)
|
| 375 |
-
- **Dataset / tasks**: RULER task families including VT, CWE, FWE, QA, and NIAH; CWE included.
|
| 376 |
-
- **Context lengths**: 4096, 8192, 16384, 32768, 65536, 131072, and 262144 tokens.
|
| 377 |
-
- **Inference**: vLLM 0.18.0
|
| 378 |
-
- **Decode**: temperature 1.0, top_p 1.0, top_k 20, min_p 0.0; task-specific output limits: NIAH 128, VT 30, CWE 120, FWE 50, QA 32.
|
| 379 |
-
|
| 380 |
---
|
| 381 |
|
| 382 |
## Languages
|
|
|
|
| 317 |
- **Workload shape:** Sweep across different levels of concurrency. 180s measure windows with ramp-up and cooldown excluded. 8k/16k workload as in the original model's card.
|
| 318 |
- **Estimated Peak Memory Usage:** KV cache utilization is monitored during the phase and we estimate memory usage as follows: `model_ weights_{gb} + kv_ cache_{usage_pct} × (nvml_used_{gb} − model_ weights_{gb})`
|
| 319 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 320 |
### Evaluation Methodology
|
| 321 |
|
| 322 |
Benchmark scores were obtained with the following setups. Methodology varies by benchmark family.
|
| 323 |
|
| 324 |
+
#### MMLU-Pro, AIME25, GPQA-D, LiveCodeBench, IFBench, AA-LCR, SciCode
|
| 325 |
|
| 326 |
- **Framework**: [Nemo-Skills](https://github.com/NVIDIA-NeMo/Skills)
|
| 327 |
- **Inference**: vLLM 0.18.0
|
| 328 |
- **Decode**: temp 1.0, top_p 1.0
|
| 329 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 330 |
#### Tau2-bench (Telecom)
|
| 331 |
|
| 332 |
- **Framework**: [EvalScope](https://github.com/EvalScope/EvalScope) 1.4.1
|
|
|
|
| 335 |
- **Decode (judge / user sim)**: temp 0.7, top_p 0.8, top_k 20, min_p 0
|
| 336 |
- **Repro**: telecom subset; max steps 100; 3 runs; judge parser hermes
|
| 337 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 338 |
#### AA-LCR
|
| 339 |
|
| 340 |
- **Framework**: [EvalScope](https://github.com/EvalScope/EvalScope) 1.4.1
|
|
|
|
| 343 |
- **Decode**: temperature 1.0, top_p 1.0
|
| 344 |
- **Decode (judge)**: judge top_p 0.8, top_k 20, min_p 0.0; pattern scoring.
|
| 345 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 346 |
---
|
| 347 |
|
| 348 |
## Languages
|