daniicruzz commited on
Commit
96b7ede
·
verified ·
1 Parent(s): ecfecfc

Drop benchmarks not present in chart

Browse files
Files changed (1) hide show
  1. README.md +1 -35
README.md CHANGED
@@ -317,27 +317,16 @@ After compression and fine-tuning, the BF16 checkpoint was quantized to FP8.
317
  - **Workload shape:** Sweep across different levels of concurrency. 180s measure windows with ramp-up and cooldown excluded. 8k/16k workload as in the original model's card.
318
  - **Estimated Peak Memory Usage:** KV cache utilization is monitored during the phase and we estimate memory usage as follows: `model_ weights_{gb} + kv_ cache_{usage_pct} × (nvml_used_{gb} − model_ weights_{gb})`
319
 
320
- ### Long Context
321
- ![Long-context benchmark results](assets/long_context_comparison.png)
322
-
323
- Pulsar preserves strong long-context behavior after compression, tracking the Nemotron-3-Nano-30B-A3B baseline closely across retrieval-heavy and full-suite long-context evaluations. Results are reported for LongBench v1, AA-LCR, NIAH, and RULER groupings up to 256k context.
324
-
325
  ### Evaluation Methodology
326
 
327
  Benchmark scores were obtained with the following setups. Methodology varies by benchmark family.
328
 
329
- #### MMLU-Pro, AIME25, GPQA:d, LiveCodeBench, IFBench, AA-LCR, SciCode
330
 
331
  - **Framework**: [Nemo-Skills](https://github.com/NVIDIA-NeMo/Skills)
332
  - **Inference**: vLLM 0.18.0
333
  - **Decode**: temp 1.0, top_p 1.0
334
 
335
- #### BFCL v4
336
-
337
- - **Framework**: [EvalScope](https://github.com/EvalScope/EvalScope) 1.4.1
338
- - **Inference**: vLLM 0.18.0
339
- - **Decode**: temp 0.6, top_p 0.95, parallel_tool_calls true
340
-
341
  #### Tau2-bench (Telecom)
342
 
343
  - **Framework**: [EvalScope](https://github.com/EvalScope/EvalScope) 1.4.1
@@ -346,13 +335,6 @@ Benchmark scores were obtained with the following setups. Methodology varies by
346
  - **Decode (judge / user sim)**: temp 0.7, top_p 0.8, top_k 20, min_p 0
347
  - **Repro**: telecom subset; max steps 100; 3 runs; judge parser hermes
348
 
349
- #### LongBench v1
350
-
351
- - **Framework**: CompactifAI evaluation interface with the LongBench harness, benchmark version v1, full suite.
352
- - **Dataset**: `Xnhyacinth/LongBench`, all LongBench v1 tasks, no subset limit.
353
- - **Inference**: vLLM 0.18.0
354
- - **Decode**: max_tokens 1024, temperature 1.0, top_p 0.95, chat template enabled.
355
-
356
  #### AA-LCR
357
 
358
  - **Framework**: [EvalScope](https://github.com/EvalScope/EvalScope) 1.4.1
@@ -361,22 +343,6 @@ Benchmark scores were obtained with the following setups. Methodology varies by
361
  - **Decode**: temperature 1.0, top_p 1.0
362
  - **Decode (judge)**: judge top_p 0.8, top_k 20, min_p 0.0; pattern scoring.
363
 
364
- #### NIAH
365
-
366
- - **Framework**: [Nemo-Skills](https://github.com/NVIDIA-NeMo/Skills)
367
- - **Dataset / tasks**: RULER NIAH variants: single needle, multi-key, multi-query, and multi-value.
368
- - **Context lengths**: 4096, 8192, 16384, 32768, 65536, 131072, and 262144 tokens.
369
- - **Inference**: vLLM 0.18.0
370
- - **Decode**: max_tokens 128, temperature 1.0, top_p 1.0, top_k 20, min_p 0.0, tokenizer `meta-llama/Llama-3.1-8B`.
371
-
372
- #### RULER
373
-
374
- - **Framework**: [Nemo-Skills](https://github.com/NVIDIA-NeMo/Skills)
375
- - **Dataset / tasks**: RULER task families including VT, CWE, FWE, QA, and NIAH; CWE included.
376
- - **Context lengths**: 4096, 8192, 16384, 32768, 65536, 131072, and 262144 tokens.
377
- - **Inference**: vLLM 0.18.0
378
- - **Decode**: temperature 1.0, top_p 1.0, top_k 20, min_p 0.0; task-specific output limits: NIAH 128, VT 30, CWE 120, FWE 50, QA 32.
379
-
380
  ---
381
 
382
  ## Languages
 
317
  - **Workload shape:** Sweep across different levels of concurrency. 180s measure windows with ramp-up and cooldown excluded. 8k/16k workload as in the original model's card.
318
  - **Estimated Peak Memory Usage:** KV cache utilization is monitored during the phase and we estimate memory usage as follows: `model_ weights_{gb} + kv_ cache_{usage_pct} × (nvml_used_{gb} − model_ weights_{gb})`
319
 
 
 
 
 
 
320
  ### Evaluation Methodology
321
 
322
  Benchmark scores were obtained with the following setups. Methodology varies by benchmark family.
323
 
324
+ #### MMLU-Pro, AIME25, GPQA-D, LiveCodeBench, IFBench, AA-LCR, SciCode
325
 
326
  - **Framework**: [Nemo-Skills](https://github.com/NVIDIA-NeMo/Skills)
327
  - **Inference**: vLLM 0.18.0
328
  - **Decode**: temp 1.0, top_p 1.0
329
 
 
 
 
 
 
 
330
  #### Tau2-bench (Telecom)
331
 
332
  - **Framework**: [EvalScope](https://github.com/EvalScope/EvalScope) 1.4.1
 
335
  - **Decode (judge / user sim)**: temp 0.7, top_p 0.8, top_k 20, min_p 0
336
  - **Repro**: telecom subset; max steps 100; 3 runs; judge parser hermes
337
 
 
 
 
 
 
 
 
338
  #### AA-LCR
339
 
340
  - **Framework**: [EvalScope](https://github.com/EvalScope/EvalScope) 1.4.1
 
343
  - **Decode**: temperature 1.0, top_p 1.0
344
  - **Decode (judge)**: judge top_p 0.8, top_k 20, min_p 0.0; pattern scoring.
345
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
346
  ---
347
 
348
  ## Languages