NullSense commited on
Commit
0d52013
·
verified ·
1 Parent(s): cd252a8

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +18 -9
README.md CHANGED
@@ -89,15 +89,24 @@ prior looped-LLM PTQ study, LoopQ arXiv:2605.16343, tested INT formats only).
89
 
90
  ## Evals (RTX 5090, vLLM v0.25.1, 64k ctx, fp8 KV)
91
 
92
- | gate | bf16 | **FP8-Dynamic** |
93
- |---|---|---|
94
- | GSM8K strict / flexible (n=100, thinking) | — | **89 / 93** |
95
- | Blind-judge summarization overall (48-article set, same-judge pair) | 4.61 | 4.55 (noise-level Δ) |
96
- | Judged faithfulness / fabrication / leaks | 5.0 / 0% / 0% | 4.95-5.0 / 0-2% / 0% |
97
- | IFEval prompt-strict / inst-strict (n=250, non-thinking) | | **76.4 / 83.5** |
98
- | MultiHop-RAG (multi-doc news QA) / MMLU-Pro / BBH (non-thinking) | | **74.2 / 62.6 / 64.9** |
99
- | Dictation-rewrite taxonomy (20 cases) | 18/20 | 18/20 |
100
- | Decode tok/s (batch-1: freeform / summarize+ngram-spec) | 96 / 136 | **154 / 206** |
 
 
 
 
 
 
 
 
 
101
 
102
  Judge protocol: blind single-judge, per-article anonymized+shuffled candidates,
103
  strict rubric (faithfulness/coverage/conciseness/coherence/concern-validity).
 
89
 
90
  ## Evals (RTX 5090, vLLM v0.25.1, 64k ctx, fp8 KV)
91
 
92
+ Public harnesses first (lm-eval, reproducible), then my closed harnesses
93
+ (relative signals within this family), then speed.
94
+
95
+ | bench | mode | bf16 | **FP8-Dynamic** |
96
+ |---|---|---|---|
97
+ | GSM8K strict (n=100) | thinking | not measured | **89** |
98
+ | GSM8K flexible | thinking | not measured | **93** |
99
+ | IFEval prompt-strict (n=250) | non-thinking | not measured | **76.4** |
100
+ | IFEval inst-strict | non-thinking | not measured | **83.5** |
101
+ | MMLU-Pro (25/category) | non-thinking | not measured | **62.6** |
102
+ | BBH CoT few-shot | non-thinking | not measured | **64.9** |
103
+ | MultiHop-RAG (gold evidence, n=248) | non-thinking | not measured | **74.2** |
104
+ | Blind-judge summarization (closed, 48 articles, same-judge pair) | non-thinking | 4.61 | 4.55 (noise-level Δ) |
105
+ | Judged faithfulness (closed) | non-thinking | 5.0 | 4.95-5.0 |
106
+ | Judged fabrication / context leaks (closed) | non-thinking | 0% / 0% | 0-2% / 0% |
107
+ | Dictation-rewrite taxonomy (closed, 20 cases) | non-thinking | 18/20 | 18/20 |
108
+ | Decode tok/s, freeform (batch-1, spec-off) | — | 96 | **154** |
109
+ | Decode tok/s, summarize (+ngram spec) | — | 136 | **206** |
110
 
111
  Judge protocol: blind single-judge, per-article anonymized+shuffled candidates,
112
  strict rubric (faithfulness/coverage/conciseness/coherence/concern-validity).