Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -89,15 +89,24 @@ prior looped-LLM PTQ study, LoopQ arXiv:2605.16343, tested INT formats only).
|
|
| 89 |
|
| 90 |
## Evals (RTX 5090, vLLM v0.25.1, 64k ctx, fp8 KV)
|
| 91 |
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
|
| 96 |
-
|
|
| 97 |
-
|
|
| 98 |
-
|
|
| 99 |
-
|
|
| 100 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 101 |
|
| 102 |
Judge protocol: blind single-judge, per-article anonymized+shuffled candidates,
|
| 103 |
strict rubric (faithfulness/coverage/conciseness/coherence/concern-validity).
|
|
|
|
| 89 |
|
| 90 |
## Evals (RTX 5090, vLLM v0.25.1, 64k ctx, fp8 KV)
|
| 91 |
|
| 92 |
+
Public harnesses first (lm-eval, reproducible), then my closed harnesses
|
| 93 |
+
(relative signals within this family), then speed.
|
| 94 |
+
|
| 95 |
+
| bench | mode | bf16 | **FP8-Dynamic** |
|
| 96 |
+
|---|---|---|---|
|
| 97 |
+
| GSM8K strict (n=100) | thinking | not measured | **89** |
|
| 98 |
+
| GSM8K flexible | thinking | not measured | **93** |
|
| 99 |
+
| IFEval prompt-strict (n=250) | non-thinking | not measured | **76.4** |
|
| 100 |
+
| IFEval inst-strict | non-thinking | not measured | **83.5** |
|
| 101 |
+
| MMLU-Pro (25/category) | non-thinking | not measured | **62.6** |
|
| 102 |
+
| BBH CoT few-shot | non-thinking | not measured | **64.9** |
|
| 103 |
+
| MultiHop-RAG (gold evidence, n=248) | non-thinking | not measured | **74.2** |
|
| 104 |
+
| Blind-judge summarization (closed, 48 articles, same-judge pair) | non-thinking | 4.61 | 4.55 (noise-level Δ) |
|
| 105 |
+
| Judged faithfulness (closed) | non-thinking | 5.0 | 4.95-5.0 |
|
| 106 |
+
| Judged fabrication / context leaks (closed) | non-thinking | 0% / 0% | 0-2% / 0% |
|
| 107 |
+
| Dictation-rewrite taxonomy (closed, 20 cases) | non-thinking | 18/20 | 18/20 |
|
| 108 |
+
| Decode tok/s, freeform (batch-1, spec-off) | — | 96 | **154** |
|
| 109 |
+
| Decode tok/s, summarize (+ngram spec) | — | 136 | **206** |
|
| 110 |
|
| 111 |
Judge protocol: blind single-judge, per-article anonymized+shuffled candidates,
|
| 112 |
strict rubric (faithfulness/coverage/conciseness/coherence/concern-validity).
|