Update README.md
Browse files
README.md
CHANGED
|
@@ -216,4 +216,15 @@ uv pip install cloudpickle msgspec zmq blake3 cachetools prometheus_client fasta
|
|
| 216 |
|
| 217 |
|
| 218 |
### Accuracy Comparison
|
| 219 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 216 |
|
| 217 |
|
| 218 |
### Accuracy Comparison
|
| 219 |
+
| Category | Benchmark | ibm-granite/granite-4.0-h-tiny | RedHatAI/granite-4.0-h-tiny-FP8-dynamic | Recovery (%) |
|
| 220 |
+
|:--|:--|:-:|:-:|:-:|
|
| 221 |
+
| **OpenLLM V1** | ARC-Challenge (Acc, 25-shot) | 62.97 | 62.37 | 99.05 |
|
| 222 |
+
| | GSM8K (Strict-Match, 5-shot) | 80.44 | 79.83 | 99.24 |
|
| 223 |
+
| | HellaSwag (Acc-Norm, 10-shot) | 61.75 | 61.56 | 99.69 |
|
| 224 |
+
| | MMLU (Acc, 5-shot) | 66.46 | 66.33 | 99.80 |
|
| 225 |
+
| | TruthfulQA (MC2, 0-shot) | 58.48 | 58.11 | 99.37 |
|
| 226 |
+
| | Winogrande (Acc, 5-shot) | 71.43 | 72.30 | 101.22 |
|
| 227 |
+
| | **Average** | **66.92** | **66.75** | **99.73** |
|
| 228 |
+
| **OpenLLM V2** | IFEval (Inst Level Strict Acc, 0-shot) | 70.62 | 71.10 | 100.68 |
|
| 229 |
+
| | MMLU-Pro (Acc, 5-shot) | 46.24 | 46.05 | 99.59 |
|
| 230 |
+
| | **Average** | **58.43** | **58.58** | **100.13** |
|