Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -111,24 +111,41 @@ These compressed representations are highly sensitive to quantization noise. NVI
|
|
| 111 |
|
| 112 |
## Evaluation
|
| 113 |
|
| 114 |
-
MMLU-Pro
|
| 115 |
-
|
| 116 |
-
|
|
| 117 |
-
|
| 118 |
-
|
|
| 119 |
-
|
|
| 120 |
-
|
|
| 121 |
-
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
|
|
| 125 |
-
|
| 126 |
-
|
|
| 127 |
-
|
|
| 128 |
-
|
|
| 129 |
-
|
|
| 130 |
-
|
| 131 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 132 |
|
| 133 |
## Citation
|
| 134 |
|
|
|
|
| 111 |
|
| 112 |
## Evaluation
|
| 113 |
|
| 114 |
+
### MMLU-Pro Overall Results
|
| 115 |
+
|
| 116 |
+
| Model | Accuracy | Correct | Total |
|
| 117 |
+
|-------|----------|---------|-------|
|
| 118 |
+
| **BF16 (baseline)** | **24.83%** | 2988 | 12032 |
|
| 119 |
+
| **NVFP4 (this model)** | **23.55%** | 2834 | 12032 |
|
| 120 |
+
| **Difference** | **-1.28%** | -154 | - |
|
| 121 |
+
|
| 122 |
+
### MMLU-Pro by Category
|
| 123 |
+
|
| 124 |
+
| Category | BF16 | NVFP4 | Difference |
|
| 125 |
+
|----------|------|-------|------------|
|
| 126 |
+
| Social Sciences | 32.70% | 31.43% | -1.27% |
|
| 127 |
+
| Other | 31.57% | 30.08% | -1.49% |
|
| 128 |
+
| Humanities | 23.78% | 22.56% | -1.22% |
|
| 129 |
+
| STEM | 19.94% | 18.70% | -1.24% |
|
| 130 |
+
|
| 131 |
+
### MMLU-Pro by Subject
|
| 132 |
+
|
| 133 |
+
| Subject | BF16 | NVFP4 | Difference |
|
| 134 |
+
|---------|------|-------|------------|
|
| 135 |
+
| Biology | 50.35% | 47.42% | -2.93% |
|
| 136 |
+
| Psychology | 44.99% | 42.48% | -2.51% |
|
| 137 |
+
| Economics | 36.37% | 34.48% | -1.89% |
|
| 138 |
+
| Health | 35.21% | 34.84% | -0.37% |
|
| 139 |
+
| History | 33.60% | 30.71% | -2.89% |
|
| 140 |
+
| Philosophy | 31.46% | 30.06% | -1.40% |
|
| 141 |
+
| Other | 28.35% | 25.87% | -2.48% |
|
| 142 |
+
| Computer Science | 26.10% | 21.46% | -4.64% |
|
| 143 |
+
| Business | 16.35% | 16.98% | +0.63% |
|
| 144 |
+
| Law | 16.89% | 16.35% | -0.54% |
|
| 145 |
+
| Engineering | 16.00% | 14.04% | -1.96% |
|
| 146 |
+
| Physics | 15.32% | 14.70% | -0.62% |
|
| 147 |
+
| Math | 14.06% | 14.29% | +0.23% |
|
| 148 |
+
| Chemistry | 14.13% | 13.34% | -0.79% |
|
| 149 |
|
| 150 |
## Citation
|
| 151 |
|