GadflyII commited on
Commit
208e33c
·
verified ·
1 Parent(s): 176f59f

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +35 -18
README.md CHANGED
@@ -111,24 +111,41 @@ These compressed representations are highly sensitive to quantization noise. NVI
111
 
112
  ## Evaluation
113
 
114
- MMLU-Pro results by category:
115
-
116
- | Category | Accuracy |
117
- |----------|----------|
118
- | Biology | 47.42% |
119
- | Psychology | 42.48% |
120
- | Health | 34.84% |
121
- | Economics | 34.48% |
122
- | History | 30.71% |
123
- | Philosophy | 30.06% |
124
- | Other | 25.87% |
125
- | Computer Science | 21.46% |
126
- | Business | 16.98% |
127
- | Law | 16.35% |
128
- | Physics | 14.70% |
129
- | Math | 14.29% |
130
- | Engineering | 14.04% |
131
- | Chemistry | 13.34% |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
132
 
133
  ## Citation
134
 
 
111
 
112
  ## Evaluation
113
 
114
+ ### MMLU-Pro Overall Results
115
+
116
+ | Model | Accuracy | Correct | Total |
117
+ |-------|----------|---------|-------|
118
+ | **BF16 (baseline)** | **24.83%** | 2988 | 12032 |
119
+ | **NVFP4 (this model)** | **23.55%** | 2834 | 12032 |
120
+ | **Difference** | **-1.28%** | -154 | - |
121
+
122
+ ### MMLU-Pro by Category
123
+
124
+ | Category | BF16 | NVFP4 | Difference |
125
+ |----------|------|-------|------------|
126
+ | Social Sciences | 32.70% | 31.43% | -1.27% |
127
+ | Other | 31.57% | 30.08% | -1.49% |
128
+ | Humanities | 23.78% | 22.56% | -1.22% |
129
+ | STEM | 19.94% | 18.70% | -1.24% |
130
+
131
+ ### MMLU-Pro by Subject
132
+
133
+ | Subject | BF16 | NVFP4 | Difference |
134
+ |---------|------|-------|------------|
135
+ | Biology | 50.35% | 47.42% | -2.93% |
136
+ | Psychology | 44.99% | 42.48% | -2.51% |
137
+ | Economics | 36.37% | 34.48% | -1.89% |
138
+ | Health | 35.21% | 34.84% | -0.37% |
139
+ | History | 33.60% | 30.71% | -2.89% |
140
+ | Philosophy | 31.46% | 30.06% | -1.40% |
141
+ | Other | 28.35% | 25.87% | -2.48% |
142
+ | Computer Science | 26.10% | 21.46% | -4.64% |
143
+ | Business | 16.35% | 16.98% | +0.63% |
144
+ | Law | 16.89% | 16.35% | -0.54% |
145
+ | Engineering | 16.00% | 14.04% | -1.96% |
146
+ | Physics | 15.32% | 14.70% | -0.62% |
147
+ | Math | 14.06% | 14.29% | +0.23% |
148
+ | Chemistry | 14.13% | 13.34% | -0.79% |
149
 
150
  ## Citation
151