Add PotentSulfurLM, as a followup to CinnabarLM!
Browse filesPer https://huggingface.co/spaces/CompactAI-O/Tiny-ML-Leaderboard/discussions/5:
"Next LLM to add will be PotentSulfurLM 500K (https://huggingface.co/MihaiPopa-1/PotentSulfurLM-500K-Base)!
It's a LLM with 587K parameters. Trained on ~200M tokens, ~4x the amount used to train CinnabarLM 1.5M!
```bash
!lm_eval --model hf \
--model_args pretrained=MihaiPopa-1/PotentSulfurLM-500K-Base \
--tasks blimp,arc_easy,wikitext \
--device cuda \
--batch_size 1024 \
--output_path results/
```
gives:
| Tasks |Version|Filter|n-shot| Metric | | Value | |Stderr|
|------------------------------------------------------------|------:|------|-----:|---------------|---|---------:|---|------|
|blimp | 2|none | 0|acc |↑ | 0.5901|± |0.0017|
|arc_easy | 1|none | 0|acc |↑ | 0.2706|± |0.0091|
|wikitext | 2|none | 0|bits_per_byte |↓ | 2.6131|± | N/A|
"
- index.html +10 -0
|
@@ -302,6 +302,16 @@ const models = [
|
|
| 302 |
tokens: '~30M',
|
| 303 |
links: { card: 'https://huggingface.co/MihaiPopa-1/CinnabarLM-1.4M-Base' }
|
| 304 |
},
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 305 |
{
|
| 306 |
name: 'StorySupra-10M',
|
| 307 |
org: 'supralabs',
|
|
|
|
| 302 |
tokens: '~30M',
|
| 303 |
links: { card: 'https://huggingface.co/MihaiPopa-1/CinnabarLM-1.4M-Base' }
|
| 304 |
},
|
| 305 |
+
{
|
| 306 |
+
name: 'PotentSulfurLM 500K',
|
| 307 |
+
org: 'mihaipopa',
|
| 308 |
+
params: '587K',
|
| 309 |
+
blimp: 59.01,
|
| 310 |
+
arc: 27.06,
|
| 311 |
+
wiki: 2.613,
|
| 312 |
+
tokens: '~200M',
|
| 313 |
+
links: { card: 'https://huggingface.co/MihaiPopa-1/PotentSulfurLM-500K-Base' }
|
| 314 |
+
},
|
| 315 |
{
|
| 316 |
name: 'StorySupra-10M',
|
| 317 |
org: 'supralabs',
|