MihaiPopa-1 commited on
Commit
d11bc14
·
verified ·
1 Parent(s): 534ac0a

Add PotentSulfurLM, as a followup to CinnabarLM!

Browse files

Per https://huggingface.co/spaces/CompactAI-O/Tiny-ML-Leaderboard/discussions/5:

"Next LLM to add will be PotentSulfurLM 500K (https://huggingface.co/MihaiPopa-1/PotentSulfurLM-500K-Base)!

It's a LLM with 587K parameters. Trained on ~200M tokens, ~4x the amount used to train CinnabarLM 1.5M!
```bash
!lm_eval --model hf \
--model_args pretrained=MihaiPopa-1/PotentSulfurLM-500K-Base \
--tasks blimp,arc_easy,wikitext \
--device cuda \
--batch_size 1024 \
--output_path results/
```
gives:

| Tasks |Version|Filter|n-shot| Metric | | Value | |Stderr|
|------------------------------------------------------------|------:|------|-----:|---------------|---|---------:|---|------|
|blimp | 2|none | 0|acc |↑ | 0.5901|± |0.0017|
|arc_easy | 1|none | 0|acc |↑ | 0.2706|± |0.0091|
|wikitext | 2|none | 0|bits_per_byte |↓ | 2.6131|± | N/A|
"

Files changed (1) hide show
  1. index.html +10 -0
index.html CHANGED
@@ -302,6 +302,16 @@ const models = [
302
  tokens: '~30M',
303
  links: { card: 'https://huggingface.co/MihaiPopa-1/CinnabarLM-1.4M-Base' }
304
  },
 
 
 
 
 
 
 
 
 
 
305
  {
306
  name: 'StorySupra-10M',
307
  org: 'supralabs',
 
302
  tokens: '~30M',
303
  links: { card: 'https://huggingface.co/MihaiPopa-1/CinnabarLM-1.4M-Base' }
304
  },
305
+ {
306
+ name: 'PotentSulfurLM 500K',
307
+ org: 'mihaipopa',
308
+ params: '587K',
309
+ blimp: 59.01,
310
+ arc: 27.06,
311
+ wiki: 2.613,
312
+ tokens: '~200M',
313
+ links: { card: 'https://huggingface.co/MihaiPopa-1/PotentSulfurLM-500K-Base' }
314
+ },
315
  {
316
  name: 'StorySupra-10M',
317
  org: 'supralabs',