Add PotentSulfurLM, as a followup to CinnabarLM!
#6
by MihaiPopa-1 - opened
Per https://huggingface.co/spaces/CompactAI-O/Tiny-ML-Leaderboard/discussions/5:
"Next LLM to add will be PotentSulfurLM 500K (https://huggingface.co/MihaiPopa-1/PotentSulfurLM-500K-Base)!
It's a LLM with 587K parameters. Trained on ~200M tokens, ~4x the amount used to train CinnabarLM 1.5M!
!lm_eval --model hf \
--model_args pretrained=MihaiPopa-1/PotentSulfurLM-500K-Base \
--tasks blimp,arc_easy,wikitext \
--device cuda \
--batch_size 1024 \
--output_path results/
gives:
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| blimp | 2 | none | 0 | acc | ↑ | 0.5901 | ± | 0.0017 |
| arc_easy | 1 | none | 0 | acc | ↑ | 0.2706 | ± | 0.0091 |
| wikitext | 2 | none | 0 | bits_per_byte | ↓ | 2.6131 | ± | N/A |
| " |
CompactAI changed pull request status to merged