Hugging Face's logo

ruhai-lin
/
MixerLoop

Transformers
Safetensors
gated-deltanet
mixerloop
Model card Files
xet
 Community
ruhai-lin commited on
Commit
f406dcc
·
verified ·
1 Parent(s): d8dc0d7

fix model card line breaks

Browse files
Files changed (1) hide show
  1. README.md +32 -1
README.md CHANGED
@@ -47,4 +47,35 @@ First-batch runs use context 1024 and global batch 128. The 1B runs contain
47
  76,294 steps (10,000,007,168 processed tokens), all seed 1337.
48
  Actual parameters: MixerLoop-13m 12,896,380; MixerLoop-100m 100,447,266;
49
  GDN-100m 100,444,194. Other directories are reserved experiment labels,
50
- not claims about completed training or measured parameter counts.\n\n## CORE v2 comparison\n\nScores are centered CORE v2 scores on the same ClimbMix 10B / seed1337 evaluation setup.\n\n| Task | 13M GDN | 13M MixerLoop | 13M FullLoop | 100M GDN | 100M MixerLoop | 100M FullLoop |\n|---|---:|---:|---:|---:|---:|---:|\n| hellaswag_zeroshot | 0.037509 | 0.032862 | 0.030472 | 0.132842 | 0.133771 | 0.167231 |\n| jeopardy | 0.000000 | 0.000000 | 0.001417 | 0.007085 | 0.006613 | 0.010392 |\n| bigbench_qa_wikidata | 0.036908 | 0.050342 | 0.071010 | 0.214950 | 0.257074 | 0.282811 |\n| arc_easy | 0.144220 | 0.153760 | 0.145342 | 0.317621 | 0.341190 | 0.385522 |\n| arc_challenge | -0.028441 | -0.053470 | -0.039818 | -0.001138 | 0.018203 | 0.045506 |\n| copa | -0.100000 | -0.120000 | -0.180000 | 0.000000 | 0.020000 | 0.020000 |\n| commonsense_qa | -0.225072 | -0.226443 | -0.186659 | -0.244278 | -0.015177 | -0.306011 |\n| piqa | 0.158868 | 0.171926 | 0.170838 | 0.316649 | 0.334059 | 0.355822 |\n| openbook_qa | 0.034667 | 0.018667 | 0.024000 | 0.090667 | 0.088000 | 0.098667 |\n| lambada_openai | 0.105376 | 0.112168 | 0.122647 | 0.253639 | 0.246264 | 0.263536 |\n| hellaswag | 0.028746 | 0.024763 | 0.022107 | 0.125938 | 0.126734 | 0.163513 |\n| winograd | 0.091575 | 0.018315 | 0.076923 | 0.172161 | 0.113553 | 0.172161 |\n| winogrande | 0.010260 | 0.043410 | 0.054459 | 0.027624 | 0.000789 | 0.029203 |\n| bigbench_dyck_languages | 0.003000 | 0.005000 | 0.005000 | 0.128000 | 0.172000 | 0.093000 |\n| agi_eval_lsat_ar | -0.055072 | 0.037681 | -0.002899 | -0.014493 | 0.072464 | 0.049275 |\n| bigbench_cs_algorithms | 0.325000 | 0.312879 | 0.312121 | 0.351515 | 0.351515 | 0.352273 |\n| bigbench_operators | 0.142857 | 0.152381 | 0.180952 | 0.095238 | 0.090476 | 0.090476 |\n| bigbench_repeat_copy_logic | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 |\n| squad | 0.004730 | 0.010028 | 0.012867 | 0.070766 | 0.047114 | 0.049669 |\n| coqa | 0.040962 | 0.023300 | 0.047225 | 0.113366 | 0.116623 | 0.123888 |\n| boolq | -0.354418 | -0.203927 | -0.229680 | -0.187832 | -0.059070 | -0.054241 |\n| bigbench_language_identification | -0.002400 | 0.008133 | 0.010533 | -0.003467 | 0.010000 | 0.002533 |\n| **Core** | **0.018149** | **0.025990** | **0.029494** | **0.089402** | **0.112373** | **0.108874** |\n\n
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
47
  76,294 steps (10,000,007,168 processed tokens), all seed 1337.
48
  Actual parameters: MixerLoop-13m 12,896,380; MixerLoop-100m 100,447,266;
49
  GDN-100m 100,444,194. Other directories are reserved experiment labels,
50
+ not claims about completed training or measured parameter counts.
51
+
52
+ ## CORE v2 comparison
53
+
54
+ Scores are centered CORE v2 scores on the same ClimbMix 10B / seed1337 evaluation setup.
55
+
56
+ | Task | 13M GDN | 13M MixerLoop | 13M FullLoop | 100M GDN | 100M MixerLoop | 100M FullLoop |
57
+ |---|---:|---:|---:|---:|---:|---:|
58
+ | hellaswag_zeroshot | 0.037509 | 0.032862 | 0.030472 | 0.132842 | 0.133771 | 0.167231 |
59
+ | jeopardy | 0.000000 | 0.000000 | 0.001417 | 0.007085 | 0.006613 | 0.010392 |
60
+ | bigbench_qa_wikidata | 0.036908 | 0.050342 | 0.071010 | 0.214950 | 0.257074 | 0.282811 |
61
+ | arc_easy | 0.144220 | 0.153760 | 0.145342 | 0.317621 | 0.341190 | 0.385522 |
62
+ | arc_challenge | -0.028441 | -0.053470 | -0.039818 | -0.001138 | 0.018203 | 0.045506 |
63
+ | copa | -0.100000 | -0.120000 | -0.180000 | 0.000000 | 0.020000 | 0.020000 |
64
+ | commonsense_qa | -0.225072 | -0.226443 | -0.186659 | -0.244278 | -0.015177 | -0.306011 |
65
+ | piqa | 0.158868 | 0.171926 | 0.170838 | 0.316649 | 0.334059 | 0.355822 |
66
+ | openbook_qa | 0.034667 | 0.018667 | 0.024000 | 0.090667 | 0.088000 | 0.098667 |
67
+ | lambada_openai | 0.105376 | 0.112168 | 0.122647 | 0.253639 | 0.246264 | 0.263536 |
68
+ | hellaswag | 0.028746 | 0.024763 | 0.022107 | 0.125938 | 0.126734 | 0.163513 |
69
+ | winograd | 0.091575 | 0.018315 | 0.076923 | 0.172161 | 0.113553 | 0.172161 |
70
+ | winogrande | 0.010260 | 0.043410 | 0.054459 | 0.027624 | 0.000789 | 0.029203 |
71
+ | bigbench_dyck_languages | 0.003000 | 0.005000 | 0.005000 | 0.128000 | 0.172000 | 0.093000 |
72
+ | agi_eval_lsat_ar | -0.055072 | 0.037681 | -0.002899 | -0.014493 | 0.072464 | 0.049275 |
73
+ | bigbench_cs_algorithms | 0.325000 | 0.312879 | 0.312121 | 0.351515 | 0.351515 | 0.352273 |
74
+ | bigbench_operators | 0.142857 | 0.152381 | 0.180952 | 0.095238 | 0.090476 | 0.090476 |
75
+ | bigbench_repeat_copy_logic | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 |
76
+ | squad | 0.004730 | 0.010028 | 0.012867 | 0.070766 | 0.047114 | 0.049669 |
77
+ | coqa | 0.040962 | 0.023300 | 0.047225 | 0.113366 | 0.116623 | 0.123888 |
78
+ | boolq | -0.354418 | -0.203927 | -0.229680 | -0.187832 | -0.059070 | -0.054241 |
79
+ | bigbench_language_identification | -0.002400 | 0.008133 | 0.010533 | -0.003467 | 0.010000 | 0.002533 |
80
+ | **Core** | **0.018149** | **0.025990** | **0.029494** | **0.089402** | **0.112373** | **0.108874** |
81
+