IFM/TxT360-v2
Viewer • Updated • 1.84B • 73.6k • 108
~50M model pretrained on an ~36B token mix of filtered general web data, math data, and in-context learning elicitation data. To the best of my knowledge, no LLM-generated synthetic data made it into the final corpus, excluding any possible contamination from the source datasets.
I'm not too happy with how this turned out, but...
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| arc_challenge | 1 | none | 0 | acc | ↑ | 0.2116 | ± | 0.0119 |
| none | 0 | acc_norm | ↑ | 0.2423 | ± | 0.0125 | ||
| arc_easy | 1 | none | 0 | acc | ↑ | 0.4949 | ± | 0.0103 |
| none | 0 | acc_norm | ↑ | 0.4436 | ± | 0.0102 | ||
| hellaswag | 1 | none | 0 | acc | ↑ | 0.3021 | ± | 0.0046 |
| none | 0 | acc_norm | ↑ | 0.3359 | ± | 0.0047 | ||
| piqa | 1 | none | 0 | acc | ↑ | 0.6496 | ± | 0.0111 |
| none | 0 | acc_norm | ↑ | 0.6366 | ± | 0.0112 |
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| babilong_longctx | 0 | none | 2 | acc | ↑ | 0.4864 | ± | 0.0061 |
| - babilong_qa1 | 0 | none | 2 | acc | ↑ | 0.4150 | ± | 0.0156 |
| - babilong_qa2 | 0 | none | 2 | acc | ↑ | 0.3153 | ± | 0.0147 |
| - babilong_qa3 | 0 | none | 2 | acc | ↑ | 0.1582 | ± | 0.0116 |
| - babilong_qa4 | 0 | none | 2 | acc | ↑ | 0.8929 | ± | 0.0098 |
| - babilong_qa5 | 0 | none | 2 | acc | ↑ | 0.6507 | ± | 0.0151 |
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| babilong_longctx | 0 | none | 2 | acc | ↑ | 0.1235 | ± | 0.0046 |
| - babilong_qa1 | 0 | none | 2 | acc | ↑ | 0.0620 | ± | 0.0076 |
| - babilong_qa2 | 0 | none | 2 | acc | ↑ | 0.0871 | ± | 0.0089 |
| - babilong_qa3 | 0 | none | 2 | acc | ↑ | 0.1854 | ± | 0.0123 |
| - babilong_qa4 | 0 | none | 2 | acc | ↑ | 0.0641 | ± | 0.0078 |
| - babilong_qa5 | 0 | none | 2 | acc | ↑ | 0.2192 | ± | 0.0131 |
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| babilong_longctx | 0 | none | 2 | acc | ↑ | 0.0258 | ± | 0.0022 |
| - babilong_qa1 | 0 | none | 2 | acc | ↑ | 0.0100 | ± | 0.0031 |
| - babilong_qa2 | 0 | none | 2 | acc | ↑ | 0.0320 | ± | 0.0056 |
| - babilong_qa3 | 0 | none | 2 | acc | ↑ | 0.0410 | ± | 0.0063 |
| - babilong_qa4 | 0 | none | 2 | acc | ↑ | 0.0150 | ± | 0.0038 |
| - babilong_qa5 | 0 | none | 2 | acc | ↑ | 0.0310 | ± | 0.0055 |
(greater than zero!)
Only clusters 2, 6, 12, and 15 of ClimbMix were used; notably this excludes the mathematics cluster and most of the STEM clusters.
Pretraining codebase: https://code.allura.moe/fizz/lobotomy Training data: https://code.allura.moe/fizz/lobotomy/src/commit/6353a73f483823eee6db1750e6a3b2fdb57e53f0/libs/data/lobotomy/data/mixes/rambley_16k/pretrain_v2.txt
====================================================================
../exported_hf (49,718,600 params) RESULTS
====================================================================
Raw continuation accuracy 36.50%
Length-normalized accuracy 36.40%
Primary (acc_norm) 36.40%
====================================================================