Quazim0t0 commited on
Commit
b849eee
·
verified ·
1 Parent(s): 57fd663

Add 68k capped-200 scores below the 40k/full tables (do not replace)

Browse files
Files changed (1) hide show
  1. README.md +18 -0
README.md CHANGED
@@ -118,6 +118,24 @@ Val loss 4.195 vs 4.231. Sampled repetition 0.011 vs 0.049. The
118
  unlooped baseline's `% % %` collapse (rep 0.92) never showed up on
119
  the looped arm.
120
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
121
  ## Usage
122
 
123
  ```bash
 
118
  unlooped baseline's `% % %` collapse (rep 0.92) never showed up on
119
  the looped arm.
120
 
121
+ ### 68k, capped 200
122
+
123
+ `checkpoints/base_68k.pt`. Same harness, `--limit 200`. Does not
124
+ replace the full table above. JSON:
125
+ [`benchmarks/Byrne-15M-Looped-base68k-cap200.json`](benchmarks/Byrne-15M-Looped-base68k-cap200.json).
126
+
127
+ | Metric | 68k cap-200 | chance |
128
+ |---|---|---|
129
+ | WikiText-2 byte_ppl ↓ | 2.937 | — |
130
+ | BLiMP acc ↑ | 0.728 | 0.50 |
131
+ | arc_easy acc | 0.360 | 0.25 |
132
+ | hellaswag acc_norm | 0.330 | 0.25 |
133
+ | piqa acc | 0.515 | 0.50 |
134
+ | boolq acc | 0.365 | 0.50 |
135
+ | ArithMark-3.0 acc_norm | 0.335 | 0.25 |
136
+
137
+ n=200 on MC / ArithMark. Full table: [BENCHMARKS.md](BENCHMARKS.md).
138
+
139
  ## Usage
140
 
141
  ```bash