Add 68k capped-200 scores below the 40k/full tables (do not replace)
Browse files
README.md
CHANGED
|
@@ -118,6 +118,24 @@ Val loss 4.195 vs 4.231. Sampled repetition 0.011 vs 0.049. The
|
|
| 118 |
unlooped baseline's `% % %` collapse (rep 0.92) never showed up on
|
| 119 |
the looped arm.
|
| 120 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 121 |
## Usage
|
| 122 |
|
| 123 |
```bash
|
|
|
|
| 118 |
unlooped baseline's `% % %` collapse (rep 0.92) never showed up on
|
| 119 |
the looped arm.
|
| 120 |
|
| 121 |
+
### 68k, capped 200
|
| 122 |
+
|
| 123 |
+
`checkpoints/base_68k.pt`. Same harness, `--limit 200`. Does not
|
| 124 |
+
replace the full table above. JSON:
|
| 125 |
+
[`benchmarks/Byrne-15M-Looped-base68k-cap200.json`](benchmarks/Byrne-15M-Looped-base68k-cap200.json).
|
| 126 |
+
|
| 127 |
+
| Metric | 68k cap-200 | chance |
|
| 128 |
+
|---|---|---|
|
| 129 |
+
| WikiText-2 byte_ppl ↓ | 2.937 | — |
|
| 130 |
+
| BLiMP acc ↑ | 0.728 | 0.50 |
|
| 131 |
+
| arc_easy acc | 0.360 | 0.25 |
|
| 132 |
+
| hellaswag acc_norm | 0.330 | 0.25 |
|
| 133 |
+
| piqa acc | 0.515 | 0.50 |
|
| 134 |
+
| boolq acc | 0.365 | 0.50 |
|
| 135 |
+
| ArithMark-3.0 acc_norm | 0.335 | 0.25 |
|
| 136 |
+
|
| 137 |
+
n=200 on MC / ArithMark. Full table: [BENCHMARKS.md](BENCHMARKS.md).
|
| 138 |
+
|
| 139 |
## Usage
|
| 140 |
|
| 141 |
```bash
|