Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -20,17 +20,39 @@ Each run is stored in its own folder and contains a single `model.pt` file. Thes
|
|
| 20 |
|
| 21 |
## Repository contents
|
| 22 |
|
| 23 |
-
Models from the ablation grid include:
|
| 24 |
|
|
|
|
| 25 |
| Path | Description |
|
| 26 |
|---|---|
|
| 27 |
-
| `run_01/model.pt` | Transformer BPE
|
| 28 |
-
| `
|
|
|
|
| 29 |
| `run_03/model.pt` | MLGRU Byte FP16 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
| `run_07/model.pt` | CPU-1 COMPLETE (V3 Ternary with DeleteGate) |
|
| 31 |
-
| `
|
| 32 |
-
| `
|
| 33 |
-
| .
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
|
| 35 |
## How to test them locally
|
| 36 |
|
|
|
|
| 20 |
|
| 21 |
## Repository contents
|
| 22 |
|
| 23 |
+
Models from the ablation grid include all trained variants across Round 1 and Round 2:
|
| 24 |
|
| 25 |
+
### FP16 Baselines
|
| 26 |
| Path | Description |
|
| 27 |
|---|---|
|
| 28 |
+
| `run_01/model.pt` | Transformer BPE (16K vocab) FP16 |
|
| 29 |
+
| `run_02a_byte_only_heads/model.pt` | Transformer Byte FP16 (Independent heads) |
|
| 30 |
+
| `run_02/model.pt` | Transformer Byte FP16 (LocalByteDecoder) — **Best Performer** |
|
| 31 |
| `run_03/model.pt` | MLGRU Byte FP16 |
|
| 32 |
+
|
| 33 |
+
### 50M Ternary Chain (V3 and Ablations)
|
| 34 |
+
| Path | Description |
|
| 35 |
+
|---|---|
|
| 36 |
+
| `run_04/model.pt` | MLGRU Byte Ternary |
|
| 37 |
+
| `run_05/model.pt` | MLGRU Byte Ternary + FPResidual (CPU-1 core) |
|
| 38 |
+
| `run_05b_kernel_strict/model.pt` | CPU-1 core without W_o |
|
| 39 |
+
| `run_06/model.pt` | CPU-1 core + Bolmo (Dynamic patches) |
|
| 40 |
| `run_07/model.pt` | CPU-1 COMPLETE (V3 Ternary with DeleteGate) |
|
| 41 |
+
| `run_08/model.pt` | Folded Transformer Byte Ternary |
|
| 42 |
+
| `run_09/model.pt` | CPU-1 COMPLETE + PFNet |
|
| 43 |
+
| `run_10/model.pt` | CPU-1 COMPLETE + PFNet + Per-Channel Decay |
|
| 44 |
+
|
| 45 |
+
### 10M Scale Runs (Ternary)
|
| 46 |
+
| Path | Description |
|
| 47 |
+
|---|---|
|
| 48 |
+
| `run_13/model.pt` | 10M BPE + Ternary (Round 1 - 2 tok/p) |
|
| 49 |
+
| `run_13_r2/model.pt` | 10M BPE + Ternary (Round 2 - 15 tok/p) |
|
| 50 |
+
| `run_14/model.pt` | 10M Byte + Ternary (Round 1 - 2 tok/p) |
|
| 51 |
+
| `run_14_r2/model.pt` | 10M Byte + Ternary (Round 2 - 15 tok/p) |
|
| 52 |
+
| `run_15/model.pt` | 10M Byte + Ternary + EmbeddingAligner (Round 1) |
|
| 53 |
+
| `run_15_r2/model.pt` | 10M Byte + Ternary + EmbeddingAligner (Round 2) |
|
| 54 |
+
| `run_16/model.pt` | 10M Byte + Ternary (No teacher logprobs) (Round 1) |
|
| 55 |
+
| `run_16_r2/model.pt` | 10M Byte + Ternary (No teacher logprobs) (Round 2) |
|
| 56 |
|
| 57 |
## How to test them locally
|
| 58 |
|