Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -113,16 +113,17 @@ After the audit, configurations were updated to apply corrections such as **bf16
|
|
| 113 |
|
| 114 |
The current trainings were interrupted prematurely due to cloud credit exhaustion (AMD MI300X, 192GB VRAM). None of them generated the final JSON. Extracted from the Hugging Face Hub (`Cukinator/cpu1-ablation-checkpoints-v3`), here are the completion percentages of each run before termination:
|
| 115 |
|
| 116 |
-
|
| 117 |
-
-
|
| 118 |
-
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
-
|
| 125 |
-
-
|
|
|
|
| 126 |
|
| 127 |
The extremely slow progress (<20% in 1 day for 50M models) was caused by a severe CPU context-switching bottleneck when orchestrating 16 concurrent processes on a single MI300X GPU.
|
| 128 |
|
|
|
|
| 113 |
|
| 114 |
The current trainings were interrupted prematurely due to cloud credit exhaustion (AMD MI300X, 192GB VRAM). None of them generated the final JSON. Extracted from the Hugging Face Hub (`Cukinator/cpu1-ablation-checkpoints-v3`), here are the completion percentages of each run before termination:
|
| 115 |
|
| 116 |
+
| Run | Architecture | Params | Target Steps | Last Step | Progress |
|
| 117 |
+
|-----|-------------|-------:|-------------:|----------:|---------:|
|
| 118 |
+
| run_01 | Transformer + BPE (16K vocab) + FP16 | 54.7M | 62,500 | 12,173 | **19.5%** |
|
| 119 |
+
| run_02 | Transformer + Byte + LocalByteDecoder | 38.8M | 62,500 | 10,963 | **17.5%** |
|
| 120 |
+
| run_04 | MLGRU + Byte + Ternary | 38.9M | 62,500 | 6,074 | **9.7%** |
|
| 121 |
+
| run_05 | + FPResidual | 39.0M | 62,500 | 4,390 | **7.0%** |
|
| 122 |
+
| run_06 | + Bolmo patch embedding | 39.0M | 62,500 | 4,237 | **6.8%** |
|
| 123 |
+
| run_08 | Folded Transformer + Byte + Ternary | 38.9M | 62,500 | 3,858 | **6.2%** |
|
| 124 |
+
| run_13 | Small CPU-1 + BPE (4K vocab) | 12.5M | 12,500 | 9,290 | **74.3%** |
|
| 125 |
+
| run_14 | Small CPU-1 byte-level (Qwen distillation) | 10.7M | 12,500 | 9,775 | **78.2%** |
|
| 126 |
+
| run_15 | Small CPU-1 byte + hidden distillation | 10.7M | 12,500 | 2,350 | **18.8%** |
|
| 127 |
|
| 128 |
The extremely slow progress (<20% in 1 day for 50M models) was caused by a severe CPU context-switching bottleneck when orchestrating 16 concurrent processes on a single MI300X GPU.
|
| 129 |
|