Cukinator commited on
Commit
8dcd097
·
verified ·
1 Parent(s): a67221b

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +11 -10
README.md CHANGED
@@ -113,16 +113,17 @@ After the audit, configurations were updated to apply corrections such as **bf16
113
 
114
  The current trainings were interrupted prematurely due to cloud credit exhaustion (AMD MI300X, 192GB VRAM). None of them generated the final JSON. Extracted from the Hugging Face Hub (`Cukinator/cpu1-ablation-checkpoints-v3`), here are the completion percentages of each run before termination:
115
 
116
- - **Target steps:** 62,500 steps (50M) / 12,500 steps (10M)
117
- - `run_01` (50M): 12,173 steps → **19.5%**
118
- - `run_02` (50M): 10,963 steps → **17.5%**
119
- - `run_04` (50M): 6,074 steps → **9.7%**
120
- - `run_05` (50M): 4,390 steps → **7.0%**
121
- - `run_06` (50M): 4,237 steps → **6.8%**
122
- - `run_08` (50M): 3,858 steps → **6.2%**
123
- - `run_13` (10M): 9,290 steps → **74.3%**
124
- - `run_14` (10M): 9,775 steps → **78.2%**
125
- - `run_15` (10M): 2,350 steps → **18.8%**
 
126
 
127
  The extremely slow progress (<20% in 1 day for 50M models) was caused by a severe CPU context-switching bottleneck when orchestrating 16 concurrent processes on a single MI300X GPU.
128
 
 
113
 
114
  The current trainings were interrupted prematurely due to cloud credit exhaustion (AMD MI300X, 192GB VRAM). None of them generated the final JSON. Extracted from the Hugging Face Hub (`Cukinator/cpu1-ablation-checkpoints-v3`), here are the completion percentages of each run before termination:
115
 
116
+ | Run | Architecture | Params | Target Steps | Last Step | Progress |
117
+ |-----|-------------|-------:|-------------:|----------:|---------:|
118
+ | run_01 | Transformer + BPE (16K vocab) + FP16 | 54.7M | 62,500 | 12,173 | **19.5%** |
119
+ | run_02 | Transformer + Byte + LocalByteDecoder | 38.8M | 62,500 | 10,963 | **17.5%** |
120
+ | run_04 | MLGRU + Byte + Ternary | 38.9M | 62,500 | 6,074 | **9.7%** |
121
+ | run_05 | + FPResidual | 39.0M | 62,500 | 4,390 | **7.0%** |
122
+ | run_06 | + Bolmo patch embedding | 39.0M | 62,500 | 4,237 | **6.8%** |
123
+ | run_08 | Folded Transformer + Byte + Ternary | 38.9M | 62,500 | 3,858 | **6.2%** |
124
+ | run_13 | Small CPU-1 + BPE (4K vocab) | 12.5M | 12,500 | 9,290 | **74.3%** |
125
+ | run_14 | Small CPU-1 byte-level (Qwen distillation) | 10.7M | 12,500 | 9,775 | **78.2%** |
126
+ | run_15 | Small CPU-1 byte + hidden distillation | 10.7M | 12,500 | 2,350 | **18.8%** |
127
 
128
  The extremely slow progress (<20% in 1 day for 50M models) was caused by a severe CPU context-switching bottleneck when orchestrating 16 concurrent processes on a single MI300X GPU.
129