Update README.md
Browse files
README.md
CHANGED
|
@@ -43,6 +43,42 @@ A small Mixtral-style Mixture-of-Experts causal language model (~432M total, ~17
|
|
| 43 |
- **Activation checkpointing**: enabled (required for 24GB VRAM)
|
| 44 |
- **Data**: C4-en, pre-tokenized to `.pt` shards (100M tokens each), cycled round-robin
|
| 45 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
## License
|
| 47 |
|
| 48 |
MIT License. Copyright (C) 2026 Michael Lee (李登淳).
|
|
|
|
| 43 |
- **Activation checkpointing**: enabled (required for 24GB VRAM)
|
| 44 |
- **Data**: C4-en, pre-tokenized to `.pt` shards (100M tokens each), cycled round-robin
|
| 45 |
|
| 46 |
+
## Results
|
| 47 |
+
|
| 48 |
+
### Training Summary
|
| 49 |
+
|
| 50 |
+
| Phase | Data | Tokens | Steps | Time | Start Loss | End Loss |
|
| 51 |
+
|-------|------|:------:|:-----:|:----:|:----------:|:--------:|
|
| 52 |
+
| Pretrain | C4-en | 4B | 177,557 | 77.1 h | 10.5 | 3.0 |
|
| 53 |
+
| Post-train | FineWeb-Edu + Cosmopedia v2 (50:50) | 1B | 44,390 | 20.8 h | 3.05 | 2.0 |
|
| 54 |
+
|
| 55 |
+
Post-training used learning rate 5e-5 with 300-step re-warmup, continuing from the pretrain checkpoint with AdamW momentum preserved.
|
| 56 |
+
|
| 57 |
+
### GLUE (zero-shot)
|
| 58 |
+
|
| 59 |
+
| Task | Metric | Pretrain (4B C4) | Post-train (5B total) |
|
| 60 |
+
|------|--------|:---:|:---:|
|
| 61 |
+
| SST2 | accuracy | 0.470 | **0.554** |
|
| 62 |
+
| MRPC | accuracy / f1 | 0.338 / 0.069 | **0.706 / 0.815** |
|
| 63 |
+
| QQP | accuracy / f1 | 0.470 / 0.412 | **0.530** / 0.342 |
|
| 64 |
+
| QNLI | accuracy | 0.494 | 0.452 |
|
| 65 |
+
| RTE | accuracy | 0.520 | 0.484 |
|
| 66 |
+
| CoLA | MCC | 0.089 | 0.006 |
|
| 67 |
+
| MNLI | accuracy | 0.348 | 0.348 |
|
| 68 |
+
| MNLI-mm | accuracy | 0.368 | 0.368 |
|
| 69 |
+
| **Mean** | — | **0.403** | **0.483** |
|
| 70 |
+
|
| 71 |
+
### ARC
|
| 72 |
+
|
| 73 |
+
| Task | Pretrain (4B C4) | Post-train (5B total) |
|
| 74 |
+
|------|:---:|:---:|
|
| 75 |
+
| ARC-C 0-shot | 0.220 | **0.233** |
|
| 76 |
+
| ARC-C 5-shot | 0.223 | **0.246** |
|
| 77 |
+
| ARC-E 0-shot | 0.311 | **0.342** |
|
| 78 |
+
| ARC-E 5-shot | 0.320 | **0.348** |
|
| 79 |
+
|
| 80 |
+
Zero-shot evaluation uses conditional log-likelihood scoring over answer spans. All evals run on a single GPU with `--limit 500 --batch-size 16 --max-length 512`. Pretrain and post-train evaluated under identical settings for fair comparison.
|
| 81 |
+
|
| 82 |
## License
|
| 83 |
|
| 84 |
MIT License. Copyright (C) 2026 Michael Lee (李登淳).
|