Update README.md
Browse files
README.md
CHANGED
|
@@ -26,12 +26,18 @@ Running under the **JiRackTrain** pipeline on the enterprise infrastructure clus
|
|
| 26 |
- **Tokenizer**: [JiRack Pro Tokenizer (128K)](https://huggingface.co/CMSManhattan/JiRack-Pro-Tokenizer-128K) — 347 active language editions of Wikipedia + specialized robotic action tokens
|
| 27 |
- **Pre-training Dataset**: [JiRack-Pretrain-Dataset](https://huggingface.co/datasets/CMSManhattan/JiRack-Pretrain-Dataset)
|
| 28 |
|
| 29 |
-
## Key Metrics (Step ~838/1000 on Shard
|
| 30 |
-
|
| 31 |
-
- **Iteration Speed**:
|
| 32 |
-
- **Current Loss**:
|
| 33 |
-
- **Moving Average Loss**:
|
| 34 |
-
- **Perplexity (PPL)**:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
|
| 36 |
Engineers anticipate a **steep drop** in the perplexity curve within the first 250 iterations of Shard 1 as the ternary weights align with the tokenizer's token distributions.
|
| 37 |
|
|
|
|
| 26 |
- **Tokenizer**: [JiRack Pro Tokenizer (128K)](https://huggingface.co/CMSManhattan/JiRack-Pro-Tokenizer-128K) — 347 active language editions of Wikipedia + specialized robotic action tokens
|
| 27 |
- **Pre-training Dataset**: [JiRack-Pretrain-Dataset](https://huggingface.co/datasets/CMSManhattan/JiRack-Pretrain-Dataset)
|
| 28 |
|
| 29 |
+
## Key Metrics (Step ~838/1000 on Shard 7) on Blackwell 96 Gb VRAM
|
| 30 |
+
|
| 31 |
+
- **Iteration Speed**: 2.36s/it (network tensor loading)
|
| 32 |
+
- **Current Loss**: 1.2665
|
| 33 |
+
- **Moving Average Loss**: 1.6637
|
| 34 |
+
- **Perplexity (PPL)**: ppl=5.3
|
| 35 |
+
|
| 36 |
+
```bash
|
| 37 |
+
[Shard 8/100] jirack_pretrain_chunk_7.pt
|
| 38 |
+
Training jirack_pretrain_chunk_7.pt: 83%|█████▊ | 208/250 [08:09<01:38, 2.36s/it, loss=1.2665, avg_loss=1.6637, ppl=5.3, lr=2.00e-04]
|
| 39 |
+
[0] 0:python3* "mc [root@809e0ca1a59e" 19:45 04-Jul-26
|
| 40 |
+
```
|
| 41 |
|
| 42 |
Engineers anticipate a **steep drop** in the perplexity curve within the first 250 iterations of Shard 1 as the ternary weights align with the tokenizer's token distributions.
|
| 43 |
|