Robotics
multilingual
ternary
multimodal
pretraining
jirack
ternarytransformer
kgrabko commited on
Commit
9959503
·
verified ·
1 Parent(s): c367088

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +12 -6
README.md CHANGED
@@ -26,12 +26,18 @@ Running under the **JiRackTrain** pipeline on the enterprise infrastructure clus
26
  - **Tokenizer**: [JiRack Pro Tokenizer (128K)](https://huggingface.co/CMSManhattan/JiRack-Pro-Tokenizer-128K) — 347 active language editions of Wikipedia + specialized robotic action tokens
27
  - **Pre-training Dataset**: [JiRack-Pretrain-Dataset](https://huggingface.co/datasets/CMSManhattan/JiRack-Pretrain-Dataset)
28
 
29
- ## Key Metrics (Step ~838/1000 on Shard 1)
30
-
31
- - **Iteration Speed**: ~242–352 seconds per iteration (network tensor loading)
32
- - **Current Loss**: 2.9678
33
- - **Moving Average Loss**: 6.0920
34
- - **Perplexity (PPL)**: 442.33
 
 
 
 
 
 
35
 
36
  Engineers anticipate a **steep drop** in the perplexity curve within the first 250 iterations of Shard 1 as the ternary weights align with the tokenizer's token distributions.
37
 
 
26
  - **Tokenizer**: [JiRack Pro Tokenizer (128K)](https://huggingface.co/CMSManhattan/JiRack-Pro-Tokenizer-128K) — 347 active language editions of Wikipedia + specialized robotic action tokens
27
  - **Pre-training Dataset**: [JiRack-Pretrain-Dataset](https://huggingface.co/datasets/CMSManhattan/JiRack-Pretrain-Dataset)
28
 
29
+ ## Key Metrics (Step ~838/1000 on Shard 7) on Blackwell 96 Gb VRAM
30
+
31
+ - **Iteration Speed**: 2.36s/it (network tensor loading)
32
+ - **Current Loss**: 1.2665
33
+ - **Moving Average Loss**: 1.6637
34
+ - **Perplexity (PPL)**: ppl=5.3
35
+
36
+ ```bash
37
+ [Shard 8/100] jirack_pretrain_chunk_7.pt
38
+ Training jirack_pretrain_chunk_7.pt: 83%|█████▊ | 208/250 [08:09<01:38, 2.36s/it, loss=1.2665, avg_loss=1.6637, ppl=5.3, lr=2.00e-04]
39
+ [0] 0:python3* "mc [root@809e0ca1a59e" 19:45 04-Jul-26
40
+ ```
41
 
42
  Engineers anticipate a **steep drop** in the perplexity curve within the first 250 iterations of Shard 1 as the ternary weights align with the tokenizer's token distributions.
43