Update README.md
Browse files
README.md
CHANGED
|
@@ -25,7 +25,7 @@ LS-Micro is 3x smaller than LS-TIny (11M vs 31M), and has 3x less active paramet
|
|
| 25 |
|
| 26 |
Pushing the limits of size to quality, LS-Micro-Base is a sparse 11.49M MoE model (A5.52M) pretrained on 5.7B tokens.
|
| 27 |
Training at a speed of 255.6k tokens/s on a 5060ti 16GB, it offers an incredible training efficiency at ~2.9x faster tokens seen compared to LS2.5's 108M-A17M model.
|
| 28 |
-
Fully
|
| 29 |
|
| 30 |
| **Model specs** | |
|
| 31 |
|----------------------|--------|
|
|
@@ -35,7 +35,7 @@ Fully trained in ~6 hours.
|
|
| 35 |
| Ngrams: | 4.21M |
|
| 36 |
| Context length: | 2048 |
|
| 37 |
| **Pretrain details** | |
|
| 38 |
-
| Tokens seen: | ~5.
|
| 39 |
| GPU time (5060ti): | 22310s |
|
| 40 |
|
| 41 |
*Note that it was trained with a 16k vocab, so 3.15M of the 5.52M active parameters are embedding.
|
|
|
|
| 25 |
|
| 26 |
Pushing the limits of size to quality, LS-Micro-Base is a sparse 11.49M MoE model (A5.52M) pretrained on 5.7B tokens.
|
| 27 |
Training at a speed of 255.6k tokens/s on a 5060ti 16GB, it offers an incredible training efficiency at ~2.9x faster tokens seen compared to LS2.5's 108M-A17M model.
|
| 28 |
+
Fully pretrained in ~6 hours.
|
| 29 |
|
| 30 |
| **Model specs** | |
|
| 31 |
|----------------------|--------|
|
|
|
|
| 35 |
| Ngrams: | 4.21M |
|
| 36 |
| Context length: | 2048 |
|
| 37 |
| **Pretrain details** | |
|
| 38 |
+
| Tokens seen: | ~5.7B |
|
| 39 |
| GPU time (5060ti): | 22310s |
|
| 40 |
|
| 41 |
*Note that it was trained with a 16k vocab, so 3.15M of the 5.52M active parameters are embedding.
|