Dsg2 commited on
Commit
0f56185
·
verified ·
1 Parent(s): 40c1e0d

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -25,7 +25,7 @@ LS-Micro is 3x smaller than LS-TIny (11M vs 31M), and has 3x less active paramet
25
 
26
  Pushing the limits of size to quality, LS-Micro-Base is a sparse 11.49M MoE model (A5.52M) pretrained on 5.7B tokens.
27
  Training at a speed of 255.6k tokens/s on a 5060ti 16GB, it offers an incredible training efficiency at ~2.9x faster tokens seen compared to LS2.5's 108M-A17M model.
28
- Fully trained in ~6 hours.
29
 
30
  | **Model specs** | |
31
  |----------------------|--------|
@@ -35,7 +35,7 @@ Fully trained in ~6 hours.
35
  | Ngrams: | 4.21M |
36
  | Context length: | 2048 |
37
  | **Pretrain details** | |
38
- | Tokens seen: | ~5.7M |
39
  | GPU time (5060ti): | 22310s |
40
 
41
  *Note that it was trained with a 16k vocab, so 3.15M of the 5.52M active parameters are embedding.
 
25
 
26
  Pushing the limits of size to quality, LS-Micro-Base is a sparse 11.49M MoE model (A5.52M) pretrained on 5.7B tokens.
27
  Training at a speed of 255.6k tokens/s on a 5060ti 16GB, it offers an incredible training efficiency at ~2.9x faster tokens seen compared to LS2.5's 108M-A17M model.
28
+ Fully pretrained in ~6 hours.
29
 
30
  | **Model specs** | |
31
  |----------------------|--------|
 
35
  | Ngrams: | 4.21M |
36
  | Context length: | 2048 |
37
  | **Pretrain details** | |
38
+ | Tokens seen: | ~5.7B |
39
  | GPU time (5060ti): | 22310s |
40
 
41
  *Note that it was trained with a 16k vocab, so 3.15M of the 5.52M active parameters are embedding.