File size: 6,522 Bytes
ef4eb29 dab01f9 ef4eb29 dab01f9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 | ---
license: apache-2.0
datasets:
- openbmb/Ultra-FineWeb-L3
- HuggingFaceFW/fineweb-edu
language:
- en
pipeline_tag: text-generation
---
This repo contains the Base (completed pretrain cudalm checkpoint) variant of LS3-Micro-Dense. The FP32 checkpoint and optimiser is provided, and can be used by tinylm chat.
- LS3-Micro-Dense-Base.bin: Weights 36.5 MB
- LS3-Micro-Dense-Base.bin.env: Training config: 355 B
- LS3-Micro-Dense-Base.bin.opt: Optimizer state: 36.4 MB
- LS3-Micro-Dense-Base.bin.opt.mu: Muon momentum: 36.4 MB
- LS3-Micro-Dense-Base.bin.opt.t1: Attention gate state 6.3 MB
# LS3-Micro-Dense: First dense model in the LS family.
<img src="https://cdn-uploads.huggingface.co/production/uploads/691aa5bf8411a45dc9e761c3/ezAUaWFkRTVfD4Y-1n-yh.png" width="900">
<img src="https://cdn-uploads.huggingface.co/production/uploads/691aa5bf8411a45dc9e761c3/2c_BenV3NKVLy1qfNaMHv.png" width="900">
| (acc norm except LAMBADA) | HellaSwag | ARC-E | ARC-C | PIQA | LAMBADA | ArithMark | \*II | \*\*Intelligence Density |
| ------------------------------- | ------------- | --------- | --------- | ------------ | ----------- | ------------- | --------- | ---------------------------- |
| LS2.5-108M-A17M-Base | 28.24 | 36.66 | 23.21 | *57.40* | *17.89* | 37.00 | 9.89 | 0.231 |
| LS-Micro-Base (11.49M) | 27.41 | 34.72 | 21.93 | 54.84 | 11.06 | 35.50 | 7.24 | 0.909 |
| LS3-Micro-Base (9.97M) | 27.31 | 34.76 | 23.29 | **57.34** | 12.87 | 38.90 | 9.64 | **1.290** |
| LS3-Micro-Dense-Base (9.89M) | **28.53** | **39.27** | 20.31 | 56.58 | **16.15** | **47.50** | **11.99** | 1.212 |
With the help of a significantly bigger and curated dataset, LS3-Micro-Dense is able to achieve significant gains in LAMBADA, ARC-E and ArithMark. ARC-C remains below chance, while HellaSwag stays largely flat.
- Intelligence Index was calculated using the formula from [Open SLM Leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard). (*Results reproducable via Cudalm's built in bench tool*)
- Intelligence density derived using the following formula: Intelligence Index / sqrt(Active Params (millions) * Total Params (millions))
- The model was benchmarked via Cudalm's LM eval tool.
---
| Model Specs | # |
| --------------- | ---------------------- |
| Shape | Dense transformer |
| Hidden Dim | 256 |
| Layers | 12 |
| Attention heads | 8 |
| KV heads | 2 |
| FFN | 660 |
| Context length | 2048 |
| Sliding window | 1024 |
| Vocab | 4096 (digits singular) |
| Optimiser | Muon 2D, AdamW else |
| Parameters | 9,890,048 (FP32) |
---
## Training summary
| | Stage A (stable) | Stage B (cooldown) |
|---------------|---------------------------------|-------------------------------------------------------------|
| Steps | 228,127 | 60,878 (total 289,005) |
| Tokens | 14.951B | 3.990B (total 18.94B) |
| Learning rate | 1.05e-3 held, 1,000-step warmup | 1.05e-3 falling to 2.1e-5 |
| Time, speed | 33 h, 128k tok/s | 8.3 h, 133k tok/s |
## Datasets used:
Stage A (warmup and stable):
- HuggingFaceFW/fineweb-edu (8 shards, 7.2B tokens), ODC-By-1.0
- openbmb/Ultra-FineWeb, data/ultrafineweb_en (40 parts, 2.4B), ODC-By-1.0
- openbmb/Ultra-FineWeb-L3, qa (11 shards, 4B), Apache-2.0
- openbmb/Ultra-FineWeb-L3, multi_style (9 shards, 2.6B), Apache-2.0
- HuggingFaceTB/cosmopedia, data/web_samples_v2 (30 shards, 0.6B), Apache-2.0
- openbmb/Ultra-FineWeb-L3, multi_style (30 shards sifted from older corpus, 0.611B), ODC-By-1.0
- nvidia/OpenMathInstruct-2, refs/convert/parquet, default/train (8 shards, 1.5B), CC-BY-4.0
- HuggingFaceTB/finemath, finemath-4plus (40 shards, 1B), ODC-By-1.0
- ajibawa-2023/Education-College-Students, refs/convert/parquet (2 shards, 0.3B), Apache-2.0
- ajibawa-2023/Education-High-School-Students, refs/convert/parquet (2 shards, 0.3B), Apache-2.0
- ajibawa-2023/Education-Young-Children, refs/convert/parquet (2 shards, 0.2B), Apache-2.0
- HuggingFaceTB/cosmopedia: stanford, openstax, khanacademy (16 shards, 0.2B), Apache-2.0
- arithb: (self generated grade school level arithmetics, 0.9B), Self
Stage B (decay) on train splits:
- Rowan/hellaswag, default, train
- ARC-Easy, ARC-Challenge allenai/ai2_arc, ARC-Easy and ARC-Challenge, train
- github.com/ybisk/ybisk.github.io, master/piqa/data/train.jsonl and train-labels.lst
- allenai/openbookqa, additional, train
- allenai/sciq, default, train
- tau/commonsense_qa, default, train
- allenai/winogrande, winogrande_xl, train
---
## Examples:
Temp 0.2
```
<|user|>
Do you like pineapple on pizza?
<|assistant|>
"Pineapple" is a pineapple-like fruit that is rich in vitamin C. It is also known as "Pineapple" because it contains vitamin C and is rich in vitamin C.
Pineapple is a fruit that contains vitamin C and is rich in vitamin C. It is also known as "Pineapple" because it contains vitamin C and is rich in vitamin C.
Pineapple is a fruit that contains vitamin C and is rich in vitamin C. It is (truncated at 120 tokens)
```
Temp 0.7
```
<|user|>
How can I turn a chess piece into a stack of 100 dollar bills?
<|assistant|>
"Each dollar is divided into two parts. "
|Adding money to a stack of 100 dollars is $100+1 = 110$.
|Having a single stack of 100 dollars is $100+100 = 200$.
|Blending dollars is $110+100 = 210$.
|Glutening dollars is $100+100=200$.
|Bl (truncated at 120 tokens)
```
To try the model out yourself, benchmark it or fine tune it, clone [Cudalm](https://github.com/Dsg2/cudalm-c) and build it with `build.py` (Blackwell 2.0 supported, or else `build.py --cpu`). Then download the LS3-Micro-Dense checkpoint in the files section and drop them into Cudalm's Models folder (folders will self init on first run). |