| --- |
| license: apache-2.0 |
| datasets: |
| - openbmb/Ultra-FineWeb-L3 |
| - HuggingFaceFW/fineweb-edu |
| language: |
| - en |
| pipeline_tag: text-generation |
| --- |
| |
| This repo contains the Base (completed pretrain cudalm checkpoint) variant of LS3-Micro-Dense. The FP32 checkpoint and optimiser is provided, and can be used by tinylm chat. |
| - LS3-Micro-Dense-Base.bin: Weights 36.5 MB |
| - LS3-Micro-Dense-Base.bin.env: Training config: 355 B |
| - LS3-Micro-Dense-Base.bin.opt: Optimizer state: 36.4 MB |
| - LS3-Micro-Dense-Base.bin.opt.mu: Muon momentum: 36.4 MB |
| - LS3-Micro-Dense-Base.bin.opt.t1: Attention gate state 6.3 MB |
|
|
| # LS3-Micro-Dense: First dense model in the LS family. |
|
|
| <img src="https://cdn-uploads.huggingface.co/production/uploads/691aa5bf8411a45dc9e761c3/ezAUaWFkRTVfD4Y-1n-yh.png" width="900"> |
|
|
| <img src="https://cdn-uploads.huggingface.co/production/uploads/691aa5bf8411a45dc9e761c3/2c_BenV3NKVLy1qfNaMHv.png" width="900"> |
|
|
| | (acc norm except LAMBADA) | HellaSwag | ARC-E | ARC-C | PIQA | LAMBADA | ArithMark | \*II | \*\*Intelligence Density | |
| | ------------------------------- | ------------- | --------- | --------- | ------------ | ----------- | ------------- | --------- | ---------------------------- | |
| | LS2.5-108M-A17M-Base | 28.24 | 36.66 | 23.21 | *57.40* | *17.89* | 37.00 | 9.89 | 0.231 | |
| | LS-Micro-Base (11.49M) | 27.41 | 34.72 | 21.93 | 54.84 | 11.06 | 35.50 | 7.24 | 0.909 | |
| | LS3-Micro-Base (9.97M) | 27.31 | 34.76 | 23.29 | **57.34** | 12.87 | 38.90 | 9.64 | **1.290** | |
| | LS3-Micro-Dense-Base (9.89M) | **28.53** | **39.27** | 20.31 | 56.58 | **16.15** | **47.50** | **11.99** | 1.212 | |
|
|
| With the help of a significantly bigger and curated dataset, LS3-Micro-Dense is able to achieve significant gains in LAMBADA, ARC-E and ArithMark. ARC-C remains below chance, while HellaSwag stays largely flat. |
|
|
| - Intelligence Index was calculated using the formula from [Open SLM Leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard). (*Results reproducable via Cudalm's built in bench tool*) |
| - Intelligence density derived using the following formula: Intelligence Index / sqrt(Active Params (millions) * Total Params (millions)) |
| - The model was benchmarked via Cudalm's LM eval tool. |
|
|
| --- |
|
|
| | Model Specs | # | |
| | --------------- | ---------------------- | |
| | Shape | Dense transformer | |
| | Hidden Dim | 256 | |
| | Layers | 12 | |
| | Attention heads | 8 | |
| | KV heads | 2 | |
| | FFN | 660 | |
| | Context length | 2048 | |
| | Sliding window | 1024 | |
| | Vocab | 4096 (digits singular) | |
| | Optimiser | Muon 2D, AdamW else | |
| | Parameters | 9,890,048 (FP32) | |
|
|
| --- |
|
|
| ## Training summary |
|
|
| | | Stage A (stable) | Stage B (cooldown) | |
| |---------------|---------------------------------|-------------------------------------------------------------| |
| | Steps | 228,127 | 60,878 (total 289,005) | |
| | Tokens | 14.951B | 3.990B (total 18.94B) | |
| | Learning rate | 1.05e-3 held, 1,000-step warmup | 1.05e-3 falling to 2.1e-5 | |
| | Time, speed | 33 h, 128k tok/s | 8.3 h, 133k tok/s | |
|
|
| ## Datasets used: |
|
|
| Stage A (warmup and stable): |
| - HuggingFaceFW/fineweb-edu (8 shards, 7.2B tokens), ODC-By-1.0 |
| - openbmb/Ultra-FineWeb, data/ultrafineweb_en (40 parts, 2.4B), ODC-By-1.0 |
| - openbmb/Ultra-FineWeb-L3, qa (11 shards, 4B), Apache-2.0 |
| - openbmb/Ultra-FineWeb-L3, multi_style (9 shards, 2.6B), Apache-2.0 |
| - HuggingFaceTB/cosmopedia, data/web_samples_v2 (30 shards, 0.6B), Apache-2.0 |
| - openbmb/Ultra-FineWeb-L3, multi_style (30 shards sifted from older corpus, 0.611B), ODC-By-1.0 |
| - nvidia/OpenMathInstruct-2, refs/convert/parquet, default/train (8 shards, 1.5B), CC-BY-4.0 |
| - HuggingFaceTB/finemath, finemath-4plus (40 shards, 1B), ODC-By-1.0 |
| - ajibawa-2023/Education-College-Students, refs/convert/parquet (2 shards, 0.3B), Apache-2.0 |
| - ajibawa-2023/Education-High-School-Students, refs/convert/parquet (2 shards, 0.3B), Apache-2.0 |
| - ajibawa-2023/Education-Young-Children, refs/convert/parquet (2 shards, 0.2B), Apache-2.0 |
| - HuggingFaceTB/cosmopedia: stanford, openstax, khanacademy (16 shards, 0.2B), Apache-2.0 |
| - arithb: (self generated grade school level arithmetics, 0.9B), Self |
| |
| Stage B (decay) on train splits: |
| - Rowan/hellaswag, default, train |
| - ARC-Easy, ARC-Challenge allenai/ai2_arc, ARC-Easy and ARC-Challenge, train |
| - github.com/ybisk/ybisk.github.io, master/piqa/data/train.jsonl and train-labels.lst |
| - allenai/openbookqa, additional, train |
| - allenai/sciq, default, train |
| - tau/commonsense_qa, default, train |
| - allenai/winogrande, winogrande_xl, train |
|
|
| --- |
|
|
| ## Examples: |
|
|
| Temp 0.2 |
| ``` |
| <|user|> |
| Do you like pineapple on pizza? |
| <|assistant|> |
| "Pineapple" is a pineapple-like fruit that is rich in vitamin C. It is also known as "Pineapple" because it contains vitamin C and is rich in vitamin C. |
| Pineapple is a fruit that contains vitamin C and is rich in vitamin C. It is also known as "Pineapple" because it contains vitamin C and is rich in vitamin C. |
| Pineapple is a fruit that contains vitamin C and is rich in vitamin C. It is (truncated at 120 tokens) |
| ``` |
|
|
| Temp 0.7 |
| ``` |
| <|user|> |
| How can I turn a chess piece into a stack of 100 dollar bills? |
| <|assistant|> |
| "Each dollar is divided into two parts. " |
| |Adding money to a stack of 100 dollars is $100+1 = 110$. |
| |Having a single stack of 100 dollars is $100+100 = 200$. |
| |Blending dollars is $110+100 = 210$. |
| |Glutening dollars is $100+100=200$. |
| |Bl (truncated at 120 tokens) |
| ``` |
|
|
| To try the model out yourself, benchmark it or fine tune it, clone [Cudalm](https://github.com/Dsg2/cudalm-c) and build it with `build.py` (Blackwell 2.0 supported, or else `build.py --cpu`). Then download the LS3-Micro-Dense checkpoint in the files section and drop them into Cudalm's Models folder (folders will self init on first run). |
