Update README.md

README.md

@@ -1,3 +1,117 @@
 ---
 license: apache-2.0
+datasets:
+- openbmb/Ultra-FineWeb-L3
+- HuggingFaceFW/fineweb-edu
+language:
+- en
+pipeline_tag: text-generation
 ---
+
+This repo contains the Base (completed pretrain cudalm checkpoint) variant of LS3-Micro-Dense. The FP32 checkpoint and optimiser is provided, and can be used by tinylm chat.
+ - LS3-Micro-Dense-Base.bin: Weights 36.5 MB
+ - LS3-Micro-Dense-Base.bin.env: Training config: 355 B
+ - LS3-Micro-Dense-Base.bin.opt: Optimizer state: 36.4 MB
+ - LS3-Micro-Dense-Base.bin.opt.mu: Muon momentum: 36.4 MB
+ - LS3-Micro-Dense-Base.bin.opt.t1: Attention gate state 6.3 MB
+
+# LS3-Micro-Dense: First dense model in the LS family.
+
+<img src="https://cdn-uploads.huggingface.co/production/uploads/691aa5bf8411a45dc9e761c3/ezAUaWFkRTVfD4Y-1n-yh.png" width="900">
+
+<img src="https://cdn-uploads.huggingface.co/production/uploads/691aa5bf8411a45dc9e761c3/2c_BenV3NKVLy1qfNaMHv.png" width="900">
+
+| (acc norm except LAMBADA)       | HellaSwag     | ARC-E     | ARC-C     | PIQA         | LAMBADA     | ArithMark     | \*II      | \*\*Intelligence Density     |
+| ------------------------------- | ------------- | --------- | --------- | ------------ | ----------- | ------------- | --------- | ---------------------------- |
+| LS2.5-108M-A17M-Base            | 28.24         | 36.66     | 23.21     | *57.40*      | *17.89*     | 37.00         | 9.89      | 0.231                        |
+| LS-Micro-Base (11.49M)          | 27.41         | 34.72     | 21.93     | 54.84        | 11.06       | 35.50         | 7.24      | 0.909                        |
+| LS3-Micro-Base (9.97M)          | 27.31         | 34.76     | 23.29     | **57.34**    | 12.87       |   38.90       | 9.64      | **1.290**                    |
+| LS3-Micro-Dense-Base (9.89M)    | **28.53**     | **39.27** | 20.31     | 56.58        | **16.15**   | **47.50**     | **11.99** | 1.212                        |
+
+With the help of a significantly bigger and curated dataset, LS3-Micro-Dense is able to achieve significant gains in LAMBADA, ARC-E and ArithMark. ARC-C remains below chance, while HellaSwag stays largely flat.
+
+ - Intelligence Index was calculated using the formula from [Open SLM Leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard). (*Results reproducable via Cudalm's built in bench tool*)
+ - Intelligence density derived using the following formula: Intelligence Index / sqrt(Active Params (millions) * Total Params (millions))
+ - The model was benchmarked via Cudalm's LM eval tool.
+
+---
+
+| Model Specs     | #                      |
+| --------------- | ---------------------- |
+| Shape           | Dense transformer      |
+| Hidden Dim      | 256                    |
+| Layers          | 12                     |
+| Attention heads | 8                      | 
+| KV heads        | 2                      |
+| FFN             | 660                    |
+| Context length  | 2048                   |
+| Sliding window  | 1024                   |
+| Vocab           | 4096 (digits singular) |
+| Optimiser       | Muon 2D, AdamW else    |
+| Parameters      | 9,890,048 (FP32)       |
+
+---
+
+## Training summary
+
+|               | Stage A (stable)                | Stage B (cooldown)                                          |
+|---------------|---------------------------------|-------------------------------------------------------------|
+| Steps         | 228,127                         | 60,878 (total 289,005)                                      |
+| Tokens        | 14.951B                         | 3.990B (total 18.94B)                                       |
+| Learning rate | 1.05e-3 held, 1,000-step warmup | 1.05e-3 falling to 2.1e-5                                   |
+| Time, speed   | 33 h, 128k tok/s                | 8.3 h, 133k tok/s                                           |
+
+## Datasets used:
+
+Stage A (warmup and stable):
+ - HuggingFaceFW/fineweb-edu (8 shards, 7.2B tokens), ODC-By-1.0
+ - openbmb/Ultra-FineWeb, data/ultrafineweb_en (40 parts, 2.4B), ODC-By-1.0
+ - openbmb/Ultra-FineWeb-L3, qa (11 shards, 4B), Apache-2.0
+ - openbmb/Ultra-FineWeb-L3, multi_style (9 shards, 2.6B), Apache-2.0
+ - HuggingFaceTB/cosmopedia, data/web_samples_v2 (30 shards, 0.6B), Apache-2.0
+ - openbmb/Ultra-FineWeb-L3, multi_style (30 shards sifted from older corpus, 0.611B), ODC-By-1.0
+ - nvidia/OpenMathInstruct-2, refs/convert/parquet, default/train (8 shards, 1.5B), CC-BY-4.0
+ - HuggingFaceTB/finemath, finemath-4plus (40 shards, 1B), ODC-By-1.0
+ - ajibawa-2023/Education-College-Students, refs/convert/parquet (2 shards, 0.3B), Apache-2.0
+ - ajibawa-2023/Education-High-School-Students, refs/convert/parquet (2 shards, 0.3B), Apache-2.0
+ - ajibawa-2023/Education-Young-Children, refs/convert/parquet (2 shards, 0.2B), Apache-2.0
+ - HuggingFaceTB/cosmopedia: stanford, openstax, khanacademy (16 shards, 0.2B), Apache-2.0
+ - arithb: (self generated grade school level arithmetics, 0.9B), Self
+
+Stage B (decay) on train splits:
+ - Rowan/hellaswag, default, train
+ - ARC-Easy, ARC-Challenge	allenai/ai2_arc, ARC-Easy and ARC-Challenge, train
+ - github.com/ybisk/ybisk.github.io, master/piqa/data/train.jsonl and train-labels.lst
+ - allenai/openbookqa, additional, train
+ - allenai/sciq, default, train
+ - tau/commonsense_qa, default, train
+ - allenai/winogrande, winogrande_xl, train
+
+---
+
+## Examples:
+
+Temp 0.2
+```
+<|user|>
+Do you like pineapple on pizza?
+<|assistant|>
+"Pineapple" is a pineapple-like fruit that is rich in vitamin C. It is also known as "Pineapple" because it contains vitamin C and is rich in vitamin C.
+Pineapple is a fruit that contains vitamin C and is rich in vitamin C. It is also known as "Pineapple" because it contains vitamin C and is rich in vitamin C.
+Pineapple is a fruit that contains vitamin C and is rich in vitamin C. It is (truncated at 120 tokens)
+```
+
+Temp 0.7
+```
+<|user|>
+How can I turn a chess piece into a stack of 100 dollar bills?
+<|assistant|>
+"Each dollar is divided into two parts. "
+|Adding money to a stack of 100 dollars is $100+1 = 110$.
+|Having a single stack of 100 dollars is $100+100 = 200$.
+|Blending dollars is $110+100 = 210$.
+|Glutening dollars is $100+100=200$.
+|Bl (truncated at 120 tokens)
+```
+
+To try the model out yourself, benchmark it or fine tune it, clone [Cudalm](https://github.com/Dsg2/cudalm-c) and build it with `build.py` (Blackwell 2.0 supported, or else `build.py --cpu`). Then download the LS3-Micro-Dense checkpoint in the files section and drop them into Cudalm's Models folder (folders will self init on first run).