Update README.md

README.md

@@ -5,73 +5,80 @@ language:
 pipeline_tag: text-generation
 ---
 
-This repo contains the Base (completed pretrain cudalm checkpoint) variant of LS-Micro. The FP32 checkpoint and optimiser is provided, and can be used by tinylm chat.
- - Model size: 28.1MB (FP32)
- - Cudalm QLM optimiser: 26.1MB
- - Ngrams container: 14.4MB
- - T1 sidecar: 48KB
- - Environment vars: 295B
+This repo contains the Base (completed pretrain cudalm checkpoint) variant of LS3-Micro. The FP32 checkpoint and optimiser is provided, and can be used by tinylm chat.
+ - Model size: 30.3MB (FP32)
+ - Environment variables for cudalm: 381B
+ - AdamW optimiser: 30.0MB
+ - Muon optimiser: 30.0MB
+ - Ngrams optimiser: 7.22MB
+ - Sidecar: 40.1KB
 
-# LS-Micro: Smallest and fastest model in the LS family.
+# LS3-Micro: The smallest model with the highest intelligence density in the LS family.
 
-|               | HellaSwag | ARC-E | ARC-C | PIQA     | LAMBADA | ArithMark | II*  |
-| ------------- | --------- | ----- | ----- | -------- | ------- | --------- | ---- |
-| LS-Micro-Base | 27.41     | 34.72 | 21.93 | 54.84    | 11.06   | 35.50     | 7.24 |
-| LS-Tiny-Base  | 27.11     | 32.24 | 24.49 | 53.81    | 2.99    | 34.00     | 6.22 |
+<img src="https://cdn-uploads.huggingface.co/production/uploads/691aa5bf8411a45dc9e761c3/azRRf7UufnElUkxnl6lst.webp" width="900">
 
-LS-Micro is 3x smaller than LS-TIny (11M vs 31M), and has 3x less active parameters (5M vs 15M), yet has a 16% higher intelligence index.
+<img src="https://cdn-uploads.huggingface.co/production/uploads/691aa5bf8411a45dc9e761c3/v_irkP1sB30tm48uVcGDH.webp" width="900">
 
-\* Intelligence Index was calculated using the formula from [Open SLM Leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard). (*Results reproducable via Cudalm's built in bench tool, see bottom quickstart*)
+| (acc norm except LAMBADA) | HellaSwag | ARC-E | ARC-C | PIQA     | LAMBADA | ArithMark | \*II  | \*\*Intelligence Density |
+| ------------------------- | --------- | ----- | ----- | -------- | ------- | --------- | ----- | ------------------------ |
+| LS2.5-108M-A17M-Base      | 28.24     | 36.66 | 23.21 | 57.40    | 17.89   | 37.00     | 9.89  | 0.231                    |
+| LS-Tiny-Base (31M)        | 27.11     | 32.24 | 24.49 | 53.81    | 2.99    | 34.00     | 6.22  | 0.288                    |
+| LS-Micro-Base (11.49M)    | 27.41     | 34.72 | 21.93 | 54.84    | 11.06   | 35.50     | 7.24  | 0.909                    |
+| LS3-Micro-Base (9.97M)    | 27.31     | 34.76 | 23.29 | 57.34    | 12.87   | 38.90     | 9.64  | 1.290                    |
 
-Pushing the limits of size to quality, LS-Micro-Base is a sparse 11.49M MoE model (A5.52M) pretrained on 5.7B tokens.
-Training at a speed of 255.6k tokens/s on a 5060ti 16GB, it offers an incredible training efficiency at ~2.9x faster tokens seen compared to LS2.5's 108M-A17M model.
-Fully pretrained in ~6 hours.
+LS3-Micro-Base has a 24.9% higher Intelligence Index than LS-Micro Base, while having 13% less total parameters. It also nearly matches LS2.5-108M-A17M with almost 11x less parameters.
+
+ - Intelligence Index was calculated using the formula from [Open SLM Leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard). (*Results reproducable via Cudalm's built in bench tool, see bottom quickstart*)
+ - Compute density derived using the following formula: Intelligence Index / sqrt(Active Params (millions) * Total Params (millions))
+
+---
+
+LS3 Experiments on narrow but deeper model architectures for SLMs. For a sub 10M model, the sweet spot found was 11 layers with a dimension of 160. 
+To maximise efficiency while training, LS3 is a not a dense model, but rather a sparse MoE model with ngrams. With a total of 9.97M parameters, ~2.11M of parameters are ngrams. The remaining ~7.87M of parameters make up the MoE backbone. The model has 2 experts and only activates one per token, leading to only 5.60M active per token.  
 
 | **Model specs**      |        |
 |----------------------|--------|
-| Total parameters:    | 11.49M |
-| Backbone:            | 7.28M  |
-| Active:              | 5.52M  |
-| Ngrams:              | 4.21M  |
+| Total parameters:    | 9.97M  |
+| Backbone:            | 7.87M  |
+| Active:              | 5.60M  |
+| Ngrams:              | 2.11M  |
 | Context length:      | 2048   |
 | **Pretrain details** |        |
 | Tokens seen:         | ~5.7B  |
-| GPU time (5060ti):   | 22310s |
+| GPU time (5060ti):   | 8h     |
 
-*Note that it was trained with a 16k vocab, so 3.15M of the 5.52M active parameters are embedding.
+*Note that it was trained with a 16k vocab, so 2.62M of the 5.52M active parameters are embedding.
 
 ## Chatbench results:
 
 Multiple choice:
-| category      | n   | acc       | accnorm     | chance |
-| ------------- | --- | --------- | ----------- | ------ |
-| fact          | 25  | 48.0%     | 60.0%       | 25.0%  |
-| commonsense   | 20  | 65.0%     | 55.0%       | 25.0%  |
-| instruct      | 20  | 55.0%     | 50.0%       | 25.0%  |
-| relevance     | 20  | 5.0%      | 30.0%       | 25.0%  |
-| consistency   | 18  | 61.1%     | 72.2%       | 25.0%  |
-| arithmetic    | 15  | 46.7%     | 46.7%       | 25.0%  |
-| LS-Micro-Base | 118 | **46.6%** | **52.5%**   | 25.0%  |
-| LS-Tiny-Base  | 118 | 40.7%     | 47.5%       | 25.0%  |
-
-Example: (Greedy)
+| Model          | acc       | accnorm     | chance |
+| -------------- | --------- | ----------- | ------ |
+| LS-Tiny-Base   | 40.7%     | 47.5%       | 25.0%  |
+| LS-Micro-Base  | **46.6%** | 52.5%       | 25.0%  |
+| LS3-Micro-Base | 45.8%     | **55.1%**   | 25.0%  |
+
+Example:
 ```
 <|user|>
-Hi
+What is your opinion on Strawberries?
 <|assistant|>
-I'm sorry, but I'm unable to assist with the problem. My current capabilities are limited to the functions provided to me. In this case, I can help you solve the problem.
+A strawberry, which is a type of paprika, or strawberry, is a type of meat-based food that is commonly used in culinary arts. Strawberry is a type of lion's food, also known as lime juice, which is a staple in the food. This type of strawberry is not only a vegetable but also a sweet and hearty treat.
+
+The idea is that strawberry is a type of meat-based food, which is a type of lion's food, like paprika or...
 ```
 
-## The base model is horrendous at chat, and is intended to be used as a base to be finetuned for specialised tasks. 
-To try LS-Micro or to train/finetune the model, follow this quickstart guide:
+## To try LS3-Micro or to train/finetune the model, follow this quickstart guide:
 1. Clone the latest version of Cudalm [here](https://github.com/Dsg2/cudalm-c) (only windows supported currently, optimised for Blackwell 2.0/CPU).
 2. Build with `build.py`/`build.py` --cpu'. Run cudalm to init all folders.
-3. Download all LS-Micro files from `LS-Micro-Base-Checkpoint\` and drop them in `<cudalm.exe>\models\`. Cudalm then automatically detects the model.
+3. Download all LS3-Micro files from `LS3-Micro-Base-Checkpoint\` and drop them in `<cudalm.exe>\models\`. Cudalm then automatically detects the model.
 
 Voila, you can now start experimenting with the model. Some basic stuff you can try first:
 
- - Check the model's config, and settings used to train it. `> Model info`
- - Run the model: `> Generate`
+ - Check the model's config, and settings used to train it. `> Model info`.
+ - Run the model: `> Generate`.
+
+To reproduce the benchmark results, use `> LM eval`. Run all benchmarks.
 
 ## Finetune guide:
 To finetune the model, you first need to prepare some data.
@@ -84,10 +91,10 @@ This guide will use the dataset [MiniCPM5-2B-distill](https://huggingface.co/dat
 
 Navigate to `> Data tools... > Browse Hugging Face`, and search for "MiniCPM5-2B-distill". Cudalm automatically detects the format, so all you need to do is download it (~250MB).
 
-Then you will need to tokenize the text so that the model can be trained on it. To do this, you will need a trained tokenizer. Luckily, you can extract an existing model's tokenizer. Use `> Data tools... > Tokenizer from model` and pull out LS-Micro-Base's tokenizer.
+Then you will need to tokenize the text so that the model can be trained on it. To do this, you will need a trained tokenizer. Luckily, you can extract an existing model's tokenizer. Use `> Data tools... > Tokenizer from model` and pull out LS3-Micro-Base's tokenizer.
 
 Now you can tokenize your data. Go to `> Data tools... > Build corpus` and set `mode` to "new" and give it a corpus name. For your own datasets or text, navigate to the folder the text files are in. Otherwise, select `<cudalm.exe>\data\hf_MiniCPM5-2B-distill`, then use the extracted tokenizer and select `Run`. It should take ~10s and produce `<cudalm.exe>\data\<corpus name>`
 
 You're now ready to start fine tuning!
 
-Go to `> Train` and select "LS-Micro-Base" as the model, and the tokenized corpus as the data directory (both auto-detected). Select Run to start training the model on the selected corpus. Progress prints and periodic saves will occur every few hundred iters.
+Go to `> Train` and select "LS3-Micro-Base" as the model, and the tokenized corpus as the data directory (both auto-detected). Select Run to start training the model on the selected corpus. Progress prints and periodic saves will occur every few hundred iters.