Blame for README.md

---
license: apache-2.0
language:
- en
pipeline_tag: text-generation
---

This repo contains the Base (completed pretrain cudalm checkpoint) variant of LS3-Micro. The FP32 checkpoint and optimiser is provided, and can be used by tinylm chat.
 - Model size: 30.3MB (FP32)
 - Environment variables for cudalm: 381B
 - AdamW optimiser: 30.0MB
 - Muon optimiser: 30.0MB
 - Ngrams optimiser: 7.22MB
 - Sidecar: 40.1KB

# LS3-Micro: The smallest model with the highest intelligence density in the LS family.

<img src="https://cdn-uploads.huggingface.co/production/uploads/691aa5bf8411a45dc9e761c3/azRRf7UufnElUkxnl6lst.webp" width="900">

<img src="https://cdn-uploads.huggingface.co/production/uploads/691aa5bf8411a45dc9e761c3/v_irkP1sB30tm48uVcGDH.webp" width="900">

| (acc norm except LAMBADA) | HellaSwag | ARC-E     | ARC-C     | PIQA         | LAMBADA     | ArithMark     | \*II      | \*\*Intelligence Density     |
| ------------------------- | --------- | --------- | --------- | ------------ | ----------- | ------------- | --------- | ---------------------------- |
| LS2.5-108M-A17M-Base      | *28.24*   | *36.66*   | 23.21     | *57.40*      | *17.89*     | 37.00         | *9.89*    | 0.231                        |
| LS-Tiny-Base (31M)        | 27.11     | 32.24     | **24.49** | 53.81        | 2.99        | 34.00         | 6.22      | 0.288                        |
| LS-Micro-Base (11.49M)    | **27.41** | 34.72     | 21.93     | 54.84        | 11.06       | 35.50         | 7.24      | 0.909                        |
| LS3-Micro-Base (9.97M)    | 27.31     | **34.76** | 23.29     | **57.34**    | **12.87**   | **38.90**     | **9.64**  | **1.290**                    |

LS3-Micro-Base has a 24.9% higher Intelligence Index than LS-Micro Base, while having 13% less total parameters. It also nearly matches LS2.5-108M-A17M with almost 11x less parameters.

 - Intelligence Index was calculated using the formula from [Open SLM Leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard). (*Results reproducable via Cudalm's built in bench tool, see bottom quickstart*)
 - Intelligence density derived using the following formula: Intelligence Index / sqrt(Active Params (millions) * Total Params (millions))

---

LS3 Experiments on narrow but deeper model architectures for SLMs. For a sub 10M model, the sweet spot found was 11 layers with a dimension of 160. 
To maximise efficiency while training, LS3 is a not a dense model, but rather a sparse MoE model with ngrams. With a total of 9.97M parameters, ~2.11M of parameters are ngrams. The remaining ~7.87M of parameters make up the MoE backbone. The model has 2 experts and only activates one per token, leading to only 5.60M active per token.  

| **Model specs**      |        |
|----------------------|--------|
| Total parameters:    | 9.97M  |
| Backbone:            | 7.87M  |
| Active:              | 5.60M  |
| Ngrams:              | 2.11M  |
| Context length:      | 2048   |
| **Pretrain details** |        |
| Tokens seen:         | ~5.7B  |
| GPU time (5060ti):   | 8h     |

\*Note that it was trained with a 16k vocab, so 2.62M of the 5.52M active parameters are embedding.

## Chatbench results:

Multiple choice:
| Model          | acc       | accnorm     | chance |
| -------------- | --------- | ----------- | ------ |
| LS-Tiny-Base   | 40.7%     | 47.5%       | 25.0%  |
| LS-Micro-Base  | **46.6%** | 52.5%       | 25.0%  |
| LS3-Micro-Base | 45.8%     | **55.1%**   | 25.0%  |

Example:
```
<|user|>
What is your opinion on Strawberries?
<|assistant|>
A strawberry, which is a type of paprika, or strawberry, is a type of meat-based food that is commonly used in culinary arts. Strawberry is a type of lion's food, also known as lime juice, which is a staple in the food. This type of strawberry is not only a vegetable but also a sweet and hearty treat.

The idea is that strawberry is a type of meat-based food, which is a type of lion's food, like paprika or...
```

## To try LS3-Micro or to train/finetune the model, follow this quickstart guide:
1. Clone the latest version of Cudalm [here](https://github.com/Dsg2/cudalm-c) (only windows supported currently, optimised for Blackwell 2.0/CPU).
2. Build with `build.py`/`build.py` --cpu'. Run cudalm to init all folders.
3. Download all LS3-Micro files from `LS3-Micro-Base-Checkpoint\` and drop them in `<cudalm.exe>\models\`. Cudalm then automatically detects the model.

Voila, you can now start experimenting with the model. Some basic stuff you can try first:

 - Check the model's config, and settings used to train it. `> Model info`.
 - Run the model: `> Generate`.

To reproduce the benchmark results, use `> LM eval`. Run all benchmarks.

## Finetune guide:
To finetune the model, you first need to prepare some data.

Navigate to `> Data tools...`. You can download preconfigured mixtures of data, use files on your system or nitpick your own datasets. 
 - To use a preset corpus, navigate into `> Data tools... > Download text`. 
 - Otherwise, find a dataset on hugginface and navigate to `> Data tools... > Browse Hugging Face`, and pick out your chosen dataset from there. Note that datasets with gated access requires your `HF_TOKEN` to be set.

This guide will use the dataset [MiniCPM5-2B-distill](https://huggingface.co/datasets/Dsg2/MiniCPM5-2B-distill) as an example. For your own text or other datasets, simply select the folder while tokenizing.

Navigate to `> Data tools... > Browse Hugging Face`, and search for "MiniCPM5-2B-distill". Cudalm automatically detects the format, so all you need to do is download it (~250MB).

Then you will need to tokenize the text so that the model can be trained on it. To do this, you will need a trained tokenizer. Luckily, you can extract an existing model's tokenizer. Use `> Data tools... > Tokenizer from model` and pull out LS3-Micro-Base's tokenizer.

Now you can tokenize your data. Go to `> Data tools... > Build corpus` and set `mode` to "new" and give it a corpus name. For your own datasets or text, navigate to the folder the text files are in. Otherwise, select `<cudalm.exe>\data\hf_MiniCPM5-2B-distill`, then use the extracted tokenizer and select `Run`. It should take ~10s and produce `<cudalm.exe>\data\<corpus name>`

You're now ready to start fine tuning!

Go to `> Train` and select "LS3-Micro-Base" as the model, and the tokenized corpus as the data directory (both auto-detected). Select Run to start training the model on the selected corpus. Progress prints and periodic saves will occur every few hundred iters.