| --- |
| license: apache-2.0 |
| language: |
| - en |
| pipeline_tag: text-generation |
| --- |
| |
| This repo contains the Base (completed pretrain cudalm checkpoint) variant of LS-Micro. The FP32 checkpoint and optimiser is provided, and can be used by tinylm chat. |
| - Model size: 28.1MB (FP32) |
| - Cudalm QLM optimiser: 26.1MB |
| - Ngrams container: 14.4MB |
| - T1 sidecar: 48KB |
| - Environment vars: 295B |
|
|
| # LS-Micro: Smallest and fastest model in the LS family. |
|
|
| | | HellaSwag | ARC-E | ARC-C | PIQA | LAMBADA | ArithMark | II* | |
| | ------------- | --------- | ----- | ----- | -------- | ------- | --------- | ---- | |
| | LS-Micro-Base | 27.41 | 34.72 | 21.93 | 54.84 | 11.06 | 35.50 | 7.24 | |
| | LS-Tiny-Base | 27.11 | 32.24 | 24.49 | 53.81 | 2.99 | 34.00 | 6.22 | |
|
|
| LS-Micro is 3x smaller than LS-TIny (11M vs 31M), and has 3x less active parameters (5M vs 15M), yet has a 16% higher intelligence index. |
|
|
| \* Intelligence Index was calculated using the formula from [Open SLM Leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard). (*Results reproducable via Cudalm's built in bench tool, see bottom quickstart*) |
|
|
| Pushing the limits of size to quality, LS-Micro-Base is a sparse 11.49M MoE model (A5.52M) pretrained on 5.7B tokens. |
| Training at a speed of 255.6k tokens/s on a 5060ti 16GB, it offers an incredible training efficiency at ~2.9x faster tokens seen compared to LS2.5's 108M-A17M model. |
| Fully pretrained in ~6 hours. |
|
|
| | **Model specs** | | |
| |----------------------|--------| |
| | Total parameters: | 11.49M | |
| | Backbone: | 7.28M | |
| | Active: | 5.52M | |
| | Ngrams: | 4.21M | |
| | Context length: | 2048 | |
| | **Pretrain details** | | |
| | Tokens seen: | ~5.7B | |
| | GPU time (5060ti): | 22310s | |
|
|
| *Note that it was trained with a 16k vocab, so 3.15M of the 5.52M active parameters are embedding. |
| |
| ## Chatbench results: |
| |
| Multiple choice: |
| | category | n | acc | accnorm | chance | |
| | ------------- | --- | --------- | ----------- | ------ | |
| | fact | 25 | 48.0% | 60.0% | 25.0% | |
| | commonsense | 20 | 65.0% | 55.0% | 25.0% | |
| | instruct | 20 | 55.0% | 50.0% | 25.0% | |
| | relevance | 20 | 5.0% | 30.0% | 25.0% | |
| | consistency | 18 | 61.1% | 72.2% | 25.0% | |
| | arithmetic | 15 | 46.7% | 46.7% | 25.0% | |
| | LS-Micro-Base | 118 | **46.6%** | **52.5%** | 25.0% | |
| | LS-Tiny-Base | 118 | 40.7% | 47.5% | 25.0% | |
| |
| Example: (Greedy) |
| ``` |
| <|user|> |
| Hi |
| <|assistant|> |
| I'm sorry, but I'm unable to assist with the problem. My current capabilities are limited to the functions provided to me. In this case, I can help you solve the problem. |
| ``` |
| |
| ## The base model is horrendous at chat, and is intended to be used as a base to be finetuned for specialised tasks. |
| To try LS-Micro or to train/finetune the model, follow this quickstart guide: |
| 1. Clone the latest version of Cudalm [here](https://github.com/Dsg2/cudalm-c) (only windows supported currently, optimised for Blackwell 2.0/CPU). |
| 2. Build with `build.py`/`build.py` --cpu'. Run cudalm to init all folders. |
| 3. Download all LS-Micro files from `LS-Micro-Base-Checkpoint\` and drop them in `<cudalm.exe>\models\`. Cudalm then automatically detects the model. |
| |
| Voila, you can now start experimenting with the model. Some basic stuff you can try first: |
| |
| - Check the model's config, and settings used to train it. `> Model info` |
| - Run the model: `> Generate` |
| |
| ## Finetune guide: |
| To finetune the model, you first need to prepare some data. |
| |
| Navigate to `> Data tools...`. You can download preconfigured mixtures of data, use files on your system or nitpick your own datasets. |
| - To use a preset corpus, navigate into `> Data tools... > Download text`. |
| - Otherwise, find a dataset on hugginface and navigate to `> Data tools... > Browse Hugging Face`, and pick out your chosen dataset from there. Note that datasets with gated access requires your `HF_TOKEN` to be set. |
| |
| This guide will use the dataset [MiniCPM5-2B-distill](https://huggingface.co/datasets/Dsg2/MiniCPM5-2B-distill) as an example. For your own text or other datasets, simply select the folder while tokenizing. |
| |
| Navigate to `> Data tools... > Browse Hugging Face`, and search for "MiniCPM5-2B-distill". Cudalm automatically detects the format, so all you need to do is download it (~250MB). |
| |
| Then you will need to tokenize the text so that the model can be trained on it. To do this, you will need a trained tokenizer. Luckily, you can extract an existing model's tokenizer. Use `> Data tools... > Tokenizer from model` and pull out LS-Micro-Base's tokenizer. |
| |
| Now you can tokenize your data. Go to `> Data tools... > Build corpus` and set `mode` to "new" and give it a corpus name. For your own datasets or text, navigate to the folder the text files are in. Otherwise, select `<cudalm.exe>\data\hf_MiniCPM5-2B-distill`, then use the extracted tokenizer and select `Run`. It should take ~10s and produce `<cudalm.exe>\data\<corpus name>` |
| |
| You're now ready to start fine tuning! |
| |
| Go to `> Train` and select "LS-Micro-Base" as the model, and the tokenized corpus as the data directory (both auto-detected). Select Run to start training the model on the selected corpus. Progress prints and periodic saves will occur every few hundred iters. |
