Duplicate from Dsg2/LS-Micro-Base

.gitattributes

@@ -0,0 +1,37 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+LS-Micro-Base-Checkpoint/LS-Micro-Base.bin.opt.ng filter=lfs diff=lfs merge=lfs -text
+LS-Micro-Base-Checkpoint/LS-Micro-Base.bin.opt.q filter=lfs diff=lfs merge=lfs -text

LS-Micro-Base-Checkpoint/LS-Micro-Base.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:72321fc00c03831c83673c79dbefde1d26be43a45ce56d247dbf49d0a92cdc66
+size 29474320

LS-Micro-Base-Checkpoint/LS-Micro-Base.bin.env

@@ -0,0 +1,17 @@
+# cudalm model config (written by new/train). KEY=VALUE; the environment overrides these.
+CUDALM_BENCH_EVERY=0
+CUDALM_BENCH_N=0
+CUDALM_QLM=1
+CUDALM_ATTN_GATE=0
+CUDALM_TOPK=1
+CUDALM_UNIPOOL=0
+NGRAM_Q4_FOLD=1
+CUDALM_NGRAM=1
+CUDALM_ENGRAM=0
+NGRAM_ROWS=131072
+NGRAM_DIM=32
+NGRAM_Q4=1
+NGRAM_GATE=1
+NGRAM_LAYER=0
+NGRAM_ORDER=2
+NGRAM_TRI=0

LS-Micro-Base-Checkpoint/LS-Micro-Base.bin.opt.ng

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:14114027d057cbab5a7b19a35ca96954d7497aad2c559ee0ed1b994309d5a62b
+size 15122460

LS-Micro-Base-Checkpoint/LS-Micro-Base.bin.opt.q

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:abd18fc908e8adb1961c99249c25b4c5a13b3214d621d14bd929c4724a0f5af5
+size 27439924

README.md

@@ -0,0 +1,93 @@
+---
+license: apache-2.0
+language:
+- en
+pipeline_tag: text-generation
+---
+
+This repo contains the Base (completed pretrain cudalm checkpoint) variant of LS-Micro. The FP32 checkpoint and optimiser is provided, and can be used by tinylm chat.
+ - Model size: 28.1MB (FP32)
+ - Cudalm QLM optimiser: 26.1MB
+ - Ngrams container: 14.4MB
+ - T1 sidecar: 48KB
+ - Environment vars: 295B
+
+# LS-Micro: Smallest and fastest model in the LS family.
+
+|               | HellaSwag | ARC-E | ARC-C | PIQA     | LAMBADA | ArithMark | II*  |
+| ------------- | --------- | ----- | ----- | -------- | ------- | --------- | ---- |
+| LS-Micro-Base | 27.41     | 34.72 | 21.93 | 54.84    | 11.06   | 35.50     | 7.24 |
+| LS-Tiny-Base  | 27.11     | 32.24 | 24.49 | 53.81    | 2.99    | 34.00     | 6.22 |
+
+LS-Micro is 3x smaller than LS-TIny (11M vs 31M), and has 3x less active parameters (5M vs 15M), yet has a 16% higher intelligence index.
+
+\* Intelligence Index was calculated using the formula from [Open SLM Leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard). (*Results reproducable via Cudalm's built in bench tool, see bottom quickstart*)
+
+Pushing the limits of size to quality, LS-Micro-Base is a sparse 11.49M MoE model (A5.52M) pretrained on 5.7B tokens.
+Training at a speed of 255.6k tokens/s on a 5060ti 16GB, it offers an incredible training efficiency at ~2.9x faster tokens seen compared to LS2.5's 108M-A17M model.
+Fully pretrained in ~6 hours.
+
+| **Model specs**      |        |
+|----------------------|--------|
+| Total parameters:    | 11.49M |
+| Backbone:            | 7.28M  |
+| Active:              | 5.52M  |
+| Ngrams:              | 4.21M  |
+| Context length:      | 2048   |
+| **Pretrain details** |        |
+| Tokens seen:         | ~5.7B  |
+| GPU time (5060ti):   | 22310s |
+
+*Note that it was trained with a 16k vocab, so 3.15M of the 5.52M active parameters are embedding.
+
+## Chatbench results:
+
+Multiple choice:
+| category      | n   | acc       | accnorm     | chance |
+| ------------- | --- | --------- | ----------- | ------ |
+| fact          | 25  | 48.0%     | 60.0%       | 25.0%  |
+| commonsense   | 20  | 65.0%     | 55.0%       | 25.0%  |
+| instruct      | 20  | 55.0%     | 50.0%       | 25.0%  |
+| relevance     | 20  | 5.0%      | 30.0%       | 25.0%  |
+| consistency   | 18  | 61.1%     | 72.2%       | 25.0%  |
+| arithmetic    | 15  | 46.7%     | 46.7%       | 25.0%  |
+| LS-Micro-Base | 118 | **46.6%** | **52.5%**   | 25.0%  |
+| LS-Tiny-Base  | 118 | 40.7%     | 47.5%       | 25.0%  |
+
+Example: (Greedy)
+```
+<|user|>
+Hi
+<|assistant|>
+I'm sorry, but I'm unable to assist with the problem. My current capabilities are limited to the functions provided to me. In this case, I can help you solve the problem.
+```
+
+## The base model is horrendous at chat, and is intended to be used as a base to be finetuned for specialised tasks. 
+To try LS-Micro or to train/finetune the model, follow this quickstart guide:
+1. Clone the latest version of Cudalm [here](https://github.com/Dsg2/cudalm-c) (only windows supported currently, optimised for Blackwell 2.0/CPU).
+2. Build with `build.py`/`build.py` --cpu'. Run cudalm to init all folders.
+3. Download all LS-Micro files from `LS-Micro-Base-Checkpoint\` and drop them in `<cudalm.exe>\models\`. Cudalm then automatically detects the model.
+
+Voila, you can now start experimenting with the model. Some basic stuff you can try first:
+
+ - Check the model's config, and settings used to train it. `> Model info`
+ - Run the model: `> Generate`
+
+## Finetune guide:
+To finetune the model, you first need to prepare some data.
+
+Navigate to `> Data tools...`. You can download preconfigured mixtures of data, use files on your system or nitpick your own datasets. 
+ - To use a preset corpus, navigate into `> Data tools... > Download text`. 
+ - Otherwise, find a dataset on hugginface and navigate to `> Data tools... > Browse Hugging Face`, and pick out your chosen dataset from there. Note that datasets with gated access requires your `HF_TOKEN` to be set.
+
+This guide will use the dataset [MiniCPM5-2B-distill](https://huggingface.co/datasets/Dsg2/MiniCPM5-2B-distill) as an example. For your own text or other datasets, simply select the folder while tokenizing.
+
+Navigate to `> Data tools... > Browse Hugging Face`, and search for "MiniCPM5-2B-distill". Cudalm automatically detects the format, so all you need to do is download it (~250MB).
+
+Then you will need to tokenize the text so that the model can be trained on it. To do this, you will need a trained tokenizer. Luckily, you can extract an existing model's tokenizer. Use `> Data tools... > Tokenizer from model` and pull out LS-Micro-Base's tokenizer.
+
+Now you can tokenize your data. Go to `> Data tools... > Build corpus` and set `mode` to "new" and give it a corpus name. For your own datasets or text, navigate to the folder the text files are in. Otherwise, select `<cudalm.exe>\data\hf_MiniCPM5-2B-distill`, then use the extracted tokenizer and select `Run`. It should take ~10s and produce `<cudalm.exe>\data\<corpus name>`
+
+You're now ready to start fine tuning!
+
+Go to `> Train` and select "LS-Micro-Base" as the model, and the tokenized corpus as the data directory (both auto-detected). Select Run to start training the model on the selected corpus. Progress prints and periodic saves will occur every few hundred iters.