Dsg2/LS2-266M-A14M

LS-266M-A14M

LS2-266M-A14M is the first model using the new LaoShu2 architecture and training pipeline. The key differences are:

• Top-2 routing instead of top-1 experts.
• Order=2 Engram table.
• Pretrained first on 2B fineweb-edu, then 3B of code, general, chat, tool, agentic and instruct.
• Trained fully QAT (4-bit) with no float master weights.

Overall, this allows squeezing much more knowledge into a sparse model while ensuring training is efficient.

task (acc_norm)	LS-220M-A25M	LS2-266M-A14M	chance
arc_easy	35.8%	36.2%	25.1%
hellaswag	31.8%	30.0%	25.0%
piqa	60.8%	60.0%	50.0%
lambada	19.2%	19.6%	0%

Trained entirely on a 1660 super over ~124 hours.

QAT training:

Without the need for Fp32 master weights, the model trains at ~4.3bpw instead of 16/32. This frees up Vram for a larger batch number.

Q4/Q3/Q2 was tested, but memory savings of Q2 was marginal compared to Q4 and resulted in large losses. Q3 could be used for tighter memory constraints for slightly more loss.

Furthermore, int8 AdamW was used instead of F16.

Total parameters: 266M

Model parameters: 137M

N-gram parameters: 129M

Active parameters: 14M

context length: 1024

Examples:

Chat: (temp 0.0)

you> why is the sky blue?
bot> The sky blue is blue because it is blue.
      [10 tok, 88.5 tok/s, ctx 26/4096]

you> hi
bot> Hello! How are you?
      [6 tok, 1000.0 tok/s, ctx 43/4096]

Code: (temp 0.2 used to stop loops)

you> write a python function that reverses a string
bot> Here is the Python function to reverse a given string:
```python
def reverse_string(s):
    return s[::-1] + 'a' * 2
end
```
This function takes two arguments, s and s. It returns the reversed version of the input string.
      [61 tok, 1070.2 tok/s, ctx 79/4096]

To try it yourself:

Llama.cpp is not compatible with LS2. LS2 can only be run with tinylm.

To run LS2, download the .gguf and use the command tinylm chat LS2-266M-A14M-Q4_K_M.gguf 256 0.2.

For the latest version, check here for the source code of tinylm and windows prebuilts.