Dsg2/LS3-300M-A18M-Base

This repo contains the Base (completed pretrain cudalm checkpoint) variant of LS3-300M-A18M. The FP32 checkpoint and optimiser is provided, and can be used by tinylm chat.

• LS3-300M-A18M.bin 627 MB: Model weights
• LS3-300M-A18M.bin.opt 627 MB: Adam state
• LS3-300M-A18M.bin.opt.mu 627 MB: Muon momentum
• LS3-300M-A18M.bin.opt.ng 456 MB: n-gram table optimizer state
• LS3-300M-A18M.bin.opt.t1 5.77 MB: optimizer sidecar state

LS3-300M-A18M

LS family's ultrasparse flagship model with a narrow but deep architecture. With ~300M parameters total and only 6% of them active per token (top-3 experts from 32 total), LS3-300M-A18M is the most capable model in the LS family.

(acc norm except LAMBADA)	HellaSwag	ARC-E	ARC-C	PIQA	LAMBADA	ArithMark	*II	**Intelligence Density
LS2.5-108M-A17M-Base	28.24	36.66	23.21	57.40	17.89	37.00	9.89	0.231
LS-Micro-Base (11.49M)	27.41	34.72	21.93	54.84	11.06	35.50	7.24	0.909
LS3-Micro-Base (9.97M)	27.31	34.76	23.29	57.34	12.87	38.90	9.64	1.290
LS3-Micro-Dense-Base (9.89M)	28.53	39.27	20.31	56.58	16.15	47.50	11.99	1.212
LS3-300M-A18M-Base (298M A28M)	30.31	44.28	23.98	60.39	17.56	38.80	14.53	0.198

*II derived from the Open SLM Leaderboard formula (Results reproducable via Cudalm's built in bench tool) **Indelligence density calculated using the formula II / sqrt(Active params * Total params).

LS3-300M-A18M achieved a much higher II when using a long stage B decay, rather than a short decay seen in LS3-Micro-Dense. The long stage B decay increased II from 7.46 to 14.53, and improved the margin from -8.21 to -4.54.

---

Model Specs	#
Shape	Sparse MoE + ngrams
Hidden Dim	160
Layers	24
Attention heads	5
KV heads	1
FFN	440
Experts	32
Top-k routing	3
Context length	2048
Vocab	4096 (digits singular)
Optimiser	Muon 2D, AdamW else
Backbone params	164,462,240
Attension gate	614,400
Ngram	133,140,480 (Q4)
Total params	298,217,120
Active params	18,102,048

---

Training summary

	Stage A (stable)	Stage B (cooldown)
Steps	365,004	97,405 (total 462,409)
Tokens	14.951B	3.990B (total 18.94B)
Learning rate	1.05e-3 held, 1,000-step warmup	1.05e-3 falling to 2.1e-5
Time, speed	104 h, 44.5 tok/s	25 h, 44.5k tok/s

Datasets used:

Stage A (warmup and stable):

• HuggingFaceFW/fineweb-edu (8 shards, 7.2B tokens), ODC-By-1.0
• openbmb/Ultra-FineWeb, data/ultrafineweb_en (40 parts, 2.4B), ODC-By-1.0
• openbmb/Ultra-FineWeb-L3, qa (11 shards, 4B), Apache-2.0
• openbmb/Ultra-FineWeb-L3, multi_style (9 shards, 2.6B), Apache-2.0
• HuggingFaceTB/cosmopedia, data/web_samples_v2 (30 shards, 0.6B), Apache-2.0
• openbmb/Ultra-FineWeb-L3, multi_style (30 shards sifted from older corpus, 0.611B), ODC-By-1.0
• nvidia/OpenMathInstruct-2, refs/convert/parquet, default/train (8 shards, 1.5B), CC-BY-4.0
• HuggingFaceTB/finemath, finemath-4plus (40 shards, 1B), ODC-By-1.0
• ajibawa-2023/Education-College-Students, refs/convert/parquet (2 shards, 0.3B), Apache-2.0
• ajibawa-2023/Education-High-School-Students, refs/convert/parquet (2 shards, 0.3B), Apache-2.0
• ajibawa-2023/Education-Young-Children, refs/convert/parquet (2 shards, 0.2B), Apache-2.0
• HuggingFaceTB/cosmopedia: stanford, openstax, khanacademy (16 shards, 0.2B), Apache-2.0
• arithb: (self generated grade school level arithmetics, 0.9B), Self

Stage B (decay) on train splits:

• Rowan/hellaswag, default, train
• ARC-Easy, ARC-Challenge allenai/ai2_arc, ARC-Easy and ARC-Challenge, train
• github.com/ybisk/ybisk.github.io, master/piqa/data/train.jsonl and train-labels.lst
• allenai/openbookqa, additional, train
• allenai/sciq, default, train
• tau/commonsense_qa, default, train
• allenai/winogrande, winogrande_xl, train