code/Taotern_LLM_Experiments/docs/showcase/GSM_Scaling_Law_Notes.md

GSM Scaling-Law Notes

Date: 2026-05-12

These notes summarize the current empirical scaling lessons from the Taotern GSM/SSM-to-LLM experiments. They are intended as a short companion to GSM_RnD_Showcase_Report.md.

Current Empirical Rule

The most repeatable finding is:

At the tested small LLM scale, pure SSM is promising but still trails attention on token-task loss, while a roughly 50/50 SSM-attention hybrid gives the best quality.

This is not yet a universal scaling law. It is a project-specific empirical law from the current TaoNet/GSM experiments.

Attention-to-SSM Ratio

Four-layer experiments tested these rough ratios:

SSM ratio	Example	Finding
0%	pure attention TaoNet	strong baseline
25%	single SSM middle	useful efficiency/quality compromise
25% late	single SSM late	weak; not recommended
50%	alternating SSM/attention	best quality
100%	pure SSM TaoNet	project target, but still behind attention on token loss

Current rule:

For the current DPLR GSM block, use two SSM blocks in four layers for best quality; do not place the only SSM block late.

SSM Capacity Scaling

The best early pure SSM point was:

ssm_hidden_dim = 16
ssm_mixer_dim  = 128

Later high-scale sweeps moved toward:

ssm_hidden_dim = 16 or 32
ssm_mixer_dim  = 256
lanes          = 2 split lanes

Current rule:

Increase SSM mixer capacity only while token quality improves enough to justify slower DPLR compute.

Locality Rule

Removing local shift sharply reduced real-token quality.

Current rule:

Pure SSM token models require a cheap local path in addition to long-range state-space mixing.

Lane Rule

Full multi-lane SSM improves quality but costs too much throughput. Split lanes recover speed and memory.

Current rule:

Prefer split/grouped lane diversity over duplicated full-width SSM lanes.

Hardware Rule

Several algebraically exact DPLR rewrites reduced apparent operation count but ran slower on GPU.

Current rule:

For GSM acceleration, benchmark forward, backward, memory, and token quality; do not trust symbolic operation count alone.

Token-to-Parameter Rule

For a 200M-parameter base model:

Training token budget	Use
300M tokens	candidate filtering
1B tokens	stronger selection and trend check
4B-5B tokens	Chinchilla-style serious base pretraining range
beyond 5B tokens	better if compute allows, especially with high-quality data

Current rule:

Use 300M-1B tokens to select the architecture, then 4B-5B tokens for the serious 200M base model.

Chatbot Readiness Rule

A base pretrained model is not automatically a chatbot.

Current rule:

For chatbot behavior, add SFT, evaluation prompts, safety/instruction data, and possibly distillation or preference optimization after base pretraining.

Best Current Scaling Hypothesis

The next serious hypothesis to test is:

At 200M parameters, pure GSM may improve with more training tokens, but the hybrid GSM-attention model is the most likely near-term showcase winner.

This preserves the project direction:

• pure GSM remains the primary research target,
• hybrid GSM-attention remains the practical quality fallback,
• attention TaoNet remains the untouched baseline.