File size: 3,538 Bytes
e2bfccc | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 | # GSM Scaling-Law Notes
Date: 2026-05-12
These notes summarize the current empirical scaling lessons from the Taotern GSM/SSM-to-LLM experiments. They are intended as a short companion to `GSM_RnD_Showcase_Report.md`.
## Current Empirical Rule
The most repeatable finding is:
```text
At the tested small LLM scale, pure SSM is promising but still trails attention on token-task loss, while a roughly 50/50 SSM-attention hybrid gives the best quality.
```
This is not yet a universal scaling law. It is a project-specific empirical law from the current TaoNet/GSM experiments.
## Attention-to-SSM Ratio
Four-layer experiments tested these rough ratios:
| SSM ratio | Example | Finding |
|---:|---|---|
| 0% | pure attention TaoNet | strong baseline |
| 25% | single SSM middle | useful efficiency/quality compromise |
| 25% late | single SSM late | weak; not recommended |
| 50% | alternating SSM/attention | best quality |
| 100% | pure SSM TaoNet | project target, but still behind attention on token loss |
Current rule:
```text
For the current DPLR GSM block, use two SSM blocks in four layers for best quality; do not place the only SSM block late.
```
## SSM Capacity Scaling
The best early pure SSM point was:
```text
ssm_hidden_dim = 16
ssm_mixer_dim = 128
```
Later high-scale sweeps moved toward:
```text
ssm_hidden_dim = 16 or 32
ssm_mixer_dim = 256
lanes = 2 split lanes
```
Current rule:
```text
Increase SSM mixer capacity only while token quality improves enough to justify slower DPLR compute.
```
## Locality Rule
Removing local shift sharply reduced real-token quality.
Current rule:
```text
Pure SSM token models require a cheap local path in addition to long-range state-space mixing.
```
## Lane Rule
Full multi-lane SSM improves quality but costs too much throughput. Split lanes recover speed and memory.
Current rule:
```text
Prefer split/grouped lane diversity over duplicated full-width SSM lanes.
```
## Hardware Rule
Several algebraically exact DPLR rewrites reduced apparent operation count but ran slower on GPU.
Current rule:
```text
For GSM acceleration, benchmark forward, backward, memory, and token quality; do not trust symbolic operation count alone.
```
## Token-to-Parameter Rule
For a 200M-parameter base model:
| Training token budget | Use |
|---:|---|
| 300M tokens | candidate filtering |
| 1B tokens | stronger selection and trend check |
| 4B-5B tokens | Chinchilla-style serious base pretraining range |
| beyond 5B tokens | better if compute allows, especially with high-quality data |
Current rule:
```text
Use 300M-1B tokens to select the architecture, then 4B-5B tokens for the serious 200M base model.
```
## Chatbot Readiness Rule
A base pretrained model is not automatically a chatbot.
Current rule:
```text
For chatbot behavior, add SFT, evaluation prompts, safety/instruction data, and possibly distillation or preference optimization after base pretraining.
```
## Best Current Scaling Hypothesis
The next serious hypothesis to test is:
```text
At 200M parameters, pure GSM may improve with more training tokens, but the hybrid GSM-attention model is the most likely near-term showcase winner.
```
This preserves the project direction:
- pure GSM remains the primary research target,
- hybrid GSM-attention remains the practical quality fallback,
- attention TaoNet remains the untouched baseline.
|