# GSM Scaling-Law Notes

Date: 2026-05-12

These notes summarize the current empirical scaling lessons from the Taotern GSM/SSM-to-LLM experiments. They are intended as a short companion to `GSM_RnD_Showcase_Report.md`.

## Current Empirical Rule

The most repeatable finding is:

```text
At the tested small LLM scale, pure SSM is promising but still trails attention on token-task loss, while a roughly 50/50 SSM-attention hybrid gives the best quality.
```

This is not yet a universal scaling law. It is a project-specific empirical law from the current TaoNet/GSM experiments.

## Attention-to-SSM Ratio

Four-layer experiments tested these rough ratios:

| SSM ratio | Example | Finding |
|---:|---|---|
| 0% | pure attention TaoNet | strong baseline |
| 25% | single SSM middle | useful efficiency/quality compromise |
| 25% late | single SSM late | weak; not recommended |
| 50% | alternating SSM/attention | best quality |
| 100% | pure SSM TaoNet | project target, but still behind attention on token loss |

Current rule:

```text
For the current DPLR GSM block, use two SSM blocks in four layers for best quality; do not place the only SSM block late.
```

## SSM Capacity Scaling

The best early pure SSM point was:

```text
ssm_hidden_dim = 16
ssm_mixer_dim  = 128
```

Later high-scale sweeps moved toward:

```text
ssm_hidden_dim = 16 or 32
ssm_mixer_dim  = 256
lanes          = 2 split lanes
```

Current rule:

```text
Increase SSM mixer capacity only while token quality improves enough to justify slower DPLR compute.
```

## Locality Rule

Removing local shift sharply reduced real-token quality.

Current rule:

```text
Pure SSM token models require a cheap local path in addition to long-range state-space mixing.
```

## Lane Rule

Full multi-lane SSM improves quality but costs too much throughput. Split lanes recover speed and memory.

Current rule:

```text
Prefer split/grouped lane diversity over duplicated full-width SSM lanes.
```

## Hardware Rule

Several algebraically exact DPLR rewrites reduced apparent operation count but ran slower on GPU.

Current rule:

```text
For GSM acceleration, benchmark forward, backward, memory, and token quality; do not trust symbolic operation count alone.
```

## Token-to-Parameter Rule

For a 200M-parameter base model:

| Training token budget | Use |
|---:|---|
| 300M tokens | candidate filtering |
| 1B tokens | stronger selection and trend check |
| 4B-5B tokens | Chinchilla-style serious base pretraining range |
| beyond 5B tokens | better if compute allows, especially with high-quality data |

Current rule:

```text
Use 300M-1B tokens to select the architecture, then 4B-5B tokens for the serious 200M base model.
```

## Chatbot Readiness Rule

A base pretrained model is not automatically a chatbot.

Current rule:

```text
For chatbot behavior, add SFT, evaluation prompts, safety/instruction data, and possibly distillation or preference optimization after base pretraining.
```

## Best Current Scaling Hypothesis

The next serious hypothesis to test is:

```text
At 200M parameters, pure GSM may improve with more training tokens, but the hybrid GSM-attention model is the most likely near-term showcase winner.
```

This preserves the project direction:

- pure GSM remains the primary research target,
- hybrid GSM-attention remains the practical quality fallback,
- attention TaoNet remains the untouched baseline.