docs/formula_coefficient_methodology.md

Formula and Coefficient Methodology

Date created: 2026-05-30

This document explains how we use the dropout-pressure formula, how its coefficients are derived from experiments, and how the resulting formula is tested as a streaming dropout schedule.

Purpose

The goal is not to find one universal dropout value.

The goal is to learn a rule that maps the current training pressure to a useful dropout rate:

model size + available unique data + cumulative sampled training
        |
        v
recommended dropout

In a streaming setting, available data and cumulative training change over time, so the formula produces a sequence of dropout values rather than one fixed dropout.

Formula Family

The current leading formula is the interaction pressure law:

p_t = clamp(p_min, p_max,
            A * log10(P / U_t)
          + B * log10(C_t / U_t)
          + D * log10(P / U_t) * log10(C_t / U_t)
          + C0)

The first-order ablation is:

p_t = clamp(p_min, p_max,
            A * log10(P / U_t)
          + B * log10(C_t / U_t)
          + C0)

Where:

Symbol	Meaning
P	model parameter count
U_t	unique tokens available at stage t
C_t	cumulative sampled training tokens consumed by stage t
p_t	active dropout rate at stage t
A	coefficient for model/data pressure
B	coefficient for sampled-token pressure
D	interaction coefficient
C0	regime baseline offset

The two pressure variables are:

x_t = log10(P / U_t)
y_t = log10(C_t / U_t)

So the interaction formula can be written compactly as:

p_t = clamp(p_min, p_max, A*x_t + B*y_t + D*x_t*y_t + C0)

Clamp

clamp keeps the formula output inside a valid dropout range:

clamp(p_min, p_max, z) = max(p_min, min(p_max, z))

In the current schedule-generation experiments:

p_min = 0.02
p_max = 0.65

Static sweeps may still test dropout=0.0 as an ablation. The clamp is mainly used when turning fitted coefficients into a deployment/training schedule, so a bad extrapolation cannot produce negative dropout or an unusably large dropout.

Regime-Specific Coefficients

The coefficients are not assumed to be universal constants.

A regime is defined by:

architecture family
+ tokenizer
+ corpus family
+ optimizer and learning-rate protocol
+ dropout placement and semantics
+ streaming protocol
+ evaluation distribution

Inside one regime, the formula inputs P, U_t, and C_t should explain how dropout changes. If the regime changes, the coefficients may need to be refitted.

Current TinyStories-regime coefficients:

A  = -0.089261
B  = -0.129754
D  =  0.255069
C0 =  0.081525

Absolute TinyStories-regime formula:

p_t = clamp(p_min, p_max,
          -0.089261 * log10(P / U_t)
        - 0.129754 * log10(C_t / U_t)
        + 0.255069 * log10(P / U_t) * log10(C_t / U_t)
        + 0.081525)

The research claim we are testing is:

the pressure-law structure transfers across regimes;
the coefficient values are calibrated per regime.

How Coefficients Are Derived

Coefficient fitting starts from static dropout sweeps.

A calibration cell is one fixed experimental setting whose best dropout we want the formula to explain.

One cell is:

one model architecture and parameter count P
+ one unique-token prefix U
+ one sampled-token training budget C
+ one validation setup
+ a sweep over static dropout rates

Example cell:

model: L12_H8_D320
parameters P: 17,367,040
unique tokens U: 1,000,000
sampled training tokens C: 10,240,000
dropout rates tested: 0.00, 0.04, 0.08, 0.12, 0.18, 0.26

The dropout sweep for that cell might look like:

0.00 -> validation loss 3.1074
0.04 -> validation loss 2.9188
0.08 -> validation loss 2.8721
0.12 -> validation loss 2.8454  best
0.18 -> validation loss 2.8623
0.26 -> validation loss 2.9006

That cell contributes one supervised training row for the coefficient fit:

input:  P, U, C
target: p_star ~= 0.12

So "cell" means one row in the coefficient-fitting dataset.

For each calibration cell, we choose:

model P
unique-token prefix U
sampled training-token budget C
dropout grid

Then we train/evaluate the model at several fixed dropout rates and record the validation loss curve:

dropout -> validation loss

Example shape:

0.00 -> high loss
0.04 -> lower loss
0.08 -> lower loss
0.12 -> best loss
0.18 -> higher loss
0.26 -> higher loss

This gives one target value for the cell:

p_star = observed useful static dropout for (P, U, C)

Target Extraction

The fitting script supports two target choices:

Target	Meaning
grid best	dropout rate with the lowest observed validation loss
quadratic optimum	local parabolic minimum around the best grid point

The quadratic target is preferred when the curve is bracketed:

left dropout has higher loss
middle dropout is best
right dropout has higher loss

If the best dropout is at the edge of the tested grid, the optimum is marked as a boundary optimum. Boundary cells are useful but weaker evidence because the true optimum may lie outside the tested rates.

Feature Construction

For each cell, compute:

x = log10(P / U)
y = log10(C / U)
xy = x * y

Then fit:

p_star ~= A*x + B*y + D*xy + C0

In plain language, the fitting step asks:

What values of A, B, D, and C0 make the formula's predicted dropout
as close as possible to the observed best dropout values across all cells?

Suppose we have many cells:

cell 1 observed best dropout: 0.12
cell 2 observed best dropout: 0.18
cell 3 observed best dropout: 0.08
...

For each cell, the formula predicts a dropout:

predicted_p_i = A*x_i + B*y_i + D*x_i*y_i + C0

The error for that cell is:

error_i = predicted_p_i - observed_p_star_i

Ordinary least squares chooses the coefficients that minimize the sum of squared errors:

minimize sum_i error_i^2

We use squared error because large misses should matter more than tiny misses. This is the standard linear-regression solution.

Why Some Cells Get Lower Weight

Not every observed p_star is equally reliable. Some dropout sweeps identify a clear optimum; others only give a rough hint. The weighted fit keeps all cells, but lets cleaner cells influence the coefficients more than uncertain cells.

Weighted least squares minimizes:

minimize sum_i w_i * error_i^2

Where:

w_i = confidence weight for cell i

If a cell has weight 1.0, it has full influence. If it has weight 0.3, it still contributes, but only weakly.

The current fitting script lowers a cell's weight in these cases:

Condition	Meaning	Why it is less reliable
boundary optimum	the best tested dropout is the smallest or largest dropout in the grid	the real optimum may be outside the tested range
not bracketed	the best point does not have worse points on both sides	we cannot confidently fit a local parabola
very flat curve	many dropout rates have almost the same validation loss	the exact best dropout is weakly identified
noisy best loss	validation loss has high variance across seeds/eval batches	the selected best point may move with more samples

Example boundary optimum:

dropout: 0.00  0.04  0.08  0.12
loss:    3.20  3.05  2.96  2.90

The best tested value is 0.12, but the curve is still improving at the edge. The true optimum might be 0.18 or 0.26, so this cell should not dominate the fit.

Example bracketed optimum:

dropout: 0.04  0.08  0.12  0.18
loss:    2.92  2.87  2.85  2.86

The best tested value is 0.12, and both neighboring sides are worse. This is a cleaner target because the bottom of the curve is visible.

Example flat curve:

dropout: 0.04  0.08  0.12  0.18
loss:    2.851 2.849 2.850 2.852

The grid best might be 0.08, but 0.04, 0.12, and 0.18 are almost tied. The correct conclusion is a plateau, not a sharply known optimum.

So the weighting is not changing the observed results. It only tells the coefficient fit how much confidence to place in each row.

The main implementation is:

scripts/fit_dropout_coefficients.py

Its main outputs are:

Output	Purpose
coefficients.json	fitted coefficients and fit metrics
calibration_cells.csv	per-cell target, prediction, residual, pressure variables
fit_diagnostics.md	human-readable report
next_dropout_suggestions.csv	suggested extra dropout points if a curve needs refinement

Solving For A, B, D, And C0

After target extraction, every cell gives one equation:

p_star_i ~= A*x_i + B*y_i + D*x_i*y_i + C0

Where:

x_i = log10(P_i / U_i)
y_i = log10(C_i / U_i)

For n cells, stack those equations into a matrix:

X =
[
  x_1  y_1  x_1*y_1  1
  x_2  y_2  x_2*y_2  1
  x_3  y_3  x_3*y_3  1
  ...
  x_n  y_n  x_n*y_n  1
]

theta =
[
  A
  B
  D
  C0
]

p =
[
  p_star_1
  p_star_2
  p_star_3
  ...
  p_star_n
]

The coefficient fit solves:

X * theta ~= p

In least-squares form:

theta_hat = argmin_theta ||X * theta - p||^2

With heuristic weights, the objective becomes:

theta_hat = argmin_theta sum_i w_i * (A*x_i + B*y_i + D*x_i*y_i + C0 - p_star_i)^2

Cells with clean bracketed dropout optima get higher weight. Boundary, flat, or noisy cells get lower weight.

The implementation uses NumPy least squares:

coef, *_ = np.linalg.lstsq(X_weighted, p_weighted, rcond=None)

For the first-order ABC ablation, the matrix drops the interaction column:

X =
[
  x_1  y_1  1
  x_2  y_2  1
  ...
]

theta =
[
  A
  B
  C0
]

Then the fit solves:

p_star_i ~= A*x_i + B*y_i + C0

What The Coefficients Mean

The coefficients are not magic constants; they are slopes in the pressure space.

For the interaction formula:

p = A*x + B*y + D*x*y + C0

A controls how dropout changes as model size grows relative to available unique tokens.

B controls how dropout changes as cumulative sampled training grows relative to available unique tokens.

D controls whether those two effects amplify or damp each other.

The interaction term was added because our TinyStories results showed that the effect of repeated sampled training is not independent of model/data pressure. The simple ABC formula underfit those changes.

How We Validate Coefficients

After fitting coefficients, we do not immediately launch new training.

First we backtest offline against existing saved results.

Within-Regime Fit

Fit coefficients using cells from one regime and measure:

predicted dropout - observed target dropout

Report:

MAE
RMSE
bias
weighted MAE
weighted RMSE

Held-Out Validation

When enough cells exist, run grouped validation:

Validation	Test
leave-model-out	can the formula predict a held-out model size?
leave-prefix-out	can it predict a held-out unique-token prefix?
leave-source-out	can it predict cells from another run source?

This tests whether the formula is learning a pressure relationship rather than memorizing one grid.

Cross-Regime Backtest

For each saved regime:

1. fit coefficients inside that regime;
2. compare base_abc and interaction;
3. test whether coefficients from one regime transfer numerically to another;
4. decide whether the structure transfers but coefficients differ.

This is the next required step before new MPS training.

How The Formula Becomes A Decay Schedule

Static fitting gives a useful dropout estimate for one (P, U, C) point.

Streaming creates a sequence of points:

stage 0: P fixed, U_0 small, C_0 small
stage 1: P fixed, U_1 larger, C_1 larger
stage 2: P fixed, U_2 larger, C_2 larger
stage 3: P fixed, U_3 larger, C_3 larger

At each stage:

raw_p_t = A*x_t + B*y_t + D*x_t*y_t + C0
p_t = clamp(p_min, p_max, raw_p_t)

The generated values become stage anchors:

U_0=p_0, U_1=p_1, U_2=p_2, U_3=p_3

The helper script is:

scripts/make_streaming_anchors.py

For the latest L12 TinyStories streaming setup, the interaction schedule was:

500k -> 0.184
1M   -> 0.141
2M   -> 0.084
4M   -> 0.045

That schedule is then tested against static dropout baselines using locked_stream.

How We Test The Decay Hypothesis

The decay hypothesis is not proven by fitting coefficients.

Fitting coefficients proves only this:

the formula estimates useful static dropout for a given pressure point

The streaming experiment tests this stronger claim:

using those estimated dropout values as a schedule helps during streaming

For streaming validation, compare:

formula-derived decay schedules
static dropout baselines
schedule-shape controls

Measure:

Metric	Why it matters
final validation loss	whether the model uses the largest stream effectively
mean trajectory validation loss	whether the full stream path is good
stage-wise validation loss	where each schedule wins or loses
train-validation gap	whether dropout is controlling overfit
paired seed deltas	whether wins survive initialization noise

Current narrowed streaming comparison:

interaction decay
baseabc decay
smooth_low decay
static_dropout_0.08
static_dropout_0.12
static_dropout_0.18

Current Multi-Seed Streaming Result

Latest TinyStories L12 3-seed final-loss result:

Condition	Mean final 4M validation loss	Std
interaction decay	2.5392	0.0020
smooth_low decay	2.5405	0.0018
baseabc decay	2.5418	0.0019
static 0.08	2.5511	0.0112
static 0.12	2.5541	0.0041
static 0.18	2.5690	0.0069

Interpretation:

interaction decay has the best 3-seed mean final loss;
5 seeds would make the TinyStories result more paper-grade.

The final proof path should be streaming multi-seed validation reports per regime. Static coefficient backtests are supporting gates, not final evidence.

Pass And Fail Conditions

Coefficient Pass

The coefficient formula passes a regime if:

within-regime MAE is low
held-out model/prefix error is low
residuals do not show obvious systematic bias
the fitted coefficients have a defensible interpretation

For our current scale, a useful target is:

dropout MAE below about 0.05

Streaming Strong Pass

The schedule strongly passes if:

mean final validation loss beats the best static baseline across seeds
and most paired seeds favor the decay schedule

Streaming Weak Pass

The schedule weakly passes if:

it matches the best hand-picked static dropout
while avoiding clearly bad static dropout choices across stages

This is still scientifically useful because it means the formula can choose a competitive schedule without manually searching a fixed dropout for every stream size.

Streaming Fail

The schedule fails if:

it loses to a simple static baseline in most seeds
or improves early stages by sacrificing final-stage loss

If that happens, do not launch a larger sweep immediately. First fit a static-to-streaming correction offline and backtest it on saved results.

Immediate Next Work

The active proof artifact is:

docs/tinystories_streaming_report.md
runs/streaming_tinystories_multiseed_validation_l12/combined_5seed_summary/

TinyStories has already been regenerated at n=5. The next paper-grade streaming validation target is WikiText-103, after reconciling the TinyStories and OpenWebText10K reports.

For any later regime, repeat the same pattern: first use static backtests to choose coefficients, then create a streaming multi-seed validation report as the end proof.