# Formula and Coefficient Methodology Date created: 2026-05-30 This document explains how we use the dropout-pressure formula, how its coefficients are derived from experiments, and how the resulting formula is tested as a streaming dropout schedule. ## Purpose The goal is not to find one universal dropout value. The goal is to learn a rule that maps the current training pressure to a useful dropout rate: ```text model size + available unique data + cumulative sampled training | v recommended dropout ``` In a streaming setting, available data and cumulative training change over time, so the formula produces a sequence of dropout values rather than one fixed dropout. ## Formula Family The current leading formula is the interaction pressure law: ```text p_t = clamp(p_min, p_max, A * log10(P / U_t) + B * log10(C_t / U_t) + D * log10(P / U_t) * log10(C_t / U_t) + C0) ``` The first-order ablation is: ```text p_t = clamp(p_min, p_max, A * log10(P / U_t) + B * log10(C_t / U_t) + C0) ``` Where: | Symbol | Meaning | |---|---| | `P` | model parameter count | | `U_t` | unique tokens available at stage `t` | | `C_t` | cumulative sampled training tokens consumed by stage `t` | | `p_t` | active dropout rate at stage `t` | | `A` | coefficient for model/data pressure | | `B` | coefficient for sampled-token pressure | | `D` | interaction coefficient | | `C0` | regime baseline offset | The two pressure variables are: ```text x_t = log10(P / U_t) y_t = log10(C_t / U_t) ``` So the interaction formula can be written compactly as: ```text p_t = clamp(p_min, p_max, A*x_t + B*y_t + D*x_t*y_t + C0) ``` ## Clamp `clamp` keeps the formula output inside a valid dropout range: ```text clamp(p_min, p_max, z) = max(p_min, min(p_max, z)) ``` In the current schedule-generation experiments: ```text p_min = 0.02 p_max = 0.65 ``` Static sweeps may still test `dropout=0.0` as an ablation. The clamp is mainly used when turning fitted coefficients into a deployment/training schedule, so a bad extrapolation cannot produce negative dropout or an unusably large dropout. ## Regime-Specific Coefficients The coefficients are not assumed to be universal constants. A regime is defined by: ```text architecture family + tokenizer + corpus family + optimizer and learning-rate protocol + dropout placement and semantics + streaming protocol + evaluation distribution ``` Inside one regime, the formula inputs `P`, `U_t`, and `C_t` should explain how dropout changes. If the regime changes, the coefficients may need to be refitted. Current TinyStories-regime coefficients: ```text A = -0.089261 B = -0.129754 D = 0.255069 C0 = 0.081525 ``` Absolute TinyStories-regime formula: ```text p_t = clamp(p_min, p_max, -0.089261 * log10(P / U_t) - 0.129754 * log10(C_t / U_t) + 0.255069 * log10(P / U_t) * log10(C_t / U_t) + 0.081525) ``` The research claim we are testing is: ```text the pressure-law structure transfers across regimes; the coefficient values are calibrated per regime. ``` ## How Coefficients Are Derived Coefficient fitting starts from static dropout sweeps. A calibration cell is one fixed experimental setting whose best dropout we want the formula to explain. One cell is: ```text one model architecture and parameter count P + one unique-token prefix U + one sampled-token training budget C + one validation setup + a sweep over static dropout rates ``` Example cell: ```text model: L12_H8_D320 parameters P: 17,367,040 unique tokens U: 1,000,000 sampled training tokens C: 10,240,000 dropout rates tested: 0.00, 0.04, 0.08, 0.12, 0.18, 0.26 ``` The dropout sweep for that cell might look like: ```text 0.00 -> validation loss 3.1074 0.04 -> validation loss 2.9188 0.08 -> validation loss 2.8721 0.12 -> validation loss 2.8454 best 0.18 -> validation loss 2.8623 0.26 -> validation loss 2.9006 ``` That cell contributes one supervised training row for the coefficient fit: ```text input: P, U, C target: p_star ~= 0.12 ``` So "cell" means one row in the coefficient-fitting dataset. For each calibration cell, we choose: ```text model P unique-token prefix U sampled training-token budget C dropout grid ``` Then we train/evaluate the model at several fixed dropout rates and record the validation loss curve: ```text dropout -> validation loss ``` Example shape: ```text 0.00 -> high loss 0.04 -> lower loss 0.08 -> lower loss 0.12 -> best loss 0.18 -> higher loss 0.26 -> higher loss ``` This gives one target value for the cell: ```text p_star = observed useful static dropout for (P, U, C) ``` ### Target Extraction The fitting script supports two target choices: | Target | Meaning | |---|---| | grid best | dropout rate with the lowest observed validation loss | | quadratic optimum | local parabolic minimum around the best grid point | The quadratic target is preferred when the curve is bracketed: ```text left dropout has higher loss middle dropout is best right dropout has higher loss ``` If the best dropout is at the edge of the tested grid, the optimum is marked as a boundary optimum. Boundary cells are useful but weaker evidence because the true optimum may lie outside the tested rates. ### Feature Construction For each cell, compute: ```text x = log10(P / U) y = log10(C / U) xy = x * y ``` Then fit: ```text p_star ~= A*x + B*y + D*xy + C0 ``` In plain language, the fitting step asks: ```text What values of A, B, D, and C0 make the formula's predicted dropout as close as possible to the observed best dropout values across all cells? ``` Suppose we have many cells: ```text cell 1 observed best dropout: 0.12 cell 2 observed best dropout: 0.18 cell 3 observed best dropout: 0.08 ... ``` For each cell, the formula predicts a dropout: ```text predicted_p_i = A*x_i + B*y_i + D*x_i*y_i + C0 ``` The error for that cell is: ```text error_i = predicted_p_i - observed_p_star_i ``` Ordinary least squares chooses the coefficients that minimize the sum of squared errors: ```text minimize sum_i error_i^2 ``` We use squared error because large misses should matter more than tiny misses. This is the standard linear-regression solution. ### Why Some Cells Get Lower Weight Not every observed `p_star` is equally reliable. Some dropout sweeps identify a clear optimum; others only give a rough hint. The weighted fit keeps all cells, but lets cleaner cells influence the coefficients more than uncertain cells. Weighted least squares minimizes: ```text minimize sum_i w_i * error_i^2 ``` Where: ```text w_i = confidence weight for cell i ``` If a cell has weight `1.0`, it has full influence. If it has weight `0.3`, it still contributes, but only weakly. The current fitting script lowers a cell's weight in these cases: | Condition | Meaning | Why it is less reliable | |---|---|---| | boundary optimum | the best tested dropout is the smallest or largest dropout in the grid | the real optimum may be outside the tested range | | not bracketed | the best point does not have worse points on both sides | we cannot confidently fit a local parabola | | very flat curve | many dropout rates have almost the same validation loss | the exact best dropout is weakly identified | | noisy best loss | validation loss has high variance across seeds/eval batches | the selected best point may move with more samples | Example boundary optimum: ```text dropout: 0.00 0.04 0.08 0.12 loss: 3.20 3.05 2.96 2.90 ``` The best tested value is `0.12`, but the curve is still improving at the edge. The true optimum might be `0.18` or `0.26`, so this cell should not dominate the fit. Example bracketed optimum: ```text dropout: 0.04 0.08 0.12 0.18 loss: 2.92 2.87 2.85 2.86 ``` The best tested value is `0.12`, and both neighboring sides are worse. This is a cleaner target because the bottom of the curve is visible. Example flat curve: ```text dropout: 0.04 0.08 0.12 0.18 loss: 2.851 2.849 2.850 2.852 ``` The grid best might be `0.08`, but `0.04`, `0.12`, and `0.18` are almost tied. The correct conclusion is a plateau, not a sharply known optimum. So the weighting is not changing the observed results. It only tells the coefficient fit how much confidence to place in each row. The main implementation is: ```text scripts/fit_dropout_coefficients.py ``` Its main outputs are: | Output | Purpose | |---|---| | `coefficients.json` | fitted coefficients and fit metrics | | `calibration_cells.csv` | per-cell target, prediction, residual, pressure variables | | `fit_diagnostics.md` | human-readable report | | `next_dropout_suggestions.csv` | suggested extra dropout points if a curve needs refinement | ### Solving For A, B, D, And C0 After target extraction, every cell gives one equation: ```text p_star_i ~= A*x_i + B*y_i + D*x_i*y_i + C0 ``` Where: ```text x_i = log10(P_i / U_i) y_i = log10(C_i / U_i) ``` For `n` cells, stack those equations into a matrix: ```text X = [ x_1 y_1 x_1*y_1 1 x_2 y_2 x_2*y_2 1 x_3 y_3 x_3*y_3 1 ... x_n y_n x_n*y_n 1 ] theta = [ A B D C0 ] p = [ p_star_1 p_star_2 p_star_3 ... p_star_n ] ``` The coefficient fit solves: ```text X * theta ~= p ``` In least-squares form: ```text theta_hat = argmin_theta ||X * theta - p||^2 ``` With heuristic weights, the objective becomes: ```text theta_hat = argmin_theta sum_i w_i * (A*x_i + B*y_i + D*x_i*y_i + C0 - p_star_i)^2 ``` Cells with clean bracketed dropout optima get higher weight. Boundary, flat, or noisy cells get lower weight. The implementation uses NumPy least squares: ```text coef, *_ = np.linalg.lstsq(X_weighted, p_weighted, rcond=None) ``` For the first-order ABC ablation, the matrix drops the interaction column: ```text X = [ x_1 y_1 1 x_2 y_2 1 ... ] theta = [ A B C0 ] ``` Then the fit solves: ```text p_star_i ~= A*x_i + B*y_i + C0 ``` ## What The Coefficients Mean The coefficients are not magic constants; they are slopes in the pressure space. For the interaction formula: ```text p = A*x + B*y + D*x*y + C0 ``` `A` controls how dropout changes as model size grows relative to available unique tokens. `B` controls how dropout changes as cumulative sampled training grows relative to available unique tokens. `D` controls whether those two effects amplify or damp each other. The interaction term was added because our TinyStories results showed that the effect of repeated sampled training is not independent of model/data pressure. The simple ABC formula underfit those changes. ## How We Validate Coefficients After fitting coefficients, we do not immediately launch new training. First we backtest offline against existing saved results. ### Within-Regime Fit Fit coefficients using cells from one regime and measure: ```text predicted dropout - observed target dropout ``` Report: ```text MAE RMSE bias weighted MAE weighted RMSE ``` ### Held-Out Validation When enough cells exist, run grouped validation: | Validation | Test | |---|---| | leave-model-out | can the formula predict a held-out model size? | | leave-prefix-out | can it predict a held-out unique-token prefix? | | leave-source-out | can it predict cells from another run source? | This tests whether the formula is learning a pressure relationship rather than memorizing one grid. ### Cross-Regime Backtest For each saved regime: 1. fit coefficients inside that regime; 2. compare `base_abc` and `interaction`; 3. test whether coefficients from one regime transfer numerically to another; 4. decide whether the structure transfers but coefficients differ. This is the next required step before new MPS training. ## How The Formula Becomes A Decay Schedule Static fitting gives a useful dropout estimate for one `(P, U, C)` point. Streaming creates a sequence of points: ```text stage 0: P fixed, U_0 small, C_0 small stage 1: P fixed, U_1 larger, C_1 larger stage 2: P fixed, U_2 larger, C_2 larger stage 3: P fixed, U_3 larger, C_3 larger ``` At each stage: ```text raw_p_t = A*x_t + B*y_t + D*x_t*y_t + C0 p_t = clamp(p_min, p_max, raw_p_t) ``` The generated values become stage anchors: ```text U_0=p_0, U_1=p_1, U_2=p_2, U_3=p_3 ``` The helper script is: ```text scripts/make_streaming_anchors.py ``` For the latest L12 TinyStories streaming setup, the interaction schedule was: ```text 500k -> 0.184 1M -> 0.141 2M -> 0.084 4M -> 0.045 ``` That schedule is then tested against static dropout baselines using `locked_stream`. ## How We Test The Decay Hypothesis The decay hypothesis is not proven by fitting coefficients. Fitting coefficients proves only this: ```text the formula estimates useful static dropout for a given pressure point ``` The streaming experiment tests this stronger claim: ```text using those estimated dropout values as a schedule helps during streaming ``` For streaming validation, compare: ```text formula-derived decay schedules static dropout baselines schedule-shape controls ``` Measure: | Metric | Why it matters | |---|---| | final validation loss | whether the model uses the largest stream effectively | | mean trajectory validation loss | whether the full stream path is good | | stage-wise validation loss | where each schedule wins or loses | | train-validation gap | whether dropout is controlling overfit | | paired seed deltas | whether wins survive initialization noise | Current narrowed streaming comparison: ```text interaction decay baseabc decay smooth_low decay static_dropout_0.08 static_dropout_0.12 static_dropout_0.18 ``` ## Current Multi-Seed Streaming Result Latest TinyStories L12 3-seed final-loss result: | Condition | Mean final 4M validation loss | Std | |---|---:|---:| | `interaction` decay | 2.5392 | 0.0020 | | `smooth_low` decay | 2.5405 | 0.0018 | | `baseabc` decay | 2.5418 | 0.0019 | | static `0.08` | 2.5511 | 0.0112 | | static `0.12` | 2.5541 | 0.0041 | | static `0.18` | 2.5690 | 0.0069 | Interpretation: ```text interaction decay has the best 3-seed mean final loss; 5 seeds would make the TinyStories result more paper-grade. ``` The final proof path should be streaming multi-seed validation reports per regime. Static coefficient backtests are supporting gates, not final evidence. ## Pass And Fail Conditions ### Coefficient Pass The coefficient formula passes a regime if: ```text within-regime MAE is low held-out model/prefix error is low residuals do not show obvious systematic bias the fitted coefficients have a defensible interpretation ``` For our current scale, a useful target is: ```text dropout MAE below about 0.05 ``` ### Streaming Strong Pass The schedule strongly passes if: ```text mean final validation loss beats the best static baseline across seeds and most paired seeds favor the decay schedule ``` ### Streaming Weak Pass The schedule weakly passes if: ```text it matches the best hand-picked static dropout while avoiding clearly bad static dropout choices across stages ``` This is still scientifically useful because it means the formula can choose a competitive schedule without manually searching a fixed dropout for every stream size. ### Streaming Fail The schedule fails if: ```text it loses to a simple static baseline in most seeds or improves early stages by sacrificing final-stage loss ``` If that happens, do not launch a larger sweep immediately. First fit a static-to-streaming correction offline and backtest it on saved results. ## Immediate Next Work The active proof artifact is: ```text docs/tinystories_streaming_report.md runs/streaming_tinystories_multiseed_validation_l12/combined_5seed_summary/ ``` TinyStories has already been regenerated at `n=5`. The next paper-grade streaming validation target is WikiText-103, after reconciling the TinyStories and OpenWebText10K reports. For any later regime, repeat the same pattern: first use static backtests to choose coefficients, then create a streaming multi-seed validation report as the end proof.