C5 width experiment: final model weights

Final checkpoints (final_model.pt) of the C5 RoPE 8-layer 8-head transformers (FF_DIM=1024) trained with d_model 128 and 512, baseline and energy-balance regularized (lambda=0.5, final layer), seeds 1–30, 200k-step budget.

file contents
d128_baseline.zip width_results/d_model_128/baseline/training_archives//final_model.pt (30 runs)
d128_regularized.zip width_results/d_model_128/regularized/training_archives//final_model.pt (30 runs)
d512_baseline.zip width_results/d_model_512/baseline/training_archives//final_model.pt (30 runs)
d512_regularized.zip width_results/d_model_512/regularized/training_archives//final_model.pt (30 runs)
d128_baseline_ext400k.zip width_results_ext400k/d_model_128/baseline/training_archives//final_model.pt (seeds 1, 30 continued to stable hit within 400k)

d128 baseline seeds 1 and 30 in d128_baseline.zip are the 200k-budget (censored) endpoints. SHA-256 checksums are in SHA256SUMS.

Preliminary endpoint analysis (run by the uploader)

The paper's endpoint analyses were applied unchanged to all 120 final models in this repository (200k-budget endpoints; d128 baseline seeds 1 and 30 are the censored 200k models).

  • Energy allocation: endpoint harmonic energy fractions at the final residual (4096-sequence representation probe, positionwise fractions averaged), N^E = 1 / sum p_m^2, two-mode = N^E >= 1.5.
  • Interventions (regularized runs in the table below; baseline runs in the next table): final-MLP source-mode removal and semantic rotation, forward (dominant pre-MLP mode as source) and reverse, with four norm-matched random controls, using the Figure 4 probes and settings. Directionality = forward minus reverse specific effect.
  • Fast/Slow: stable-hit step < 30k / >= 30k (the paper's fixed boundary). The d256 column is the paper's cohort, recomputed with the same summary code.
d128 d256 (paper) d512
baseline two-mode runs 1/30 10/30 26/30
baseline stable-hit: two-mode / single-mode 18.8k / 43.2-146.6k (+2 runs >200k) 14.6-19.6k / 40.2-54.0k 14.6-21.4k / 30.8-44.8k
regularized two-mode runs (min N^E) 30/30 (1.935) 30/30 (1.968) 30/30 (1.986)
regularized Fast / Slow 9 / 21 25 / 5 30 / 0
Fast / Slow stable-hit range 13.0-21.2k / 52.6-106.6k 11.0-19.2k / 36.4-48.8k 11.4-22.6k / -
removal directionality: Fast median [max] / Slow median [min] -0.056 [0.157] / 0.482 [0.406] -0.075 [0.069] / 0.745 [0.619] -0.029 [0.065] / -
rotation directionality: Fast median [max] / Slow median [min] -0.014 [0.060] / 0.696 [0.538] -0.023 [0.030] / 0.768 [0.702] -0.007 [0.028] / -

At d128, all 21 Slow regularized runs show strong positive removal and rotation directionality, with no overlap with the 9 Fast runs; their stable-hit steps are separated by an empty interval (21.2-52.6k) that contains the 30k boundary. At d512 there are no Slow runs, and directionality is near zero in all 30 runs. These numbers are provided for cross-checking.

Baseline vs regularized interventions

The same final-MLP removal and rotation assays were also applied to all 60 baseline models (forward = dominant pre-MLP mode as source; all 60 assayable). Values are median [min, max] of forward minus reverse; "both > 0" counts runs with positive removal and rotation directionality. The d256 baseline row is the rebuttal analysis of the 10 two-mode baseline runs (analyses/baseline_directionality.py); single-mode d256 baselines were not assayed.

group d128 d256 d512
baseline two-mode: removal n=1: -0.091 n=10: -0.113 [-0.404, 0.042] n=26: 0.000 [-0.132, 0.084]
baseline two-mode: rotation -0.020 -0.034 [-0.125, 0.008] -0.002 [-0.052, 0.031]
baseline single-mode: removal n=29: 0.333 [-0.243, 2.057] not assayed n=4: 0.306 [0.203, 0.363]
baseline single-mode: rotation 0.279 [-0.301, 0.554] (both > 0: 20/29) 0.173 [0.155, 0.193] (both > 0: 4/4)
regularized Fast: removal n=9: -0.056 [-0.219, 0.157] n=25: -0.075 [-0.301, 0.069] n=30: -0.029 [-0.168, 0.065]
regularized Fast: rotation -0.014 [-0.065, 0.060] -0.023 [-0.122, 0.030] -0.007 [-0.057, 0.028]
regularized Slow: removal n=21: 0.482 [0.406, 0.558] n=5: 0.745 [0.619, 0.767] n=0
regularized Slow: rotation 0.696 [0.538, 0.760] 0.768 [0.702, 0.782]

Two-mode baselines, like Fast regularized runs, show no directionality at any width. Single-mode baselines (all slow) mostly show positive directionality, but their weak mode carries little endpoint energy (N^E 1.02-1.48), so these values are less stable (one d128 run has removal 2.06) and have no d256 reference. Only the regularized Slow runs combine balanced endpoint energy with a consistent one-way dependence.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support