C5 width experiment: final model weights
Final checkpoints (final_model.pt) of the C5 RoPE 8-layer 8-head transformers (FF_DIM=1024) trained with d_model 128 and 512, baseline and energy-balance regularized (lambda=0.5, final layer), seeds 1β30, 200k-step budget.
| file | contents |
|---|---|
| d128_baseline.zip | width_results/d_model_128/baseline/training_archives//final_model.pt (30 runs) |
| d128_regularized.zip | width_results/d_model_128/regularized/training_archives//final_model.pt (30 runs) |
| d512_baseline.zip | width_results/d_model_512/baseline/training_archives//final_model.pt (30 runs) |
| d512_regularized.zip | width_results/d_model_512/regularized/training_archives//final_model.pt (30 runs) |
| d128_baseline_ext400k.zip | width_results_ext400k/d_model_128/baseline/training_archives//final_model.pt (seeds 1, 30 continued to stable hit within 400k) |
d128 baseline seeds 1 and 30 in d128_baseline.zip are the 200k-budget (censored) endpoints. SHA-256 checksums are in SHA256SUMS.
Preliminary endpoint analysis (run by the uploader)
The paper's endpoint analyses were applied unchanged to all 120 final models in this repository (200k-budget endpoints; d128 baseline seeds 1 and 30 are the censored 200k models).
- Energy allocation: endpoint harmonic energy fractions at the final residual (4096-sequence representation probe, positionwise fractions averaged), N^E = 1 / sum p_m^2, two-mode = N^E >= 1.5.
- Interventions (regularized runs in the table below; baseline runs in the next table): final-MLP source-mode removal and semantic rotation, forward (dominant pre-MLP mode as source) and reverse, with four norm-matched random controls, using the Figure 4 probes and settings. Directionality = forward minus reverse specific effect.
- Fast/Slow: stable-hit step < 30k / >= 30k (the paper's fixed boundary). The d256 column is the paper's cohort, recomputed with the same summary code.
| d128 | d256 (paper) | d512 | |
|---|---|---|---|
| baseline two-mode runs | 1/30 | 10/30 | 26/30 |
| baseline stable-hit: two-mode / single-mode | 18.8k / 43.2-146.6k (+2 runs >200k) | 14.6-19.6k / 40.2-54.0k | 14.6-21.4k / 30.8-44.8k |
| regularized two-mode runs (min N^E) | 30/30 (1.935) | 30/30 (1.968) | 30/30 (1.986) |
| regularized Fast / Slow | 9 / 21 | 25 / 5 | 30 / 0 |
| Fast / Slow stable-hit range | 13.0-21.2k / 52.6-106.6k | 11.0-19.2k / 36.4-48.8k | 11.4-22.6k / - |
| removal directionality: Fast median [max] / Slow median [min] | -0.056 [0.157] / 0.482 [0.406] | -0.075 [0.069] / 0.745 [0.619] | -0.029 [0.065] / - |
| rotation directionality: Fast median [max] / Slow median [min] | -0.014 [0.060] / 0.696 [0.538] | -0.023 [0.030] / 0.768 [0.702] | -0.007 [0.028] / - |
At d128, all 21 Slow regularized runs show strong positive removal and rotation directionality, with no overlap with the 9 Fast runs; their stable-hit steps are separated by an empty interval (21.2-52.6k) that contains the 30k boundary. At d512 there are no Slow runs, and directionality is near zero in all 30 runs. These numbers are provided for cross-checking.
Baseline vs regularized interventions
The same final-MLP removal and rotation assays were also applied to all 60 baseline models (forward = dominant pre-MLP mode as source; all 60 assayable). Values are median [min, max] of forward minus reverse; "both > 0" counts runs with positive removal and rotation directionality. The d256 baseline row is the rebuttal analysis of the 10 two-mode baseline runs (analyses/baseline_directionality.py); single-mode d256 baselines were not assayed.
| group | d128 | d256 | d512 |
|---|---|---|---|
| baseline two-mode: removal | n=1: -0.091 | n=10: -0.113 [-0.404, 0.042] | n=26: 0.000 [-0.132, 0.084] |
| baseline two-mode: rotation | -0.020 | -0.034 [-0.125, 0.008] | -0.002 [-0.052, 0.031] |
| baseline single-mode: removal | n=29: 0.333 [-0.243, 2.057] | not assayed | n=4: 0.306 [0.203, 0.363] |
| baseline single-mode: rotation | 0.279 [-0.301, 0.554] (both > 0: 20/29) | 0.173 [0.155, 0.193] (both > 0: 4/4) | |
| regularized Fast: removal | n=9: -0.056 [-0.219, 0.157] | n=25: -0.075 [-0.301, 0.069] | n=30: -0.029 [-0.168, 0.065] |
| regularized Fast: rotation | -0.014 [-0.065, 0.060] | -0.023 [-0.122, 0.030] | -0.007 [-0.057, 0.028] |
| regularized Slow: removal | n=21: 0.482 [0.406, 0.558] | n=5: 0.745 [0.619, 0.767] | n=0 |
| regularized Slow: rotation | 0.696 [0.538, 0.760] | 0.768 [0.702, 0.782] |
Two-mode baselines, like Fast regularized runs, show no directionality at any width. Single-mode baselines (all slow) mostly show positive directionality, but their weak mode carries little endpoint energy (N^E 1.02-1.48), so these values are less stable (one d128 run has removal 2.06) and have no d256 reference. Only the regularized Slow runs combine balanced endpoint energy with a consistent one-way dependence.