Add training-loss scenario chart (backtest failed; not a forecast); fix decay shape (1 - sqrt)

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+analysis/scaling-training-loss.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -104,10 +104,29 @@ removal of every personal datum.
 | Final training loss | 2.6969 (step 1430, nats per token) |
 | Checkpoints | every 150 steps (10 in total), each copied off the training node with sha256 |
 | Optimizer | Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest |
-| Schedule | warmup 150 steps, constant, linear decay over the last 20 % of steps to lr / 10 |
+| Schedule | warmup 150 steps, constant, then 1 − sqrt decay over the last 20 % of steps (from step 1144) to lr / 10 |
 | Data order | each 4096-token window of the stream used at most once (epoch sampler), data seed 20261023 |
 | Precision | bf16 autocast, torch.compile, activation checkpointing |
 
+## Training-loss scenarios (not a forecast)
+
+![Training loss and conditional scenarios](analysis/scaling-training-loss.png)
+
+`analysis/scaling-training-loss.png` shows the measured training loss and conditional scenarios for 3, 6 and 12 billion
+tokens: a new warmup-stable-decay run of the same recipe (solid) and resuming the final checkpoint at lr / 10 (dashed).
+The scenarios come from the learning-rate-annealing scaling law (Tissue et al., arXiv:2408.11029) fitted to steps
+200–1430, for λ = 0.99 / 0.995 / 0.999.
+
+- **The backtest of this method failed** on this run: fitted on earlier steps, it missed the measured loss by
+  0.077–0.088 nats per token (threshold 0.03). The bars of ±0.09 show that historical error; they are not a confidence
+  or prediction interval.
+- The spread between λ values (up to 0.6 nats at 12 B tokens) is larger than the gain being estimated. One 1.5 B-token
+  run does not determine it.
+- Assumptions: training loss (not validation loss), new data from the same distribution beyond the 1.89 B tokens that
+  exist, one seed. Benchmarks are not projected.
+- Numbers: `analysis/scaling-training-loss.json`; input loss log: `analysis/training_metrics.jsonl`. Two independent
+  implementations agree to 3.4e-7 nats.
+
 ## Limitations
 
 - 1 073 912 851 parameters trained on 1 499 463 680 tokens (about 1.4 tokens per parameter). Compare with the token

analysis/SHA256SUMS

@@ -0,0 +1,3 @@
+d688a6df1d28da423065094a666a8f8ce1135bef507d22950426a4223a0d767a  scaling-training-loss.png
+1e3d582f25712cacc8a2eb19e5098c53d126d693c52cc970036b624751630866  scaling-training-loss.json
+d568379d614d9581c64bb095a0b8477781e128affc07261a446a37af2d406a89  training_metrics.jsonl

analysis/scaling-training-loss.json

@@ -0,0 +1,65 @@
+{
+ "wejscie_sha256": "d568379d614d9581c64bb095a0b8477781e128affc07261a446a37af2d406a89",
+ "status": "SCENARIUSZE (test wsteczny nie przeszedł)",
+ "pasmo_pm": 0.09,
+ "okna_fit": 25,
+ "lambdy": {
+  "0.99": {
+   "fit": {
+    "L0": 2.7176897151567374,
+    "A": 4021.2636775897254,
+    "alfa": 1.4605386227846933,
+    "C": 0.002494840277031487
+   },
+   "rmse": 0.02194506426342817,
+   "W": {
+    "3": 2.558643845359285,
+    "6": 2.5192136684084825,
+    "12": 2.5038974367628724
+   },
+   "R": {
+    "3": 2.5951149142957837,
+    "6": 2.570892840164922,
+    "12": 2.5439097409677127
+   }
+  },
+  "0.995": {
+   "fit": {
+    "L0": 2.7102361499411547,
+    "A": 3688.4946224522505,
+    "alfa": 1.4438300628317653,
+    "C": 0.001728893917261872
+   },
+   "rmse": 0.023290885082249434,
+   "W": {
+    "3": 2.5132075456187684,
+    "6": 2.4464141581852736,
+    "12": 2.4194519558198984
+   },
+   "R": {
+    "3": 2.504696233418353,
+    "6": 2.479767886720681,
+    "12": 2.4520033166280295
+   }
+  },
+  "0.999": {
+   "fit": {
+    "L0": 2.7019973750885717,
+    "A": 3358.820680436602,
+    "alfa": 1.4257070782142964,
+    "C": 0.0011942258519509795
+   },
+   "rmse": 0.024834686663436946,
+   "W": {
+    "3": 2.4197142341118845,
+    "6": 2.177962085814373,
+    "12": 1.9353969001764642
+   },
+   "R": {
+    "3": 1.9498791924777694,
+    "6": 1.723482469279064,
+    "12": 1.6827141876221703
+   }
+  }
+ }
+}

analysis/training_metrics.jsonl

@@ -0,0 +1,143 @@
+{"step": 10, "loss": 10.558928048610687, "lr": 3.9999999999999996e-05, "tok_s": 57751.26294201406}
+{"step": 20, "loss": 9.436821150779725, "lr": 7.999999999999999e-05, "tok_s": 100884.4599635125}
+{"step": 30, "loss": 8.319514799118043, "lr": 0.00011999999999999999, "tok_s": 100833.53929754537}
+{"step": 40, "loss": 7.838142824172974, "lr": 0.00015999999999999999, "tok_s": 101081.1784334697}
+{"step": 50, "loss": 7.553256011009216, "lr": 0.00019999999999999998, "tok_s": 100774.87408020499}
+{"step": 60, "loss": 7.184791326522827, "lr": 0.00023999999999999998, "tok_s": 100710.54432974351}
+{"step": 70, "loss": 6.705662786960602, "lr": 0.00028, "tok_s": 100549.5921565801}
+{"step": 80, "loss": 6.277644205093384, "lr": 0.00031999999999999997, "tok_s": 100891.40099534027}
+{"step": 90, "loss": 6.0171788215637205, "lr": 0.00035999999999999997, "tok_s": 100701.70811769294}
+{"step": 100, "loss": 5.8424641489982605, "lr": 0.00039999999999999996, "tok_s": 100215.07324317082}
+{"step": 110, "loss": 5.713338726758957, "lr": 0.0004399999999999999, "tok_s": 100702.97768444012}
+{"step": 120, "loss": 5.6118501961231235, "lr": 0.00047999999999999996, "tok_s": 100644.04254403408}
+{"step": 130, "loss": 5.51015482544899, "lr": 0.00052, "tok_s": 99816.29323487202}
+{"step": 140, "loss": 5.410903519392013, "lr": 0.00056, "tok_s": 100381.76601220365}
+{"step": 150, "loss": 5.2926738381385805, "lr": 0.0006, "tok_s": 100281.57027631332}
+{"step": 160, "loss": 5.134509360790252, "lr": 0.0006, "tok_s": 93072.21049814693}
+{"step": 170, "loss": 4.978759688138962, "lr": 0.0006, "tok_s": 100491.10753252104}
+{"step": 180, "loss": 4.811357623338699, "lr": 0.0006, "tok_s": 100284.86463024934}
+{"step": 190, "loss": 4.682711988687515, "lr": 0.0006, "tok_s": 100441.47560816094}
+{"step": 200, "loss": 4.53830453157425, "lr": 0.0006, "tok_s": 100738.70030676374}
+{"step": 210, "loss": 4.394386672973633, "lr": 0.0006, "tok_s": 100571.50240582418}
+{"step": 220, "loss": 4.263681900501251, "lr": 0.0006, "tok_s": 100923.51592699431}
+{"step": 230, "loss": 4.155369570851326, "lr": 0.0006, "tok_s": 100884.67332722961}
+{"step": 240, "loss": 4.047501584887504, "lr": 0.0006, "tok_s": 100335.96362118336}
+{"step": 250, "loss": 3.9694435477256773, "lr": 0.0006, "tok_s": 100539.24196462665}
+{"step": 260, "loss": 3.884575602412224, "lr": 0.0006, "tok_s": 100426.14970602097}
+{"step": 270, "loss": 3.823830580711365, "lr": 0.0006, "tok_s": 100654.37792934928}
+{"step": 280, "loss": 3.7185207545757293, "lr": 0.0006, "tok_s": 100435.92455150445}
+{"step": 290, "loss": 3.6873651176691054, "lr": 0.0006, "tok_s": 100386.63169298533}
+{"step": 300, "loss": 3.626361387968063, "lr": 0.0006, "tok_s": 100659.55574857818}
+{"step": 310, "loss": 3.6048438757658006, "lr": 0.0006, "tok_s": 93197.34633267309}
+{"step": 320, "loss": 3.552429750561714, "lr": 0.0006, "tok_s": 100261.03729336945}
+{"step": 330, "loss": 3.510664466023445, "lr": 0.0006, "tok_s": 100530.96454005908}
+{"step": 340, "loss": 3.485000064969063, "lr": 0.0006, "tok_s": 99364.67089083602}
+{"step": 350, "loss": 3.446547359228134, "lr": 0.0006, "tok_s": 100383.1436879417}
+{"step": 360, "loss": 3.4383565574884414, "lr": 0.0006, "tok_s": 100328.5575202722}
+{"step": 370, "loss": 3.402011752128601, "lr": 0.0006, "tok_s": 100711.98293499982}
+{"step": 380, "loss": 3.3815925151109694, "lr": 0.0006, "tok_s": 100612.65023614647}
+{"step": 390, "loss": 3.3673422038555145, "lr": 0.0006, "tok_s": 98685.50748380071}
+{"step": 400, "loss": 3.348368188738823, "lr": 0.0006, "tok_s": 100769.08731985553}
+{"step": 410, "loss": 3.3255837202072143, "lr": 0.0006, "tok_s": 100634.79176646235}
+{"step": 420, "loss": 3.2951862037181856, "lr": 0.0006, "tok_s": 100599.12529834628}
+{"step": 430, "loss": 3.2823459267616273, "lr": 0.0006, "tok_s": 100472.07488748315}
+{"step": 440, "loss": 3.2495147228240966, "lr": 0.0006, "tok_s": 100537.51457341926}
+{"step": 450, "loss": 3.2421604454517365, "lr": 0.0006, "tok_s": 100546.56610997692}
+{"step": 460, "loss": 3.2565063536167145, "lr": 0.0006, "tok_s": 92967.6989556629}
+{"step": 470, "loss": 3.2139015913009645, "lr": 0.0006, "tok_s": 100243.17924641138}
+{"step": 480, "loss": 3.2065516859292984, "lr": 0.0006, "tok_s": 100438.83567746296}
+{"step": 490, "loss": 3.223135444521904, "lr": 0.0006, "tok_s": 100482.15596986027}
+{"step": 500, "loss": 3.1827252596616744, "lr": 0.0006, "tok_s": 100638.6077076409}
+{"step": 510, "loss": 3.17941058576107, "lr": 0.0006, "tok_s": 100451.8166921828}
+{"step": 520, "loss": 3.1631668627262117, "lr": 0.0006, "tok_s": 100559.95505644038}
+{"step": 530, "loss": 3.168537271022797, "lr": 0.0006, "tok_s": 100281.07249486496}
+{"step": 540, "loss": 3.147646015882492, "lr": 0.0006, "tok_s": 100751.68394141551}
+{"step": 550, "loss": 3.1297860950231553, "lr": 0.0006, "tok_s": 100549.44181541628}
+{"step": 560, "loss": 3.1166075110435485, "lr": 0.0006, "tok_s": 100261.74332723948}
+{"step": 570, "loss": 3.134752944111824, "lr": 0.0006, "tok_s": 100804.65793604114}
+{"step": 580, "loss": 3.1094461649656298, "lr": 0.0006, "tok_s": 100547.59224210691}
+{"step": 590, "loss": 3.108005365729332, "lr": 0.0006, "tok_s": 100551.6487033244}
+{"step": 600, "loss": 3.097949135303497, "lr": 0.0006, "tok_s": 100720.75959394994}
+{"step": 610, "loss": 3.0965460538864136, "lr": 0.0006, "tok_s": 93277.87862540164}
+{"step": 620, "loss": 3.086359590291977, "lr": 0.0006, "tok_s": 100798.75132029096}
+{"step": 630, "loss": 3.0684080839157106, "lr": 0.0006, "tok_s": 100720.74044891441}
+{"step": 640, "loss": 3.0619209706783295, "lr": 0.0006, "tok_s": 100749.15923113434}
+{"step": 650, "loss": 3.058165040612221, "lr": 0.0006, "tok_s": 100978.06394390087}
+{"step": 660, "loss": 3.0585276931524277, "lr": 0.0006, "tok_s": 100749.46295592526}
+{"step": 670, "loss": 3.0499379098415376, "lr": 0.0006, "tok_s": 99320.94484772583}
+{"step": 680, "loss": 3.031423282623291, "lr": 0.0006, "tok_s": 100375.67632740687}
+{"step": 690, "loss": 3.034598910808563, "lr": 0.0006, "tok_s": 100531.92118421877}
+{"step": 700, "loss": 3.013563647866249, "lr": 0.0006, "tok_s": 100501.78679464728}
+{"step": 710, "loss": 3.020408755540848, "lr": 0.0006, "tok_s": 100400.61455812691}
+{"step": 720, "loss": 3.0080485552549363, "lr": 0.0006, "tok_s": 100626.51817147095}
+{"step": 730, "loss": 3.010661116242409, "lr": 0.0006, "tok_s": 100560.31673320402}
+{"step": 740, "loss": 2.985303795337677, "lr": 0.0006, "tok_s": 100594.11843351876}
+{"step": 750, "loss": 2.9911435574293135, "lr": 0.0006, "tok_s": 100719.24346806774}
+{"step": 760, "loss": 2.993647199869156, "lr": 0.0006, "tok_s": 92694.93586499312}
+{"step": 770, "loss": 2.991041973233223, "lr": 0.0006, "tok_s": 100411.09876164561}
+{"step": 780, "loss": 2.986131799221039, "lr": 0.0006, "tok_s": 100584.46022805343}
+{"step": 790, "loss": 2.9761282563209535, "lr": 0.0006, "tok_s": 100769.66845956571}
+{"step": 800, "loss": 2.9691010802984237, "lr": 0.0006, "tok_s": 100267.10507498478}
+{"step": 810, "loss": 2.9632755160331725, "lr": 0.0006, "tok_s": 100483.79146422203}
+{"step": 820, "loss": 2.9471588104963304, "lr": 0.0006, "tok_s": 100624.9167054258}
+{"step": 830, "loss": 2.9456286638975144, "lr": 0.0006, "tok_s": 100345.73373853264}
+{"step": 840, "loss": 2.9343577355146406, "lr": 0.0006, "tok_s": 100796.49152859482}
+{"step": 850, "loss": 2.937390160560608, "lr": 0.0006, "tok_s": 100521.5555603355}
+{"step": 860, "loss": 2.9361020803451536, "lr": 0.0006, "tok_s": 100744.30011428859}
+{"step": 870, "loss": 2.929842785000801, "lr": 0.0006, "tok_s": 100557.68710583313}
+{"step": 880, "loss": 2.921016755700111, "lr": 0.0006, "tok_s": 100533.28345118709}
+{"step": 890, "loss": 2.908094030618668, "lr": 0.0006, "tok_s": 100429.13617330971}
+{"step": 900, "loss": 2.915668472647667, "lr": 0.0006, "tok_s": 99995.44207278691}
+{"step": 910, "loss": 2.9192551016807555, "lr": 0.0006, "tok_s": 93241.89946835527}
+{"step": 920, "loss": 2.912376421689987, "lr": 0.0006, "tok_s": 100597.82153270121}
+{"step": 930, "loss": 2.919869974255562, "lr": 0.0006, "tok_s": 100551.00455790506}
+{"step": 940, "loss": 2.9144054144620894, "lr": 0.0006, "tok_s": 100592.46508110054}
+{"step": 950, "loss": 2.9140206307172773, "lr": 0.0006, "tok_s": 100397.97631651394}
+{"step": 960, "loss": 2.88630553483963, "lr": 0.0006, "tok_s": 100567.73110532905}
+{"step": 970, "loss": 2.8786509573459624, "lr": 0.0006, "tok_s": 100087.77366207258}
+{"step": 980, "loss": 2.887146371603012, "lr": 0.0006, "tok_s": 100603.54192329201}
+{"step": 990, "loss": 2.8839745849370955, "lr": 0.0006, "tok_s": 100008.96685521863}
+{"step": 1000, "loss": 2.891139271855354, "lr": 0.0006, "tok_s": 100479.05478346854}
+{"step": 1010, "loss": 2.870338073372841, "lr": 0.0006, "tok_s": 100561.80393493171}
+{"step": 1020, "loss": 2.875291308760643, "lr": 0.0006, "tok_s": 100625.90345665337}
+{"step": 1030, "loss": 2.8735448092222216, "lr": 0.0006, "tok_s": 100316.55494130778}
+{"step": 1040, "loss": 2.853999066352844, "lr": 0.0006, "tok_s": 100613.428210747}
+{"step": 1050, "loss": 2.853374880552292, "lr": 0.0006, "tok_s": 100421.40217802356}
+{"step": 1060, "loss": 2.8355960071086885, "lr": 0.0006, "tok_s": 93269.71954240384}
+{"step": 1070, "loss": 2.868672314286232, "lr": 0.0006, "tok_s": 100303.63013389682}
+{"step": 1080, "loss": 2.8497932821512224, "lr": 0.0006, "tok_s": 100377.78533064568}
+{"step": 1090, "loss": 2.855304646492004, "lr": 0.0006, "tok_s": 100553.07610564643}
+{"step": 1100, "loss": 2.849302512407303, "lr": 0.0006, "tok_s": 99987.75222334897}
+{"step": 1110, "loss": 2.848549008369446, "lr": 0.0006, "tok_s": 100530.79472211229}
+{"step": 1120, "loss": 2.84353244304657, "lr": 0.0006, "tok_s": 100201.24770568225}
+{"step": 1130, "loss": 2.8420894235372542, "lr": 0.0006, "tok_s": 100575.62289620684}
+{"step": 1140, "loss": 2.8188397914171217, "lr": 0.0006, "tok_s": 100381.05737051132}
+{"step": 1150, "loss": 2.8355686396360396, "lr": 0.0005286004054780274, "tok_s": 100421.70026114526}
+{"step": 1160, "loss": 2.836396670341492, "lr": 0.00047633227464812693, "tok_s": 100335.48612965432}
+{"step": 1170, "loss": 2.8013833552598952, "lr": 0.0004403456530829476, "tok_s": 100441.48271911831}
+{"step": 1180, "loss": 2.7967331647872924, "lr": 0.00041109442918401085, "tok_s": 100521.98703573096}
+{"step": 1190, "loss": 2.783522140979767, "lr": 0.0003858012164340822, "tok_s": 99864.53469411423}
+{"step": 1200, "loss": 2.8081950902938844, "lr": 0.00036319433478710207, "tok_s": 99727.40740473339}
+{"step": 1210, "loss": 2.771678701043129, "lr": 0.00034256510090369, "tok_s": 92779.58668970785}
+{"step": 1220, "loss": 2.776586925983429, "lr": 0.00032347055949043783, "tok_s": 99871.56355511003}
+{"step": 1230, "loss": 2.7670613795518877, "lr": 0.0003056119301596349, "tok_s": 100189.3965312251}
+{"step": 1240, "loss": 2.7639532893896104, "lr": 0.00028877638286894117, "tok_s": 100251.38923646363}
+{"step": 1250, "loss": 2.7678561240434645, "lr": 0.0002728059535139063, "tok_s": 100067.8661624419}
+{"step": 1260, "loss": 2.7602756023406982, "lr": 0.0002575795738397777, "tok_s": 100559.05673895142}
+{"step": 1270, "loss": 2.7430291563272475, "lr": 0.000243002027390137, "tok_s": 100429.29280549876}
+{"step": 1280, "loss": 2.741029241681099, "lr": 0.00022899682394438096, "tok_s": 100206.27261538581}
+{"step": 1290, "loss": 2.732442146539688, "lr": 0.00021550141633659966, "tok_s": 100266.00168343422}
+{"step": 1300, "loss": 2.6943935334682463, "lr": 0.00020246388210004733, "tok_s": 100394.61196316617}
+{"step": 1310, "loss": 2.703908529877663, "lr": 0.000189840556331115, "tok_s": 100486.53592685594}
+{"step": 1320, "loss": 2.7091004699468613, "lr": 0.00017759430232704788, "tok_s": 100606.48808964455}
+{"step": 1330, "loss": 2.7243851006031035, "lr": 0.00016569322176325906, "tok_s": 100216.50229516492}
+{"step": 1340, "loss": 2.71903018951416, "lr": 0.00015410967512382394, "tok_s": 100037.34963684995}
+{"step": 1350, "loss": 2.7019290685653687, "lr": 0.00014281952577347963, "tok_s": 100086.58811765567}
+{"step": 1360, "loss": 2.715005224943161, "lr": 0.00013180154826207484, "tok_s": 90787.59303400978}
+{"step": 1370, "loss": 2.6950105369091033, "lr": 0.00012103695924884308, "tok_s": 100544.85087942333}
+{"step": 1380, "loss": 2.691245433688164, "lr": 0.00011050904134866944, "tok_s": 100008.1163323947}
+{"step": 1390, "loss": 2.6923380970954893, "lr": 0.0001002028383461919, "tok_s": 100149.29923623533}
+{"step": 1400, "loss": 2.689820957183838, "lr": 9.010490589435262e-05, "tok_s": 100079.63394488336}
+{"step": 1410, "loss": 2.6837279438972472, "lr": 8.020310582996826e-05, "tok_s": 99837.14824462736}
+{"step": 1420, "loss": 2.7054353982210158, "lr": 7.048643512690308e-05, "tok_s": 99750.98831031937}
+{"step": 1430, "loss": 2.6968517333269117, "lr": 6.0944882613642584e-05, "tok_s": 100106.40645600921}