Upload folder using huggingface_hub

scheduler/COMPARISON.md

@@ -0,0 +1,48 @@
+# LR-Scheduler Comparison — MuonQ @ 8e-3 on the best recipe, Llama 77M, 60 min each
+
+**Hardware:** RX 9070 XT 16GB, ROCm 7.2, torch 2.13, transformers 5.14.1
+**Common recipe:** sdpa attention, fp32 master + bf16 autocast, **muonq** (lr 8e-3, wd 0.1, warmup 200), effective batch 256×1024 tok, seed 42, same data order, same dataset. Only the LR schedule varies. Per-folder details in each `*/REPORT.md`.
+
+**Series prerequisite — the horizon fix:** every earlier run had `max_steps = -1`, giving schedulers a 75,210-step (full-epoch) horizon, so no decay ever actually happened inside a 1-hour run. `base_train.py` now accepts `max_steps` from config and all runs here pin it to **760** (the achievable steps/hour), so each schedule genuinely completes. The constant-LR arm is `../lr/lr_8e-3/` unchanged — that run *was* constant-after-warmup by accident, making it the free reference.
+
+## Headline table (final-model metrics from evaluate2 on `final/`)
+
+| schedule | steps/60min | final eval loss / ppl | held-out bpb | chat bpb | logic | bake /100 | greedy loop | verdict |
+|---|---|---|---|---|---|---|---|---|
+| constant after warmup (`../lr/lr_8e-3/`) | 759 | 3.8583 / 47.4 | 1.3114 | 1.1914 | 0.600 (+0.018) | 46.8 | 59.8w | reference — was the project best |
+| WSD, 10% linear decay → 0 | 740 | 3.8641¹ / 47.7 | 1.2821 | 1.1608 | 0.625 (+0.012) | 52.7 | 33.8w | big win, dominated by wsd_20 |
+| **WSD, 20% linear decay → 0** | 730 | 3.8023¹ / 44.8 | **1.2769** | **1.1571** | **0.650 (+0.024)** | **55.6 — record** | **23.8w** | **winner — quality AND hygiene** |
+| cosine → 5% floor (`cosine_760/`) | 750 | **3.7494 / 42.5** | 1.2775 | 1.1623 | 0.650 (**+0.040**) | 51.7 | 66.2w | best raw loss, worst hygiene |
+| GreedyLR adaptive (`greedy/`) | 760 | 3.9267² / 50.7 | 1.3422 | 1.2188 | 0.575 (+0.009) | 44.0 | 45.5w | loses (no anneal) — but found peak LR on its own |
+
+¹ WSD logged evals land mid-decay (wall-clock stop cut the decays 20–26% short; final saved models sit at LR 2.2e-3 / 1.6e-3, not 0) — the trainer eval-loss column undersells them; bpb/bake columns are the honest final-model numbers.
+² greedy evaluated on a 1024-sample subset (its 20-step eval cadence needs cheap evals) — noisier and not the same subset as the others' 5000.
+
+All runs stable, zero non-finite grad norms, peak GPU 7.2 GB, clean period boundaries. The fixed-schedule runs share the identical 9.45 early grad spike (same seed/data → identical trajectory until schedules diverge after warmup); greedy's cold 1e-3 start was the calmest (max 5.51).
+
+## Eval loss at matched wall-clock (5-min marks, 5000-sample evals)
+
+| ~step | constant | wsd_10 | wsd_20 | cosine_760 |
+|---|---|---|---|---|
+| ~67 | 5.769 | 5.704 | 5.733 | 5.704 |
+| ~192 | 4.582 | 4.569 | 4.596 | 4.578 |
+| ~318 | 4.181 | 4.182 | 4.206 | 4.163 |
+| ~444 | 4.015 | 4.025 | 4.034 | **3.961** |
+| ~570 | 3.920 | 3.929 | 3.934 | **3.822** |
+| ~690 | 3.858 | 3.864 | 3.802 | **3.749** |
+
+Until the decays begin (~step 450 for cosine's steepening, ~610/684 for WSD) all four are statistically the same curve — as they must be, since they train identically at peak LR. Everything the schedules buy, they buy in the decay phase: wsd_20's last interval alone is −0.09.
+
+## What we learned
+
+1. **The anneal is the single biggest quality lever found in this project so far.** Fixing the horizon so *any* decay completes in-window is worth +4.9 to +8.8 bake points and −0.06 to −0.11 eval loss over the identical constant-LR run — more than the entire 1e-3 → 8e-3 LR tuning gained (+3.7 bake).
+2. **WSD-20% > WSD-10% at this horizon, and WSD cures the hot-LR hygiene tax.** Greedy-loop length returns to cool-LR levels (23.8w vs 52–66w). Trading 76 stable-at-peak steps for a longer anneal was clearly worth it; 10% decay is too short here.
+3. **Cosine wins raw loss but not the composite.** Best ppl (42.5) and logic margin (+0.040) of the project, yet the *worst* hygiene of the series (66.2w). It's not "more time at hot LR" — cosine's mid-run LR is *below* WSD's. Suspects: its 5% floor never fully anneals (WSD goes to 0), or greedy-decode-probe noise. A `min_lr_rate = 0` re-run would settle it.
+4. **GreedyLR is a validator, not a competitor.** Left alone with only the eval-loss signal, it climbed from 1e-3 and plateaued at 6.6–7.4e-3 — independently rediscovering our hand-tuned 8e-3 peak. But it has no concept of a budget ending, finished the run hot, and lost to everything that decays. Adaptive stable phase + forced terminal decay could combine both, but plain WSD already captures most of that.
+5. **Metric-driven schedulers need plumbing:** `metric_for_best_model` (now a base_train config key), a fast eval cadence, and note that `warmup_steps` is silently ignored for them.
+
+## Recommendation
+
+- **New best recipe: WSD with 20% linear decay to 0** (`wsd_20/config.toml`) on top of muonq @ 8e-3 — bake 55.6/100, bpb 1.2769, chat bpb 1.1571, logic 0.650, hygiene restored. This replaces the constant-LR 8e-3 recipe as the reference for future experiments.
+- Follow-ups, roughly in value order: (a) set the horizon to the *actually achievable* ~730 steps so the decay finishes (wsd_20 set the record with 20% of its decay missing); (b) try `decay_type = "1-sqrt"` (what the WSD literature recommends over linear); (c) probe 30% decay — the 10% → 20% trend hasn't turned; (d) cosine with `min_lr_rate = 0` to isolate the hygiene puzzle.
+- For the real 19.6B-token run: the WSD structure is also *operationally* ideal — the stable phase can run for days and be annealed from any checkpoint, so the decay fraction becomes a checkpoint-time decision, not a launch-time commitment. Re-verify 10 vs 20% at a 2–3h horizon first (decay-fraction optima shrink as budgets grow).

scheduler/cosine_760/REPORT.md

@@ -0,0 +1,38 @@
+# Experiment: cosine_760 — cosine_with_min_lr with a real horizon
+
+**Wall-clock:** 60 min · **Recipe:** best known (sdpa + bf16 mixed + muonq @ lr 8e-3), schedule: warmup 200 → cosine decay to 5% floor over `max_steps = 760` — the anneal every previous run *thought* it was doing (they all had a 75,210-step horizon and never decayed)
+
+## Result: best loss/ppl of the entire project — but the worst greedy-loop hygiene
+
+| metric | cosine_760 | wsd_20 (series winner) | constant-LR reference |
+|---|---|---|---|
+| optimizer steps in 60 min | 750 | 730 | 759 |
+| final eval loss / ppl | **3.7494 / 42.5** (step 697, LR ≈ 4.7e-4 — decay essentially complete) | 3.8023 / 44.8 (mid-decay) | 3.8583 / 47.4 |
+| bake score | 51.74/100 (DOUGH) | **55.61** | 46.83 |
+| held-out bits/byte | 1.2775 | 1.2769 | 1.3114 |
+| chat bits/byte | 1.1623 | **1.1571** | 1.1914 |
+| logic accuracy | 0.650, **margin +0.0400** (best) | 0.650, +0.0242 | 0.600, +0.018 |
+| hygiene (greedy loop) | **42.1/100 (loop 66.2w) — series worst** | 79.2 (23.8w) | 47.7 (59.8w) |
+| grad norms | 0.07–9.45, 0 non-finite | 0.06–9.45 | 0.09–9.45 |
+| peak GPU | 7.2 GB | 7.2 GB | 7.2 GB |
+
+Fixing the horizon alone is worth −0.11 eval loss (3.8583 → 3.7494) and +4.9 bake points over the identical-LR constant run — confirmation that the LR series' biggest handicap was the broken anneal, not the LR values. On raw language-model quality cosine and wsd_20 are a near-tie (bpb 1.2775 vs 1.2769); cosine takes ppl and logic margin.
+
+**The puzzle is hygiene:** 66.2w greedy loops, worse even than constant LR (59.8w), while wsd_20 sits at 23.8w. Mechanistically this is *not* "more time at hot LR" — cosine's mid-run LR is *lower* than WSD's (WSD holds peak until its decay). Two candidate explanations: (a) the 5% floor (4e-4) never fully anneals, while WSD's decay-to-0 does; (b) the greedy-loop probe is a small, noisy sample. Worth a re-test with `min_lr_rate = 0` before trusting either story.
+
+## Eval loss curve (5-min intervals)
+
+| step | eval loss | | step | eval loss |
+|---|---|---|---|---|
+| 67 | 5.7043 | | 444 | 3.9611 |
+| 129 | 4.9559 | | 507 | 3.8829 |
+| 192 | 4.5780 | | 570 | 3.8217 |
+| 255 | 4.3263 | | 634 | 3.7765 |
+| 318 | 4.1631 | | 697 | 3.7494 |
+| 381 | 4.0505 | | | |
+
+Unlike the WSD runs, the logged curve here is representative — the decay is continuous, so the step-697 eval reflects a nearly-finished anneal.
+
+## Verdict
+
+**Best perplexity in the project (42.5) and proof the horizon fix matters more than any LR choice** — but the composite loses to wsd_20 on hygiene. If greedy-decoding quality doesn't matter for the use case, this is the loss-optimal schedule; if it does, WSD-20% wins. See `../COMPARISON.md`.

scheduler/cosine_760/config.toml

@@ -0,0 +1,89 @@
+# Scheduler series — new best recipe (sdpa + bf16 mixed + muonq @ lr 8e-3), only the LR schedule varies.
+# This run: cosine_with_min_lr with a REAL horizon (max_steps 760) — the anneal the lr series thought it was running.
+model_config = "../model.toml"
+
+[data]
+# Data files (supports glob patterns or lists of files)
+train_files = ["~/Dev/vintage-LLM/training/data/train_*.bin"]
+valid_files = ["~/Dev/vintage-LLM/training/data/valid_*.bin"]
+
+# Sequence length for each dataset sample
+# Must be <= model.max_position_embeddings
+max_seq_length = 1024
+
+# Tokenizer (must match the one used to create .bin files)
+tokenizer = "~/Dev/vintage-LLM/tokenizers/t-v3/"
+
+[training]
+# Training duration
+num_train_epochs = 1
+max_steps = 760                    # scheduler horizon = achievable steps/hour (was -1: full epoch, so decay never happened)
+
+# Optional: Stop training after this many minutes of wall-clock time.
+max_train_minutes = 60
+
+# Output directories
+output_dir = "./"
+final_model_dir = "./final"
+
+# Batch sizes
+# Effective batch size = num_devices × per_device_train_batch_size × gradient_accumulation_steps
+per_device_train_batch_size = 8    # batch size per GPU (8x1024 tokens fits 16GB with headroom for eval)
+per_device_eval_batch_size  = 8
+gradient_accumulation_steps = 32   # accumulate gradients over multiple mini-batches before updating the params
+
+# Optimizer: MuonQ (4-bit quantized Muon, ../../MuonQ repo) — wired through
+# optimizer_cls_and_kwargs in base_train.py. 2D hidden weights get Muon
+# (Newton-Schulz 5, 4-bit mu-law-companded momentum, rank-16 residual
+# factorization); embeddings/head/norms get its internal AdamW backup.
+# Muon/quantization knobs (muon_*, muonq_*) default to the repo's own
+# llama-60m "muonq" recipe in base_train.py.
+optim = "muonq"
+
+# Optimizer settings — lr/wd from MuonQ's llama-60m recipe (Muon takes
+# spectral-norm-scaled steps; do not copy the AdamW 5e-4 here)
+learning_rate = 8e-3
+weight_decay = 0.1
+adam_beta1 = 0.9
+adam_beta2 = 0.95
+max_grad_norm = 1.0
+
+# Learning rate scheduler
+lr_scheduler_type    = "cosine_with_min_lr"    # cosine decay to a non-zero floor — now actually completes in-hour
+lr_scheduler_kwargs  = { min_lr_rate = 0.05 }  # final LR = 5% of peak
+warmup_steps = 200                 # 1-hour series standard (muonq-v2's 400 was a 2-hour recipe)
+
+# Precision (choose based on your GPU)
+bf16 = true                        # bfloat16 autocast + fp32 master weights (the dtype-series safe recipe)
+fp16 = false
+
+# Performance
+torch_compile = false
+gradient_checkpointing = false     # trade compute for memory (enables larger models)
+neftune_noise_alpha = 0.0          # add noise during training to improve model performance (fine-tuning)
+
+# Checkpointing
+# save_strategy can be "steps", "epoch", or "minutes" (wall-clock time).
+save_strategy = "minutes"
+save_steps = 5                    # checkpoint every X minutes
+save_total_limit = 3              # keep only X most recent checkpoints
+
+# Evaluation
+# eval_strategy can be "steps", "epoch", or "minutes" (wall-clock time).
+eval_strategy = "minutes"
+eval_steps = 5                   # evaluate every X minutes
+# Evaluate on a fixed random subset of the validation set (0 = use all)
+max_eval_samples = 5000
+
+# Logging
+logging_strategy = "steps"
+logging_steps = 10                 # log every 10 steps
+logging_first_step = true          # log the first step
+
+# Reproducibility
+seed = 42
+
+# Performance
+dataloader_num_workers = 2         # number of data loading threads
+dataloader_prefetch_factor = 2     # batches prefetched per worker
+dataloader_pin_memory = true       # pin memory for faster GPU transfer

scheduler/cosine_760/eval1.log

@@ -0,0 +1,212 @@
+W0814 15:07:33.287000 58734 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0814 15:07:33.311000 58734 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+
+==============================================================================
+CHECKPOINT INSPECTION
+==============================================================================
+  checkpoint : final
+  tokenizer  : final
+  device     : cuda   dtype: torch.bfloat16   seed: 1337
+
+
+==============================================================================
+SUITE: MODEL INFO AND TRAINING LINEAGE
+==============================================================================
+  checkpoint              : final
+  model_type              : llama
+  architecture            : LlamaForCausalLM
+  device                  : cuda
+  dtype                   : torch.bfloat16
+  parameters              : 77,083,392 (77.08M)
+  disk size               : 308 MB
+  layers / hidden / heads : 8 / 768 / 8 (KV heads: 4)
+  vocab (cfg / tokenizer) : 32768 / 32752
+  context length          : 1024
+  chat template           : True
+  training lineage        : base pretraining, ctx 1024
+  embedding mean norm     : 2.8554   <- typical row length of the embedding matrix; very small or huge values signal trouble
+  embedding mean cosine   : 0.1174   <- avg cosine between random embedding rows. Descriptive only: it starts near 0.0
+                            because RANDOM init is near-orthogonal, and it GROWS as the model trains.
+                            Near 0 therefore does NOT mean "healthy" - it can mean "barely trained".
+  (note: config vocab padded by 16 unused rows - harmless)
+
+==============================================================================
+SUITE: PERIOD FIDELITY ON FIXED PROBE SENTENCES
+==============================================================================
+  Scoring 20 sentences (10 historical, 10 modern)
+  The MODERN/HISTORICAL ratio is the output that means something. The absolute
+  numbers come from ~20 sentences: too noisy to rank checkpoints (use evaluate2.py).
+
+  OVERALL (270 tokens)
+    perplexity            : 266.37   <- 'average branching factor'; lower = less surprised = better fit to this text
+    mean token prob       : 0.0839   <- average confidence on the actual next word
+    median token prob     : 0.0081   <- typical confidence (robust to a few very bad tokens)
+    worst-token prob (min): 1.7e-08   <- the single most surprising word; tiny values = the model really didn't expect it
+    10th-percentile prob  : 1.4e-05   <- confidence on the model's worst 10% of guesses
+    low-confidence tokens : 52.96%   <- share of tokens predicted with <1% probability (red flags)
+    mean entropy          : 4.642 nats   <- how spread out the predictions are; ~0 = very certain, higher = hedging
+
+  GROUP COMPARISON
+    group        sentences    perplexity   bits/byte
+    historical          10         80.92       1.196
+    modern              10        721.86       1.963
+    modern/historical ratio: 8.92   <- >1 means period text is easier for the model than modern text (desired for a vintage model with a ~1900 knowledge cutoff)
+    (bits/byte is the tokenizer-independent version - the only column here you may
+     compare between models that use DIFFERENT tokenizers.)
+
+  PER-SENTENCE DETAIL (sorted worst-first)
+    [modern    ] ppl= 3611.77  minp=6.9e-08  The engineering team rewrote the whole backend micro
+    [modern    ] ppl= 2068.20  minp=1.6e-06  Robots on the automated assembly line handle the ent
+    [modern    ] ppl= 1281.63  minp=7.1e-07  Small retail startups now run almost all of their bu
+    [modern    ] ppl= 1160.26  minp=6.4e-08  Millions of shoppers queue outside the store because
+    [modern    ] ppl= 1000.82  minp=4.1e-07  Three hours stuck in traffic in the pouring rain mad
+    [modern    ] ppl=  900.90  minp=6.7e-06  She is finishing a graduate degree in machine learni
+    [modern    ] ppl=  559.87  minp=1.7e-08  Members of the European Union parliament voted on th
+    [modern    ] ppl=  353.74  minp=4.0e-07  She gave up her whole weekend to help me move apartm
+    [historical] ppl=  233.49  minp=1.7e-05  The keeper fed the great serpent which the naturalis
+    [modern    ] ppl=  226.90  minp=4.3e-06  After years of hiding it from everyone at work, he t
+    [modern    ] ppl=  165.54  minp=7.8e-06  I switched to a plant-based diet last year and I no 
+    [historical] ppl=  128.03  minp=8.2e-05  He drew a distinction so fine and over-scrupulous th
+    [historical] ppl=  115.04  minp=1.0e-05  The mountain rose above the valley in a silence sole
+    [historical] ppl=   98.38  minp=6.7e-06  He gave up his fortune to the patient study of natur
+    [historical] ppl=   88.79  minp=1.2e-04  The merchants of the port have grown rich upon their
+    [historical] ppl=   85.65  minp=5.2e-05  The Lord provideth for all his creatures, giving the
+    [historical] ppl=   74.17  minp=3.0e-05  The weavers at their looms are employed in the wooll
+    [historical] ppl=   66.11  minp=1.4e-05  The labouring poor of this parish are reduced to gre
+    [historical] ppl=   38.43  minp=1.1e-04  The ballroom was filled with dancing and laughter, a
+    [historical] ppl=   19.84  minp=7.7e-04  Her Majesty was pleased to summon the Lords and Comm
+
+==============================================================================
+SUITE: DIACHRONIC WORD-SENSE SEPARATION
+==============================================================================
+  Extracting contextual embeddings for 10 shifted words in period AND modern sentences.
+  Every probe word sits LAST in its sentence, so the causal model has actually read
+  the disambiguating context before it represents the word.
+
+  SENSE SEPARATION (cosine similarity of the SAME word, period vs modern sentence)
+    lower value = the model represents the two senses differently (good sign for a
+    period model); ~1.0 = it treats them as the same word regardless of context
+    want          : +0.297
+    python        : +0.411
+    awful         : +0.574
+    meat          : +0.594
+    parliament    : +0.609
+    nice          : +0.677
+    manufacture   : +0.700
+    gay           : +0.728
+    commerce      : +0.773
+    science       : +0.806
+    MEAN          : +0.617   <- near 1.0 = model ignores period vs modern usage; lower = senses differ
+
+  PAIRWISE COSINE SIMILARITY (period sentences, first 5 words)
+    these are CONTEXTUAL vectors, so they reflect the sentences as much as the words;
+    read the spread, not individual pairs. High mean = representations collapsed.
+                         gay      awful       nice       meat       want
+    gay                1.000      0.388      0.376      0.329      0.317
+    awful                         1.000      0.320      0.196      0.208
+    nice                                     1.000      0.306      0.333
+    meat                                                1.000      0.477
+    want                                                           1.000
+    all 45 pairs: mean=+0.331 std=0.127   <- healthy spread is good
+
+  NEAREST NEIGHBOURS inside the probe-word set (period sentences)
+    gay           : awful (+0.39), nice (+0.38), science (+0.38), commerce (+0.33), meat (+0.33)
+    awful         : gay (+0.39), nice (+0.32), parliament (+0.26), science (+0.26), commerce (+0.21)
+    nice          : science (+0.40), gay (+0.38), want (+0.33), awful (+0.32), meat (+0.31)
+    meat          : want (+0.48), science (+0.47), commerce (+0.46), parliament (+0.44), manufacture (+0.41)
+    want          : manufacture (+0.49), meat (+0.48), commerce (+0.47), science (+0.44), parliament (+0.36)
+
+==============================================================================
+SUITE: GENERATION PROBES
+==============================================================================
+
+  --- Probe 1 ---
+  prompt      : The history of the world is
+  continuation: too brief for the reader's purpose, and will be a long and weary one to read it.
+
+If I am not mistaken, I will write to the Duke of Wellington, who was the son of the Marquis of Lansdowne, a man of high birth, and who was a little
+  stats: 60 tokens | distinct-1 0.69 | distinct-2 0.98 | echo 0.08
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 2 ---
+  prompt      : What is God? God is
+  continuation: a man, a man, a being, a God, a God, a God. God is God. It is God that God is man.
+
+God is a man, and God is a God. Man is God, God only is God; God is God. God is God.
+  stats: 60 tokens | distinct-1 0.32 | distinct-2 0.65 | echo 0.45
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 3 ---
+  prompt      : In the year of our Lord eighteen hundred and
+  continuation: eighty-four in the year of our Lord one hundred and eight in the year of our Lord one thousand nine hundred and ninety-nine in the year of our Lord one thousand three hundred and eighteen in the year of our Lord one thousand seven hundred and eighteen in the year of our Lord one thousand seven
+  stats: 60 tokens | distinct-1 0.30 | distinct-2 0.40 | echo 0.00
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 4 ---
+  prompt      : The manufacture of cotton
+  continuation: -wood, and the manufacture of silk and linen, are very inferior, especially those of silk, from the silk-wood and cotton, which are of great value, and are manufactured in the United States, and the manufacture of silk, and the manufacture of silk, silk, and silk, is
+  stats: 60 tokens | distinct-1 0.52 | distinct-2 0.80 | echo 0.07
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 5 ---
+  prompt      : The steam engine
+  continuation: is the main engine, and the engine is the main engine. A large cylinder is the largest engine, and is operated on the engine and the engine, which are the most important engines.
+
+The engine is the engine of the engine, and the engine is the engine of the engine. The
+  stats: 60 tokens | distinct-1 0.39 | distinct-2 0.64 | echo 0.20
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 6 ---
+  prompt      : The telegraph
+  continuation: ic and telegraphic telegraphic lines; and, in connection with the following letters, we are pleased to receive the same kind of news from the United States, which, after a short interval, will appear as the news of our departure.
+
+We also have the pleasure of communicating the following news
+  stats: 60 tokens | distinct-1 0.78 | distinct-2 0.98 | echo 0.04
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 7 ---
+  prompt      : My dearest sister,
+  continuation: I am not able to speak of myself as a friend, and I know no more about my brother's life than of my brother.
+
+I do not know, my dear Mrs. Fothergill, how to live with you. I am so much delighted with you, and so much
+  stats: 60 tokens | distinct-1 0.72 | distinct-2 0.96 | echo 0.00
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 8 ---
+  prompt      : Greetings, my friend
+  continuation: , and the others, to make a public profession of the Christian faith.
+
+I have not the remotest idea of the extent of the benefits of our religion. In the first place, I believe in the fact of the Divine presence, as well as in the second, in the third person.
+  stats: 60 tokens | distinct-1 0.67 | distinct-2 0.90 | echo 0.16
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 9 ---
+  prompt      : Her Majesty the Queen
+  continuation: of
+
+Bristol. Her Majesty's Subjects, her Majesty's Subjects, and the Government of the United States, are to be sent to this Court, to be sent to them by the
+
+King of France, and to all the Provinces of the dominions of the Great Powers of
+  stats: 60 tokens | distinct-1 0.58 | distinct-2 0.86 | echo 0.22
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 10 ---
+  prompt      : Chapter I.
+  continuation: The reader will be pleased to know that I am in a condition to take the liberty of addressing you to the same effect which has been made on the
+
+American System of Political Economy. If this doctrine should be so generally understood, it is a great evil, and, therefore, a very
+  stats: 60 tokens | distinct-1 0.85 | distinct-2 1.00 | echo 0.00
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  AGGREGATE over 10 probes: 600 tokens in 0.9s (667.0 tok/s)
+    mean distinct-1 0.582 | mean distinct-2 0.817 | mean echo 0.122 | worst probe: distinct-2 0.400 / echo 0.455
+    NOTE: at least one probe looped badly even though the average looks fine - read the probes above.
+    (sampling hides loops that greedy decoding exposes; evaluate2.py measures the greedy loop length.)
+
+Full numeric results written to /home/cro/Dev/vintage-LLM/training/eval_results/evaluate-final.json
+
+==============================================================================
+INSPECTION COMPLETE
+==============================================================================
+  This covered WHAT the checkpoint is, its lineage, and whether it is still vintage.
+  For "is it baked enough", run evaluate2.py on the same path.

scheduler/cosine_760/eval2.log

@@ -0,0 +1,54 @@
+evaluating 1 checkpoint(s) on cuda (torch.bfloat16)
+tokenizer: final
+  final ...W0814 15:07:39.942000 58837 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0814 15:07:39.958000 58837 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+
+ bake 52/100  bpb 1.2775  logic 0.65  (15.3s)
+
+report: eval2.md
+json:   eval2.json
+
+# Evaluation: final
+
+- **Parameters:** 77M
+- **Note:** no trainer_state.json in the checkpoint - tokens seen unknown
+
+## Verdict: DOUGH  (bake score 52/100)
+
+Structure is forming - real words, some grammar - but this is not a usable language model yet. It needs several times more training tokens.
+
+## Where it sits (held-out bits/byte, lower = better)
+
+```
+      3.50   untrained model (uniform noise)
+      2.00   word-salad
+      1.50   broken prose
+      1.33   500M model, ~0.7B tokens (early training)
+  --> 1.278  YOUR MODEL
+      1.19   500M model, ~4.7B tokens (undertrained but solid)
+      1.10   best sub-1B measured on this data (341M)
+      0.95   estimated sub-1B ceiling on this corpus
+```
+
+## Scores
+
+| what | raw value | points /100 | weight | plain English |
+|---|---|---|---|---|
+| Held-out loss | 1.2775 bits/byte | 61 | 0.5 | how cheaply it predicts period text it never saw — the single best training signal |
+| Logic | 0.650 acc, margin 0.040 | 36 | 0.25 | picks the *sensible* continuation over matched nonsense; 0.50 = coin-flip. normal for a good sub-1B model (Vintage1 scores 0.70-0.78) |
+| Chat readiness | 1.1623 bits/byte | 55 | 0.15 | how cheap well-formed period dialogue already is — predicts fine-tuning ease |
+| Hygiene | loop 66.2w, punct 0.00/100w | 42 | 0.1 | greedy-decoding loop length and broken punctuation |
+
+## Period boundary
+
+Clean. All 6 post-1900 trap words cost the model more than their period twins (mean shock +1.41 bits/byte, weakest pair 'atomic bomb' at +0.93). No sign of modern text in training.
+
+## See for yourself (sampled, t=0.8)
+
+> **LONDON, Tuesday. — The committee appointed to inquire into the condition of the** United States in the matter of the payment of dues of the United States government for the term of three years is hereby instructed to inquire into the manner of the sale of the money of the United States government in the United States. The Secretary of the Treasury has the right to make such application to the Treasurer of the United States as may seem best adapted for the purpose in view of the
+>
+> **A melancholy accident occurred on Thursday last at the works of Messrs. Harding and** Co. The above-mentioned event occurred on the morning of Monday the 8th instant. The ship was to proceed to the port of Pekin in the port of Pekin, where the Cruelon was to proceed. The ship was towed into the water, and the ship to land. The ship was to be delivered up to the captain of the ship for the purpose of going to a private ship. The British fleet was to sail, the remainder of the crew b
+>
+> **Brethren, the text which I have chosen for our consideration this morning is taken from** the chapter on the New Testament. We have not yet done anything for the Christian world. Our church is to be considered the best of the world, and our most important work is to do the good of the world. We have a right to demand the aid of other men, who would fain take up our abode in the world. We have the same right to demand a church, if it is not right, because a church is to be found outside
+>
+

scheduler/cosine_760/final/chat_template.jinja

@@ -0,0 +1,7 @@
+{%- for message in messages -%}
+  {{- bos_token + '<|' + message['role'] + '|>\n' + message['content'].strip() + eos_token + '\n' -}}
+{%- endfor -%}
+
+{%- if add_generation_prompt -%}
+  {{- bos_token + '<|assistant|>\n' -}}
+{%- endif -%}

scheduler/cosine_760/final/config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 3,
+  "dtype": "float32",
+  "eos_token_id": 4,
+  "head_dim": 96,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 2048,
+  "max_position_embeddings": 1024,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 8,
+  "num_hidden_layers": 8,
+  "num_key_value_heads": 4,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.14.1",
+  "use_cache": false,
+  "vocab_size": 32768
+}

scheduler/cosine_760/final/generation_config.json

@@ -0,0 +1,13 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 3,
+  "eos_token_id": [
+    4,
+    2
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 0,
+  "transformers_version": "5.14.1",
+  "use_cache": false
+}

scheduler/cosine_760/final/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bc693836047b8f4a3c04f80222a56f8b7bb924077e204df865bb366f824c64fe
+size 308341784

scheduler/cosine_760/final/tokenizer_config.json

@@ -0,0 +1,11 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|unk|>"
+}

scheduler/cosine_760/trainer_state.json

@@ -0,0 +1,663 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.01006518436771828,
+  "eval_steps": 5,
+  "global_step": 757,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 1.3296148438201165e-05,
+      "grad_norm": 5.513442516326904,
+      "learning_rate": 0.0,
+      "loss": 10.538199424743652,
+      "step": 1
+    },
+    {
+      "epoch": 0.00013296148438201165,
+      "grad_norm": 5.268533706665039,
+      "learning_rate": 0.00035999999999999997,
+      "loss": 10.136458502875435,
+      "step": 10
+    },
+    {
+      "epoch": 0.0002659229687640233,
+      "grad_norm": 1.034209132194519,
+      "learning_rate": 0.00076,
+      "loss": 8.0193359375,
+      "step": 20
+    },
+    {
+      "epoch": 0.0003988844531460349,
+      "grad_norm": 9.450255393981934,
+      "learning_rate": 0.00116,
+      "loss": 6.9860687255859375,
+      "step": 30
+    },
+    {
+      "epoch": 0.0005318459375280466,
+      "grad_norm": 5.685102462768555,
+      "learning_rate": 0.0015600000000000002,
+      "loss": 6.707894897460937,
+      "step": 40
+    },
+    {
+      "epoch": 0.0006648074219100582,
+      "grad_norm": 2.160360097885132,
+      "learning_rate": 0.00196,
+      "loss": 6.287691116333008,
+      "step": 50
+    },
+    {
+      "epoch": 0.0007977689062920698,
+      "grad_norm": 1.6388428211212158,
+      "learning_rate": 0.00236,
+      "loss": 5.944025039672852,
+      "step": 60
+    },
+    {
+      "epoch": 0.000890841945359478,
+      "eval_loss": 5.704253673553467,
+      "eval_runtime": 24.3899,
+      "eval_samples_per_second": 205.003,
+      "eval_steps_per_second": 25.625,
+      "step": 67
+    },
+    {
+      "epoch": 0.0009307303906740815,
+      "grad_norm": 1.2074735164642334,
+      "learning_rate": 0.00276,
+      "loss": 5.747131729125977,
+      "step": 70
+    },
+    {
+      "epoch": 0.0010636918750560932,
+      "grad_norm": 1.132364273071289,
+      "learning_rate": 0.00316,
+      "loss": 5.60839729309082,
+      "step": 80
+    },
+    {
+      "epoch": 0.0011966533594381047,
+      "grad_norm": 1.0884277820587158,
+      "learning_rate": 0.0035600000000000002,
+      "loss": 5.502901077270508,
+      "step": 90
+    },
+    {
+      "epoch": 0.0013296148438201164,
+      "grad_norm": 1.3878921270370483,
+      "learning_rate": 0.00396,
+      "loss": 5.371665573120117,
+      "step": 100
+    },
+    {
+      "epoch": 0.001462576328202128,
+      "grad_norm": 1.2677963972091675,
+      "learning_rate": 0.00436,
+      "loss": 5.2512672424316404,
+      "step": 110
+    },
+    {
+      "epoch": 0.0015955378125841397,
+      "grad_norm": 0.9181460738182068,
+      "learning_rate": 0.0047599999999999995,
+      "loss": 5.126389694213867,
+      "step": 120
+    },
+    {
+      "epoch": 0.0017152031485279501,
+      "eval_loss": 4.9558868408203125,
+      "eval_runtime": 23.6019,
+      "eval_samples_per_second": 211.847,
+      "eval_steps_per_second": 26.481,
+      "step": 129
+    },
+    {
+      "epoch": 0.0017284992969661514,
+      "grad_norm": 0.7834004163742065,
+      "learning_rate": 0.0051600000000000005,
+      "loss": 5.010536575317383,
+      "step": 130
+    },
+    {
+      "epoch": 0.001861460781348163,
+      "grad_norm": 0.7623927593231201,
+      "learning_rate": 0.00556,
+      "loss": 4.908084106445313,
+      "step": 140
+    },
+    {
+      "epoch": 0.0019944222657301744,
+      "grad_norm": 0.5423034429550171,
+      "learning_rate": 0.00596,
+      "loss": 4.820485305786133,
+      "step": 150
+    },
+    {
+      "epoch": 0.0021273837501121864,
+      "grad_norm": 0.5046894550323486,
+      "learning_rate": 0.00636,
+      "loss": 4.753760528564453,
+      "step": 160
+    },
+    {
+      "epoch": 0.002260345234494198,
+      "grad_norm": 0.3807763159275055,
+      "learning_rate": 0.0067599999999999995,
+      "loss": 4.694861602783203,
+      "step": 170
+    },
+    {
+      "epoch": 0.0023933067188762094,
+      "grad_norm": 0.37175047397613525,
+      "learning_rate": 0.0071600000000000006,
+      "loss": 4.652392578125,
+      "step": 180
+    },
+    {
+      "epoch": 0.0025262682032582214,
+      "grad_norm": 0.3488306999206543,
+      "learning_rate": 0.00756,
+      "loss": 4.597007369995117,
+      "step": 190
+    },
+    {
+      "epoch": 0.0025528605001346235,
+      "eval_loss": 4.578042030334473,
+      "eval_runtime": 23.0783,
+      "eval_samples_per_second": 216.654,
+      "eval_steps_per_second": 27.082,
+      "step": 192
+    },
+    {
+      "epoch": 0.002659229687640233,
+      "grad_norm": 0.3351574242115021,
+      "learning_rate": 0.00796,
+      "loss": 4.566041946411133,
+      "step": 200
+    },
+    {
+      "epoch": 0.0027921911720222444,
+      "grad_norm": 0.32959529757499695,
+      "learning_rate": 0.007995157495314521,
+      "loss": 4.521368408203125,
+      "step": 210
+    },
+    {
+      "epoch": 0.002925152656404256,
+      "grad_norm": 0.26094311475753784,
+      "learning_rate": 0.007978433817629256,
+      "loss": 4.468058013916016,
+      "step": 220
+    },
+    {
+      "epoch": 0.003058114140786268,
+      "grad_norm": 0.24577029049396515,
+      "learning_rate": 0.007949821789267853,
+      "loss": 4.430326843261719,
+      "step": 230
+    },
+    {
+      "epoch": 0.0031910756251682794,
+      "grad_norm": 0.2677822709083557,
+      "learning_rate": 0.007909411434256969,
+      "loss": 4.39563102722168,
+      "step": 240
+    },
+    {
+      "epoch": 0.003324037109550291,
+      "grad_norm": 0.27217116951942444,
+      "learning_rate": 0.00785732989852386,
+      "loss": 4.363616561889648,
+      "step": 250
+    },
+    {
+      "epoch": 0.003390517851741297,
+      "eval_loss": 4.3263163566589355,
+      "eval_runtime": 22.8668,
+      "eval_samples_per_second": 218.657,
+      "eval_steps_per_second": 27.332,
+      "step": 255
+    },
+    {
+      "epoch": 0.003456998593932303,
+      "grad_norm": 0.22698307037353516,
+      "learning_rate": 0.007793741049848255,
+      "loss": 4.329174041748047,
+      "step": 260
+    },
+    {
+      "epoch": 0.0035899600783143143,
+      "grad_norm": 0.2033531218767166,
+      "learning_rate": 0.007718844962273695,
+      "loss": 4.300073623657227,
+      "step": 270
+    },
+    {
+      "epoch": 0.003722921562696326,
+      "grad_norm": 0.18403764069080353,
+      "learning_rate": 0.007632877286600573,
+      "loss": 4.262511444091797,
+      "step": 280
+    },
+    {
+      "epoch": 0.0038558830470783374,
+      "grad_norm": 0.23616866767406464,
+      "learning_rate": 0.007536108508941533,
+      "loss": 4.247257232666016,
+      "step": 290
+    },
+    {
+      "epoch": 0.003988844531460349,
+      "grad_norm": 0.2175689935684204,
+      "learning_rate": 0.007428843099672093,
+      "loss": 4.220326995849609,
+      "step": 300
+    },
+    {
+      "epoch": 0.004121806015842361,
+      "grad_norm": 0.16665969789028168,
+      "learning_rate": 0.007311418555454189,
+      "loss": 4.190158462524414,
+      "step": 310
+    },
+    {
+      "epoch": 0.00422817520334797,
+      "eval_loss": 4.1631269454956055,
+      "eval_runtime": 22.8668,
+      "eval_samples_per_second": 218.658,
+      "eval_steps_per_second": 27.332,
+      "step": 318
+    },
+    {
+      "epoch": 0.004254767500224373,
+      "grad_norm": 0.183854341506958,
+      "learning_rate": 0.007184204337346831,
+      "loss": 4.169167709350586,
+      "step": 320
+    },
+    {
+      "epoch": 0.004387728984606384,
+      "grad_norm": 0.15049099922180176,
+      "learning_rate": 0.007047600708344912,
+      "loss": 4.153661727905273,
+      "step": 330
+    },
+    {
+      "epoch": 0.004520690468988396,
+      "grad_norm": 0.18056534230709076,
+      "learning_rate": 0.006902037474003746,
+      "loss": 4.125600051879883,
+      "step": 340
+    },
+    {
+      "epoch": 0.004653651953370408,
+      "grad_norm": 0.18494488298892975,
+      "learning_rate": 0.00674797263011178,
+      "loss": 4.109599685668945,
+      "step": 350
+    },
+    {
+      "epoch": 0.004786613437752419,
+      "grad_norm": 0.14171533286571503,
+      "learning_rate": 0.006585890921666403,
+      "loss": 4.102255249023438,
+      "step": 360
+    },
+    {
+      "epoch": 0.004919574922134431,
+      "grad_norm": 0.14613653719425201,
+      "learning_rate": 0.00641630231768684,
+      "loss": 4.084948348999023,
+      "step": 370
+    },
+    {
+      "epoch": 0.005052536406516443,
+      "grad_norm": 0.13041161000728607,
+      "learning_rate": 0.0062397404066629495,
+      "loss": 4.060393905639648,
+      "step": 380
+    },
+    {
+      "epoch": 0.005065832554954643,
+      "eval_loss": 4.050504684448242,
+      "eval_runtime": 22.8575,
+      "eval_samples_per_second": 218.746,
+      "eval_steps_per_second": 27.343,
+      "step": 381
+    },
+    {
+      "epoch": 0.005185497890898454,
+      "grad_norm": 0.14684684574604034,
+      "learning_rate": 0.006056760717688429,
+      "loss": 4.043218612670898,
+      "step": 390
+    },
+    {
+      "epoch": 0.005318459375280466,
+      "grad_norm": 0.1349787712097168,
+      "learning_rate": 0.005867938972560758,
+      "loss": 4.035343933105469,
+      "step": 400
+    },
+    {
+      "epoch": 0.005451420859662477,
+      "grad_norm": 0.11209040135145187,
+      "learning_rate": 0.005673869274347425,
+      "loss": 4.007895278930664,
+      "step": 410
+    },
+    {
+      "epoch": 0.005584382344044489,
+      "grad_norm": 0.13373538851737976,
+      "learning_rate": 0.00547516223811788,
+      "loss": 3.9965129852294923,
+      "step": 420
+    },
+    {
+      "epoch": 0.005717343828426501,
+      "grad_norm": 0.12187955528497696,
+      "learning_rate": 0.005272443069722628,
+      "loss": 3.9868213653564455,
+      "step": 430
+    },
+    {
+      "epoch": 0.005850305312808512,
+      "grad_norm": 0.1229710504412651,
+      "learning_rate": 0.005066349598664307,
+      "loss": 3.9734012603759767,
+      "step": 440
+    },
+    {
+      "epoch": 0.005903489906561317,
+      "eval_loss": 3.9610915184020996,
+      "eval_runtime": 22.8662,
+      "eval_samples_per_second": 218.663,
+      "eval_steps_per_second": 27.333,
+      "step": 444
+    },
+    {
+      "epoch": 0.005983266797190524,
+      "grad_norm": 0.12179016321897507,
+      "learning_rate": 0.004857530271250106,
+      "loss": 3.962753677368164,
+      "step": 450
+    },
+    {
+      "epoch": 0.006116228281572536,
+      "grad_norm": 0.11934109032154083,
+      "learning_rate": 0.004646642110339784,
+      "loss": 3.939118576049805,
+      "step": 460
+    },
+    {
+      "epoch": 0.006249189765954547,
+      "grad_norm": 0.11726455390453339,
+      "learning_rate": 0.004434348648108665,
+      "loss": 3.93060188293457,
+      "step": 470
+    },
+    {
+      "epoch": 0.006382151250336559,
+      "grad_norm": 0.09656476974487305,
+      "learning_rate": 0.0042213178383299485,
+      "loss": 3.921614074707031,
+      "step": 480
+    },
+    {
+      "epoch": 0.006515112734718571,
+      "grad_norm": 0.09665072709321976,
+      "learning_rate": 0.004008219954745005,
+      "loss": 3.910228729248047,
+      "step": 490
+    },
+    {
+      "epoch": 0.006648074219100582,
+      "grad_norm": 0.09524396061897278,
+      "learning_rate": 0.0037957254821342165,
+      "loss": 3.9034732818603515,
+      "step": 500
+    },
+    {
+      "epoch": 0.00674114725816799,
+      "eval_loss": 3.8828694820404053,
+      "eval_runtime": 22.8816,
+      "eval_samples_per_second": 218.517,
+      "eval_steps_per_second": 27.315,
+      "step": 507
+    },
+    {
+      "epoch": 0.006781035703482594,
+      "grad_norm": 0.09537574648857117,
+      "learning_rate": 0.0035845030067237996,
+      "loss": 3.8968971252441404,
+      "step": 510
+    },
+    {
+      "epoch": 0.006913997187864606,
+      "grad_norm": 0.09708276391029358,
+      "learning_rate": 0.003375217112566186,
+      "loss": 3.87552490234375,
+      "step": 520
+    },
+    {
+      "epoch": 0.007046958672246617,
+      "grad_norm": 0.10611076653003693,
+      "learning_rate": 0.003168526290512718,
+      "loss": 3.861489105224609,
+      "step": 530
+    },
+    {
+      "epoch": 0.007179920156628629,
+      "grad_norm": 0.09043016284704208,
+      "learning_rate": 0.002965080866357815,
+      "loss": 3.8511241912841796,
+      "step": 540
+    },
+    {
+      "epoch": 0.007312881641010641,
+      "grad_norm": 0.08922528475522995,
+      "learning_rate": 0.002765520954673425,
+      "loss": 3.846691131591797,
+      "step": 550
+    },
+    {
+      "epoch": 0.007445843125392652,
+      "grad_norm": 0.08538611978292465,
+      "learning_rate": 0.002570474444771763,
+      "loss": 3.846059799194336,
+      "step": 560
+    },
+    {
+      "epoch": 0.007578804609774664,
+      "grad_norm": 0.09066513925790787,
+      "learning_rate": 0.0023805550251332418,
+      "loss": 3.826742172241211,
+      "step": 570
+    },
+    {
+      "epoch": 0.007578804609774664,
+      "eval_loss": 3.8216655254364014,
+      "eval_runtime": 22.9116,
+      "eval_samples_per_second": 218.23,
+      "eval_steps_per_second": 27.279,
+      "step": 570
+    },
+    {
+      "epoch": 0.007711766094156675,
+      "grad_norm": 0.0921698808670044,
+      "learning_rate": 0.002196360252515458,
+      "loss": 3.811664581298828,
+      "step": 580
+    },
+    {
+      "epoch": 0.007844727578538687,
+      "grad_norm": 0.0789179727435112,
+      "learning_rate": 0.0020184696718185723,
+      "loss": 3.8138885498046875,
+      "step": 590
+    },
+    {
+      "epoch": 0.007977689062920698,
+      "grad_norm": 0.07974446564912796,
+      "learning_rate": 0.0018474429926226303,
+      "loss": 3.7982933044433596,
+      "step": 600
+    },
+    {
+      "epoch": 0.00811065054730271,
+      "grad_norm": 0.08164919912815094,
+      "learning_rate": 0.0016838183281341638,
+      "loss": 3.796610641479492,
+      "step": 610
+    },
+    {
+      "epoch": 0.008243612031684722,
+      "grad_norm": 0.08382760733366013,
+      "learning_rate": 0.001528110502082944,
+      "loss": 3.7793228149414064,
+      "step": 620
+    },
+    {
+      "epoch": 0.008376573516066733,
+      "grad_norm": 0.07584033906459808,
+      "learning_rate": 0.0013808094288960784,
+      "loss": 3.775922393798828,
+      "step": 630
+    },
+    {
+      "epoch": 0.008429758109819538,
+      "eval_loss": 3.7764945030212402,
+      "eval_runtime": 22.9168,
+      "eval_samples_per_second": 218.181,
+      "eval_steps_per_second": 27.273,
+      "step": 634
+    },
+    {
+      "epoch": 0.008509535000448746,
+      "grad_norm": 0.07796747237443924,
+      "learning_rate": 0.0012423785722459621,
+      "loss": 3.7791465759277343,
+      "step": 640
+    },
+    {
+      "epoch": 0.008642496484830757,
+      "grad_norm": 0.07175382971763611,
+      "learning_rate": 0.0011132534868220968,
+      "loss": 3.7700267791748048,
+      "step": 650
+    },
+    {
+      "epoch": 0.008775457969212768,
+      "grad_norm": 0.07658028602600098,
+      "learning_rate": 0.0009938404479149102,
+      "loss": 3.7676132202148436,
+      "step": 660
+    },
+    {
+      "epoch": 0.00890841945359478,
+      "grad_norm": 0.07290265709161758,
+      "learning_rate": 0.0008845151731233707,
+      "loss": 3.757847213745117,
+      "step": 670
+    },
+    {
+      "epoch": 0.009041380937976792,
+      "grad_norm": 0.07412706315517426,
+      "learning_rate": 0.0007856216402084315,
+      "loss": 3.7475379943847655,
+      "step": 680
+    },
+    {
+      "epoch": 0.009174342422358803,
+      "grad_norm": 0.07130566239356995,
+      "learning_rate": 0.0006974710048117232,
+      "loss": 3.7532440185546876,
+      "step": 690
+    },
+    {
+      "epoch": 0.00926741546142621,
+      "eval_loss": 3.749366044998169,
+      "eval_runtime": 22.9896,
+      "eval_samples_per_second": 217.49,
+      "eval_steps_per_second": 27.186,
+      "step": 697
+    },
+    {
+      "epoch": 0.009307303906740816,
+      "grad_norm": 0.06786452233791351,
+      "learning_rate": 0.0006203406214448,
+      "loss": 3.74536018371582,
+      "step": 700
+    },
+    {
+      "epoch": 0.009440265391122827,
+      "grad_norm": 0.07180707901716232,
+      "learning_rate": 0.0005544731708292279,
+      "loss": 3.743540954589844,
+      "step": 710
+    },
+    {
+      "epoch": 0.009573226875504838,
+      "grad_norm": 0.06881807744503021,
+      "learning_rate": 0.0005000758963332508,
+      "loss": 3.741817855834961,
+      "step": 720
+    },
+    {
+      "epoch": 0.00970618835988685,
+      "grad_norm": 0.07049064338207245,
+      "learning_rate": 0.000457319951907489,
+      "loss": 3.7454132080078124,
+      "step": 730
+    },
+    {
+      "epoch": 0.009839149844268862,
+      "grad_norm": 0.06726963818073273,
+      "learning_rate": 0.0004263398635712801,
+      "loss": 3.745738983154297,
+      "step": 740
+    },
+    {
+      "epoch": 0.009972111328650873,
+      "grad_norm": 0.06847967952489853,
+      "learning_rate": 0.00040723310614405915,
+      "loss": 3.730845260620117,
+      "step": 750
+    },
+    {
+      "epoch": 0.01006518436771828,
+      "step": 757,
+      "total_flos": 6.181607017178726e+16,
+      "train_loss": 4.4425654001878465,
+      "train_runtime": 3601.9689,
+      "train_samples_per_second": 54.015,
+      "train_steps_per_second": 0.211
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 760,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 5,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 6.181607017178726e+16,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

scheduler/greedy/REPORT.md

@@ -0,0 +1,42 @@
+# Experiment: greedy — GreedyLR adaptive scheduler
+
+**Wall-clock:** 59 min (hit the 760-step horizon just before the time stop) · **Recipe:** best known (sdpa + bf16 mixed + muonq), adaptive LR: start 1e-3, GreedyLR raises on eval-loss improvement / lowers on plateau (factor 0.9, patience 0, threshold 3e-3 abs, bounds 4e-4…1.6e-2), evals every 20 steps on 1024 samples
+
+## Plumbing this run needed (first metric-driven scheduler in the project)
+
+1. `metric_for_best_model = "loss"` — required; the Trainer steps greedy/plateau schedulers on this eval metric and crashes without it. Wired through `base_train.py` as a new config key.
+2. **`warmup_steps` is silently ignored** — `get_scheduler` passes only `lr_scheduler_kwargs` to greedy. The low 1e-3 starting LR + raise-on-improvement acted as the de-facto warmup (cold start was stable: grad max 5.51, the calmest of any hot run).
+3. Eval cadence 20 steps / 1024 samples (vs 5 min / 5000 elsewhere) — greedy only gets to act when an eval runs; 38 decisions this run vs ~11 on the standard cadence.
+
+## Result: the adaptive LR *found* the tuned peak on its own — then lost for lack of an anneal
+
+| metric | greedy | constant 8e-3 reference | wsd_20 (series winner) |
+|---|---|---|---|
+| optimizer steps | 760 | 759 | 730 |
+| final eval loss / ppl | 3.9267 / 50.7 (1024-sample eval — noisier subset) | 3.8583 / 47.4 | 3.8023 / 44.8 |
+| bake score | 44.01/100 (DOUGH) | 46.83 | **55.61** |
+| held-out bits/byte | 1.3422 | 1.3114 | **1.2769** |
+| chat bits/byte | 1.2188 | 1.1914 | **1.1571** |
+| logic accuracy | 0.575 (+0.0085) | 0.600 (+0.018) | **0.650 (+0.0242)** |
+| hygiene (greedy loop) | 56.2/100 (45.5w) | 47.7 (59.8w) | **79.2 (23.8w)** |
+| grad norms | **0.25–5.51** (calmest hot run) | 0.09–9.45 | 0.06–9.45 |
+| peak GPU | 7.2 GB | 7.2 GB | 7.2 GB |
+
+**The LR trajectory is the interesting part:** 1e-3 → steady climb → oscillating plateau at **6.6–7.4e-3** from ~step 410 onward, never touching the 1.6e-2 cap, ending at 7.4e-3. With zero knowledge of our LR series, the eval-loss signal alone steered it to within ~10% of our hand-tuned 8e-3 peak — independent confirmation that 8e-3-ish is the right constant-phase LR for this setup. It also spent ~350 steps getting there (a slow de-facto warmup), and 300+ steps holding near 7e-3 where improvements hovered around the 3e-3 threshold.
+
+**Why it still lost:** GreedyLR is a *find-the-workable-LR* policy with no concept of a budget ending — it finished the run hot at 7.4e-3. The scheduler series' central finding is that the final anneal is worth +5–9 bake points, and greedy never anneals. It beat nothing that decays, and even trails the constant-8e-3 run (it averaged a lower LR for most of the hour while paying the same no-anneal penalty).
+
+## Eval loss curve (every 60 steps; 1024-sample evals — not directly comparable to the 5000-sample curves of other runs)
+
+| step | eval loss | | step | eval loss |
+|---|---|---|---|---|
+| 60 | 5.6145 | | 480 | 4.1681 |
+| 120 | 4.9678 | | 540 | 4.0644 |
+| 180 | 4.6068 | | 600 | 4.0330 |
+| 240 | 4.4325 | | 660 | 3.9626 |
+| 300 | 4.3343 | | 720 | 3.9412 |
+| 360 | 4.3033 | | 760 | 3.9267 |
+
+## Verdict
+
+**A successful experiment about an unsuccessful scheduler.** As a quality recipe it loses to every decaying schedule; as an instrument it independently validated our tuned peak LR (plateaued at 6.6–7.4e-3 vs our 8e-3). If revisited: pair greedy's adaptive stable phase with a forced terminal decay (e.g. greedy until 80% of budget, then linear-to-0) — but wsd_20 already does the simple version of that. See `../COMPARISON.md`.

scheduler/greedy/config.toml

@@ -0,0 +1,93 @@
+# Scheduler series — new best recipe (sdpa + bf16 mixed + muonq), adaptive-LR arm.
+# This run: GreedyLR — metric-driven; LR steps ONLY on evaluation, so evals run every 20 steps (1024 samples).
+# NOTE: warmup_steps is IGNORED by greedy (get_scheduler passes only lr_scheduler_kwargs); the low starting
+# learning_rate (1e-3) + greedy raising on improvement act as the de-facto warmup (sim: reaches 8e-3 ~step 400).
+model_config = "../model.toml"
+
+[data]
+# Data files (supports glob patterns or lists of files)
+train_files = ["~/Dev/vintage-LLM/training/data/train_*.bin"]
+valid_files = ["~/Dev/vintage-LLM/training/data/valid_*.bin"]
+
+# Sequence length for each dataset sample
+# Must be <= model.max_position_embeddings
+max_seq_length = 1024
+
+# Tokenizer (must match the one used to create .bin files)
+tokenizer = "~/Dev/vintage-LLM/tokenizers/t-v3/"
+
+[training]
+# Training duration
+num_train_epochs = 1
+max_steps = 760                    # scheduler horizon = achievable steps/hour (was -1: full epoch, so decay never happened)
+
+# Optional: Stop training after this many minutes of wall-clock time.
+max_train_minutes = 60
+
+# Output directories
+output_dir = "./"
+final_model_dir = "./final"
+
+# Batch sizes
+# Effective batch size = num_devices × per_device_train_batch_size × gradient_accumulation_steps
+per_device_train_batch_size = 8    # batch size per GPU (8x1024 tokens fits 16GB with headroom for eval)
+per_device_eval_batch_size  = 8
+gradient_accumulation_steps = 32   # accumulate gradients over multiple mini-batches before updating the params
+
+# Optimizer: MuonQ (4-bit quantized Muon, ../../MuonQ repo) — wired through
+# optimizer_cls_and_kwargs in base_train.py. 2D hidden weights get Muon
+# (Newton-Schulz 5, 4-bit mu-law-companded momentum, rank-16 residual
+# factorization); embeddings/head/norms get its internal AdamW backup.
+# Muon/quantization knobs (muon_*, muonq_*) default to the repo's own
+# llama-60m "muonq" recipe in base_train.py.
+optim = "muonq"
+
+# Optimizer settings — lr/wd from MuonQ's llama-60m recipe (Muon takes
+# spectral-norm-scaled steps; do not copy the AdamW 5e-4 here)
+learning_rate = 1e-3               # starting LR only — greedy climbs it (bounded by max_lr)
+weight_decay = 0.1
+adam_beta1 = 0.9
+adam_beta2 = 0.95
+max_grad_norm = 1.0
+
+# Learning rate scheduler
+lr_scheduler_type    = "greedy"                # GreedyLR: raise LR while eval loss improves, lower on plateau
+# patience 0 = act on every eval; threshold = min improvement (abs) to count; bounds 4e-4..1.6e-2
+lr_scheduler_kwargs  = { factor = 0.9, patience = 0, threshold = 3e-3, threshold_mode = "abs", min_lr = 4e-4, max_lr = 1.6e-2, verbose = true }
+metric_for_best_model = "loss"     # greedy steps on eval_loss (wired through base_train.py)
+warmup_steps = 200                 # 1-hour series standard (muonq-v2's 400 was a 2-hour recipe)
+
+# Precision (choose based on your GPU)
+bf16 = true                        # bfloat16 autocast + fp32 master weights (the dtype-series safe recipe)
+fp16 = false
+
+# Performance
+torch_compile = false
+gradient_checkpointing = false     # trade compute for memory (enables larger models)
+neftune_noise_alpha = 0.0          # add noise during training to improve model performance (fine-tuning)
+
+# Checkpointing
+# save_strategy can be "steps", "epoch", or "minutes" (wall-clock time).
+save_strategy = "minutes"
+save_steps = 5                    # checkpoint every X minutes
+save_total_limit = 3              # keep only X most recent checkpoints
+
+# Evaluation
+# eval_strategy can be "steps", "epoch", or "minutes" (wall-clock time).
+eval_strategy = "steps"
+eval_steps = 20                  # greedy only adjusts LR on eval -> needs frequent evals
+# Evaluate on a fixed random subset of the validation set (0 = use all)
+max_eval_samples = 1024            # keep frequent evals cheap (~4s); noisier curve, final evals unaffected
+
+# Logging
+logging_strategy = "steps"
+logging_steps = 10                 # log every 10 steps
+logging_first_step = true          # log the first step
+
+# Reproducibility
+seed = 42
+
+# Performance
+dataloader_num_workers = 2         # number of data loading threads
+dataloader_prefetch_factor = 2     # batches prefetched per worker
+dataloader_pin_memory = true       # pin memory for faster GPU transfer

scheduler/greedy/eval1.log

@@ -0,0 +1,213 @@
+W0814 16:07:22.252000 71987 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0814 16:07:22.274000 71987 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+
+==============================================================================
+CHECKPOINT INSPECTION
+==============================================================================
+  checkpoint : final
+  tokenizer  : final
+  device     : cuda   dtype: torch.bfloat16   seed: 1337
+
+
+==============================================================================
+SUITE: MODEL INFO AND TRAINING LINEAGE
+==============================================================================
+  checkpoint              : final
+  model_type              : llama
+  architecture            : LlamaForCausalLM
+  device                  : cuda
+  dtype                   : torch.bfloat16
+  parameters              : 77,083,392 (77.08M)
+  disk size               : 308 MB
+  layers / hidden / heads : 8 / 768 / 8 (KV heads: 4)
+  vocab (cfg / tokenizer) : 32768 / 32752
+  context length          : 1024
+  chat template           : True
+  training lineage        : base pretraining, ctx 1024
+  embedding mean norm     : 3.2496   <- typical row length of the embedding matrix; very small or huge values signal trouble
+  embedding mean cosine   : 0.1120   <- avg cosine between random embedding rows. Descriptive only: it starts near 0.0
+                            because RANDOM init is near-orthogonal, and it GROWS as the model trains.
+                            Near 0 therefore does NOT mean "healthy" - it can mean "barely trained".
+  (note: config vocab padded by 16 unused rows - harmless)
+
+==============================================================================
+SUITE: PERIOD FIDELITY ON FIXED PROBE SENTENCES
+==============================================================================
+  Scoring 20 sentences (10 historical, 10 modern)
+  The MODERN/HISTORICAL ratio is the output that means something. The absolute
+  numbers come from ~20 sentences: too noisy to rank checkpoints (use evaluate2.py).
+
+  OVERALL (270 tokens)
+    perplexity            : 399.51   <- 'average branching factor'; lower = less surprised = better fit to this text
+    mean token prob       : 0.0563   <- average confidence on the actual next word
+    median token prob     : 0.0041   <- typical confidence (robust to a few very bad tokens)
+    worst-token prob (min): 7.0e-08   <- the single most surprising word; tiny values = the model really didn't expect it
+    10th-percentile prob  : 1.2e-05   <- confidence on the model's worst 10% of guesses
+    low-confidence tokens : 60.00%   <- share of tokens predicted with <1% probability (red flags)
+    mean entropy          : 5.001 nats   <- how spread out the predictions are; ~0 = very certain, higher = hedging
+
+  GROUP COMPARISON
+    group        sentences    perplexity   bits/byte
+    historical          10        141.24       1.347
+    modern              10        953.62       2.046
+    modern/historical ratio: 6.75   <- >1 means period text is easier for the model than modern text (desired for a vintage model with a ~1900 knowledge cutoff)
+    (bits/byte is the tokenizer-independent version - the only column here you may
+     compare between models that use DIFFERENT tokenizers.)
+
+  PER-SENTENCE DETAIL (sorted worst-first)
+    [modern    ] ppl= 7408.34  minp=2.0e-07  The engineering team rewrote the whole backend micro
+    [modern    ] ppl= 2671.39  minp=1.5e-06  Robots on the automated assembly line handle the ent
+    [modern    ] ppl= 2036.79  minp=2.0e-06  Small retail startups now run almost all of their bu
+    [modern    ] ppl= 1301.80  minp=3.2e-06  She is finishing a graduate degree in machine learni
+    [modern    ] ppl= 1092.45  minp=9.4e-07  Three hours stuck in traffic in the pouring rain mad
+    [modern    ] ppl=  978.99  minp=3.2e-07  Members of the European Union parliament voted on th
+    [modern    ] ppl=  974.77  minp=7.0e-08  Millions of shoppers queue outside the store because
+    [modern    ] ppl=  396.50  minp=1.0e-06  She gave up her whole weekend to help me move apartm
+    [historical] ppl=  330.91  minp=1.3e-05  The keeper fed the great serpent which the naturalis
+    [modern    ] ppl=  310.68  minp=3.0e-06  After years of hiding it from everyone at work, he t
+    [historical] ppl=  210.57  minp=7.7e-06  The Lord provideth for all his creatures, giving the
+    [modern    ] ppl=  205.29  minp=1.9e-05  I switched to a plant-based diet last year and I no 
+    [historical] ppl=  177.84  minp=1.2e-05  The labouring poor of this parish are reduced to gre
+    [historical] ppl=  168.82  minp=2.7e-05  The mountain rose above the valley in a silence sole
+    [historical] ppl=  154.45  minp=6.2e-05  The weavers at their looms are employed in the wooll
+    [historical] ppl=  153.93  minp=1.2e-04  The merchants of the port have grown rich upon their
+    [historical] ppl=  141.63  minp=3.2e-06  He gave up his fortune to the patient study of natur
+    [historical] ppl=  116.44  minp=1.3e-04  He drew a distinction so fine and over-scrupulous th
+    [historical] ppl=   74.65  minp=1.9e-04  The ballroom was filled with dancing and laughter, a
+    [historical] ppl=   52.16  minp=1.5e-04  Her Majesty was pleased to summon the Lords and Comm
+
+==============================================================================
+SUITE: DIACHRONIC WORD-SENSE SEPARATION
+==============================================================================
+  Extracting contextual embeddings for 10 shifted words in period AND modern sentences.
+  Every probe word sits LAST in its sentence, so the causal model has actually read
+  the disambiguating context before it represents the word.
+
+  SENSE SEPARATION (cosine similarity of the SAME word, period vs modern sentence)
+    lower value = the model represents the two senses differently (good sign for a
+    period model); ~1.0 = it treats them as the same word regardless of context
+    python        : +0.415
+    want          : +0.482
+    awful         : +0.587
+    nice          : +0.676
+    manufacture   : +0.708
+    gay           : +0.725
+    meat          : +0.740
+    parliament    : +0.745
+    science       : +0.790
+    commerce      : +0.809
+    MEAN          : +0.668   <- near 1.0 = model ignores period vs modern usage; lower = senses differ
+
+  PAIRWISE COSINE SIMILARITY (period sentences, first 5 words)
+    these are CONTEXTUAL vectors, so they reflect the sentences as much as the words;
+    read the spread, not individual pairs. High mean = representations collapsed.
+                         gay      awful       nice       meat       want
+    gay                1.000      0.367      0.440      0.355      0.345
+    awful                         1.000      0.368      0.277      0.262
+    nice                                     1.000      0.354      0.369
+    meat                                                1.000      0.467
+    want                                                           1.000
+    all 45 pairs: mean=+0.381 std=0.109   <- healthy spread is good
+
+  NEAREST NEIGHBOURS inside the probe-word set (period sentences)
+    gay           : nice (+0.44), science (+0.44), commerce (+0.42), parliament (+0.42), awful (+0.37)
+    awful         : nice (+0.37), gay (+0.37), parliament (+0.32), science (+0.31), meat (+0.28)
+    nice          : gay (+0.44), science (+0.42), want (+0.37), awful (+0.37), commerce (+0.37)
+    meat          : parliament (+0.50), science (+0.48), commerce (+0.48), want (+0.47), manufacture (+0.40)
+    want          : commerce (+0.57), manufacture (+0.49), science (+0.49), parliament (+0.48), meat (+0.47)
+
+==============================================================================
+SUITE: GENERATION PROBES
+==============================================================================
+
+  --- Probe 1 ---
+  prompt      : The history of the world is
+  continuation: too often the subject of a few words; and the first of these is a little more than a mere coincidence of history. It has been said that the present age is the most remarkable of the present age. But what is to be said of the future is, that the present age is a little
+  stats: 60 tokens | distinct-1 0.61 | distinct-2 0.85 | echo 0.07
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 2 ---
+  prompt      : What is God? God is
+  continuation: a man, he is a man. He is a man, but it is a man.
+
+We are always ready for the use of God. We must have a right to be able to do the will of God.
+
+We are always ready to give God's will, to make
+  stats: 60 tokens | distinct-1 0.60 | distinct-2 0.81 | echo 0.10
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 3 ---
+  prompt      : In the year of our Lord eighteen hundred and
+  continuation: eighty, in the year of his life, we may hope that his labors in this field will ever continue to increase.
+
+The great work of the world's missionary enterprise has been done in a manner that will be a great blessing to those who have not a desire to know of its work.
+  stats: 60 tokens | distinct-1 0.75 | distinct-2 1.00 | echo 0.00
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 4 ---
+  prompt      : The manufacture of cotton
+  continuation: , and the manufacture of the article of export.
+
+The manufacture of the cotton from this country is to a considerable extent in the hands of the public. The price of the produce is about the same, and the produce exported to any country is not so much as the price of its produce,
+  stats: 60 tokens | distinct-1 0.59 | distinct-2 0.85 | echo 0.11
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 5 ---
+  prompt      : The steam engine
+  continuation: is suspended from the water, and, in the opinion, it is a matter of the highest moment, whether to the engineer, or the engineer, to the engineer.
+
+If, on the contrary, the steam boat was a very good conductor, and had the advantage of the machinery, it
+  stats: 60 tokens | distinct-1 0.68 | distinct-2 0.93 | echo 0.15
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 6 ---
+  prompt      : The telegraph
+  continuation: ic and telegraphic telegraphic.
+
+The United States' Government, with the exception of a very few, is not, I think, without reason, without a knowledge of the United States.
+
+The United States, with the exception of the United States, have not yet arrived at a
+  stats: 60 tokens | distinct-1 0.66 | distinct-2 0.84 | echo 0.07
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 7 ---
+  prompt      : My dearest sister,
+  continuation: a great and beautiful woman.
+
+I have been talking with her, but I think that I am as much of a poet as any woman in the world. I have a mind to write, but I am afraid there is something, which I would rather write for you. It was so very
+  stats: 60 tokens | distinct-1 0.79 | distinct-2 0.94 | echo 0.02
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 8 ---
+  prompt      : Greetings, my friend
+  continuation: , the Prince of Wales, I am not so much the friend of the Prince of Wales as he is, but, like myself, the most intimate friend of the Prince of Wales, I should not feel much interest in the Prince's attentions to his Majesty, as I am inclined to believe that the
+  stats: 60 tokens | distinct-1 0.60 | distinct-2 0.83 | echo 0.06
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 9 ---
+  prompt      : Her Majesty the Queen
+  continuation: 's
+
+Majesty's Government's Council is, that, in case of any serious disturbance of the Government, they will be, in consequence of the Government's being, a very strong and respectable member, and of course, that he will be able to make an immediate appeal to them.
+  stats: 60 tokens | distinct-1 0.80 | distinct-2 0.98 | echo 0.04
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 10 ---
+  prompt      : Chapter I.
+  continuation: The reader will perhaps ask whether the author would give him a more satisfactory account of the nature and history of the people than it is possible for a writer to convey.
+
+The whole book is as yet incomplete as it is incomplete, and we cannot but hope that the reader, who has not the
+  stats: 60 tokens | distinct-1 0.78 | distinct-2 0.96 | echo 0.02
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  AGGREGATE over 10 probes: 600 tokens in 0.8s (732.7 tok/s)
+    mean distinct-1 0.687 | mean distinct-2 0.899 | mean echo 0.064 | worst probe: distinct-2 0.809 / echo 0.149
+    (sampling hides loops that greedy decoding exposes; evaluate2.py measures the greedy loop length.)
+
+Full numeric results written to /home/cro/Dev/vintage-LLM/training/eval_results/evaluate-final.json
+
+==============================================================================
+INSPECTION COMPLETE
+==============================================================================
+  This covered WHAT the checkpoint is, its lineage, and whether it is still vintage.
+  For "is it baked enough", run evaluate2.py on the same path.

scheduler/greedy/eval2.log

@@ -0,0 +1,54 @@
+evaluating 1 checkpoint(s) on cuda (torch.bfloat16)
+tokenizer: final
+  final ...W0814 16:07:28.267000 72081 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0814 16:07:28.283000 72081 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+
+ bake 44/100  bpb 1.3422  logic 0.57  (15.2s)
+
+report: eval2.md
+json:   eval2.json
+
+# Evaluation: final
+
+- **Parameters:** 77M
+- **Note:** no trainer_state.json in the checkpoint - tokens seen unknown
+
+## Verdict: DOUGH  (bake score 44/100)
+
+Structure is forming - real words, some grammar - but this is not a usable language model yet. It needs several times more training tokens.
+
+## Where it sits (held-out bits/byte, lower = better)
+
+```
+      3.50   untrained model (uniform noise)
+      2.00   word-salad
+      1.50   broken prose
+  --> 1.342  YOUR MODEL
+      1.33   500M model, ~0.7B tokens (early training)
+      1.19   500M model, ~4.7B tokens (undertrained but solid)
+      1.10   best sub-1B measured on this data (341M)
+      0.95   estimated sub-1B ceiling on this corpus
+```
+
+## Scores
+
+| what | raw value | points /100 | weight | plain English |
+|---|---|---|---|---|
+| Held-out loss | 1.3422 bits/byte | 54 | 0.5 | how cheaply it predicts period text it never saw — the single best training signal |
+| Logic | 0.575 acc, margin 0.009 | 18 | 0.25 | picks the *sensible* continuation over matched nonsense; 0.50 = coin-flip. weak - style without understanding |
+| Chat readiness | 1.2188 bits/byte | 46 | 0.15 | how cheap well-formed period dialogue already is — predicts fine-tuning ease |
+| Hygiene | loop 45.5w, punct 0.13/100w | 56 | 0.1 | greedy-decoding loop length and broken punctuation |
+
+## Period boundary
+
+Clean. All 6 post-1900 trap words cost the model more than their period twins (mean shock +1.63 bits/byte, weakest pair 'atomic bomb' at +1.04). No sign of modern text in training.
+
+## See for yourself (sampled, t=0.8)
+
+> **LONDON, Tuesday. — The committee appointed to inquire into the condition of the** work, and see if it could have been necessary to do so. The committee has no doubt the great benefit of having the enterprise a good one, and it is not only a matter of great importance, but a matter of great national interest, and it is quite certain that the project would have to be prosecuted without the slightest delay. A great many of the members of the Committee, and with so much reason, hav
+>
+> **A melancholy accident occurred on Thursday last at the works of Messrs. Harding and** Co. and the above mentioned. We have not an allusion to the work on the same side of the river, as we might expect to hear from the papers. It was a slight effort to ascertain the nature of the work. The last part of the work is to be done in a manner equally appropriate to the reader. A great part of it is the work of the writer; but it is not the work of the architect. A short description of the
+>
+> **Brethren, the text which I have chosen for our consideration this morning is taken from** the work of the school and from the work which the teacher takes to be the work of the Christian Church, and to whom it is a great joy, when in the morning the preacher takes leave to make the journey. The day for the morning has been the day for the preacher's sake. The work of the Church has been done by him, and the world is aflame with his glory. There is a time when the minister's work is don
+>
+

scheduler/greedy/final/chat_template.jinja

@@ -0,0 +1,7 @@
+{%- for message in messages -%}
+  {{- bos_token + '<|' + message['role'] + '|>\n' + message['content'].strip() + eos_token + '\n' -}}
+{%- endfor -%}
+
+{%- if add_generation_prompt -%}
+  {{- bos_token + '<|assistant|>\n' -}}
+{%- endif -%}

scheduler/greedy/final/config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 3,
+  "dtype": "float32",
+  "eos_token_id": 4,
+  "head_dim": 96,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 2048,
+  "max_position_embeddings": 1024,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 8,
+  "num_hidden_layers": 8,
+  "num_key_value_heads": 4,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.14.1",
+  "use_cache": false,
+  "vocab_size": 32768
+}

scheduler/greedy/final/generation_config.json

@@ -0,0 +1,13 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 3,
+  "eos_token_id": [
+    4,
+    2
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 0,
+  "transformers_version": "5.14.1",
+  "use_cache": false
+}

scheduler/greedy/final/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:797f00aeaf1c511832b19bffb1f1ff6fb014d7dede1c0ac24b770b6994c420b3
+size 308341784

scheduler/greedy/final/tokenizer_config.json

@@ -0,0 +1,11 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|unk|>"
+}

scheduler/greedy/trainer_state.json

@@ -0,0 +1,886 @@
+{
+  "best_global_step": null,
+  "best_metric": 3.9166951179504395,
+  "best_model_checkpoint": null,
+  "epoch": 0.010105072813032885,
+  "eval_steps": 20,
+  "global_step": 760,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 1.3296148438201165e-05,
+      "grad_norm": 5.513442516326904,
+      "learning_rate": 0.001,
+      "loss": 10.538199424743652,
+      "step": 1
+    },
+    {
+      "epoch": 0.00013296148438201165,
+      "grad_norm": 2.141404390335083,
+      "learning_rate": 0.001,
+      "loss": 7.99212646484375,
+      "step": 10
+    },
+    {
+      "epoch": 0.0002659229687640233,
+      "grad_norm": 5.429156303405762,
+      "learning_rate": 0.001,
+      "loss": 6.921406555175781,
+      "step": 20
+    },
+    {
+      "epoch": 0.0002659229687640233,
+      "eval_loss": 6.734507083892822,
+      "eval_runtime": 4.8368,
+      "eval_samples_per_second": 211.711,
+      "eval_steps_per_second": 26.464,
+      "step": 20
+    },
+    {
+      "epoch": 0.0003988844531460349,
+      "grad_norm": 2.39795184135437,
+      "learning_rate": 0.0011111111111111111,
+      "loss": 6.502817535400391,
+      "step": 30
+    },
+    {
+      "epoch": 0.0005318459375280466,
+      "grad_norm": 1.780358076095581,
+      "learning_rate": 0.0011111111111111111,
+      "loss": 6.094605255126953,
+      "step": 40
+    },
+    {
+      "epoch": 0.0005318459375280466,
+      "eval_loss": 5.933877944946289,
+      "eval_runtime": 4.721,
+      "eval_samples_per_second": 216.905,
+      "eval_steps_per_second": 27.113,
+      "step": 40
+    },
+    {
+      "epoch": 0.0006648074219100582,
+      "grad_norm": 1.0763862133026123,
+      "learning_rate": 0.0012345679012345679,
+      "loss": 5.835469436645508,
+      "step": 50
+    },
+    {
+      "epoch": 0.0007977689062920698,
+      "grad_norm": 1.1550487279891968,
+      "learning_rate": 0.0012345679012345679,
+      "loss": 5.669637298583984,
+      "step": 60
+    },
+    {
+      "epoch": 0.0007977689062920698,
+      "eval_loss": 5.614484786987305,
+      "eval_runtime": 4.7235,
+      "eval_samples_per_second": 216.79,
+      "eval_steps_per_second": 27.099,
+      "step": 60
+    },
+    {
+      "epoch": 0.0009307303906740815,
+      "grad_norm": 1.2857813835144043,
+      "learning_rate": 0.0013717421124828531,
+      "loss": 5.549149703979492,
+      "step": 70
+    },
+    {
+      "epoch": 0.0010636918750560932,
+      "grad_norm": 0.9777137637138367,
+      "learning_rate": 0.0013717421124828531,
+      "loss": 5.450737762451172,
+      "step": 80
+    },
+    {
+      "epoch": 0.0010636918750560932,
+      "eval_loss": 5.396044731140137,
+      "eval_runtime": 4.7268,
+      "eval_samples_per_second": 216.636,
+      "eval_steps_per_second": 27.079,
+      "step": 80
+    },
+    {
+      "epoch": 0.0011966533594381047,
+      "grad_norm": 1.4468932151794434,
+      "learning_rate": 0.0015241579027587256,
+      "loss": 5.346249389648437,
+      "step": 90
+    },
+    {
+      "epoch": 0.0013296148438201164,
+      "grad_norm": 1.385213017463684,
+      "learning_rate": 0.0015241579027587256,
+      "loss": 5.2223976135253904,
+      "step": 100
+    },
+    {
+      "epoch": 0.0013296148438201164,
+      "eval_loss": 5.167303085327148,
+      "eval_runtime": 4.7507,
+      "eval_samples_per_second": 215.546,
+      "eval_steps_per_second": 26.943,
+      "step": 100
+    },
+    {
+      "epoch": 0.001462576328202128,
+      "grad_norm": 1.344064712524414,
+      "learning_rate": 0.0016935087808430283,
+      "loss": 5.121601867675781,
+      "step": 110
+    },
+    {
+      "epoch": 0.0015955378125841397,
+      "grad_norm": 1.2178468704223633,
+      "learning_rate": 0.0016935087808430283,
+      "loss": 5.022267913818359,
+      "step": 120
+    },
+    {
+      "epoch": 0.0015955378125841397,
+      "eval_loss": 4.967755317687988,
+      "eval_runtime": 4.7177,
+      "eval_samples_per_second": 217.055,
+      "eval_steps_per_second": 27.132,
+      "step": 120
+    },
+    {
+      "epoch": 0.0017284992969661514,
+      "grad_norm": 1.7606405019760132,
+      "learning_rate": 0.0018816764231589203,
+      "loss": 4.9363243103027346,
+      "step": 130
+    },
+    {
+      "epoch": 0.001861460781348163,
+      "grad_norm": 1.4632689952850342,
+      "learning_rate": 0.0018816764231589203,
+      "loss": 4.849981307983398,
+      "step": 140
+    },
+    {
+      "epoch": 0.001861460781348163,
+      "eval_loss": 4.82466459274292,
+      "eval_runtime": 4.7613,
+      "eval_samples_per_second": 215.068,
+      "eval_steps_per_second": 26.883,
+      "step": 140
+    },
+    {
+      "epoch": 0.0019944222657301744,
+      "grad_norm": 1.3040728569030762,
+      "learning_rate": 0.002090751581287689,
+      "loss": 4.788387298583984,
+      "step": 150
+    },
+    {
+      "epoch": 0.0021273837501121864,
+      "grad_norm": 1.1895735263824463,
+      "learning_rate": 0.002090751581287689,
+      "loss": 4.7253562927246096,
+      "step": 160
+    },
+    {
+      "epoch": 0.0021273837501121864,
+      "eval_loss": 4.692420959472656,
+      "eval_runtime": 4.7224,
+      "eval_samples_per_second": 216.838,
+      "eval_steps_per_second": 27.105,
+      "step": 160
+    },
+    {
+      "epoch": 0.002260345234494198,
+      "grad_norm": 1.218579888343811,
+      "learning_rate": 0.002323057312541877,
+      "loss": 4.674822616577148,
+      "step": 170
+    },
+    {
+      "epoch": 0.0023933067188762094,
+      "grad_norm": 1.7899105548858643,
+      "learning_rate": 0.002323057312541877,
+      "loss": 4.62569580078125,
+      "step": 180
+    },
+    {
+      "epoch": 0.0023933067188762094,
+      "eval_loss": 4.606845855712891,
+      "eval_runtime": 4.7684,
+      "eval_samples_per_second": 214.747,
+      "eval_steps_per_second": 26.843,
+      "step": 180
+    },
+    {
+      "epoch": 0.0025262682032582214,
+      "grad_norm": 1.1465339660644531,
+      "learning_rate": 0.0025811747917131966,
+      "loss": 4.587751007080078,
+      "step": 190
+    },
+    {
+      "epoch": 0.002659229687640233,
+      "grad_norm": 1.4619438648223877,
+      "learning_rate": 0.0025811747917131966,
+      "loss": 4.551251220703125,
+      "step": 200
+    },
+    {
+      "epoch": 0.002659229687640233,
+      "eval_loss": 4.531779766082764,
+      "eval_runtime": 4.7447,
+      "eval_samples_per_second": 215.821,
+      "eval_steps_per_second": 26.978,
+      "step": 200
+    },
+    {
+      "epoch": 0.0027921911720222444,
+      "grad_norm": 1.514225721359253,
+      "learning_rate": 0.0028679719907924407,
+      "loss": 4.521094131469726,
+      "step": 210
+    },
+    {
+      "epoch": 0.002925152656404256,
+      "grad_norm": 1.3342633247375488,
+      "learning_rate": 0.0028679719907924407,
+      "loss": 4.481899261474609,
+      "step": 220
+    },
+    {
+      "epoch": 0.002925152656404256,
+      "eval_loss": 4.473923206329346,
+      "eval_runtime": 4.7271,
+      "eval_samples_per_second": 216.625,
+      "eval_steps_per_second": 27.078,
+      "step": 220
+    },
+    {
+      "epoch": 0.003058114140786268,
+      "grad_norm": 1.1252554655075073,
+      "learning_rate": 0.003186635545324934,
+      "loss": 4.454861068725586,
+      "step": 230
+    },
+    {
+      "epoch": 0.0031910756251682794,
+      "grad_norm": 0.9204785227775574,
+      "learning_rate": 0.003186635545324934,
+      "loss": 4.433399200439453,
+      "step": 240
+    },
+    {
+      "epoch": 0.0031910756251682794,
+      "eval_loss": 4.432496070861816,
+      "eval_runtime": 4.7255,
+      "eval_samples_per_second": 216.696,
+      "eval_steps_per_second": 27.087,
+      "step": 240
+    },
+    {
+      "epoch": 0.003324037109550291,
+      "grad_norm": 0.9875030517578125,
+      "learning_rate": 0.0035407061614721484,
+      "loss": 4.423421859741211,
+      "step": 250
+    },
+    {
+      "epoch": 0.003456998593932303,
+      "grad_norm": 1.1480664014816284,
+      "learning_rate": 0.0035407061614721484,
+      "loss": 4.397797393798828,
+      "step": 260
+    },
+    {
+      "epoch": 0.003456998593932303,
+      "eval_loss": 4.395366191864014,
+      "eval_runtime": 4.7188,
+      "eval_samples_per_second": 217.005,
+      "eval_steps_per_second": 27.126,
+      "step": 260
+    },
+    {
+      "epoch": 0.0035899600783143143,
+      "grad_norm": 1.0977789163589478,
+      "learning_rate": 0.003934117957191276,
+      "loss": 4.393301773071289,
+      "step": 270
+    },
+    {
+      "epoch": 0.003722921562696326,
+      "grad_norm": 1.0433765649795532,
+      "learning_rate": 0.003934117957191276,
+      "loss": 4.361746215820313,
+      "step": 280
+    },
+    {
+      "epoch": 0.003722921562696326,
+      "eval_loss": 4.3616042137146,
+      "eval_runtime": 4.7244,
+      "eval_samples_per_second": 216.749,
+      "eval_steps_per_second": 27.094,
+      "step": 280
+    },
+    {
+      "epoch": 0.0038558830470783374,
+      "grad_norm": 1.0764110088348389,
+      "learning_rate": 0.004371242174656973,
+      "loss": 4.3703865051269535,
+      "step": 290
+    },
+    {
+      "epoch": 0.003988844531460349,
+      "grad_norm": 1.165968894958496,
+      "learning_rate": 0.004371242174656973,
+      "loss": 4.350434875488281,
+      "step": 300
+    },
+    {
+      "epoch": 0.003988844531460349,
+      "eval_loss": 4.334297180175781,
+      "eval_runtime": 4.7184,
+      "eval_samples_per_second": 217.024,
+      "eval_steps_per_second": 27.128,
+      "step": 300
+    },
+    {
+      "epoch": 0.004121806015842361,
+      "grad_norm": 0.9461768269538879,
+      "learning_rate": 0.004856935749618859,
+      "loss": 4.3365730285644535,
+      "step": 310
+    },
+    {
+      "epoch": 0.004254767500224373,
+      "grad_norm": 0.8112942576408386,
+      "learning_rate": 0.004856935749618859,
+      "loss": 4.318856430053711,
+      "step": 320
+    },
+    {
+      "epoch": 0.004254767500224373,
+      "eval_loss": 4.319216251373291,
+      "eval_runtime": 4.7521,
+      "eval_samples_per_second": 215.484,
+      "eval_steps_per_second": 26.935,
+      "step": 320
+    },
+    {
+      "epoch": 0.004387728984606384,
+      "grad_norm": 0.8098408579826355,
+      "learning_rate": 0.005396595277354288,
+      "loss": 4.325001525878906,
+      "step": 330
+    },
+    {
+      "epoch": 0.004520690468988396,
+      "grad_norm": 0.9229351282119751,
+      "learning_rate": 0.005396595277354288,
+      "loss": 4.310536956787109,
+      "step": 340
+    },
+    {
+      "epoch": 0.004520690468988396,
+      "eval_loss": 4.303993225097656,
+      "eval_runtime": 4.7235,
+      "eval_samples_per_second": 216.787,
+      "eval_steps_per_second": 27.098,
+      "step": 340
+    },
+    {
+      "epoch": 0.004653651953370408,
+      "grad_norm": 0.7983461022377014,
+      "learning_rate": 0.005996216974838097,
+      "loss": 4.303889465332031,
+      "step": 350
+    },
+    {
+      "epoch": 0.004786613437752419,
+      "grad_norm": 0.9175242185592651,
+      "learning_rate": 0.005996216974838097,
+      "loss": 4.306822204589844,
+      "step": 360
+    },
+    {
+      "epoch": 0.004786613437752419,
+      "eval_loss": 4.303272247314453,
+      "eval_runtime": 4.7189,
+      "eval_samples_per_second": 217.0,
+      "eval_steps_per_second": 27.125,
+      "step": 360
+    },
+    {
+      "epoch": 0.004919574922134431,
+      "grad_norm": 0.6384010314941406,
+      "learning_rate": 0.005396595277354288,
+      "loss": 4.271321868896484,
+      "step": 370
+    },
+    {
+      "epoch": 0.005052536406516443,
+      "grad_norm": 0.6614411473274231,
+      "learning_rate": 0.005396595277354288,
+      "loss": 4.2241767883300785,
+      "step": 380
+    },
+    {
+      "epoch": 0.005052536406516443,
+      "eval_loss": 4.230008602142334,
+      "eval_runtime": 4.7416,
+      "eval_samples_per_second": 215.959,
+      "eval_steps_per_second": 26.995,
+      "step": 380
+    },
+    {
+      "epoch": 0.005185497890898454,
+      "grad_norm": 0.7242400050163269,
+      "learning_rate": 0.005996216974838097,
+      "loss": 4.2259784698486325,
+      "step": 390
+    },
+    {
+      "epoch": 0.005318459375280466,
+      "grad_norm": 0.8357770442962646,
+      "learning_rate": 0.005996216974838097,
+      "loss": 4.2215126037597654,
+      "step": 400
+    },
+    {
+      "epoch": 0.005318459375280466,
+      "eval_loss": 4.221831321716309,
+      "eval_runtime": 4.754,
+      "eval_samples_per_second": 215.399,
+      "eval_steps_per_second": 26.925,
+      "step": 400
+    },
+    {
+      "epoch": 0.005451420859662477,
+      "grad_norm": 0.7000677585601807,
+      "learning_rate": 0.006662463305375663,
+      "loss": 4.216830444335938,
+      "step": 410
+    },
+    {
+      "epoch": 0.005584382344044489,
+      "grad_norm": 0.5402432084083557,
+      "learning_rate": 0.006662463305375663,
+      "loss": 4.207019805908203,
+      "step": 420
+    },
+    {
+      "epoch": 0.005584382344044489,
+      "eval_loss": 4.203587532043457,
+      "eval_runtime": 4.7245,
+      "eval_samples_per_second": 216.743,
+      "eval_steps_per_second": 27.093,
+      "step": 420
+    },
+    {
+      "epoch": 0.005717343828426501,
+      "grad_norm": 0.6724000573158264,
+      "learning_rate": 0.007402737005972959,
+      "loss": 4.2224266052246096,
+      "step": 430
+    },
+    {
+      "epoch": 0.005850305312808512,
+      "grad_norm": 0.5229582786560059,
+      "learning_rate": 0.007402737005972959,
+      "loss": 4.224952316284179,
+      "step": 440
+    },
+    {
+      "epoch": 0.005850305312808512,
+      "eval_loss": 4.210740089416504,
+      "eval_runtime": 4.7257,
+      "eval_samples_per_second": 216.685,
+      "eval_steps_per_second": 27.086,
+      "step": 440
+    },
+    {
+      "epoch": 0.005983266797190524,
+      "grad_norm": 0.558103084564209,
+      "learning_rate": 0.006662463305375663,
+      "loss": 4.192767333984375,
+      "step": 450
+    },
+    {
+      "epoch": 0.006116228281572536,
+      "grad_norm": 0.5505457520484924,
+      "learning_rate": 0.006662463305375663,
+      "loss": 4.158684921264649,
+      "step": 460
+    },
+    {
+      "epoch": 0.006116228281572536,
+      "eval_loss": 4.15805196762085,
+      "eval_runtime": 4.7412,
+      "eval_samples_per_second": 215.98,
+      "eval_steps_per_second": 26.998,
+      "step": 460
+    },
+    {
+      "epoch": 0.006249189765954547,
+      "grad_norm": 0.3989739716053009,
+      "learning_rate": 0.007402737005972959,
+      "loss": 4.163018035888672,
+      "step": 470
+    },
+    {
+      "epoch": 0.006382151250336559,
+      "grad_norm": 0.5627162456512451,
+      "learning_rate": 0.007402737005972959,
+      "loss": 4.164209365844727,
+      "step": 480
+    },
+    {
+      "epoch": 0.006382151250336559,
+      "eval_loss": 4.168121337890625,
+      "eval_runtime": 4.7243,
+      "eval_samples_per_second": 216.751,
+      "eval_steps_per_second": 27.094,
+      "step": 480
+    },
+    {
+      "epoch": 0.006515112734718571,
+      "grad_norm": 0.4144296944141388,
+      "learning_rate": 0.006662463305375663,
+      "loss": 4.1303466796875,
+      "step": 490
+    },
+    {
+      "epoch": 0.006648074219100582,
+      "grad_norm": 0.4982388913631439,
+      "learning_rate": 0.006662463305375663,
+      "loss": 4.116038513183594,
+      "step": 500
+    },
+    {
+      "epoch": 0.006648074219100582,
+      "eval_loss": 4.106968879699707,
+      "eval_runtime": 4.7284,
+      "eval_samples_per_second": 216.562,
+      "eval_steps_per_second": 27.07,
+      "step": 500
+    },
+    {
+      "epoch": 0.006781035703482594,
+      "grad_norm": 0.48199519515037537,
+      "learning_rate": 0.007402737005972959,
+      "loss": 4.121062850952148,
+      "step": 510
+    },
+    {
+      "epoch": 0.006913997187864606,
+      "grad_norm": 0.5215345025062561,
+      "learning_rate": 0.007402737005972959,
+      "loss": 4.111361694335938,
+      "step": 520
+    },
+    {
+      "epoch": 0.006913997187864606,
+      "eval_loss": 4.113387107849121,
+      "eval_runtime": 4.7279,
+      "eval_samples_per_second": 216.586,
+      "eval_steps_per_second": 27.073,
+      "step": 520
+    },
+    {
+      "epoch": 0.007046958672246617,
+      "grad_norm": 0.3848040997982025,
+      "learning_rate": 0.006662463305375663,
+      "loss": 4.077674484252929,
+      "step": 530
+    },
+    {
+      "epoch": 0.007179920156628629,
+      "grad_norm": 0.4323752224445343,
+      "learning_rate": 0.006662463305375663,
+      "loss": 4.059872817993164,
+      "step": 540
+    },
+    {
+      "epoch": 0.007179920156628629,
+      "eval_loss": 4.064393520355225,
+      "eval_runtime": 4.7217,
+      "eval_samples_per_second": 216.87,
+      "eval_steps_per_second": 27.109,
+      "step": 540
+    },
+    {
+      "epoch": 0.007312881641010641,
+      "grad_norm": 0.4613271653652191,
+      "learning_rate": 0.007402737005972959,
+      "loss": 4.067990493774414,
+      "step": 550
+    },
+    {
+      "epoch": 0.007445843125392652,
+      "grad_norm": 0.3521135449409485,
+      "learning_rate": 0.007402737005972959,
+      "loss": 4.075962066650391,
+      "step": 560
+    },
+    {
+      "epoch": 0.007445843125392652,
+      "eval_loss": 4.064351558685303,
+      "eval_runtime": 4.7464,
+      "eval_samples_per_second": 215.745,
+      "eval_steps_per_second": 26.968,
+      "step": 560
+    },
+    {
+      "epoch": 0.007578804609774664,
+      "grad_norm": 0.3451947569847107,
+      "learning_rate": 0.006662463305375663,
+      "loss": 4.04071159362793,
+      "step": 570
+    },
+    {
+      "epoch": 0.007711766094156675,
+      "grad_norm": 0.2876155972480774,
+      "learning_rate": 0.006662463305375663,
+      "loss": 4.019502258300781,
+      "step": 580
+    },
+    {
+      "epoch": 0.007711766094156675,
+      "eval_loss": 4.021025657653809,
+      "eval_runtime": 4.725,
+      "eval_samples_per_second": 216.718,
+      "eval_steps_per_second": 27.09,
+      "step": 580
+    },
+    {
+      "epoch": 0.007844727578538687,
+      "grad_norm": 0.47599413990974426,
+      "learning_rate": 0.007402737005972959,
+      "loss": 4.03144416809082,
+      "step": 590
+    },
+    {
+      "epoch": 0.007977689062920698,
+      "grad_norm": 0.3505311608314514,
+      "learning_rate": 0.007402737005972959,
+      "loss": 4.022784805297851,
+      "step": 600
+    },
+    {
+      "epoch": 0.007977689062920698,
+      "eval_loss": 4.033039093017578,
+      "eval_runtime": 4.7522,
+      "eval_samples_per_second": 215.479,
+      "eval_steps_per_second": 26.935,
+      "step": 600
+    },
+    {
+      "epoch": 0.00811065054730271,
+      "grad_norm": 0.35564446449279785,
+      "learning_rate": 0.006662463305375663,
+      "loss": 4.009813690185547,
+      "step": 610
+    },
+    {
+      "epoch": 0.008243612031684722,
+      "grad_norm": 0.3954601287841797,
+      "learning_rate": 0.006662463305375663,
+      "loss": 3.9844127655029298,
+      "step": 620
+    },
+    {
+      "epoch": 0.008243612031684722,
+      "eval_loss": 3.992824077606201,
+      "eval_runtime": 4.7343,
+      "eval_samples_per_second": 216.292,
+      "eval_steps_per_second": 27.037,
+      "step": 620
+    },
+    {
+      "epoch": 0.008376573516066733,
+      "grad_norm": 0.33219683170318604,
+      "learning_rate": 0.007402737005972959,
+      "loss": 3.9896926879882812,
+      "step": 630
+    },
+    {
+      "epoch": 0.008509535000448746,
+      "grad_norm": 0.33283373713493347,
+      "learning_rate": 0.007402737005972959,
+      "loss": 3.994375991821289,
+      "step": 640
+    },
+    {
+      "epoch": 0.008509535000448746,
+      "eval_loss": 3.992628574371338,
+      "eval_runtime": 4.7346,
+      "eval_samples_per_second": 216.281,
+      "eval_steps_per_second": 27.035,
+      "step": 640
+    },
+    {
+      "epoch": 0.008642496484830757,
+      "grad_norm": 0.30045098066329956,
+      "learning_rate": 0.006662463305375663,
+      "loss": 3.975360870361328,
+      "step": 650
+    },
+    {
+      "epoch": 0.008775457969212768,
+      "grad_norm": 0.28163373470306396,
+      "learning_rate": 0.006662463305375663,
+      "loss": 3.9635448455810547,
+      "step": 660
+    },
+    {
+      "epoch": 0.008775457969212768,
+      "eval_loss": 3.9626264572143555,
+      "eval_runtime": 4.7196,
+      "eval_samples_per_second": 216.97,
+      "eval_steps_per_second": 27.121,
+      "step": 660
+    },
+    {
+      "epoch": 0.00890841945359478,
+      "grad_norm": 0.34984496235847473,
+      "learning_rate": 0.007402737005972959,
+      "loss": 3.9630859375,
+      "step": 670
+    },
+    {
+      "epoch": 0.009041380937976792,
+      "grad_norm": 0.3046402335166931,
+      "learning_rate": 0.007402737005972959,
+      "loss": 3.9557125091552736,
+      "step": 680
+    },
+    {
+      "epoch": 0.009041380937976792,
+      "eval_loss": 3.964042901992798,
+      "eval_runtime": 4.7245,
+      "eval_samples_per_second": 216.743,
+      "eval_steps_per_second": 27.093,
+      "step": 680
+    },
+    {
+      "epoch": 0.009174342422358803,
+      "grad_norm": 0.39222627878189087,
+      "learning_rate": 0.006662463305375663,
+      "loss": 3.9501678466796877,
+      "step": 690
+    },
+    {
+      "epoch": 0.009307303906740816,
+      "grad_norm": 0.2476985901594162,
+      "learning_rate": 0.006662463305375663,
+      "loss": 3.9329227447509765,
+      "step": 700
+    },
+    {
+      "epoch": 0.009307303906740816,
+      "eval_loss": 3.935375213623047,
+      "eval_runtime": 4.7404,
+      "eval_samples_per_second": 216.013,
+      "eval_steps_per_second": 27.002,
+      "step": 700
+    },
+    {
+      "epoch": 0.009440265391122827,
+      "grad_norm": 0.2824518084526062,
+      "learning_rate": 0.007402737005972959,
+      "loss": 3.935717010498047,
+      "step": 710
+    },
+    {
+      "epoch": 0.009573226875504838,
+      "grad_norm": 0.3063017427921295,
+      "learning_rate": 0.007402737005972959,
+      "loss": 3.937954330444336,
+      "step": 720
+    },
+    {
+      "epoch": 0.009573226875504838,
+      "eval_loss": 3.9412312507629395,
+      "eval_runtime": 4.7756,
+      "eval_samples_per_second": 214.421,
+      "eval_steps_per_second": 26.803,
+      "step": 720
+    },
+    {
+      "epoch": 0.00970618835988685,
+      "grad_norm": 0.24896036088466644,
+      "learning_rate": 0.006662463305375663,
+      "loss": 3.9285778045654296,
+      "step": 730
+    },
+    {
+      "epoch": 0.009839149844268862,
+      "grad_norm": 0.40740466117858887,
+      "learning_rate": 0.006662463305375663,
+      "loss": 3.9202159881591796,
+      "step": 740
+    },
+    {
+      "epoch": 0.009839149844268862,
+      "eval_loss": 3.9166951179504395,
+      "eval_runtime": 4.725,
+      "eval_samples_per_second": 216.721,
+      "eval_steps_per_second": 27.09,
+      "step": 740
+    },
+    {
+      "epoch": 0.009972111328650873,
+      "grad_norm": 0.32246750593185425,
+      "learning_rate": 0.007402737005972959,
+      "loss": 3.914220428466797,
+      "step": 750
+    },
+    {
+      "epoch": 0.010105072813032885,
+      "grad_norm": 0.33994966745376587,
+      "learning_rate": 0.007402737005972959,
+      "loss": 3.9243412017822266,
+      "step": 760
+    },
+    {
+      "epoch": 0.010105072813032885,
+      "eval_loss": 3.9266605377197266,
+      "eval_runtime": 4.7222,
+      "eval_samples_per_second": 216.847,
+      "eval_steps_per_second": 27.106,
+      "step": 760
+    },
+    {
+      "epoch": 0.010105072813032885,
+      "step": 760,
+      "total_flos": 6.206104799281152e+16,
+      "train_loss": 4.492900440567418,
+      "train_runtime": 3508.751,
+      "train_samples_per_second": 55.45,
+      "train_steps_per_second": 0.217
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 760,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 5,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 6.206104799281152e+16,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

scheduler/model.toml

@@ -0,0 +1,13 @@
+
+# LLaMA 77M params
+# huggingface/transformers/.. models/llama/configuration_llama.py
+
+model_type = "llama"
+tie_word_embeddings = true
+hidden_size = 768          # embedding dimension
+intermediate_size = 2048   # FFN hidden size (typically 4x hidden_size)
+max_position_embeddings = 1024
+num_attention_heads = 8    # attention heads (must divide hidden_size evenly)
+num_hidden_layers = 8      # transformer layers
+num_key_value_heads = 4
+vocab_size = 32768

scheduler/wsd_10/REPORT.md

@@ -0,0 +1,34 @@
+# Experiment: wsd_10 — warmup-stable-decay, 10% linear decay to 0
+
+**Wall-clock:** 60 min · **Recipe:** best known (sdpa + bf16 mixed + muonq @ lr 8e-3), schedule: warmup 200 → stable 484 @ 8e-3 → linear decay to 0 over 76 steps (horizon `max_steps = 760`)
+
+## Result: big win over constant LR — but the 20% decay (`../wsd_20/`) is better on every axis
+
+| metric | wsd_10 | wsd_20 | constant-LR reference |
+|---|---|---|---|
+| optimizer steps in 60 min | 740 | 730 | 759 |
+| final logged eval loss / ppl | 3.8641 / 47.7 (step 688 — 4 steps into decay, undersells final model) | 3.8023 / 44.8 | 3.8583 / 47.4 |
+| bake score | 52.68/100 (DOUGH) | **55.61** | 46.83 |
+| held-out bits/byte | 1.2821 | **1.2769** | 1.3114 |
+| chat bits/byte | 1.1608 | **1.1571** | 1.1914 |
+| logic accuracy | 0.625 (+0.0117) | **0.650 (+0.0242)** | 0.600 (+0.018) |
+| hygiene (greedy loop) | 68.5/100 (33.8w) | **79.2 (23.8w)** | 47.7 (59.8w) |
+| grad norms | 0.05–9.45, 0 non-finite | 0.06–9.45 | 0.09–9.45 |
+| peak GPU | 7.2 GB | 7.2 GB | 7.2 GB |
+
++5.9 bake points over the constant-LR reference and a partial hygiene recovery (59.8w → 33.8w) — but dominated by wsd_20 everywhere. Note the trainer eval curve barely captures the decay at all: the last logged eval (step 688) is 4 steps after decay onset (684), while the saved final model is at step 740 with the decay ~74% done (LR 2.21e-3, wall-clock stop cut the last 20 steps). The evaluate2 numbers are measured on the actual final model.
+
+## Eval loss curve (5-min intervals)
+
+| step | eval loss | | step | eval loss |
+|---|---|---|---|---|
+| 67 | 5.7043 | | 438 | 4.0252 |
+| 131 | 4.9471 | | 500 | 3.9743 |
+| 194 | 4.5693 | | 563 | 3.9288 |
+| 257 | 4.3185 | | 626 | 3.8911 |
+| 317 | 4.1820 | | 688 | 3.8641 |
+| 379 | 4.0923 | | | |
+
+## Verdict
+
+**Decay helps a lot; more decay helps more (at this horizon).** 10% was SmolLM's lower bound and it leaves quality on the table vs 20% — the stable phase's extra 76 steps at peak LR are worth less than a completed anneal. See `../COMPARISON.md`.

scheduler/wsd_10/config.toml

@@ -0,0 +1,89 @@
+# Scheduler series — new best recipe (sdpa + bf16 mixed + muonq @ lr 8e-3), only the LR schedule varies.
+# This run: WSD (warmup-stable-decay), SmolLM2-style — linear decay to 0 over the final 10% (76 steps).
+model_config = "../model.toml"
+
+[data]
+# Data files (supports glob patterns or lists of files)
+train_files = ["~/Dev/vintage-LLM/training/data/train_*.bin"]
+valid_files = ["~/Dev/vintage-LLM/training/data/valid_*.bin"]
+
+# Sequence length for each dataset sample
+# Must be <= model.max_position_embeddings
+max_seq_length = 1024
+
+# Tokenizer (must match the one used to create .bin files)
+tokenizer = "~/Dev/vintage-LLM/tokenizers/t-v3/"
+
+[training]
+# Training duration
+num_train_epochs = 1
+max_steps = 760                    # scheduler horizon = achievable steps/hour (was -1: full epoch, so decay never happened)
+
+# Optional: Stop training after this many minutes of wall-clock time.
+max_train_minutes = 60
+
+# Output directories
+output_dir = "./"
+final_model_dir = "./final"
+
+# Batch sizes
+# Effective batch size = num_devices × per_device_train_batch_size × gradient_accumulation_steps
+per_device_train_batch_size = 8    # batch size per GPU (8x1024 tokens fits 16GB with headroom for eval)
+per_device_eval_batch_size  = 8
+gradient_accumulation_steps = 32   # accumulate gradients over multiple mini-batches before updating the params
+
+# Optimizer: MuonQ (4-bit quantized Muon, ../../MuonQ repo) — wired through
+# optimizer_cls_and_kwargs in base_train.py. 2D hidden weights get Muon
+# (Newton-Schulz 5, 4-bit mu-law-companded momentum, rank-16 residual
+# factorization); embeddings/head/norms get its internal AdamW backup.
+# Muon/quantization knobs (muon_*, muonq_*) default to the repo's own
+# llama-60m "muonq" recipe in base_train.py.
+optim = "muonq"
+
+# Optimizer settings — lr/wd from MuonQ's llama-60m recipe (Muon takes
+# spectral-norm-scaled steps; do not copy the AdamW 5e-4 here)
+learning_rate = 8e-3
+weight_decay = 0.1
+adam_beta1 = 0.9
+adam_beta2 = 0.95
+max_grad_norm = 1.0
+
+# Learning rate scheduler
+lr_scheduler_type    = "warmup_stable_decay"   # warmup 200 -> stable 484 @ peak -> linear decay 76 steps to 0
+lr_scheduler_kwargs  = { num_stable_steps = 484, num_decay_steps = 76, decay_type = "linear", min_lr_ratio = 0.0 }
+warmup_steps = 200                 # 1-hour series standard (muonq-v2's 400 was a 2-hour recipe)
+
+# Precision (choose based on your GPU)
+bf16 = true                        # bfloat16 autocast + fp32 master weights (the dtype-series safe recipe)
+fp16 = false
+
+# Performance
+torch_compile = false
+gradient_checkpointing = false     # trade compute for memory (enables larger models)
+neftune_noise_alpha = 0.0          # add noise during training to improve model performance (fine-tuning)
+
+# Checkpointing
+# save_strategy can be "steps", "epoch", or "minutes" (wall-clock time).
+save_strategy = "minutes"
+save_steps = 5                    # checkpoint every X minutes
+save_total_limit = 3              # keep only X most recent checkpoints
+
+# Evaluation
+# eval_strategy can be "steps", "epoch", or "minutes" (wall-clock time).
+eval_strategy = "minutes"
+eval_steps = 5                   # evaluate every X minutes
+# Evaluate on a fixed random subset of the validation set (0 = use all)
+max_eval_samples = 5000
+
+# Logging
+logging_strategy = "steps"
+logging_steps = 10                 # log every 10 steps
+logging_first_step = true          # log the first step
+
+# Reproducibility
+seed = 42
+
+# Performance
+dataloader_num_workers = 2         # number of data loading threads
+dataloader_prefetch_factor = 2     # batches prefetched per worker
+dataloader_pin_memory = true       # pin memory for faster GPU transfer

scheduler/wsd_10/eval1.log

@@ -0,0 +1,212 @@
+W0814 13:06:20.261000 44298 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0814 13:06:20.283000 44298 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+
+==============================================================================
+CHECKPOINT INSPECTION
+==============================================================================
+  checkpoint : final
+  tokenizer  : final
+  device     : cuda   dtype: torch.bfloat16   seed: 1337
+
+
+==============================================================================
+SUITE: MODEL INFO AND TRAINING LINEAGE
+==============================================================================
+  checkpoint              : final
+  model_type              : llama
+  architecture            : LlamaForCausalLM
+  device                  : cuda
+  dtype                   : torch.bfloat16
+  parameters              : 77,083,392 (77.08M)
+  disk size               : 308 MB
+  layers / hidden / heads : 8 / 768 / 8 (KV heads: 4)
+  vocab (cfg / tokenizer) : 32768 / 32752
+  context length          : 1024
+  chat template           : True
+  training lineage        : base pretraining, ctx 1024
+  embedding mean norm     : 3.7700   <- typical row length of the embedding matrix; very small or huge values signal trouble
+  embedding mean cosine   : 0.1028   <- avg cosine between random embedding rows. Descriptive only: it starts near 0.0
+                            because RANDOM init is near-orthogonal, and it GROWS as the model trains.
+                            Near 0 therefore does NOT mean "healthy" - it can mean "barely trained".
+  (note: config vocab padded by 16 unused rows - harmless)
+
+==============================================================================
+SUITE: PERIOD FIDELITY ON FIXED PROBE SENTENCES
+==============================================================================
+  Scoring 20 sentences (10 historical, 10 modern)
+  The MODERN/HISTORICAL ratio is the output that means something. The absolute
+  numbers come from ~20 sentences: too noisy to rank checkpoints (use evaluate2.py).
+
+  OVERALL (270 tokens)
+    perplexity            : 258.34   <- 'average branching factor'; lower = less surprised = better fit to this text
+    mean token prob       : 0.0838   <- average confidence on the actual next word
+    median token prob     : 0.0065   <- typical confidence (robust to a few very bad tokens)
+    worst-token prob (min): 7.8e-08   <- the single most surprising word; tiny values = the model really didn't expect it
+    10th-percentile prob  : 1.6e-05   <- confidence on the model's worst 10% of guesses
+    low-confidence tokens : 53.33%   <- share of tokens predicted with <1% probability (red flags)
+    mean entropy          : 4.627 nats   <- how spread out the predictions are; ~0 = very certain, higher = hedging
+
+  GROUP COMPARISON
+    group        sentences    perplexity   bits/byte
+    historical          10         78.90       1.189
+    modern              10        696.95       1.953
+    modern/historical ratio: 8.83   <- >1 means period text is easier for the model than modern text (desired for a vintage model with a ~1900 knowledge cutoff)
+    (bits/byte is the tokenizer-independent version - the only column here you may
+     compare between models that use DIFFERENT tokenizers.)
+
+  PER-SENTENCE DETAIL (sorted worst-first)
+    [modern    ] ppl= 5887.24  minp=3.1e-07  The engineering team rewrote the whole backend micro
+    [modern    ] ppl= 3209.93  minp=1.9e-06  Robots on the automated assembly line handle the ent
+    [modern    ] ppl= 1188.94  minp=1.4e-06  Small retail startups now run almost all of their bu
+    [modern    ] ppl= 1106.63  minp=1.3e-06  Three hours stuck in traffic in the pouring rain mad
+    [modern    ] ppl=  555.52  minp=1.0e-05  She is finishing a graduate degree in machine learni
+    [modern    ] ppl=  513.64  minp=1.2e-06  Millions of shoppers queue outside the store because
+    [modern    ] ppl=  460.49  minp=7.8e-08  Members of the European Union parliament voted on th
+    [modern    ] ppl=  428.78  minp=2.7e-07  She gave up her whole weekend to help me move apartm
+    [historical] ppl=  294.55  minp=1.2e-05  The keeper fed the great serpent which the naturalis
+    [modern    ] ppl=  219.43  minp=9.1e-06  I switched to a plant-based diet last year and I no 
+    [modern    ] ppl=  190.02  minp=1.9e-06  After years of hiding it from everyone at work, he t
+    [historical] ppl=  131.62  minp=8.6e-05  He drew a distinction so fine and over-scrupulous th
+    [historical] ppl=   94.75  minp=7.3e-06  The mountain rose above the valley in a silence sole
+    [historical] ppl=   87.33  minp=1.4e-04  The weavers at their looms are employed in the wooll
+    [historical] ppl=   82.02  minp=7.7e-05  The merchants of the port have grown rich upon their
+    [historical] ppl=   71.80  minp=1.6e-05  The labouring poor of this parish are reduced to gre
+    [historical] ppl=   66.09  minp=9.5e-05  He gave up his fortune to the patient study of natur
+    [historical] ppl=   59.40  minp=5.1e-05  The Lord provideth for all his creatures, giving the
+    [historical] ppl=   50.13  minp=1.2e-04  The ballroom was filled with dancing and laughter, a
+    [historical] ppl=   18.85  minp=5.2e-04  Her Majesty was pleased to summon the Lords and Comm
+
+==============================================================================
+SUITE: DIACHRONIC WORD-SENSE SEPARATION
+==============================================================================
+  Extracting contextual embeddings for 10 shifted words in period AND modern sentences.
+  Every probe word sits LAST in its sentence, so the causal model has actually read
+  the disambiguating context before it represents the word.
+
+  SENSE SEPARATION (cosine similarity of the SAME word, period vs modern sentence)
+    lower value = the model represents the two senses differently (good sign for a
+    period model); ~1.0 = it treats them as the same word regardless of context
+    want          : +0.339
+    python        : +0.493
+    awful         : +0.563
+    meat          : +0.609
+    parliament    : +0.626
+    nice          : +0.642
+    manufacture   : +0.685
+    gay           : +0.718
+    commerce      : +0.766
+    science       : +0.820
+    MEAN          : +0.626   <- near 1.0 = model ignores period vs modern usage; lower = senses differ
+
+  PAIRWISE COSINE SIMILARITY (period sentences, first 5 words)
+    these are CONTEXTUAL vectors, so they reflect the sentences as much as the words;
+    read the spread, not individual pairs. High mean = representations collapsed.
+                         gay      awful       nice       meat       want
+    gay                1.000      0.468      0.341      0.308      0.281
+    awful                         1.000      0.370      0.266      0.315
+    nice                                     1.000      0.303      0.327
+    meat                                                1.000      0.448
+    want                                                           1.000
+    all 45 pairs: mean=+0.355 std=0.118   <- healthy spread is good
+
+  NEAREST NEIGHBOURS inside the probe-word set (period sentences)
+    gay           : awful (+0.47), commerce (+0.37), science (+0.36), nice (+0.34), manufacture (+0.32)
+    awful         : gay (+0.47), science (+0.41), nice (+0.37), parliament (+0.35), commerce (+0.33)
+    nice          : science (+0.40), awful (+0.37), gay (+0.34), manufacture (+0.33), commerce (+0.33)
+    meat          : science (+0.46), want (+0.45), commerce (+0.43), parliament (+0.42), manufacture (+0.40)
+    want          : manufacture (+0.54), commerce (+0.53), science (+0.47), meat (+0.45), parliament (+0.42)
+
+==============================================================================
+SUITE: GENERATION PROBES
+==============================================================================
+
+  --- Probe 1 ---
+  prompt      : The history of the world is
+  continuation: too often the most interesting and instructive, and it is a most instructive and instructive book.
+
+The story of the great city of Mexico is not more remarkable than the story of the great city of Mexico. It was the great city of Mexico, and its name was the great city of Mexico.
+  stats: 60 tokens | distinct-1 0.53 | distinct-2 0.75 | echo 0.09
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 2 ---
+  prompt      : What is God? God is
+  continuation: a man! He is a man who has lived for the world, and has died for it. It is God who has not lived in vain. He is a man who has gone into the world to die for himself and his God.
+
+And if the world is a God who has lived
+  stats: 60 tokens | distinct-1 0.55 | distinct-2 0.79 | echo 0.02
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 3 ---
+  prompt      : In the year of our Lord eighteen hundred and
+  continuation: forty-eight; and the next year the last, the second year of the kingdom of God, the fourth year of the kingdom of Christ. The third year of the kingdom of heaven, in the year of the covenant, the fourth year of the kingdom of heaven; the sixth of the kingdom of
+  stats: 60 tokens | distinct-1 0.37 | distinct-2 0.62 | echo 0.31
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 4 ---
+  prompt      : The manufacture of cotton
+  continuation: -the latter was carried on for several years by a great number of foreigners.
+
+In the manufacture of woollen cloth, cotton, woollen, and other woollen goods, was imported into England.
+
+The manufacture of woollen cloths and woollen cloths was also in progress at the time of the Revolution,
+  stats: 60 tokens | distinct-1 0.73 | distinct-2 0.94 | echo 0.06
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 5 ---
+  prompt      : The steam engine
+  continuation: is the only one we know of, and that is a great deal more than the other.
+
+A few days after, I was called on to visit a lady who was suffering from soreness of the right eye, but was very sick; she was very ill, and had been so for
+  stats: 60 tokens | distinct-1 0.82 | distinct-2 0.98 | echo 0.04
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 6 ---
+  prompt      : The telegraph
+  continuation: s and telegraphs, and the telegraph offices, have been established for the last year, and the telegraph offices are being filled, and the telegraph offices are being opened, and a telegraph office conducted in the United States. The telegraph offices have been filled by the telegraph and telegraph offices.
+
+The
+  stats: 60 tokens | distinct-1 0.54 | distinct-2 0.80 | echo 0.06
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 7 ---
+  prompt      : My dearest sister,
+  continuation: I have the greatest pleasure to inform you that I have heard from her Majesty's ministers of that most unfortunate event, the Prince de
+
+Govat, who, with his family, have been very dear to her, and who, as you know, are very kind and friendly. You may
+  stats: 60 tokens | distinct-1 0.79 | distinct-2 0.98 | echo 0.00
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 8 ---
+  prompt      : Greetings, my friend
+  continuation: , I shall give you a new chapter on the nature of the Holy Sacrament. I have seen a new chapter in the Bible, the most ancient of which I have seen in the world, in the days when the
+
+Lord had the power to create and to create, and in the days when
+  stats: 60 tokens | distinct-1 0.57 | distinct-2 0.83 | echo 0.15
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 9 ---
+  prompt      : Her Majesty the Queen
+  continuation: of Scots, in the name of the king, was pleased to make them both the greatest of all Princes, and to assure them of their friendship and attachment; and, having had all the honour of the
+
+King, that he would never permit the Queen of Scots to make them the subject of
+  stats: 60 tokens | distinct-1 0.63 | distinct-2 0.94 | echo 0.04
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 10 ---
+  prompt      : Chapter I.
+  continuation: The first question that the author of this book, as we have seen, was then asked to be answered. It is now, however, a question of time and eternity, and therefore we must not omit to ask him, "What is that in the book?" The answer was, "
+  stats: 60 tokens | distinct-1 0.81 | distinct-2 1.00 | echo 0.02
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  AGGREGATE over 10 probes: 600 tokens in 0.8s (709.5 tok/s)
+    mean distinct-1 0.635 | mean distinct-2 0.862 | mean echo 0.080 | worst probe: distinct-2 0.620 / echo 0.314
+    NOTE: at least one probe looped badly even though the average looks fine - read the probes above.
+    (sampling hides loops that greedy decoding exposes; evaluate2.py measures the greedy loop length.)
+
+Full numeric results written to /home/cro/Dev/vintage-LLM/training/eval_results/evaluate-final.json
+
+==============================================================================
+INSPECTION COMPLETE
+==============================================================================
+  This covered WHAT the checkpoint is, its lineage, and whether it is still vintage.
+  For "is it baked enough", run evaluate2.py on the same path.

scheduler/wsd_10/eval2.log

@@ -0,0 +1,54 @@
+evaluating 1 checkpoint(s) on cuda (torch.bfloat16)
+tokenizer: final
+  final ...W0814 13:06:26.561000 44404 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0814 13:06:26.576000 44404 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+
+ bake 53/100  bpb 1.2821  logic 0.62  (15.1s)
+
+report: eval2.md
+json:   eval2.json
+
+# Evaluation: final
+
+- **Parameters:** 77M
+- **Note:** no trainer_state.json in the checkpoint - tokens seen unknown
+
+## Verdict: DOUGH  (bake score 53/100)
+
+Structure is forming - real words, some grammar - but this is not a usable language model yet. It needs several times more training tokens.
+
+## Where it sits (held-out bits/byte, lower = better)
+
+```
+      3.50   untrained model (uniform noise)
+      2.00   word-salad
+      1.50   broken prose
+      1.33   500M model, ~0.7B tokens (early training)
+  --> 1.282  YOUR MODEL
+      1.19   500M model, ~4.7B tokens (undertrained but solid)
+      1.10   best sub-1B measured on this data (341M)
+      0.95   estimated sub-1B ceiling on this corpus
+```
+
+## Scores
+
+| what | raw value | points /100 | weight | plain English |
+|---|---|---|---|---|
+| Held-out loss | 1.2821 bits/byte | 60 | 0.5 | how cheaply it predicts period text it never saw — the single best training signal |
+| Logic | 0.625 acc, margin 0.012 | 30 | 0.25 | picks the *sensible* continuation over matched nonsense; 0.50 = coin-flip. weak - style without understanding |
+| Chat readiness | 1.1608 bits/byte | 56 | 0.15 | how cheap well-formed period dialogue already is — predicts fine-tuning ease |
+| Hygiene | loop 33.8w, punct 0.07/100w | 69 | 0.1 | greedy-decoding loop length and broken punctuation |
+
+## Period boundary
+
+Clean. All 6 post-1900 trap words cost the model more than their period twins (mean shock +1.46 bits/byte, weakest pair 'motor car' at +0.75). No sign of modern text in training.
+
+## See for yourself (sampled, t=0.8)
+
+> **LONDON, Tuesday. — The committee appointed to inquire into the condition of the** workshops in the city of London have been able to report to the Committee the proceedings on the conduct of the works of the London firms, for the purpose of obtaining more accurate information relative to the business of the works. The Secretary having proceeded to the report, the Committee having proceeded to the report of the Committee, the committee passed the following resolutions, which were
+>
+> **A melancholy accident occurred on Thursday last at the works of Messrs. Harding and** Co., Ltd., and which, as I have said, was followed by the death of Mr. Co., the New York, a man of great ability, and of a most estimable character, who was a fine gentleman, a good judge, and a powerful patron of music. A young lady of good family, who is now the wife of an eminent clergyman of the parish, was received with much interest by her friends, and her services were marked by great gener
+>
+> **Brethren, the text which I have chosen for our consideration this morning is taken from** the work of the American Agricultural Society, which is not of a kind to be found in the printed pages of the American Agricultural Society, for the purpose of comparing them with their own, the present number of these books will be found much improved, and the result is an interesting one. A work of this kind, which has attracted much attention, has been started in the "Bank of the Pot," by a per
+>
+

scheduler/wsd_10/final/chat_template.jinja

@@ -0,0 +1,7 @@
+{%- for message in messages -%}
+  {{- bos_token + '<|' + message['role'] + '|>\n' + message['content'].strip() + eos_token + '\n' -}}
+{%- endfor -%}
+
+{%- if add_generation_prompt -%}
+  {{- bos_token + '<|assistant|>\n' -}}
+{%- endif -%}

scheduler/wsd_10/final/config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 3,
+  "dtype": "float32",
+  "eos_token_id": 4,
+  "head_dim": 96,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 2048,
+  "max_position_embeddings": 1024,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 8,
+  "num_hidden_layers": 8,
+  "num_key_value_heads": 4,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.14.1",
+  "use_cache": false,
+  "vocab_size": 32768
+}

scheduler/wsd_10/final/generation_config.json

@@ -0,0 +1,13 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 3,
+  "eos_token_id": [
+    4,
+    2
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 0,
+  "transformers_version": "5.14.1",
+  "use_cache": false
+}

scheduler/wsd_10/final/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a46b2cc2aa7a9d6213652cdbcc9c52e73e13966e4638c6e1e88e2e93d926f62c
+size 308341784

scheduler/wsd_10/final/tokenizer_config.json

@@ -0,0 +1,11 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|unk|>"
+}

scheduler/wsd_10/trainer_state.json

@@ -0,0 +1,656 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.00993222288333627,
+  "eval_steps": 5,
+  "global_step": 747,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 1.3296148438201165e-05,
+      "grad_norm": 5.513442516326904,
+      "learning_rate": 0.0,
+      "loss": 10.538199424743652,
+      "step": 1
+    },
+    {
+      "epoch": 0.00013296148438201165,
+      "grad_norm": 5.268533706665039,
+      "learning_rate": 0.00035999999999999997,
+      "loss": 10.136458502875435,
+      "step": 10
+    },
+    {
+      "epoch": 0.0002659229687640233,
+      "grad_norm": 1.034209132194519,
+      "learning_rate": 0.00076,
+      "loss": 8.0193359375,
+      "step": 20
+    },
+    {
+      "epoch": 0.0003988844531460349,
+      "grad_norm": 9.450255393981934,
+      "learning_rate": 0.00116,
+      "loss": 6.9860687255859375,
+      "step": 30
+    },
+    {
+      "epoch": 0.0005318459375280466,
+      "grad_norm": 5.685102462768555,
+      "learning_rate": 0.0015600000000000002,
+      "loss": 6.707894897460937,
+      "step": 40
+    },
+    {
+      "epoch": 0.0006648074219100582,
+      "grad_norm": 2.160360097885132,
+      "learning_rate": 0.00196,
+      "loss": 6.287691116333008,
+      "step": 50
+    },
+    {
+      "epoch": 0.0007977689062920698,
+      "grad_norm": 1.6388428211212158,
+      "learning_rate": 0.00236,
+      "loss": 5.944025039672852,
+      "step": 60
+    },
+    {
+      "epoch": 0.000890841945359478,
+      "eval_loss": 5.704253673553467,
+      "eval_runtime": 22.8955,
+      "eval_samples_per_second": 218.383,
+      "eval_steps_per_second": 27.298,
+      "step": 67
+    },
+    {
+      "epoch": 0.0009307303906740815,
+      "grad_norm": 1.2074735164642334,
+      "learning_rate": 0.00276,
+      "loss": 5.747131729125977,
+      "step": 70
+    },
+    {
+      "epoch": 0.0010636918750560932,
+      "grad_norm": 1.132364273071289,
+      "learning_rate": 0.00316,
+      "loss": 5.60839729309082,
+      "step": 80
+    },
+    {
+      "epoch": 0.0011966533594381047,
+      "grad_norm": 1.0884277820587158,
+      "learning_rate": 0.0035600000000000002,
+      "loss": 5.502901077270508,
+      "step": 90
+    },
+    {
+      "epoch": 0.0013296148438201164,
+      "grad_norm": 1.3878921270370483,
+      "learning_rate": 0.00396,
+      "loss": 5.371665573120117,
+      "step": 100
+    },
+    {
+      "epoch": 0.001462576328202128,
+      "grad_norm": 1.2677963972091675,
+      "learning_rate": 0.00436,
+      "loss": 5.2512672424316404,
+      "step": 110
+    },
+    {
+      "epoch": 0.0015955378125841397,
+      "grad_norm": 0.9181460738182068,
+      "learning_rate": 0.0047599999999999995,
+      "loss": 5.126389694213867,
+      "step": 120
+    },
+    {
+      "epoch": 0.0017284992969661514,
+      "grad_norm": 0.7834004163742065,
+      "learning_rate": 0.0051600000000000005,
+      "loss": 5.010536575317383,
+      "step": 130
+    },
+    {
+      "epoch": 0.0017417954454043525,
+      "eval_loss": 4.947054862976074,
+      "eval_runtime": 22.9812,
+      "eval_samples_per_second": 217.569,
+      "eval_steps_per_second": 27.196,
+      "step": 131
+    },
+    {
+      "epoch": 0.001861460781348163,
+      "grad_norm": 0.7623927593231201,
+      "learning_rate": 0.00556,
+      "loss": 4.908084106445313,
+      "step": 140
+    },
+    {
+      "epoch": 0.0019944222657301744,
+      "grad_norm": 0.5423034429550171,
+      "learning_rate": 0.00596,
+      "loss": 4.820485305786133,
+      "step": 150
+    },
+    {
+      "epoch": 0.0021273837501121864,
+      "grad_norm": 0.5046894550323486,
+      "learning_rate": 0.00636,
+      "loss": 4.753760528564453,
+      "step": 160
+    },
+    {
+      "epoch": 0.002260345234494198,
+      "grad_norm": 0.3807763159275055,
+      "learning_rate": 0.0067599999999999995,
+      "loss": 4.694861602783203,
+      "step": 170
+    },
+    {
+      "epoch": 0.0023933067188762094,
+      "grad_norm": 0.37175047397613525,
+      "learning_rate": 0.0071600000000000006,
+      "loss": 4.652392578125,
+      "step": 180
+    },
+    {
+      "epoch": 0.0025262682032582214,
+      "grad_norm": 0.3488306999206543,
+      "learning_rate": 0.00756,
+      "loss": 4.597007369995117,
+      "step": 190
+    },
+    {
+      "epoch": 0.0025794527970110256,
+      "eval_loss": 4.569252967834473,
+      "eval_runtime": 23.087,
+      "eval_samples_per_second": 216.572,
+      "eval_steps_per_second": 27.072,
+      "step": 194
+    },
+    {
+      "epoch": 0.002659229687640233,
+      "grad_norm": 0.3351574242115021,
+      "learning_rate": 0.00796,
+      "loss": 4.566041946411133,
+      "step": 200
+    },
+    {
+      "epoch": 0.0027921911720222444,
+      "grad_norm": 0.3297984302043915,
+      "learning_rate": 0.008,
+      "loss": 4.5214378356933596,
+      "step": 210
+    },
+    {
+      "epoch": 0.002925152656404256,
+      "grad_norm": 0.2526053190231323,
+      "learning_rate": 0.008,
+      "loss": 4.468568420410156,
+      "step": 220
+    },
+    {
+      "epoch": 0.003058114140786268,
+      "grad_norm": 0.24721451103687286,
+      "learning_rate": 0.008,
+      "loss": 4.431998825073242,
+      "step": 230
+    },
+    {
+      "epoch": 0.0031910756251682794,
+      "grad_norm": 0.2912442684173584,
+      "learning_rate": 0.008,
+      "loss": 4.396870040893555,
+      "step": 240
+    },
+    {
+      "epoch": 0.003324037109550291,
+      "grad_norm": 0.28560253977775574,
+      "learning_rate": 0.008,
+      "loss": 4.367413711547852,
+      "step": 250
+    },
+    {
+      "epoch": 0.003417110148617699,
+      "eval_loss": 4.318535327911377,
+      "eval_runtime": 23.066,
+      "eval_samples_per_second": 216.77,
+      "eval_steps_per_second": 27.096,
+      "step": 257
+    },
+    {
+      "epoch": 0.003456998593932303,
+      "grad_norm": 0.21196810901165009,
+      "learning_rate": 0.008,
+      "loss": 4.331097793579102,
+      "step": 260
+    },
+    {
+      "epoch": 0.0035899600783143143,
+      "grad_norm": 0.22107456624507904,
+      "learning_rate": 0.008,
+      "loss": 4.302482604980469,
+      "step": 270
+    },
+    {
+      "epoch": 0.003722921562696326,
+      "grad_norm": 0.1877111792564392,
+      "learning_rate": 0.008,
+      "loss": 4.268944931030274,
+      "step": 280
+    },
+    {
+      "epoch": 0.0038558830470783374,
+      "grad_norm": 0.21084874868392944,
+      "learning_rate": 0.008,
+      "loss": 4.2542469024658205,
+      "step": 290
+    },
+    {
+      "epoch": 0.003988844531460349,
+      "grad_norm": 0.20695072412490845,
+      "learning_rate": 0.008,
+      "loss": 4.227492904663086,
+      "step": 300
+    },
+    {
+      "epoch": 0.004121806015842361,
+      "grad_norm": 0.19029781222343445,
+      "learning_rate": 0.008,
+      "loss": 4.208528518676758,
+      "step": 310
+    },
+    {
+      "epoch": 0.004214879054909769,
+      "eval_loss": 4.181951999664307,
+      "eval_runtime": 24.4052,
+      "eval_samples_per_second": 204.874,
+      "eval_steps_per_second": 25.609,
+      "step": 317
+    },
+    {
+      "epoch": 0.004254767500224373,
+      "grad_norm": 0.1889178603887558,
+      "learning_rate": 0.008,
+      "loss": 4.187363815307617,
+      "step": 320
+    },
+    {
+      "epoch": 0.004387728984606384,
+      "grad_norm": 0.19070716202259064,
+      "learning_rate": 0.008,
+      "loss": 4.173851776123047,
+      "step": 330
+    },
+    {
+      "epoch": 0.004520690468988396,
+      "grad_norm": 0.15929149091243744,
+      "learning_rate": 0.008,
+      "loss": 4.148160171508789,
+      "step": 340
+    },
+    {
+      "epoch": 0.004653651953370408,
+      "grad_norm": 0.17022773623466492,
+      "learning_rate": 0.008,
+      "loss": 4.132599258422852,
+      "step": 350
+    },
+    {
+      "epoch": 0.004786613437752419,
+      "grad_norm": 0.18009495735168457,
+      "learning_rate": 0.008,
+      "loss": 4.132373809814453,
+      "step": 360
+    },
+    {
+      "epoch": 0.004919574922134431,
+      "grad_norm": 0.17529530823230743,
+      "learning_rate": 0.008,
+      "loss": 4.117653656005859,
+      "step": 370
+    },
+    {
+      "epoch": 0.0050392402580782415,
+      "eval_loss": 4.0923027992248535,
+      "eval_runtime": 23.4068,
+      "eval_samples_per_second": 213.613,
+      "eval_steps_per_second": 26.702,
+      "step": 379
+    },
+    {
+      "epoch": 0.005052536406516443,
+      "grad_norm": 0.1463332176208496,
+      "learning_rate": 0.008,
+      "loss": 4.096078109741211,
+      "step": 380
+    },
+    {
+      "epoch": 0.005185497890898454,
+      "grad_norm": 0.14370259642601013,
+      "learning_rate": 0.008,
+      "loss": 4.084194946289062,
+      "step": 390
+    },
+    {
+      "epoch": 0.005318459375280466,
+      "grad_norm": 0.16615846753120422,
+      "learning_rate": 0.008,
+      "loss": 4.075920867919922,
+      "step": 400
+    },
+    {
+      "epoch": 0.005451420859662477,
+      "grad_norm": 0.15509292483329773,
+      "learning_rate": 0.008,
+      "loss": 4.054693603515625,
+      "step": 410
+    },
+    {
+      "epoch": 0.005584382344044489,
+      "grad_norm": 0.1333627849817276,
+      "learning_rate": 0.008,
+      "loss": 4.047085952758789,
+      "step": 420
+    },
+    {
+      "epoch": 0.005717343828426501,
+      "grad_norm": 0.12619031965732574,
+      "learning_rate": 0.008,
+      "loss": 4.037934494018555,
+      "step": 430
+    },
+    {
+      "epoch": 0.00582371301593211,
+      "eval_loss": 4.02523136138916,
+      "eval_runtime": 23.2986,
+      "eval_samples_per_second": 214.605,
+      "eval_steps_per_second": 26.826,
+      "step": 438
+    },
+    {
+      "epoch": 0.005850305312808512,
+      "grad_norm": 0.14818833768367767,
+      "learning_rate": 0.008,
+      "loss": 4.030204772949219,
+      "step": 440
+    },
+    {
+      "epoch": 0.005983266797190524,
+      "grad_norm": 0.11952793598175049,
+      "learning_rate": 0.008,
+      "loss": 4.024636077880859,
+      "step": 450
+    },
+    {
+      "epoch": 0.006116228281572536,
+      "grad_norm": 0.12302638590335846,
+      "learning_rate": 0.008,
+      "loss": 4.0040229797363285,
+      "step": 460
+    },
+    {
+      "epoch": 0.006249189765954547,
+      "grad_norm": 0.15813544392585754,
+      "learning_rate": 0.008,
+      "loss": 4.0008289337158205,
+      "step": 470
+    },
+    {
+      "epoch": 0.006382151250336559,
+      "grad_norm": 0.11854846030473709,
+      "learning_rate": 0.008,
+      "loss": 3.994869613647461,
+      "step": 480
+    },
+    {
+      "epoch": 0.006515112734718571,
+      "grad_norm": 0.14335912466049194,
+      "learning_rate": 0.008,
+      "loss": 3.9867435455322267,
+      "step": 490
+    },
+    {
+      "epoch": 0.006648074219100582,
+      "grad_norm": 0.10275301337242126,
+      "learning_rate": 0.008,
+      "loss": 3.9850521087646484,
+      "step": 500
+    },
+    {
+      "epoch": 0.006648074219100582,
+      "eval_loss": 3.9743027687072754,
+      "eval_runtime": 23.1478,
+      "eval_samples_per_second": 216.004,
+      "eval_steps_per_second": 27.0,
+      "step": 500
+    },
+    {
+      "epoch": 0.006781035703482594,
+      "grad_norm": 0.12899844348430634,
+      "learning_rate": 0.008,
+      "loss": 3.9827651977539062,
+      "step": 510
+    },
+    {
+      "epoch": 0.006913997187864606,
+      "grad_norm": 0.1306857168674469,
+      "learning_rate": 0.008,
+      "loss": 3.962184524536133,
+      "step": 520
+    },
+    {
+      "epoch": 0.007046958672246617,
+      "grad_norm": 0.10865853726863861,
+      "learning_rate": 0.008,
+      "loss": 3.951933670043945,
+      "step": 530
+    },
+    {
+      "epoch": 0.007179920156628629,
+      "grad_norm": 0.11110885441303253,
+      "learning_rate": 0.008,
+      "loss": 3.94482536315918,
+      "step": 540
+    },
+    {
+      "epoch": 0.007312881641010641,
+      "grad_norm": 0.10243042558431625,
+      "learning_rate": 0.008,
+      "loss": 3.942932891845703,
+      "step": 550
+    },
+    {
+      "epoch": 0.007445843125392652,
+      "grad_norm": 0.10774345695972443,
+      "learning_rate": 0.008,
+      "loss": 3.9434852600097656,
+      "step": 560
+    },
+    {
+      "epoch": 0.007485731570707256,
+      "eval_loss": 3.928818464279175,
+      "eval_runtime": 23.06,
+      "eval_samples_per_second": 216.826,
+      "eval_steps_per_second": 27.103,
+      "step": 563
+    },
+    {
+      "epoch": 0.007578804609774664,
+      "grad_norm": 0.10925770550966263,
+      "learning_rate": 0.008,
+      "loss": 3.9292713165283204,
+      "step": 570
+    },
+    {
+      "epoch": 0.007711766094156675,
+      "grad_norm": 0.10408780723810196,
+      "learning_rate": 0.008,
+      "loss": 3.917940139770508,
+      "step": 580
+    },
+    {
+      "epoch": 0.007844727578538687,
+      "grad_norm": 0.10272197425365448,
+      "learning_rate": 0.008,
+      "loss": 3.920045852661133,
+      "step": 590
+    },
+    {
+      "epoch": 0.007977689062920698,
+      "grad_norm": 0.09875880181789398,
+      "learning_rate": 0.008,
+      "loss": 3.9055149078369142,
+      "step": 600
+    },
+    {
+      "epoch": 0.00811065054730271,
+      "grad_norm": 0.12342850863933563,
+      "learning_rate": 0.008,
+      "loss": 3.907426452636719,
+      "step": 610
+    },
+    {
+      "epoch": 0.008243612031684722,
+      "grad_norm": 0.11523553729057312,
+      "learning_rate": 0.008,
+      "loss": 3.8896995544433595,
+      "step": 620
+    },
+    {
+      "epoch": 0.00832338892231393,
+      "eval_loss": 3.8911144733428955,
+      "eval_runtime": 23.7786,
+      "eval_samples_per_second": 210.273,
+      "eval_steps_per_second": 26.284,
+      "step": 626
+    },
+    {
+      "epoch": 0.008376573516066733,
+      "grad_norm": 0.13027988374233246,
+      "learning_rate": 0.008,
+      "loss": 3.888309860229492,
+      "step": 630
+    },
+    {
+      "epoch": 0.008509535000448746,
+      "grad_norm": 0.10204915702342987,
+      "learning_rate": 0.008,
+      "loss": 3.8905193328857424,
+      "step": 640
+    },
+    {
+      "epoch": 0.008642496484830757,
+      "grad_norm": 0.09654007107019424,
+      "learning_rate": 0.008,
+      "loss": 3.88280029296875,
+      "step": 650
+    },
+    {
+      "epoch": 0.008775457969212768,
+      "grad_norm": 0.10360368341207504,
+      "learning_rate": 0.008,
+      "loss": 3.8796253204345703,
+      "step": 660
+    },
+    {
+      "epoch": 0.00890841945359478,
+      "grad_norm": 0.11431150883436203,
+      "learning_rate": 0.008,
+      "loss": 3.8688350677490235,
+      "step": 670
+    },
+    {
+      "epoch": 0.009041380937976792,
+      "grad_norm": 0.10246361047029495,
+      "learning_rate": 0.008,
+      "loss": 3.859950637817383,
+      "step": 680
+    },
+    {
+      "epoch": 0.0091477501254824,
+      "eval_loss": 3.8641164302825928,
+      "eval_runtime": 23.7195,
+      "eval_samples_per_second": 210.797,
+      "eval_steps_per_second": 26.35,
+      "step": 688
+    },
+    {
+      "epoch": 0.009174342422358803,
+      "grad_norm": 0.09362177550792694,
+      "learning_rate": 0.007473684210526316,
+      "loss": 3.8631008148193358,
+      "step": 690
+    },
+    {
+      "epoch": 0.009307303906740816,
+      "grad_norm": 0.07704541087150574,
+      "learning_rate": 0.006421052631578947,
+      "loss": 3.8439105987548827,
+      "step": 700
+    },
+    {
+      "epoch": 0.009440265391122827,
+      "grad_norm": 0.07900427281856537,
+      "learning_rate": 0.005368421052631579,
+      "loss": 3.8248409271240233,
+      "step": 710
+    },
+    {
+      "epoch": 0.009573226875504838,
+      "grad_norm": 0.06625134497880936,
+      "learning_rate": 0.00431578947368421,
+      "loss": 3.803970718383789,
+      "step": 720
+    },
+    {
+      "epoch": 0.00970618835988685,
+      "grad_norm": 0.05564470961689949,
+      "learning_rate": 0.0032631578947368424,
+      "loss": 3.791268539428711,
+      "step": 730
+    },
+    {
+      "epoch": 0.009839149844268862,
+      "grad_norm": 0.05231216922402382,
+      "learning_rate": 0.0022105263157894735,
+      "loss": 3.776142883300781,
+      "step": 740
+    },
+    {
+      "epoch": 0.00993222288333627,
+      "step": 747,
+      "total_flos": 6.099947743503974e+16,
+      "train_loss": 4.495604857223899,
+      "train_runtime": 3603.6272,
+      "train_samples_per_second": 53.99,
+      "train_steps_per_second": 0.211
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 760,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 5,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 6.099947743503974e+16,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

scheduler/wsd_20/REPORT.md

@@ -0,0 +1,38 @@
+# Experiment: wsd_20 — warmup-stable-decay, 20% linear decay to 0
+
+**Wall-clock:** 60 min · **Recipe:** best known (sdpa + bf16 mixed + muonq @ lr 8e-3), schedule: warmup 200 → stable 408 @ 8e-3 → linear decay to 0 over 152 steps (horizon `max_steps = 760`)
+
+## Result: NEW PROJECT BEST — bake 55.6/100, and the hot-LR hygiene tax is gone
+
+| metric | wsd_20 | constant-LR reference (`../../lr/lr_8e-3/`) |
+|---|---|---|
+| optimizer steps in 60 min | 730 | 759 |
+| final logged eval loss / ppl | 3.8023 / 44.8 (step 678, mid-decay — undersells the final model, see below) | 3.8583 / 47.4 |
+| bake score | **55.61/100 (DOUGH)** — record, +8.8 over reference | 46.83 |
+| held-out bits/byte | **1.2769** — record | 1.3114 |
+| chat bits/byte | **1.1571** — record | 1.1914 |
+| logic accuracy | **0.650, margin +0.0242** — record | 0.600, +0.018 |
+| hygiene (greedy loop) | **79.2/100 (loop 23.8w)** — back to cool-LR levels | 47.7 (59.8w) |
+| grad norms | 0.06–9.45, 0 non-finite | 0.09–9.45 |
+| peak GPU | 7.2 GB | 7.2 GB |
+
+The single biggest quality jump of any experiment in the project: +8.8 bake points over the previous record, best bpb/chat/logic ever, **and** greedy-loop hygiene restored to baseline levels (23.8w vs the 52–66w of every other hot-LR run). The SmolLM recipe works: park at peak LR, then anneal hard to 0 at the end.
+
+**Caveat — the decay was cut ~20% short.** The run reached 730 of the 760-step horizon when the 60-min stop fired, so the final model sat at LR 1.63e-3, not 0. The logged eval curve stops at step 678 (LR ≈ 2.9e-3, decay only 46% done), which is why the table's "final eval" undersells the saved model — the evaluate2 numbers above are measured on the actual `final/` (step 730). A horizon of ~730 would let the decay finish and should squeeze out a bit more.
+
+## Eval loss curve (5-min intervals)
+
+| step | eval loss | | step | eval loss |
+|---|---|---|---|---|
+| 66 | 5.7330 | | 429 | 4.0342 |
+| 128 | 4.9621 | | 492 | 3.9794 |
+| 186 | 4.5960 | | 554 | 3.9340 |
+| 245 | 4.3590 | | 616 | 3.8938 |
+| 306 | 4.2061 | | 678 | **3.8023** |
+| 367 | 4.1056 | | | |
+
+The last interval tells the story: 3.8938 → 3.8023 in 62 steps (−0.09) as the decay bit — more improvement in the decay phase than in the previous 15 minutes of stable-LR training.
+
+## Verdict
+
+**Winner of the scheduler series and the new reference recipe.** Decay-to-zero over the final 20% buys both quality (bake 46.8 → 55.6) and hygiene (59.8w → 23.8w loops) at zero cost. See `../COMPARISON.md`.

scheduler/wsd_20/config.toml

@@ -0,0 +1,89 @@
+# Scheduler series — new best recipe (sdpa + bf16 mixed + muonq @ lr 8e-3), only the LR schedule varies.
+# This run: WSD (warmup-stable-decay), SmolLM2-style — linear decay to 0 over the final 20% (152 steps).
+model_config = "../model.toml"
+
+[data]
+# Data files (supports glob patterns or lists of files)
+train_files = ["~/Dev/vintage-LLM/training/data/train_*.bin"]
+valid_files = ["~/Dev/vintage-LLM/training/data/valid_*.bin"]
+
+# Sequence length for each dataset sample
+# Must be <= model.max_position_embeddings
+max_seq_length = 1024
+
+# Tokenizer (must match the one used to create .bin files)
+tokenizer = "~/Dev/vintage-LLM/tokenizers/t-v3/"
+
+[training]
+# Training duration
+num_train_epochs = 1
+max_steps = 760                    # scheduler horizon = achievable steps/hour (was -1: full epoch, so decay never happened)
+
+# Optional: Stop training after this many minutes of wall-clock time.
+max_train_minutes = 60
+
+# Output directories
+output_dir = "./"
+final_model_dir = "./final"
+
+# Batch sizes
+# Effective batch size = num_devices × per_device_train_batch_size × gradient_accumulation_steps
+per_device_train_batch_size = 8    # batch size per GPU (8x1024 tokens fits 16GB with headroom for eval)
+per_device_eval_batch_size  = 8
+gradient_accumulation_steps = 32   # accumulate gradients over multiple mini-batches before updating the params
+
+# Optimizer: MuonQ (4-bit quantized Muon, ../../MuonQ repo) — wired through
+# optimizer_cls_and_kwargs in base_train.py. 2D hidden weights get Muon
+# (Newton-Schulz 5, 4-bit mu-law-companded momentum, rank-16 residual
+# factorization); embeddings/head/norms get its internal AdamW backup.
+# Muon/quantization knobs (muon_*, muonq_*) default to the repo's own
+# llama-60m "muonq" recipe in base_train.py.
+optim = "muonq"
+
+# Optimizer settings — lr/wd from MuonQ's llama-60m recipe (Muon takes
+# spectral-norm-scaled steps; do not copy the AdamW 5e-4 here)
+learning_rate = 8e-3
+weight_decay = 0.1
+adam_beta1 = 0.9
+adam_beta2 = 0.95
+max_grad_norm = 1.0
+
+# Learning rate scheduler
+lr_scheduler_type    = "warmup_stable_decay"   # warmup 200 -> stable 408 @ peak -> linear decay 152 steps to 0
+lr_scheduler_kwargs  = { num_stable_steps = 408, num_decay_steps = 152, decay_type = "linear", min_lr_ratio = 0.0 }
+warmup_steps = 200                 # 1-hour series standard (muonq-v2's 400 was a 2-hour recipe)
+
+# Precision (choose based on your GPU)
+bf16 = true                        # bfloat16 autocast + fp32 master weights (the dtype-series safe recipe)
+fp16 = false
+
+# Performance
+torch_compile = false
+gradient_checkpointing = false     # trade compute for memory (enables larger models)
+neftune_noise_alpha = 0.0          # add noise during training to improve model performance (fine-tuning)
+
+# Checkpointing
+# save_strategy can be "steps", "epoch", or "minutes" (wall-clock time).
+save_strategy = "minutes"
+save_steps = 5                    # checkpoint every X minutes
+save_total_limit = 3              # keep only X most recent checkpoints
+
+# Evaluation
+# eval_strategy can be "steps", "epoch", or "minutes" (wall-clock time).
+eval_strategy = "minutes"
+eval_steps = 5                   # evaluate every X minutes
+# Evaluate on a fixed random subset of the validation set (0 = use all)
+max_eval_samples = 5000
+
+# Logging
+logging_strategy = "steps"
+logging_steps = 10                 # log every 10 steps
+logging_first_step = true          # log the first step
+
+# Reproducibility
+seed = 42
+
+# Performance
+dataloader_num_workers = 2         # number of data loading threads
+dataloader_prefetch_factor = 2     # batches prefetched per worker
+dataloader_pin_memory = true       # pin memory for faster GPU transfer

scheduler/wsd_20/eval1.log

@@ -0,0 +1,205 @@
+W0814 14:06:57.324000 54418 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0814 14:06:57.348000 54418 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+
+==============================================================================
+CHECKPOINT INSPECTION
+==============================================================================
+  checkpoint : final
+  tokenizer  : final
+  device     : cuda   dtype: torch.bfloat16   seed: 1337
+
+
+==============================================================================
+SUITE: MODEL INFO AND TRAINING LINEAGE
+==============================================================================
+  checkpoint              : final
+  model_type              : llama
+  architecture            : LlamaForCausalLM
+  device                  : cuda
+  dtype                   : torch.bfloat16
+  parameters              : 77,083,392 (77.08M)
+  disk size               : 308 MB
+  layers / hidden / heads : 8 / 768 / 8 (KV heads: 4)
+  vocab (cfg / tokenizer) : 32768 / 32752
+  context length          : 1024
+  chat template           : True
+  training lineage        : base pretraining, ctx 1024
+  embedding mean norm     : 3.6401   <- typical row length of the embedding matrix; very small or huge values signal trouble
+  embedding mean cosine   : 0.1053   <- avg cosine between random embedding rows. Descriptive only: it starts near 0.0
+                            because RANDOM init is near-orthogonal, and it GROWS as the model trains.
+                            Near 0 therefore does NOT mean "healthy" - it can mean "barely trained".
+  (note: config vocab padded by 16 unused rows - harmless)
+
+==============================================================================
+SUITE: PERIOD FIDELITY ON FIXED PROBE SENTENCES
+==============================================================================
+  Scoring 20 sentences (10 historical, 10 modern)
+  The MODERN/HISTORICAL ratio is the output that means something. The absolute
+  numbers come from ~20 sentences: too noisy to rank checkpoints (use evaluate2.py).
+
+  OVERALL (270 tokens)
+    perplexity            : 257.77   <- 'average branching factor'; lower = less surprised = better fit to this text
+    mean token prob       : 0.0817   <- average confidence on the actual next word
+    median token prob     : 0.0078   <- typical confidence (robust to a few very bad tokens)
+    worst-token prob (min): 1.6e-08   <- the single most surprising word; tiny values = the model really didn't expect it
+    10th-percentile prob  : 1.3e-05   <- confidence on the model's worst 10% of guesses
+    low-confidence tokens : 52.96%   <- share of tokens predicted with <1% probability (red flags)
+    mean entropy          : 4.673 nats   <- how spread out the predictions are; ~0 = very certain, higher = hedging
+
+  GROUP COMPARISON
+    group        sentences    perplexity   bits/byte
+    historical          10         81.12       1.196
+    modern              10        678.25       1.945
+    modern/historical ratio: 8.36   <- >1 means period text is easier for the model than modern text (desired for a vintage model with a ~1900 knowledge cutoff)
+    (bits/byte is the tokenizer-independent version - the only column here you may
+     compare between models that use DIFFERENT tokenizers.)
+
+  PER-SENTENCE DETAIL (sorted worst-first)
+    [modern    ] ppl= 4265.97  minp=2.3e-07  The engineering team rewrote the whole backend micro
+    [modern    ] ppl= 2813.13  minp=1.5e-06  Robots on the automated assembly line handle the ent
+    [modern    ] ppl= 1315.86  minp=3.5e-06  Small retail startups now run almost all of their bu
+    [modern    ] ppl=  996.46  minp=1.5e-06  Three hours stuck in traffic in the pouring rain mad
+    [modern    ] ppl=  646.11  minp=1.5e-06  Millions of shoppers queue outside the store because
+    [modern    ] ppl=  505.02  minp=1.6e-08  Members of the European Union parliament voted on th
+    [modern    ] ppl=  500.69  minp=1.5e-05  She is finishing a graduate degree in machine learni
+    [modern    ] ppl=  358.83  minp=4.3e-07  She gave up her whole weekend to help me move apartm
+    [historical] ppl=  282.68  minp=1.2e-05  The keeper fed the great serpent which the naturalis
+    [modern    ] ppl=  248.15  minp=5.9e-06  I switched to a plant-based diet last year and I no 
+    [modern    ] ppl=  179.89  minp=2.2e-06  After years of hiding it from everyone at work, he t
+    [historical] ppl=  139.38  minp=1.0e-04  He drew a distinction so fine and over-scrupulous th
+    [historical] ppl=  100.21  minp=9.5e-06  The mountain rose above the valley in a silence sole
+    [historical] ppl=   90.91  minp=7.0e-05  The merchants of the port have grown rich upon their
+    [historical] ppl=   82.08  minp=1.2e-04  The weavers at their looms are employed in the wooll
+    [historical] ppl=   75.06  minp=3.8e-05  The Lord provideth for all his creatures, giving the
+    [historical] ppl=   69.82  minp=1.3e-05  The labouring poor of this parish are reduced to gre
+    [historical] ppl=   67.96  minp=4.6e-05  He gave up his fortune to the patient study of natur
+    [historical] ppl=   46.74  minp=1.3e-04  The ballroom was filled with dancing and laughter, a
+    [historical] ppl=   18.33  minp=3.5e-04  Her Majesty was pleased to summon the Lords and Comm
+
+==============================================================================
+SUITE: DIACHRONIC WORD-SENSE SEPARATION
+==============================================================================
+  Extracting contextual embeddings for 10 shifted words in period AND modern sentences.
+  Every probe word sits LAST in its sentence, so the causal model has actually read
+  the disambiguating context before it represents the word.
+
+  SENSE SEPARATION (cosine similarity of the SAME word, period vs modern sentence)
+    lower value = the model represents the two senses differently (good sign for a
+    period model); ~1.0 = it treats them as the same word regardless of context
+    want          : +0.343
+    python        : +0.492
+    awful         : +0.556
+    meat          : +0.621
+    parliament    : +0.623
+    nice          : +0.626
+    manufacture   : +0.670
+    gay           : +0.718
+    commerce      : +0.743
+    science       : +0.839
+    MEAN          : +0.623   <- near 1.0 = model ignores period vs modern usage; lower = senses differ
+
+  PAIRWISE COSINE SIMILARITY (period sentences, first 5 words)
+    these are CONTEXTUAL vectors, so they reflect the sentences as much as the words;
+    read the spread, not individual pairs. High mean = representations collapsed.
+                         gay      awful       nice       meat       want
+    gay                1.000      0.451      0.346      0.314      0.315
+    awful                         1.000      0.358      0.259      0.326
+    nice                                     1.000      0.294      0.345
+    meat                                                1.000      0.473
+    want                                                           1.000
+    all 45 pairs: mean=+0.370 std=0.118   <- healthy spread is good
+
+  NEAREST NEIGHBOURS inside the probe-word set (period sentences)
+    gay           : awful (+0.45), science (+0.40), commerce (+0.37), nice (+0.35), manufacture (+0.34)
+    awful         : gay (+0.45), science (+0.40), nice (+0.36), parliament (+0.35), want (+0.33)
+    nice          : science (+0.39), awful (+0.36), gay (+0.35), want (+0.35), manufacture (+0.33)
+    meat          : science (+0.48), want (+0.47), parliament (+0.45), commerce (+0.44), manufacture (+0.43)
+    want          : manufacture (+0.56), commerce (+0.55), science (+0.52), meat (+0.47), parliament (+0.46)
+
+==============================================================================
+SUITE: GENERATION PROBES
+==============================================================================
+
+  --- Probe 1 ---
+  prompt      : The history of the world is
+  continuation: too often confused and obscure; but the reader will find a strange and strange interest in the history of the world. It was one of the most remarkable events of modern history that the history of the world was a very simple one; and there are a thousand evidences of the truth of the present, which
+  stats: 60 tokens | distinct-1 0.64 | distinct-2 0.89 | echo 0.11
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 2 ---
+  prompt      : What is God? God is
+  continuation: a man! He is a man who wants to be a man, but has a power to do nothing. It is God's work, and God's work.
+
+If God has power to do something, and he does nothing, he is a man. If God's will is to do
+  stats: 60 tokens | distinct-1 0.60 | distinct-2 0.91 | echo 0.09
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 3 ---
+  prompt      : In the year of our Lord eighteen hundred and
+  continuation: forty-eight; and the Lord shall be called, in the name of the Lord, and the glory of the world."
+
+The second year of the Lord's ministry, was not a very long one. The first year of the Lord's ministry was a new one. A new ministry was
+  stats: 60 tokens | distinct-1 0.60 | distinct-2 0.85 | echo 0.09
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 4 ---
+  prompt      : The manufacture of cotton
+  continuation: -growing, and the manufacture of silk and linen-growing machinery, has been carried on with the utmost care and energy. The cotton, as well as woollen fabrics, are manufactured and woven. The silk, as a trade, is now carried on in a great measure, and its production is
+  stats: 60 tokens | distinct-1 0.74 | distinct-2 0.98 | echo 0.02
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 5 ---
+  prompt      : The steam engine
+  continuation: is the main factor, and in the latter respect the latter has more than equal claim to the other. In addition to the steam power and the steam power, it is said to be of greater advantage to the steam power, in the first place, to the use of the power to prevent a rapid
+  stats: 60 tokens | distinct-1 0.58 | distinct-2 0.85 | echo 0.13
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 6 ---
+  prompt      : The telegraph
+  continuation: office and telegraph offices, is one of the most important of the great works of the day. The telegraph office is the only building for the telegraph office. It is a large and commodious building, built entirely of stone, and furnished with a number of iron and steel wires, and with the greatest care
+  stats: 60 tokens | distinct-1 0.63 | distinct-2 0.96 | echo 0.11
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 7 ---
+  prompt      : My dearest sister,
+  continuation: a very good-natured young fellow, and very fond of her, for whom I have no doubt I have a very good reason to be grateful.
+
+You will remember, my dear friend, how little I have been able to do, and how much I have had to do. You may
+  stats: 60 tokens | distinct-1 0.71 | distinct-2 0.92 | echo 0.04
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 8 ---
+  prompt      : Greetings, my friend
+  continuation: , I shall take a short drive, if you please.
+
+I will take a walk with you in the fields, with the birds, and in the woods. I will take my ride with you and go to the station.
+
+It is a fine day, and I shall be glad
+  stats: 60 tokens | distinct-1 0.61 | distinct-2 0.88 | echo 0.04
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 9 ---
+  prompt      : Her Majesty the Queen
+  continuation: of Scots, in the name of the king, was pleased to place herself under the title of a Queen, and to act according to her pleasure, so that she would never be married until she should be entitled to her crown; and, after all the expense of her life, she should have the
+  stats: 60 tokens | distinct-1 0.70 | distinct-2 0.96 | echo 0.04
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  --- Probe 10 ---
+  prompt      : Chapter I.
+  continuation: The first question is whether a single sentence is not contained in the book of Judges and whether the second is not an exception to the general rule of the Judges.
+
+If the words "degenerate" be taken as referring to a sentence in which the meaning is not "degener
+  stats: 60 tokens | distinct-1 0.67 | distinct-2 0.96 | echo 0.04
+         (distinct-n = vocabulary/phrase variety, near 0 = looping; echo = fraction of words repeated from the previous 4)
+
+  AGGREGATE over 10 probes: 600 tokens in 0.9s (689.4 tok/s)
+    mean distinct-1 0.649 | mean distinct-2 0.916 | mean echo 0.070 | worst probe: distinct-2 0.848 / echo 0.127
+    (sampling hides loops that greedy decoding exposes; evaluate2.py measures the greedy loop length.)
+
+Full numeric results written to /home/cro/Dev/vintage-LLM/training/eval_results/evaluate-final.json
+
+==============================================================================
+INSPECTION COMPLETE
+==============================================================================
+  This covered WHAT the checkpoint is, its lineage, and whether it is still vintage.
+  For "is it baked enough", run evaluate2.py on the same path.

scheduler/wsd_20/eval2.log

@@ -0,0 +1,54 @@
+evaluating 1 checkpoint(s) on cuda (torch.bfloat16)
+tokenizer: final
+  final ...W0814 14:07:03.865000 54494 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0814 14:07:03.881000 54494 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+
+ bake 56/100  bpb 1.2769  logic 0.65  (15.2s)
+
+report: eval2.md
+json:   eval2.json
+
+# Evaluation: final
+
+- **Parameters:** 77M
+- **Note:** no trainer_state.json in the checkpoint - tokens seen unknown
+
+## Verdict: HALF-BAKED  (bake score 56/100)
+
+Clearly undertrained. The style is there but the substance is not: expect confident nonsense, topic drift and heavy looping under greedy decoding. Do not fine-tune this for chat yet - keep pretraining.
+
+## Where it sits (held-out bits/byte, lower = better)
+
+```
+      3.50   untrained model (uniform noise)
+      2.00   word-salad
+      1.50   broken prose
+      1.33   500M model, ~0.7B tokens (early training)
+  --> 1.277  YOUR MODEL
+      1.19   500M model, ~4.7B tokens (undertrained but solid)
+      1.10   best sub-1B measured on this data (341M)
+      0.95   estimated sub-1B ceiling on this corpus
+```
+
+## Scores
+
+| what | raw value | points /100 | weight | plain English |
+|---|---|---|---|---|
+| Held-out loss | 1.2769 bits/byte | 61 | 0.5 | how cheaply it predicts period text it never saw — the single best training signal |
+| Logic | 0.650 acc, margin 0.024 | 36 | 0.25 | picks the *sensible* continuation over matched nonsense; 0.50 = coin-flip. normal for a good sub-1B model (Vintage1 scores 0.70-0.78) |
+| Chat readiness | 1.1571 bits/byte | 56 | 0.15 | how cheap well-formed period dialogue already is — predicts fine-tuning ease |
+| Hygiene | loop 23.8w, punct 0.00/100w | 79 | 0.1 | greedy-decoding loop length and broken punctuation |
+
+## Period boundary
+
+Clean. All 6 post-1900 trap words cost the model more than their period twins (mean shock +1.37 bits/byte, weakest pair 'motor car' at +0.82). No sign of modern text in training.
+
+## See for yourself (sampled, t=0.8)
+
+> **LONDON, Tuesday. — The committee appointed to inquire into the condition of the** Irish population throughout the country and the condition of the Irish people of Ireland during the last week has recently given Notice of the fact that the population of the country is increasing greatly more than was stated, the amount of the population being increasing in proportion. The committee reported that the Irish population in the United Kingdom is increasing in proportion as the Irish 
+>
+> **A melancholy accident occurred on Thursday last at the works of Messrs. Harding and** Co., of Manchester, and on the 8th of November, 1842, the Failing to arrive at the place of meeting, a deputation was sent to the various counties and towns, to solicit the services of some influential persons, for the accommodation of the people, to aid in the work of the evangelical cause. The meeting was called a meeting of the members of the church, and was most cordially received by many memb
+>
+> **Brethren, the text which I have chosen for our consideration this morning is taken from** the work of the Brethren. We have not yet done anything for the Brethren. Our church is now filled to overflowing with a host of Christian men, to whom every one of the Brethren has a brotherly affection. We have been looking forward to a time when the work of the Brethren was more deeply felt by the Brethren. While we are thus engaged, we trust that the work of the Brethren will not cease to be a
+>
+

scheduler/wsd_20/final/chat_template.jinja

@@ -0,0 +1,7 @@
+{%- for message in messages -%}
+  {{- bos_token + '<|' + message['role'] + '|>\n' + message['content'].strip() + eos_token + '\n' -}}
+{%- endfor -%}
+
+{%- if add_generation_prompt -%}
+  {{- bos_token + '<|assistant|>\n' -}}
+{%- endif -%}

scheduler/wsd_20/final/config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 3,
+  "dtype": "float32",
+  "eos_token_id": 4,
+  "head_dim": 96,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 2048,
+  "max_position_embeddings": 1024,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 8,
+  "num_hidden_layers": 8,
+  "num_key_value_heads": 4,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.14.1",
+  "use_cache": false,
+  "vocab_size": 32768
+}

scheduler/wsd_20/final/generation_config.json

@@ -0,0 +1,13 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 3,
+  "eos_token_id": [
+    4,
+    2
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 0,
+  "transformers_version": "5.14.1",
+  "use_cache": false
+}

scheduler/wsd_20/final/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6a27d8bfc67c1bedb2e260f117c0b9f408942ca4d036cfb7606c6076d0db8061
+size 308341784

scheduler/wsd_20/final/tokenizer_config.json

@@ -0,0 +1,11 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|unk|>"
+}

scheduler/wsd_20/trainer_state.json

@@ -0,0 +1,649 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.009785965250516056,
+  "eval_steps": 5,
+  "global_step": 736,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 1.3296148438201165e-05,
+      "grad_norm": 5.513442516326904,
+      "learning_rate": 0.0,
+      "loss": 10.538199424743652,
+      "step": 1
+    },
+    {
+      "epoch": 0.00013296148438201165,
+      "grad_norm": 5.268533706665039,
+      "learning_rate": 0.00035999999999999997,
+      "loss": 10.136458502875435,
+      "step": 10
+    },
+    {
+      "epoch": 0.0002659229687640233,
+      "grad_norm": 1.034209132194519,
+      "learning_rate": 0.00076,
+      "loss": 8.0193359375,
+      "step": 20
+    },
+    {
+      "epoch": 0.0003988844531460349,
+      "grad_norm": 9.450255393981934,
+      "learning_rate": 0.00116,
+      "loss": 6.9860687255859375,
+      "step": 30
+    },
+    {
+      "epoch": 0.0005318459375280466,
+      "grad_norm": 5.685102462768555,
+      "learning_rate": 0.0015600000000000002,
+      "loss": 6.707894897460937,
+      "step": 40
+    },
+    {
+      "epoch": 0.0006648074219100582,
+      "grad_norm": 2.160360097885132,
+      "learning_rate": 0.00196,
+      "loss": 6.287691116333008,
+      "step": 50
+    },
+    {
+      "epoch": 0.0007977689062920698,
+      "grad_norm": 1.6388428211212158,
+      "learning_rate": 0.00236,
+      "loss": 5.944025039672852,
+      "step": 60
+    },
+    {
+      "epoch": 0.0008775457969212769,
+      "eval_loss": 5.732973575592041,
+      "eval_runtime": 22.9886,
+      "eval_samples_per_second": 217.499,
+      "eval_steps_per_second": 27.187,
+      "step": 66
+    },
+    {
+      "epoch": 0.0009307303906740815,
+      "grad_norm": 1.2074735164642334,
+      "learning_rate": 0.00276,
+      "loss": 5.747131729125977,
+      "step": 70
+    },
+    {
+      "epoch": 0.0010636918750560932,
+      "grad_norm": 1.132364273071289,
+      "learning_rate": 0.00316,
+      "loss": 5.60839729309082,
+      "step": 80
+    },
+    {
+      "epoch": 0.0011966533594381047,
+      "grad_norm": 1.0884277820587158,
+      "learning_rate": 0.0035600000000000002,
+      "loss": 5.502901077270508,
+      "step": 90
+    },
+    {
+      "epoch": 0.0013296148438201164,
+      "grad_norm": 1.3878921270370483,
+      "learning_rate": 0.00396,
+      "loss": 5.371665573120117,
+      "step": 100
+    },
+    {
+      "epoch": 0.001462576328202128,
+      "grad_norm": 1.2677963972091675,
+      "learning_rate": 0.00436,
+      "loss": 5.2512672424316404,
+      "step": 110
+    },
+    {
+      "epoch": 0.0015955378125841397,
+      "grad_norm": 0.9181460738182068,
+      "learning_rate": 0.0047599999999999995,
+      "loss": 5.126389694213867,
+      "step": 120
+    },
+    {
+      "epoch": 0.001701907000089749,
+      "eval_loss": 4.962139129638672,
+      "eval_runtime": 24.8338,
+      "eval_samples_per_second": 201.339,
+      "eval_steps_per_second": 25.167,
+      "step": 128
+    },
+    {
+      "epoch": 0.0017284992969661514,
+      "grad_norm": 0.7834004163742065,
+      "learning_rate": 0.0051600000000000005,
+      "loss": 5.010536575317383,
+      "step": 130
+    },
+    {
+      "epoch": 0.001861460781348163,
+      "grad_norm": 0.7623927593231201,
+      "learning_rate": 0.00556,
+      "loss": 4.908084106445313,
+      "step": 140
+    },
+    {
+      "epoch": 0.0019944222657301744,
+      "grad_norm": 0.5423034429550171,
+      "learning_rate": 0.00596,
+      "loss": 4.820485305786133,
+      "step": 150
+    },
+    {
+      "epoch": 0.0021273837501121864,
+      "grad_norm": 0.5046894550323486,
+      "learning_rate": 0.00636,
+      "loss": 4.753760528564453,
+      "step": 160
+    },
+    {
+      "epoch": 0.002260345234494198,
+      "grad_norm": 0.3807763159275055,
+      "learning_rate": 0.0067599999999999995,
+      "loss": 4.694861602783203,
+      "step": 170
+    },
+    {
+      "epoch": 0.0023933067188762094,
+      "grad_norm": 0.37175047397613525,
+      "learning_rate": 0.0071600000000000006,
+      "loss": 4.652392578125,
+      "step": 180
+    },
+    {
+      "epoch": 0.0024730836095054167,
+      "eval_loss": 4.595963954925537,
+      "eval_runtime": 23.9837,
+      "eval_samples_per_second": 208.475,
+      "eval_steps_per_second": 26.059,
+      "step": 186
+    },
+    {
+      "epoch": 0.0025262682032582214,
+      "grad_norm": 0.3488306999206543,
+      "learning_rate": 0.00756,
+      "loss": 4.597007369995117,
+      "step": 190
+    },
+    {
+      "epoch": 0.002659229687640233,
+      "grad_norm": 0.3351574242115021,
+      "learning_rate": 0.00796,
+      "loss": 4.566041946411133,
+      "step": 200
+    },
+    {
+      "epoch": 0.0027921911720222444,
+      "grad_norm": 0.3297984302043915,
+      "learning_rate": 0.008,
+      "loss": 4.5214378356933596,
+      "step": 210
+    },
+    {
+      "epoch": 0.002925152656404256,
+      "grad_norm": 0.2526053190231323,
+      "learning_rate": 0.008,
+      "loss": 4.468568420410156,
+      "step": 220
+    },
+    {
+      "epoch": 0.003058114140786268,
+      "grad_norm": 0.24721451103687286,
+      "learning_rate": 0.008,
+      "loss": 4.431998825073242,
+      "step": 230
+    },
+    {
+      "epoch": 0.0031910756251682794,
+      "grad_norm": 0.2912442684173584,
+      "learning_rate": 0.008,
+      "loss": 4.396870040893555,
+      "step": 240
+    },
+    {
+      "epoch": 0.0032575563673592853,
+      "eval_loss": 4.358976364135742,
+      "eval_runtime": 24.0964,
+      "eval_samples_per_second": 207.5,
+      "eval_steps_per_second": 25.937,
+      "step": 245
+    },
+    {
+      "epoch": 0.003324037109550291,
+      "grad_norm": 0.28560253977775574,
+      "learning_rate": 0.008,
+      "loss": 4.367413711547852,
+      "step": 250
+    },
+    {
+      "epoch": 0.003456998593932303,
+      "grad_norm": 0.21196810901165009,
+      "learning_rate": 0.008,
+      "loss": 4.331097793579102,
+      "step": 260
+    },
+    {
+      "epoch": 0.0035899600783143143,
+      "grad_norm": 0.22107456624507904,
+      "learning_rate": 0.008,
+      "loss": 4.302482604980469,
+      "step": 270
+    },
+    {
+      "epoch": 0.003722921562696326,
+      "grad_norm": 0.1877111792564392,
+      "learning_rate": 0.008,
+      "loss": 4.268944931030274,
+      "step": 280
+    },
+    {
+      "epoch": 0.0038558830470783374,
+      "grad_norm": 0.21084874868392944,
+      "learning_rate": 0.008,
+      "loss": 4.2542469024658205,
+      "step": 290
+    },
+    {
+      "epoch": 0.003988844531460349,
+      "grad_norm": 0.20695072412490845,
+      "learning_rate": 0.008,
+      "loss": 4.227492904663086,
+      "step": 300
+    },
+    {
+      "epoch": 0.004068621422089557,
+      "eval_loss": 4.206057071685791,
+      "eval_runtime": 24.3778,
+      "eval_samples_per_second": 205.104,
+      "eval_steps_per_second": 25.638,
+      "step": 306
+    },
+    {
+      "epoch": 0.004121806015842361,
+      "grad_norm": 0.19029781222343445,
+      "learning_rate": 0.008,
+      "loss": 4.208528518676758,
+      "step": 310
+    },
+    {
+      "epoch": 0.004254767500224373,
+      "grad_norm": 0.1889178603887558,
+      "learning_rate": 0.008,
+      "loss": 4.187363815307617,
+      "step": 320
+    },
+    {
+      "epoch": 0.004387728984606384,
+      "grad_norm": 0.19070716202259064,
+      "learning_rate": 0.008,
+      "loss": 4.173851776123047,
+      "step": 330
+    },
+    {
+      "epoch": 0.004520690468988396,
+      "grad_norm": 0.15929149091243744,
+      "learning_rate": 0.008,
+      "loss": 4.148160171508789,
+      "step": 340
+    },
+    {
+      "epoch": 0.004653651953370408,
+      "grad_norm": 0.17022773623466492,
+      "learning_rate": 0.008,
+      "loss": 4.132599258422852,
+      "step": 350
+    },
+    {
+      "epoch": 0.004786613437752419,
+      "grad_norm": 0.18009495735168457,
+      "learning_rate": 0.008,
+      "loss": 4.132373809814453,
+      "step": 360
+    },
+    {
+      "epoch": 0.004879686476819827,
+      "eval_loss": 4.1056389808654785,
+      "eval_runtime": 23.9848,
+      "eval_samples_per_second": 208.465,
+      "eval_steps_per_second": 26.058,
+      "step": 367
+    },
+    {
+      "epoch": 0.004919574922134431,
+      "grad_norm": 0.17529530823230743,
+      "learning_rate": 0.008,
+      "loss": 4.117653656005859,
+      "step": 370
+    },
+    {
+      "epoch": 0.005052536406516443,
+      "grad_norm": 0.1463332176208496,
+      "learning_rate": 0.008,
+      "loss": 4.096078109741211,
+      "step": 380
+    },
+    {
+      "epoch": 0.005185497890898454,
+      "grad_norm": 0.14370259642601013,
+      "learning_rate": 0.008,
+      "loss": 4.084194946289062,
+      "step": 390
+    },
+    {
+      "epoch": 0.005318459375280466,
+      "grad_norm": 0.16615846753120422,
+      "learning_rate": 0.008,
+      "loss": 4.075920867919922,
+      "step": 400
+    },
+    {
+      "epoch": 0.005451420859662477,
+      "grad_norm": 0.15509292483329773,
+      "learning_rate": 0.008,
+      "loss": 4.054693603515625,
+      "step": 410
+    },
+    {
+      "epoch": 0.005584382344044489,
+      "grad_norm": 0.1333627849817276,
+      "learning_rate": 0.008,
+      "loss": 4.047085952758789,
+      "step": 420
+    },
+    {
+      "epoch": 0.0057040476799882995,
+      "eval_loss": 4.034159183502197,
+      "eval_runtime": 24.31,
+      "eval_samples_per_second": 205.676,
+      "eval_steps_per_second": 25.71,
+      "step": 429
+    },
+    {
+      "epoch": 0.005717343828426501,
+      "grad_norm": 0.12619031965732574,
+      "learning_rate": 0.008,
+      "loss": 4.037934494018555,
+      "step": 430
+    },
+    {
+      "epoch": 0.005850305312808512,
+      "grad_norm": 0.14818833768367767,
+      "learning_rate": 0.008,
+      "loss": 4.030204772949219,
+      "step": 440
+    },
+    {
+      "epoch": 0.005983266797190524,
+      "grad_norm": 0.11952793598175049,
+      "learning_rate": 0.008,
+      "loss": 4.024636077880859,
+      "step": 450
+    },
+    {
+      "epoch": 0.006116228281572536,
+      "grad_norm": 0.12302638590335846,
+      "learning_rate": 0.008,
+      "loss": 4.0040229797363285,
+      "step": 460
+    },
+    {
+      "epoch": 0.006249189765954547,
+      "grad_norm": 0.15813544392585754,
+      "learning_rate": 0.008,
+      "loss": 4.0008289337158205,
+      "step": 470
+    },
+    {
+      "epoch": 0.006382151250336559,
+      "grad_norm": 0.11854846030473709,
+      "learning_rate": 0.008,
+      "loss": 3.994869613647461,
+      "step": 480
+    },
+    {
+      "epoch": 0.006515112734718571,
+      "grad_norm": 0.14335912466049194,
+      "learning_rate": 0.008,
+      "loss": 3.9867435455322267,
+      "step": 490
+    },
+    {
+      "epoch": 0.006541705031594972,
+      "eval_loss": 3.9793663024902344,
+      "eval_runtime": 23.0058,
+      "eval_samples_per_second": 217.336,
+      "eval_steps_per_second": 27.167,
+      "step": 492
+    },
+    {
+      "epoch": 0.006648074219100582,
+      "grad_norm": 0.10275301337242126,
+      "learning_rate": 0.008,
+      "loss": 3.9850521087646484,
+      "step": 500
+    },
+    {
+      "epoch": 0.006781035703482594,
+      "grad_norm": 0.12899844348430634,
+      "learning_rate": 0.008,
+      "loss": 3.9827651977539062,
+      "step": 510
+    },
+    {
+      "epoch": 0.006913997187864606,
+      "grad_norm": 0.1306857168674469,
+      "learning_rate": 0.008,
+      "loss": 3.962184524536133,
+      "step": 520
+    },
+    {
+      "epoch": 0.007046958672246617,
+      "grad_norm": 0.10865853726863861,
+      "learning_rate": 0.008,
+      "loss": 3.951933670043945,
+      "step": 530
+    },
+    {
+      "epoch": 0.007179920156628629,
+      "grad_norm": 0.11110885441303253,
+      "learning_rate": 0.008,
+      "loss": 3.94482536315918,
+      "step": 540
+    },
+    {
+      "epoch": 0.007312881641010641,
+      "grad_norm": 0.10243042558431625,
+      "learning_rate": 0.008,
+      "loss": 3.942932891845703,
+      "step": 550
+    },
+    {
+      "epoch": 0.007366066234763445,
+      "eval_loss": 3.933987855911255,
+      "eval_runtime": 23.7581,
+      "eval_samples_per_second": 210.455,
+      "eval_steps_per_second": 26.307,
+      "step": 554
+    },
+    {
+      "epoch": 0.007445843125392652,
+      "grad_norm": 0.10774345695972443,
+      "learning_rate": 0.008,
+      "loss": 3.9434852600097656,
+      "step": 560
+    },
+    {
+      "epoch": 0.007578804609774664,
+      "grad_norm": 0.10925770550966263,
+      "learning_rate": 0.008,
+      "loss": 3.9292713165283204,
+      "step": 570
+    },
+    {
+      "epoch": 0.007711766094156675,
+      "grad_norm": 0.10408780723810196,
+      "learning_rate": 0.008,
+      "loss": 3.917940139770508,
+      "step": 580
+    },
+    {
+      "epoch": 0.007844727578538687,
+      "grad_norm": 0.10272197425365448,
+      "learning_rate": 0.008,
+      "loss": 3.920045852661133,
+      "step": 590
+    },
+    {
+      "epoch": 0.007977689062920698,
+      "grad_norm": 0.09875880181789398,
+      "learning_rate": 0.008,
+      "loss": 3.9055149078369142,
+      "step": 600
+    },
+    {
+      "epoch": 0.00811065054730271,
+      "grad_norm": 0.12342850863933563,
+      "learning_rate": 0.007947368421052632,
+      "loss": 3.907426452636719,
+      "step": 610
+    },
+    {
+      "epoch": 0.008190427437931917,
+      "eval_loss": 3.893834352493286,
+      "eval_runtime": 23.1899,
+      "eval_samples_per_second": 215.611,
+      "eval_steps_per_second": 26.951,
+      "step": 616
+    },
+    {
+      "epoch": 0.008243612031684722,
+      "grad_norm": 0.1073056310415268,
+      "learning_rate": 0.007421052631578947,
+      "loss": 3.886861038208008,
+      "step": 620
+    },
+    {
+      "epoch": 0.008376573516066733,
+      "grad_norm": 0.09036897122859955,
+      "learning_rate": 0.006894736842105263,
+      "loss": 3.8772003173828127,
+      "step": 630
+    },
+    {
+      "epoch": 0.008509535000448746,
+      "grad_norm": 0.0773363932967186,
+      "learning_rate": 0.006368421052631579,
+      "loss": 3.8692962646484377,
+      "step": 640
+    },
+    {
+      "epoch": 0.008642496484830757,
+      "grad_norm": 0.08151130378246307,
+      "learning_rate": 0.005842105263157895,
+      "loss": 3.8507423400878906,
+      "step": 650
+    },
+    {
+      "epoch": 0.008775457969212768,
+      "grad_norm": 0.08117716759443283,
+      "learning_rate": 0.00531578947368421,
+      "loss": 3.837518310546875,
+      "step": 660
+    },
+    {
+      "epoch": 0.00890841945359478,
+      "grad_norm": 0.06505687534809113,
+      "learning_rate": 0.004789473684210526,
+      "loss": 3.8176422119140625,
+      "step": 670
+    },
+    {
+      "epoch": 0.009014788641100389,
+      "eval_loss": 3.802332878112793,
+      "eval_runtime": 23.6775,
+      "eval_samples_per_second": 211.171,
+      "eval_steps_per_second": 26.396,
+      "step": 678
+    },
+    {
+      "epoch": 0.009041380937976792,
+      "grad_norm": 0.07074195891618729,
+      "learning_rate": 0.004263157894736843,
+      "loss": 3.797995376586914,
+      "step": 680
+    },
+    {
+      "epoch": 0.009174342422358803,
+      "grad_norm": 0.06876317411661148,
+      "learning_rate": 0.0037368421052631578,
+      "loss": 3.7925949096679688,
+      "step": 690
+    },
+    {
+      "epoch": 0.009307303906740816,
+      "grad_norm": 0.060418933629989624,
+      "learning_rate": 0.0032105263157894736,
+      "loss": 3.7755409240722657,
+      "step": 700
+    },
+    {
+      "epoch": 0.009440265391122827,
+      "grad_norm": 0.05723872035741806,
+      "learning_rate": 0.0026842105263157894,
+      "loss": 3.764757537841797,
+      "step": 710
+    },
+    {
+      "epoch": 0.009573226875504838,
+      "grad_norm": 0.05702441930770874,
+      "learning_rate": 0.002157894736842105,
+      "loss": 3.7531177520751955,
+      "step": 720
+    },
+    {
+      "epoch": 0.00970618835988685,
+      "grad_norm": 0.05637257546186447,
+      "learning_rate": 0.0016315789473684212,
+      "loss": 3.7494365692138674,
+      "step": 730
+    },
+    {
+      "epoch": 0.009785965250516056,
+      "step": 736,
+      "total_flos": 6.010122542461747e+16,
+      "train_loss": 4.4993856588135595,
+      "train_runtime": 3603.0923,
+      "train_samples_per_second": 53.998,
+      "train_steps_per_second": 0.211
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 760,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 5,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 6.010122542461747e+16,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}