# FSI Anomaly — Agent Notes (living document) > **What this file is:** the single chronological record of everything we do on > this project — what we tried, what worked, what failed, the research behind > each decision, and where we currently stand. Every session / every agent > updates it. It feeds (a) continuity between sessions and between the human > team and the AI collaborators, and (b) the end-of-project war story, the > paper, and the training-document set at the end. > > **Rule:** after ANY change, trial, measurement, or decision → update this > file AND `CHANGELOG.md` with the date and honest numbers. Never silently > re-roll a failed experiment; record it first. --- ## PICK UP HERE — 2026-08-13 (current state, for any new machine/agent) ### What this is - **FSI-Anomaly**: an on-device, offline forensic-journalism / truth-verifier LLM. Verifies claims, finds discrepancies and cross-source patterns, and answers in Spock-style verdicts: `true / false / misleading / overclaim / contradiction / abstain / unsubstantiated`. Built for researchers, journalists, whistleblowers, and rabbit-hole truth-seeking. - **Owner/team**: solo developer (licensed electrician, no funding), label FERRELL SYNTHETIC INTELLIGENCE (FSI). Goal: free open-source small-model family, HF downloads, grants/loans. - **Device**: 8-core ARMv9 tablet, 7.4GB RAM + 12.3GB swap, no GPU. Everything below was trained/reasoned on this device. ### Model line (IMPORTANT — do not mix with the old 25M line) - **Current line: `hybrid50m` + 16k tokenizer** (~50M params, d_model=320, liquid architecture with MTP in pretrain only). The old 25M/8k line is archived history (`ckpt/tiny25m*`, 8k tokenizer, old analyst stamp schema). - **Canonical pretrain base:** `ckpt/hybrid50m_v16k_pretrain/model_5000.pt`. - **Tokenizer:** `data/tokenizer16k.json`. ALWAYS pass `--tok data/tokenizer16k.json`; the default 8k tokenizer crashes on 16k checkpoints. ### Where we are (measured 2026-08-13) - **SFT v25** (`ckpt/hybrid50m_v25_lora/best.pt`, 268 rows: 119 base + 149 gold): battery main **0.184** / researcher **0.167** / red-team **0.038**. - **DPO v25** (`ckpt/hybrid50m_v25_dpo/model_final.pt`, β=5.0 LFM2, 136 schema-matched pairs, 1 epoch): loss 0.7146→0.6787 (real signal, NO v22 collapse), battery = exact parity with SFT. DPO is no longer destructive; it needs more pairs + a better SFT base. - **Merges** (`ckpt/hybrid50m_v25_merges/`): main-battery results so far — soup **0.184**, taskarith_l0p5 **0.204**, ties_k0p2 in progress (all vs 0.184 baseline). - **Release gates NOT passed:** main ≥0.40 / researcher ≥0.25 at ≥60% coverage, red-team pass, multi-turn + real-task verification. Do NOT release until these pass. ### Data (all handcrafted, no generators/scripts — discipline rule) - **Gold: 198 / 3000** authored (`data/gold_700|800|900|1000`, `gold_3000_final`). - **SFT staged:** `data/sft_v26.jsonl` = 317 rows (119 base + 198 gold, 28/class × 7 + 2 final). - **Preference pairs:** `data/prefs_v26.jsonl` = 154 schema-matched (136 v23 + 18 new). ### What is running on the tablet right now (2026-08-13) - Chained, detached, one heavy job at a time: merge candidate evals (main battery) -> `train/watchdog_lora_v26.sh` (SFT v26, ~2h) -> `train/watchdog_dpo_v26.sh` (DPO v26, ~20 min). - After v26: battery-eval SFT v26 + DPO v26, merge v26 candidates, then red-team + multi-turn pipeline verification. Author more gold toward 3,000 in parallel. ### Key commands (verified recipe) - LoRA SFT: `train/train_lora.py --base ckpt/hybrid50m_v16k_pretrain --data --tok data/tokenizer16k.json --ckpt --val-bin data/valid16k.bin --replay-bin data/train_phase2b16k.bin --replay-ratio 0.5 --epochs 3 --batch 4 --seq 512 --lr 5e-5 --r 16 --alpha 32.0 --dropout 0.05 --kl 0.1 --ppl-guard 60.0 --eval-every 25 --val-batches 4 --threads 6` - LFM2 DPO: `train/train_dpo.py --base --data --tok data/tokenizer16k.json --ckpt --val-bin data/valid16k.bin --epochs 1 --batch 4 --seq 512 --lr 8e-7 --lr-schedule cosine --lr-min 8e-8 --warmup-frac 0.01 --beta 5.0 --margin 0.1 --apo-weight 0.2 --threads 6 --ppl-guard 60.0 --lora --r 16 --alpha 32.0 --dropout 0.05` (NO `--log-every` flag exists). - Launch heavy jobs detached: `setsid nohup ./train/.sh >/dev/null 2>&1 always use resume-safe watchdogs + `.done` markers. - Never stack two heavy torch jobs (RAM tight). One heavy job at a time. - MTP is pretrain-only: post-training loads use `cfg.mtp_heads=0` + `strict=False`. - Merges: pretrain base has `mtp_heads.*` keys that folded post-training ckpts lack; merge scripts intersect keys (regression-tested in `tests/test_merges.py`). ## 1. What this project is - **Model:** `fsi-anomaly` (family: `fsi_felon-*`), a **25.4M-parameter** custom liquid-architecture language model, trained ENTIRELY on an 8-core ARM tablet, no GPU, no big-compute. - **Purpose:** an on-device, offline research/journalism partner for dark-web/.onion research — verify and debunk claims, find patterns and discrepancies, inspect history/what-was-said vs what-the-record-shows, abstain when unverifiable. Built for conspiracy researchers, whistleblowers, and anyone who wants the truth without the paywall. - **Owner/team:** solo developer (licensed electrician, no funding), building under **FERRELL SYNTHETIC INTELLIGENCE (FSI)**. Goal: a free open-source small-model family + enough HF downloads for grants/loans. - **Launch identity:** Hugging Face user `FerrellSyntheticIntelligence`, repo `fsi-anomaly` (`hf_repo_v8/` is the prepared launch repo). GGUF Q8 export at release, honest eval card, easy install. ## 2. Where we stand right now (2026-08-09) - **The fluent base is safe and intact:** `ckpt/tiny25m/model_best.pt` (coherent TinyStories-era English, ppl ~8.3). It is the only checkpoint that generates coherent free-form text. NEVER full-SFT/DPO over it again. - **Best verdict accuracy:** `ckpt/tiny25m_dpo3/model_200.pt` — main 0.286, researcher 0.167, combined 0.254, format 1.00 (the "probe champion"). Free-form is soup. - **Both-worlds gate (coherent chat AND verdicts in one checkpoint): NEVER passed in 8 measured adaptations.** This is the honest headline. - **Last experiment:** adapter-only DPO (2 runs) — lr 1e-4 diverged (ppl 913); lr 5e-5 over-abstracted (combined 0.030). Recorded as a dead end. - **Current recommendation (research + measurement):** release as **brain + suit** — fluent base (chat voice) + DPO3@200 analyst behind constrained verdict decode + orchestrator/BM25/tor/TUI as the hands, with an honest per-mode eval card. - **Device change:** the user enabled "RAM plus" — measured 2026-08-09: swap grew 4.0G → 12.3G (≈ +8G addressable). RAM still 7.4G. This raises the practical memory envelope (bigger batches/seq possible) but does not change the coherence sweet spot (~28M) or the overnight-time ceiling. ## 3. Timeline (the full journey) ### Phase 1 — 7.8M liquid pretrain + experiments (Jul 30 – Aug 5) - Built custom liquid (non-transformer) architecture: 6 gated liquid blocks, 320-dim, basis expansion, gated MLP, RoPE-style position, personas. - Pretrained 7.8M on TinyStories + code + domain mix. Coherent stories. - MoE "nanobot" experts: router collapse (51/250 used) → REJECTED. - Width upscale 320→512: val loss 6-7 vs 2.58 → REJECTED. - 7.8M LoRA experiments (v5-v8): probe accuracy 0.18-0.20, free-form mixed. - Lesson: tiny models memorize format but not reasoning below ~28M; conserve the base; balanced mixes + window shuffle are mandatory for corpora. ### Phase 2 — Growth to hybrid25m (Aug 6) - Identity-init wide-head tower growth: 16.8M → **25.4M** (`hybrid25m` config: d_model 320, 6 trunk blocks + 8 tower blocks, tower_d 512). Baseline preserved EXACTLY (up=identity, down=0). - Continue-pretrain on balanced `train_phase2b.bin` (32.5M tokens): 5,000 steps @ ~420-440 tok/s, several runs, **~1.5+ days total**. - Result: `ckpt/tiny25m/model_best.pt` — fluent English (the sacred asset). ### Stage-C SFT (Aug 7) - 230-row merged gold. Probe: main 1/49 (0.020), researcher 1/18 (0.056), format 1.00. Root cause: bespoke verdict vocab + class imbalance. ### Stage-D + process-DPO (Aug 8) - 434 normalized rows (16 canonical classes) + 25 process pairs. - Stage-D SFT: best val ppl 7.35. Process-DPO: main 5/49 (0.102), researcher 2/18 (0.111), combined 0.104 — process supervision moved the boundary 3x. ### Stage-E + DPO2 (Aug 8) - 111 process pairs, 6 epochs → overfit: main 7/49 (0.143), researcher 0/18 (0.000), combined 0.105. Head collapsed to "insufficient evidence". ### Preference gate PASS (Aug 8) - **3,004 handcrafted chosen/rejected pairs**, balanced 21/21 verdict classes, 143/class. Gate PASS. This is the preference corpus the champion uses. ### Stage-F SFT + DPO3 (Aug 8) - Stage-F: 806 gold rows from Stage-E best → best_sft 2.8461, best_ppl 7.11. - DPO 1 epoch on the 3,004 pairs → full-epoch collapsed to "unverifiable", but **model_200.pt = probe champion** (main 0.286 / res 0.167 / 0.254). ### Stage-G/H + merges (Aug 8) — all failed - Fluency restore from un-fluent base: failed by design. - last2 SFT from fluent base: partially fluent, verdicts collapsed to "true". - Weight merges (H×DPO3, pretrain×DPO3, w 0.3-0.7): both-worlds gate failed. ### LoRA run i (Aug 8-9) — no replay - Frozen fluent base + 56 adapters (1.16M trainable), r16/a32, KL 0.05, 2 epochs on 902 gold rows. ppl guard held (8.46); free-form soup; battery main 0.080 / res 0.056 / combined 0.074. Verdicts 3.4x worse than champion. ### Replay research (Aug 9) — the root-cause finding - arXiv 2502.06042: finetuning on limited target data overfits AND drifts; injecting ~1%+ pretraining data into the mixture prevents both. - arXiv 2401.05605: LoRA still forgets; perf↔forgetting is inverse-linear and NOT fixable by rank/epochs/early-stopping → replay is the lever. - **What we were doing wrong:** every adaptation ran on DOMAIN-ONLY gold (~115K assistant tokens) with zero pretraining-data replay. - Full note: `docs/replay_research.md`. Replay rule added to `skills/tiny-model-training/SKILL.md`. ### LoRA run ii (Aug 9) — replay - Replay ratio 0.5 from `train_phase2b.bin`, KL 0.1, lr 2e-4, 1 epoch. - Result: ppl **7.54** (better than base), free-form OPENS with clean domain English then degrades; main 0.020 / res **0.167** (champion-level) / combined 0.060. Replay fixed fluency, not main-verdict discrimination. ### Adapter-only DPO (Aug 9) — 2 runs, dead end - Run A (lr 1e-4, beta 0.1): dpo_loss → 0.0015, ppl EXPLODED to 913 by step 100. Adapter divergence. Abort-on-guard added to `train_dpo.py`. - Run B (lr 5e-5, beta 0.1): healthy to step 75 (ppl 5.82), diverged at 125 (ppl 72.7, guard abort). best_ppl@75: main 0.041 / res 0.000 / 0.030 — over-abstention. Preference signal taught abstain, not discrimination. ## 4. Full honest scorecard (all measured adaptations) | Checkpoint | main | researcher | combined | free-form | |---|---|---|---|---| | Fluent base (`tiny25m/model_best.pt`) | ~0 | ~0 | ~0 | coherent story (off-topic) | | DPO3@200 (`tiny25m_dpo3/model_200.pt`) | 0.286 | 0.167 | **0.254** | soup | | Stage-D + process-DPO | 0.102 | 0.111 | 0.104 | poor | | DPO2 (overfit) | 0.143 | 0.000 | 0.105 | poor | | LoRA run i (no replay) | 0.080 | 0.056 | 0.074 | soup | | LoRA run ii (replay) | 0.020 | 0.167 | 0.060 | template+story mix | | Adapter-DPO @75 | 0.041 | 0.000 | 0.030 | template fragments | ## 5. What worked vs what didn't **Worked (measured):** - Identity tower growth (baseline-preserving) — the only architecture move kept. - Pretraining-data replay in adaptation — fluency held (ppl 7.54) and the free-form output became clean domain English at the start. - DPO3@200 from stage-F best — best verdict accuracy (0.254), format 1.00. - Process supervision (reward the trace steps) — 3x boundary movement. - Constrained/structured verdict decode (`research/structured.py`) — format 1.00 even when free-form is soup. - Handcrafted-only data discipline + balanced preference corpus (3,004 pairs). - Chunked resumable eval (`/tmp/chunked_eval.py`) — survives host OOM kills. **Didn't work (measured dead ends, do not re-run without a new reason):** - Full-param SFT (last2) and full-param DPO — destroy free-form fluency. - Weight merges (H×DPO3, pretrain×DPO3, w 0.3-0.7) — both-worlds gate fails. - MoE nano-experts / router (7.8M) — router collapse. - Width upscaling (320→512) — val loss 2.58 → 6-7. - No-replay LoRA — verdicts 3.4x worse than champion. - Adapter-only DPO — lr divergence (ppl 913) or over-abstention (0.030). - eval.py scratch-90 parallel ×2 — host OOM kills (use chunked, scratch 24, threads 2, sequential). ## 6. Research log (papers that shaped decisions) - TinyStories (2305.07759) — tiny models speak coherently only inside a simple constrained domain; ~28M coherence threshold for open-ended English. - phi-1 (2306.11644) — textbook-quality curated data; small models leave clean when data is clean. - LIMA (2305.11206) — ~1k curated examples shape style IF the base is strong; quality > quantity, but format ≠ reasoning. - Don't Stop Pretraining (2004.10964) — second-phase in-domain pretraining beats fine-tuning; no restart needed (our CONTINUE decision). - LoRA (2106.09685) + EWC (1612.00796) — adapt on frozen base, don't restart. - LFM2 (2511.23404) — staged curriculum SFT → preference → merging on ONE base. - DeepSeek-R1 (2501.12948) / Let's Verify Step by Step (2305.20050) — verifiable rewards + process supervision. - Chain-of-Verification (2309.11495) — draft → verify → revise SOP. - Self-consistency (2203.11171) — majority vote at inference (in the suit). - **Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection (2502.06042)** — 1% pretrain-data injection prevents drift/overfit. - **Scaling Laws for Forgetting When Fine-Tuning LLMs (2401.05605)** — LoRA still forgets; perf↔forgetting tradeoff not fixable by rank/epochs. - Skill docs: `skills/*/SKILL.md` (18 skills) + `docs/` (BIGTECH_RECIPE, research_coherence_small, replay_research, model_card_fsi_anomaly). ## 7. Data & assets inventory - Tokenizer: custom byte-level BPE, vocab 8192 (`data/tokenizer.json`). - Pretrain: `train_full.bin` 528M tokens; `train_phase2b.bin` 32.5M tokens (balanced, used for phase-2 and replay); `valid.bin` 5.5M TinyStories canary (fluency guard — NEVER used as replay); `valid_mix.bin` mixed. - Domain gold: `data/gold_g_all.jsonl` 902 rows (806 forensic + 96 Spock dialogue), handcrafted, `<|scratchpad|>` reasoning traces. - Preference: `data/prefs_p_all.jsonl` 3,004 pairs, 21 classes, gate PASS. - Probes: `data/eval_probes.jsonl` (50) + `data/probes_researcher.jsonl` (27); canonical labels in `research/eval_labels.py`. - Checkpoints: `tiny25m/model_best.pt` (fluent base), `tiny25m_dpo3/model_200.pt` (champion), `tiny25m_sft_f/best.pt` (stage-F), `tiny25m_lora_i|_ii` (LoRA runs), `tiny25m_lora_dpo` (adapter-DPO, dead end). - Tooling: `train/train_lora.py` (LoRA + replay), `train/train_dpo.py` (adapter-only + ppl guard + abort), `/tmp/chunked_eval.py` (battery), `/tmp/conv_sample.py` (free-form), `/tmp/quick_verdict_check.py` (spread). ## 8. Device envelope (measured) - 8-core ARMv9 (4×A720 + 4×A520), 1.95 GHz max. 7.4G RAM + **12.3G swap (RAM+ added 2026-08-09)**. CPU-only, fp32. - hybrid25m: ~420-500 tok/s forward/train; ~19h per 30M-token epoch. - RAM ceiling pre-RAM+: ~30-35M params fp32. With +8G swap, larger batch/seq/configs are feasible in principle, but swap-backed training is slower and the overnight epoch ceiling still favors ≤28M. - One heavy torch job at a time. Background jobs are reaped when the launching shell/session closes — run long jobs in a foreground session and poll, or use `--resume` + checkpoint. ## 9. Decisions & rules (from the skills) - Research → create/apply skill → gate → measure → record. No guessing. - Handcrafted gold ONLY — no generators, ever (`tiny-model-kd`). - Replay is MANDATORY for adaptation stages (`tiny-model-training`). - Frozen base; adapters only; ppl guard 60 with abort (`train_dpo.py`). - DPO gated on probes learning; preference floor ≥1,500 pairs (met: 3,004). - NEVER full-SFT/DPO the fluent base. No scratch restart (CONTINUE decision). - Release honesty: report per-mode numbers exactly as measured; no hiding. ## 10. Open questions / not yet tried - Chat-pretrained 28M base (not story) — the only training lever left for open chat; big project, new pretrain (violates "never restart" → needs a documented decision first). - 16k re-tokenizer on domain corpus + continue-pretrain (recorded future scope; 8k fragments domain words like "Stepartment"). - RLVR/GRPO on verifiable verdicts (TinyZero/TINA-style) at 25M. - Replay + adapter-DPO combined (DPO on top of the replay-SFT adapter, lower beta, KL anchor). - Bigger batch/seq now that swap is +8G. - Using agent notes + changelog as final training documents (per owner). ## 12. The "Perfect-the-System" plan (2026-08-09, collaborator + agent) DECISION: NO 50M growth. Measured reasons: (a) swap-backed training is slow and 50M won't break the both-worlds tradeoff (2401.05605), (b) RAM+ swap helps batch/seq, not the coherence ceiling, (c) perfection = the SYSTEM, not the checkpoint. We perfect what we have. Phase 1 — Model optimization (measured, no shortcuts): - 16k tokenizer retrain (fix "Stepartment"/"Stepublication" artifacts) + embedding remap + continue-pretrain. AGENT CORRECTION: this resets the ANALYST too (model_200.pt is 8k-tokenizer-native) — it is a DAYS project (raw corpus must be re-downloaded: we deleted the 2.1G txt; re-encode all bins + battery; retrain the adaptation stack), not 2-4 hours. Still the right long-pole unlock; schedule as such. - BF16 training. AGENT CORRECTION: on this CPU stack the measured gain is marginal (tiny-scale: "bf16 autocast roughly recovers one line"); AdamW stays fp32 anyway. Test once, record, don't expect 30% RAM. - Replay ratio sweep 0.3/0.5/0.7/0.9 and LoRA rank sweep r=8/16/32/64 — run AFTER the tokenizer decision to avoid wasted runs; measure ppl + verdicts; expectation: these improve the CHAT side, NOT the analyst boundary (objective limit, already measured). Phase 2 — Harness perfection (no training, highest ROI/hour): - Confidence calibration for DPO3@200: map each confidence bucket (LOW/MED/ HIGH/cannot assess) to measured accuracy. First task started 2026-08-09. - Weighted agent voting by historical accuracy + disagreement flagging. - Adversarial/red-team probe set: 20+ handcrafted probes (false dichotomy, leading/emotionally loaded claims) — fits the no-generator rule. - Chain-of-custody: every verdict traceable (sources, agent agreement, abstention) — grant-ready audit trail. Suit already has provenance + artifact sandbox; extend to per-verdict logging. Phase 3 — Release (week 3): GGUF Q8 (base + analyst), HF repo with honest eval card, 3-min demo video, grant proposal ("25M, 100% auditable, on-device forensic journalism"). Rule: every change measured; every claim backed by the battery; no "feels better" — numbers only. ## 11. War-story / paper material - Solo developer, no funding, no GPU: trained a custom liquid 25.4M model from scratch on a tablet. - The 8-run both-worlds investigation (form vs reasoning; replay vs preference) — an honest small-model study matching 2401.05605's tradeoff. - The measured replay finding (2502.06042 applied): the fluency fix. - Brain + suit: the "many coordinated specialists" verdict at tiny scale. - The discipline system (research → skill → apply → gate → record) as the repeatable moat. ## 13. Phase 2 started — confidence calibration run (2026-08-09) - **Tooling fix (bug found + fixed properly):** `/tmp/chunked_eval.py` mis-keyed the researcher probes. It mapped rows by CANON-list index instead of `research/eval.py`'s official `-` scheme, so the pattern-10/11/12 rows got shifted labels and the last 3 rows became q24/q25/q26. Fixed: keys now match eval.py exactly (`expected` in row -> id field or p%02d; else `-%02d`). RESULT: logs/eval_dpo_res.jsonl (adapter-DPO run) researcher ids were mislabeled (aggregate 0/27 still representative). - **Clarification:** logs/eval_dpo_main.jsonl + eval_dpo_res.jsonl (Aug 9 00:45/ 00:59) are the adapter-DPO run `ckpt/tiny25m_lora_dpo` (main 0.040), NOT the champion. Champion calibration had never been run. - **Run started 2026-08-09 (08:34 UTC-5):** champion `ckpt/tiny25m_dpo3/ model_200.pt` full battery via fixed chunked_eval (threads 2, max-scratch 24): - 50 main -> logs/calib_dpo3_200_main.jsonl - 27 res -> logs/calib_dpo3_200_res.jsonl - ~55 s/probe, ~70 min total; session 1284. RESULTS APPENDED BELOW when done. - **New tool:** research/calibration.py — maps confidence label (HIGH/MEDIUM/LOW/ cannot assess) to MEASURED per-bucket accuracy (+ Wilson 95% CI), HIGH-bucket verdict mix, abstention stats; writes logs/calib_summary_.json. This is the basis for weighted agent voting (Phase 2). - **Harness context re-confirmed:** confidence is a constraint-decoded SELF- REPORTED label (structured.py CONFIDENCES), not a probability. fusion.py raises confidence on mind agreement (uncalibrated). orchestrator.py votes by simple majority of verdict strings, no confidence weighting. - **Red-team battery authored (handcrafted, no generators):** `data/eval_redteam.jsonl` — 26 adversarial probes (rt01..rt26) + canonical labels in `research/eval_labels.py` (CANON rt keys; battery untouched, ids disjoint). Trap classes: false dichotomy, unproven/leading premise, emotional loading, appeal to authority, single/anonymous source, fabricated citation, correlation-not-causation, cherry-picked window, false equivalence, ad hominem, absence-of-evidence, double negative/silence, anachronistic evidence, non-additive totals, failure-to-deny, unverifiable insider claim, minor-error!=fraud, retrofitted prophecy, no base rate, FOIA!=classified, word-reuse!=same-event, suspicion!=evidence, vague-overclaim vs record, duplicate single source, plus 2 record-backed sanity probes (rt25 true, rt26 false). This is the Phase-2 red-team gate: model must abstain/tag-low on traps, not pattern-match. ### Calibration results (2026-08-09) - Run 1 (max_scratch=24, threads=2 — memory-safe config): main 0.143 (7/49), res 0.056 (1/18), combined 0.119 (8/67). Per-bucket combined: HIGH 0.077 (2/26), MEDIUM 0.231 (3/13), LOW 0.118 (2/17) — HIGH is the LEAST reliable bucket (anti-calibrated). Abstain rate 2/49 main, 0/18 res. - FINDING (honest, recorded): max_scratch CHANGES the verdict distribution — the 24-scratch run gives 0.143 main vs the official scorecard's 0.286 at max_scratch=90. Scratch length is a scoring config, not a free knob. The 24-scratch numbers are saved as calib_summary_dpo3_200_scratch24.json (labeled) and are NOT comparable to the official scorecard. - Fix: calibration.py zero-abstain division bug fixed. - Run 2 (2026-08-09, session 58557): re-running at the OFFICIAL config max_scratch=90, threads=2 -> logs/calib90_dpo3_200_{main,res}.jsonl. RESULTS APPENDED when done. ## 14. Harness research + decision-spine build (2026-08-09) — "punch like 7B" ### Research (arXiv, 2026-08-09, multi-source) - Verbalized confidence is ANTI-CALIBRATED: ORCE (2026-05), Direct Confidence Alignment (2025-12), "Probabilistic vs Verbalized Confidence" (arXiv 2408.11774). => never trust a self-reported HIGH/MEDIUM/LOW label; map it to measured accuracy. - Small models need STRONG EXTERNAL verifiers to self-correct (arXiv 2404.09931 "SLMs Need Strong Verifiers"). => the verify loop is deterministic suit logic (rule spine + retrieval + value checks), never weak self-critique. - Selective prediction / governed abstention is the SLM production recipe: abstain below a calibrated threshold, publish accuracy-at-coverage (governance-ready SLM 2025-08; conformal selective prediction 2026-07). - Self-consistency: sample N, but WEIGHT by calibrated reliability instead of naive majority (arXiv 2203.11171; Universal SC 2311.08110). - RLVR: Reasoning Gym (2025-05) = library of verifiers for RLVR — supports the deterministic-spine direction (recorded; not this week). - Chain-of-Verification (Meta 2023-09) + RAG+CoVe (2024-10): draft -> verify -> revise; verification must be external (see strong-verifier finding). ### What this means for FSI-Anomaly The 25M brain cannot be trusted to grade its own answers or pick its own confidence. The suit must: (1) sample/collect votes, (2) weight them by the calibration table, (3) abstain below threshold, (4) log chain-of-custody. That is the "punch like 7B" mechanism for our narrow domain — reliability through the system, not the parameter count. ### Built this session (all recorded; unit-tested) - research/calibration.py — label -> measured accuracy (+ Wilson CI, HIGH-bucket verdict mix, abstention stats); writes logs/calib_summary_.json. - research/decision.py — decision spine: weighted_tally, decide (p_final = mean calibrated reliability behind winner; governed abstention), accuracy_vs_coverage (selective-prediction curve), bucket_abstention_curve, trace (chain-of-custody). Pure Python, no torch. - tests/test_decision.py — 9 unit tests, all PASS. - data/eval_redteam.jsonl — 26 handcrafted adversarial probes (rt01..rt26) + CANON labels in research/eval_labels.py (battery untouched). Red-team gate. - skills/tiny-model-suit updated + mirrored to ~/.codex/skills/ (research items 5-8, decision spine marked BUILT, weighted-vote plan replaces naive majority). ### Next (when calibration run finishes) - Plug logs/calib_summary_dpo3_200.jsonl into decision.py; produce the accuracy-vs-coverage curve for the champion; choose the operating threshold. - Wire calibrated fusion: replace naive "HIGH if either mind HIGH" with p = mean(calibrated(analyst_conf), calibrated(skeptic_conf)). - Red-team eval on champion (26 probes) — the release gate. ## 15. Big-tech harness research -> skill -> apply (2026-08-09) ### Research (multi-source; digest in docs/harness_research.md) - OpenAI "A practical guide to building agents": agent = MODEL + TOOLS + INSTRUCTIONS + GUARDRAILS (relevance/safety/PII/rules/output validation); single-agent first, tools standardized. - Anthropic "Building effective agents" (2024-12): simple composable patterns; workflows (deterministic) before agents (model-driven). - Anthropic "Effective context engineering" (2025-09): context is finite; curate/structure/retrieve/compact. - Anthropic "Language Models (Mostly) Know What They Know" (2022): calibration is format-dependent; smaller models worse. - arXiv: cannot self-correct (2310.01798); SLMs need strong verifiers (2404.09931); CRITIC tool-critique (2305.11738); CoVe (2309.09308); self-consistency (2203.11171, 2311.08110); multiagent debate (2305.14325); DeepSeek-R1 RLVR (2501.12948); Reasoning Gym (2025-05); Distilling Step-by-Step (2305.02301). ### Skill created + applied - NEW skill: skills/tiny-model-harness/SKILL.md (9-rule doctrine, component map, build/audit rule) + mirrored to ~/.codex/skills/. Full digest: docs/harness_research.md (9 sections, cited). - APPLIED #1: research/guardrails.py — input guardrails (injection/safety blocklist, relevance gate, PII redaction) + output validation; tests tests/test_guardrails.py (8 PASS). - APPLIED #2: research/verify_loop.py — external verification loop (draft -> plan_checks -> retrieve -> deterministic spine -> revise -> trace); tests tests/test_verify_loop.py (8 PASS). Fixed 2 regex bugs found by tests (4.2M split, 9:30am trailing-\b). - Remaining applies (planned): weighted self-consistency sampler N=3-5; calibrated fusion (replace naive confidence raise); context budget in TUI; RLVR with decision spine as verifier. ## 16. 150M-on-tablet feasibility research (2026-08-09) ### Research question Can we train a 150M parameter model on the Exynos 1580 tablet (8-core ARMv9, 7.4GB RAM + 12.3GB swap) with our existing disciplined loop? ### Sources (measured, not guessed) - **tiny-scale skill** (2026-08-06): measured throughput table, RAM ceiling, coherence floor. 25M = 430 tok/s, 19h/epoch. Extrapolation: 150M ≈ 50-100 tok/s, 95-190h/epoch (4-10 days). RAM at 150M ≈ 2.0-2.5GB per step (weights + AdamW + activations). Fits in physical RAM (7.4GB) but swap risk for AdamW state. - **tiny-model-phase2 skill** (2026-08-06): VERIFIED winning path is wide-head tower growth from trained trunk (identity-init). Width upscaling 320→512 FAILED (val loss 2.58→6.1-7.7). Depth-only 12.94M worked. Tower growth hybrid18m (16.77M) and hybrid25m (25.4M) both preserved baseline EXACTLY (val 2.5784 == baseline). SCAN_CHUNK=16 critical for numerical stability. Corpus mixing MUST be window-shuffled balanced (train_phase2b.bin), not concatenated blocks. - **tiny-model-training skill**: replay ratio 0.5 mandatory (arXiv 2502.06042, 2401.05605), LoRA on grown base preferred over full SFT. ### Findings 1. **150M is technically feasible on this device** — fits in physical RAM, training will complete. BUT iteration speed drops 5-10×. One epoch = 4-10 days. A failed hyperparameter guess = 1 week lost. 8-run study at 25M took weeks because iteration was daily; at 150M it would be monthly. 2. **Coherence ceiling is ~28M** (TinyStories class). 25M hybrid25m is AT this ceiling. Going to 150M doesn't improve coherence for open-ended generation; it only adds capacity for more specialized SFT/DPO. 3. **The harness IS the product** — decision.py, fusion.py, verify_loop.py, guardrails.py, calibration system, helix memory, dual-mind fusion, SOP agent. These apply to ANY model size. Building them on 25M is faster and produces the same grant-worthy artifacts. 4. **Growth path is proven** — hybrid25m tower from trained 320-dim trunk. If capacity is needed, the next step is hybrid28m (tower_d=512, tower_blocks=12 or tower_d=768, tower_blocks=8), not 150M from scratch. ### Decision **Do NOT pursue 150M on tablet now.** The iteration penalty is too high for disciplined development. Instead: - Perfect the 25M harness (Phase 2: calibrated fusion, redteam eval, GGUF export, HF release) - Use 25M for ablations: data mixes, LoRA ranks, replay ratios, character data scales - Only grow if 25M harness proves the model needs more capacity for the specific forensic SFT target - Document this decision in skills and roadmap ### Next (confirmed) 1. Wire calibrated fusion into fusion.py (replace naive "HIGH if either mind HIGH" with calibrated merge using decision.py's load_table + decide) 2. Run redteam battery (rt01..rt26) through full pipeline: guardrails → model → calibration → fusion → verify_loop 3. GGUF Q8 export of champion (tiny25m base + tiny25m_dpo3 analyst) 4. HF repo with honest eval card 5. Grant proposal framing: "25M parameters, 100% auditable, on-device forensic journalism" ## 17. Mandalorian Credo Discipline Skill (2026-08-09) ### Research → Skill → Apply - Researched the Mandalorian credo: "Protect the innocent, use resources responsibly, honor the mission" — a complete ethical/operational framework. - Created `skills/tiny-model-mandalorian/SKILL.md` mapping 5 Mandalorian principles to FSI model guardrails: 1. **Protect the Innocent** → PII redaction, input/output safety gates 2. **Use Resources Responsibly** → no generators, handcrafted data only, replay ratio 0.5, measured ablations 3. **Honor the Mission** → verifiable 3-way verdicts, traceable sources, no fabricated claims, process supervision 4. **Tools Must Be Clean** → first-class guardrails, chain-of-custody, weighted voting by calibration 5. **Mission Overrides Personal** → closed-loop research→skill→apply→gate→measure→record - Applied: Updated `research/fusion.py` with `_calibrated_merge()` that replaces naive "HIGH if either mind HIGH" with calibrated probability mean using `decision.py`'s `load_table` + `calibrated_prob`. This is the Mandalorian "Tools Must Be Clean" principle in code — never trust self-reported confidence, always use measured accuracy. ### Calibration Status (DPO3@200 champion, max_scratch=90) - Main: 0.26 accuracy (4 HIGH/4 MEDIUM/40 LOW/2 cannot assess) — HIGH bucket only 25% accurate - Researcher: 0.074 accuracy — HIGH/MEDIUM buckets 0% accurate, LOW 10% - Finding: verbalized confidence is systematically anti-calibrated → must use calibration table ### Next Steps (per roadmap) 1. Create unified calibration summary `logs/calib_summary_dpo3_200.json` combining main + researcher 2. Run redteam battery (rt01..rt26) through full pipeline: guardrails → model → calibration → fusion → verify_loop 3. GGUF Q8 export of champion (tiny25m base + tiny25m_dpo3 analyst) 4. HF repo `FerrellSyntheticIntelligence/fsi-anomaly` with honest eval card 5. Grant proposal: "25M parameters, 100% auditable, on-device forensic journalism" --- ## 19. Model Growth Preparation — 50M Sweet Spot (2026-08-09) ### Device Size Analysis Results Measured on Exynos 1580 (8-core ARMv9, 7.4GB RAM + 12.3GB swap): - With LoRA training (frozen base, adapter-only AdamW): ALL sizes trainable - Sweet spot = **50M parameters** (70,387,200 params) - Iteration speed: 56.5h/epoch (~2.3 days) — fast enough for disciplined iteration - RAM usage with LoRA: ~582MB — fits comfortably in 7.4GB - Capacity: ~2x current 25M — potentially breaks both-worlds tradeoff ### Red-team Battery Results (DPO3@200 champion) - 26 adversarial probes run through full pipeline (guardrails → model → calibration → fusion) - Input guardrails: 0 blocked (all on-domain) - Verdict alignment: 2/26 correct (7.7%) — model misses most logical fallacies - Main probes: 5/49 (10.2%) - Researcher probes: 0/18 (0.0%) - Combined: 7/93 (7.5%) - Format rate: 100% — harness outputs are well-formed - **Decision**: 25M lacks capacity for forensic reasoning → scale to 50M ### Size Skills Created 1. `skills/tiny-model-grow50m/SKILL.md` — 50M growth strategy + config 2. `skills/tiny-model-tracking/SKILL.md` — experiment tracking pipeline (MLflow/W&B-inspired) ### Growth Pipeline (Identity-Init Tower Expansion) 1. Start from `ckpt/tiny25m/model_best.pt` (fluent base, preserve EXACTLY) 2. New config: `hybrid50m` — d_model=512 (was 320), tower_d=896 (was 512) 3. `train/grow_weights.py --base ckpt/tiny25m/model_best.pt --config hybrid50m --mode tower` 4. Tower blocks: identity-init (zero-output, baseline preserved EXACTLY) 5. Continue-pretrain on phase-2 corpus → LoRA-adapt with replay ratio 0.5 6. Gate: val ppl down, no NaN, fluent stories, baseline preserved ### Next Steps 1. Run growth script: `./stage_grow50m.sh` 2. Verify: baseline preserved EXACTLY (forward pass comparison) 3. Continue-pretrain 50M model on phase-2 corpus 4. Apply LoRA adaptation with replay ratio 0.5 5. Re-run full eval battery (main + researcher + red-team) 6. Compare results vs 25M baseline --- ## 20. 50M Model Growth Complete (2026-08-09) ### Growth Method - Identity-init tower expansion from hybrid25m base - Script: train/grow_weights.py --config hybrid50m --mode tower - Developer's Credo applied: width before depth, baseline preserved ### Growth Config - d_model=320 (frozen trunk), tower_d=800 (widened), tower_blocks=8 - Params: 50.07M - up_proj: identity mapping; down_proj: zero-init residual ### Verification Results - Baseline val loss: 2.2525 == Grown val loss: 2.2525 - Delta: 0.0000 — baseline preserved EXACTLY - Checkpoint: ckpt/hybrid50m_grown/model_final.pt (200MB) - Status: DONE ### Next Steps 1. Continue-pretrain 50M on phase-2 corpus (56.5h/epoch, ~2.3 days) 2. Apply LoRA adaptation with 3,004 preference pairs 3. Re-run full eval battery (main + researcher + red-team) 4. Compare vs 25M baseline ### Growth Path 7.8M -> hybrid18m(16.8M) -> hybrid25m(25.4M) -> hybrid50m(50.1M) DONE ## 21. 50M Continue-Pretrain In Progress (2026-08-10) - Checkpoint: ckpt/hybrid50m_pretrain/ - Corpus: train_phase2b.bin (32.5M tokens), BF16, 5000 steps - Throughput: ~324 tok/s, Loss: 9.07 -> 6.38 -> 5.44 (step 100) ## 22. Pipeline Status (2026-08-10) DONE: 7.8M pretrain, 25.4M growth, DPO3@200, harness, calibration, red-team, 50M grown IN PROGRESS: 50M continue-pretrain (loss dropping 9.07->5.44) NEXT: LoRA adaptation (3,004 pairs, replay 0.5), then full eval vs 25M ## 23. Training Pipeline Research — Verified Against Big Tech (2026-08-10) Researched from multiple sources (SmolLM, SmolLM2, Zephyr, Phi-3, DeepSeek-R1, LIMA, forgetting papers arXiv 2401.05605/2502.06042, Liquid LFM2 technical report). Created skills/tiny-model-pipeline/SKILL.md + docs/harness_research.md section 10. VERDICT: our pipeline is already close to the gold standard. Verified matches: - AdamW (0.9,0.95), wd 0.1, grad clip 1.0, BF16, embedding tying - Cosine LR + warmup (GPT-3/LLaMA standard) - SFT then DPO 1 epoch (Zephyr/SmolLM recipe) - Handcrafted gold (LIMA/phi-1; better than synthetic for this niche) - Replay 0.5, KL anchor, ppl guard, curriculum stages, eval gates KEY FINDING (Liquid LFM2, our architecture family): the full recipe ends with THREE post-training stages: SFT -> length-normalized preference optimization -> MODEL MERGING. We have SFT + DPO but merging after preference is untested at 50M (naive averaging failed at 25M; TIES/task-vector untested). ACTIONABLE GAPS to close: 1. 16k tokenizer retrain (8k fragments "Stepartment"; SmolLM uses 49k) 2. Model merging after preference (TIES/Delta-merge, not naive avg) 3. Length-normalized preference optimization (fixes DPO abstain-collapse) 4. RLVR stage (verifier exists; DeepSeek-R1 unlock) 5. Optional cooldown tail (SmolLM 20%) 50M continue-pretrain status: step 400/5000, loss 4.10, ~355 tok/s. ## 24. The Big Question — Lab-Grade? Best Path Forward? (2026-08-10) ### Would this setup be at home in a big tech lab? METHODOLOGY: YES. Research -> skill -> apply -> gate -> measure -> record; handcrafted gold (LIMA/phi-1); calibrated harness (decision/fusion/verify/ guardrails); honest eval. This is the same discipline big tech runs. HARDWARE: No lab trains on a tablet. The methodology survives the hardware; the hardware is the story (grant angle), not the advantage. ### The single biggest power gap: pretraining data scale - 528M-token corpus (train_full.bin) is ON DISK, encoded Aug 1, ~90% unused. - Only ~24M tokens ever seen by 7.8M base; 25M saw ~10M more (phase2b). - Big tech: SmolLM 135M = 600B tokens (4,444 tok/param); SmolLM2 = 2T. - Our 50M on 32.5M = 0.65 tok/param — 4 orders below the recipe. - Our 50M on 528M = 10.6 tok/param — Chinchilla-ish, defensible. ### What this means for the path - The current 32.5M continue-pretrain is the right GROWTH step (activates new tower capacity fast). - The biggest power unlock available: after growth/LoRA/DPO iteration, run a LONGER final pretrain on train_full.bin (100-200M+ tokens = 3-6 days at ~355 tok/s) before release. More tokens = more knowledge density + cross- domain pattern vocabulary (the owner's closed-loop insight). - Close the other recipe gaps: 16k tokenizer, TIES model merge, length-norm DPO/IPO, RLVR (verifier exists). ### Bottom line On this hardware we will never match big tech's data scale. But the niche is narrow (forensic claim verification + dark-web research), the product is brain + harness, and the harness carries the reasoning. The path IS the best one available; the #1 add is more pretraining tokens from the corpus we already own. ## 25. Journalism Suite Built (2026-08-10) ### Owner-approved features (collaborator list) -> BUILT 1. Source credibility + provenance ledger -> research/provenance.py 2. Timeline gap detection -> research/timeline.py (gaps, cliffs, anachronisms) 3. Framing/language forensics -> research/framing.py (passive, loaded, hedges, nominalization, agency, omissions) 4. Cross-domain pattern synthesis -> research/patterns.py (rungs/themes/names across domains; cards = LEADS only, base-rate caveat) 5. Entity relationship graph -> research/entitygraph.py (+ DOT export) 6. Pre-publication adversarial review -> research/editorial_review.py (HOLD/REVISE/CLEAR + per-item PASS/FLAG) 7. CaseFile notebook -> research/casefile.py + research/journalism.py facade (suite_report: one call = provenance + timeline + framing + patterns + entity graph + review + notebook, saved data/casefiles/.jsonl) ### Research basis (web + repo) - Bellingcat OSINT toolkit / evidence standards: chain-of-custody + tiers - Entman 1993 framing: selection + salience -> measurable proxies - Repo decisions: suit investigation layer (2026-08-07), eval_redteam adversarial patterns, helix rung model, timeline_reconstruction SOP ### Wiring - /journal [name] in tui/cli.py + tui/engine.py (runs over case ledger + data/library; saves CaseFile) - Skill: skills/tiny-model-journalism/SKILL.md ### Tests / gates - tests/test_journalism.py: 12 tests, all passing - Full suite: 37 tests passing - Smoke: suite_report over data/library (5 docs) -> 5.8k markdown notebook ### Training status (unchanged) - 50M continue-pretrain: PID 8463 alive, step ~550/5000, val best 3.6972, ~300-355 tok/s. Next checkpoint save at step 1000. - Nothing in this build touches training (pure-Python suite, no torch). ## 26. "Most Powerful Path" — Big-Tech Playbook (2026-08-10) ### Question What would a big-tech lab do if they were in our exact situation (solo dev, 8-core ARM tablet, 50M model, 528M-token corpus on disk)? ### Answer (grounded in SmolLM2 "When Smol Goes Big" + Tokens-Per-Parameter scaling + Chinchilla + Liquid LFM2 + repo measurements): 1. FIX THE TOKENIZER FIRST. 8k vocab fragments domain words ("Stepartment"); SmolLM uses ~49k. A 16k retrain before any long run avoids re-learning garbage segmentation (re-encode is cheap; re-pretrain is not). 2. MAXIMIZE TOKENS/PARAM. Our 50M on 32.5M tokens = 0.65 tok/param. On train_full.bin (528M, TinyStories-class) = 10.6 tok/param -> the loss wall at this size is DATA, not capacity. This is the single biggest lever left; architecture work is done (growth preserved baseline exactly). 3. CURRICULUM (Liquid LFM2 family): general (TinyStories) -> domain -> forensic gold. 4. FULL POST-TRAINING RECIPE: SFT -> length-normalized DPO -> TIES model merge -> RLVR on our deterministic verifier (DeepSeek-R1 style). 5. GATE EVERYTHING: val ppl + probe batteries, ckpt every 500, resume-safe. ### Honest nuance - train_full.bin is TinyStories (fluency rail). Forensic power lives in handcrafted domain gold + the harness (journalism suite, verify loop, calibration) — those rails run in PARALLEL, no compute conflict. - Timeline math: 528M tokens @ ~300 tok/s ≈ 20 days (one long pole run). ### Staged path (ships en route, same as labs ship revisions): - STAGE 1 (this week, after current run): 16k tokenizer retrain -> re-encode phase-2 corpus -> continue-pretrain -> LoRA SFT (3,004 pairs) -> length-norm DPO -> full eval vs 25M. Fast 50M iteration + v1 candidate. - PARALLEL (no compute): author handcrafted forensic gold, build RLVR reward harness on verify_loop, extend journalism suite. - STAGE 2 (long pole): full 528M pretrain from 50M base (~3 weeks, checkpointed) -> SFT/DPO/TIES/RLVR -> v1.1 release. - GATE: if Stage-1 50M+harness clears >=0.40 main / >=0.25 researcher at >=60% coverage, ship v1 for community + grants while Stage 2 runs. ### Training status 50M continue-pretrain: step 600/5000, loss 3.61, ~250 tok/s (~18h to go). ## 27. 16k Tokenizer Pipeline Ready (2026-08-10) ### What was built (per "most powerful path" -> Stage 1 prep) - 4 new skills: tiny-model-tokenizer, tiny-model-pretrain-full, tiny-model-posttrain (length-norm + TIES), tiny-model-rlvr (verify-spine RL) - 3 scripts + 3 stage runners (all smoke-tested, no training interference): - data/retrain_tokenizer_16k.py: streams the encoded 528M corpus, decodes (byte-level BPE is lossless; no raw .txt needed), trains 16k BPE via train_from_iterator. memmap => memory-safe. - train/map_vocab.py: old->new token map by decoded-text EXACT match; tok_emb expansion only (lm_head tied); strict load; --verify parity gate. - data/reencode.py: streaming old->new corpus re-encode, flush-safe. - stage_tokenizer_16k.sh / stage_v16k_continue.sh / stage_pretrain_full16k.sh ### Smoke-test numbers (gates) - retrain on valid.bin: 16k BPE, 119,632 lines -> 13,727 merges (data-limited; the real 528M run fills 16,384) - map on real hybrid50m ckpt: 95.4% exact (7817/8192), 375 first-token fallback, 0 noise; 50.07M -> 51.85M params; strict load OK - reencode: 0.98x token ratio; 34s per 5.5M tokens (~1h estimate for 528M) ### Execution order (one heavy job at a time) 1. Current 50M continue-pretrain finishes (step ~600/5000). 2. ./stage_tokenizer_16k.sh -> tokenizer16k.json + phase2b16k.bin + full16k.bin 3. ./stage_v16k_continue.sh -> vocab-expanded init + 16k phase-2b pretrain 4. LoRA SFT (3,004 pairs) + length-norm DPO at 16k base (tiny-model-posttrain) 5. Full eval vs 25M; if >=0.40 main / >=0.25 researcher @60% coverage -> v1 6. Later: ./stage_pretrain_full16k.sh (528M, ~20 days) -> v1.1 + RLVR ### Constraints respected - No heavy job ran alongside training (smokes were seconds, mapping 39s). - Handcrafted gold rule untouched; special-token order locked; PYTHONPATH=$PWD convention followed for stage scripts. ## 28. Parallel Rail Built During 50M Run (2026-08-10) ### What and why While the 50M continue-pretrain trains (~16h left), completed the post-training tooling that needs NO compute (discipline: one heavy job at a time). 1. train/ties_merge.py — TIES merge (trim -> sign consensus -> mean of survivors -> add to base). The Liquid LFM2 ending; fixes naive-avg failure measured at 25M. Usage in docstring; unit tests pass. 2. research/rlvr.py — reward harness for the RLVR stage (DeepSeek-R1-style on OUR deterministic spine): verdict exact +1, abstain 0, contradiction -1, citation in evidence +0.2, fabricated citation -0.2. Confidence never rewarded (anti-calibrated, measured). reward_card() = audit trace. 3. stage_eval_50m.sh — full battery runner (main + researcher + red-team) for any 50M checkpoint; used at every gate from here on. 4. data/kd_gold_v21.jsonl — 12 handcrafted Spock rows teaching the journalism suite concepts; joins the SFT gold stack (v20: suit SOP, v21: journal desk). ### Gates / numbers - 44/44 tests passing. - kd_gold_v21: 12 rows, format-validated, all verifiable from prompt. - Training: step 750/5000, loss 3.97, ~262 tok/s, 1.7GB RAM. ### Next when training finishes 1. ./stage_tokenizer_16k.sh (16k BPE + re-encode both corpora) 2. ./stage_v16k_continue.sh (vocab expansion 95.4% exact -> 16k phase-2b continue-pretrain) 3. ./stage_lora_50m.sh with the 16k base + gold v20+v21 (replay 0.5, KL 0.1) 4. train_dpo.py at lr<=5e-5, 1 epoch (length-norm mean already in batch_seq_logprob) -> ties_merge.py the SFT+DPO task vectors 5. ./stage_eval_50m.sh -> gate >=0.40 main / >=0.25 researcher @60% coverage ## 29. Market / Funding Strategy Research (2026-08-10) ### Verification-AI landscape (web-verified) - Logically (largest AI anti-misinfo startup, ~$24M+ Amazon-backed) COLLAPSED after losing TikTok/Meta contracts -> platform-dependent business models are the trap; grant + sovereign + open-source is more durable. - Full Fact (nonprofit UK), NewsGuard, Ground News, ClaimBuster, Factiverse = cloud-based verification tools. All assume internet + cloud trust. - Hunchly: commercial OSINT chain-of-custody tool for journalists + law enforcement -> VALIDATES the CaseFile/provenance concept + willingness to pay. - Edge/sovereign SLM is a named 2026 trend ("on-device is the privacy shift") -> our exact framing has tailwind. ### Wedge (the real niche) NOT another fact-checking chatbot (Logically's space, platform-dependent). The niche = SOVEREIGN + AUDITABLE research desk: tiny on-device model, full chain-of-custody, offline, works where journalists cannot trust the cloud. Nobody combines small-model-on-device + audit trail + OSINT capability. ### Funding paths (ranked) 1. NGI0 Entrust (NLnet) — open-source trust-enhancing tech grant (~EUR 50k, no matching funds, small team) — BEST first target, exact fit. 2. Knight Prototype Fund ($75k, journalism tech) — prototype-stage fit. 3. Craig Newmark Philanthropies — journalism trust/safety. 4. OTF (internet freedom) — careful framing (source protection, not hacking). 5. Later: Mozilla/Sloan; SBIR/STTR for the Alaska defense angle. 6. Non-grant: community downloads + donations, newsroom consulting/contracts, commercial license of the HARNESS (model stays open). ### Unlock checklist (what a committee will demand) - 3-min demo of the FULL loop on one real claim (search -> provenance -> verify -> CaseFile audit -> adversarial review). - Honest eval card with accuracy-vs-coverage curve (already planned). - "Runs on a $300 tablet, no cloud, no telemetry" proof — nobody else has it. - Reframe: lead with verification + source protection + OSINT safety, NOT "conspiracy/rabbit holes" (dark-web framing hurts some committees). ### Training status 50M continue-pretrain step ~750/5000 (verified earlier this session). ## 30. Training Interrupted + Resumed (2026-08-10) - EVENT: the 50M continue-pretrain process died during a session break (no traceback; killed externally). Last checkpoint model_best.pt / model_500.pt at step 500 (best_val 3.6972). Steps ~501-750 (~2h of compute) were lost — not checkpointed (save-every 500). - ACTION: relaunched from step 500 via --resume (opt + step + best_val restored), same hyperparams, --steps 4500 (target total 5000). - LESSON: nohup-backgrounded children do NOT survive this harness; the proven pattern is a FOREGROUND PTY session (original ran hours that way). - STATUS: PID 12586 running, RSS 1.87GB, 22 threads, resumes at step 500. Monitor: session 89218, log-every 50 (first line at step 550). ## 31. Training Detached + MTP Implemented (2026-08-10) - Training died twice at session boundaries -> root cause: harness kills attached process trees. FIX VERIFIED: setsid + nohup + MTP [done] > longer context > KD-with-teacher [owner flag] > depth growth > RLVR). MoE = dead end, never re-run. ## 32. Watchdog Active (2026-08-10 08:01) - Third death (after detached relaunch) -> lesson: this environment kills long processes regardless of detachment. The durable fix is resume-safe auto-restart, not detachment. - train/watchdog_50m.sh: loops train_lm --resume until step >= 5000; restarts 30s after any death; losses capped at one checkpoint window (500 steps). - Current: resumed at step 1000 (val best 3.3154), PID 26785 under watchdog 26781. Threads 6. Log logs/pretrain50m.log with "=== attempt N ===" marks. - Data point: val loss improved 3.6972 -> 3.3154 between steps 500 and 1000; the resume/continue strategy is working. ## 33. Second-eyes journalism audit (2026-08-10) - Audit scope: architecture, training curriculum, post-training failures, eval, journalism suite, memory, deployment, and active processes. - Live state: 50M continue-pretrain remains active as PID 28397 with hybrid50m, BF16, batch 8, seq 512, six threads; do not interrupt or start another heavy torch job beside it. - Stale-process cleanup: terminated the leftover MTP smoke test PIDs 3552 and 4955. It was a three-step micro6m test still alive and competing for CPU; the real 50M process was preserved. - Model decision: no new architecture experiment. The identity tower growth to 50.1M is already verified (baseline val loss 2.2525 == grown 2.2525). The next model path remains 50M finish -> 16k tokenizer -> continue-pretrain -> replay LoRA -> length-normalized preference -> TIES -> RLVR -> full eval. - Journalism decision: define success as a system scorecard, not one model accuracy number. Required measures are retrieval precision, independent-source count, exact value/date/name citation, timeline extraction accuracy, discrepancy detection, gap/omission detection, safe OSINT adherence, adversarial editorial hold rate, and accuracy-vs-coverage under abstention. - Highest-leverage product insight: the model is the analyst brain; the suit is the journalism desk. Evidence retrieval, provenance, deterministic comparison, and publication review carry more reliable capability than another blind SFT or architecture change. - Gate: no production or grant claim until the 50M checkpoint is evaluated on the fixed 77-probe battery plus red-team and journalism-suite scorecards. ## 34. Audit validation (2026-08-10) - Project `.venv` was present with torch 2.13.0+cpu, but pytest was missing. - Installed pytest 9.1.1 and ran the full pytest suite while the 50M training process continued untouched. - Gate result: **48 passed in 141.64s**, 0 failures. - The prior standard-library unittest discovery ran 0 tests because this suite uses pytest-style functions; that was a test-runner gap, not a passing suite. - Post-test process check: PID 28397 remains alive on the 50M continue-pretrain, RSS 2.09 GB; log confirms resume from step 1000 with 50.09M parameters. ## 35. 50M continue-pretrain completed and audited (2026-08-11) - Watchdog completed cleanly: step **5000/5000**, exit code 0, target reached; no process remains active. - Validation loss improved from 3.1073 at step 1500 to **2.7050432384** at step 5000; best checkpoint was saved at step 5000. - Generated samples remained coherent TinyStories-era English through the final checkpoint; this is a pretraining coherence gate, not a journalism capability claim. - Checkpoint integrity gate: `model_5000.pt` and `model_best.pt` each contain **50,093,312 parameters**, 145 tensors, 0 non-finite tensors, and identical 601,281,426-byte files. - Decision: training is complete; do not restart or overwrite the checkpoint. Next heavy job is the prepared 16k tokenizer pipeline, one stage at a time, followed by mapped-baseline parity before any SFT. ## 36. 16k preparation audit (2026-08-11) - `tokenizer16k.json` exists and phase-2b re-encoding completed: 32,562,432 old tokens -> 31,238,190 new tokens (0.96x). - Full-corpus re-encoding was attempted three times but did not complete; `data/train_full16k.bin` was **0 bytes**. No 16k model training started. - Fix: `stage_tokenizer_16k.sh` now writes `train_full16k.bin.partial` and only renames it to the canonical output after successful completion. The 50M checkpoint remains untouched. - First real vocab mapping parity attempt exposed a gate bug: it evaluated the 16k model on old-tokenizer `valid.bin`, producing a meaningless loss jump 2.7400 -> 9.2114. Mapping was not accepted or trained. `map_vocab.py` now compares old model/old validation with mapped model/`valid16k.bin`. - Corrected mapping measurement: old loss 2.7400 vs new-tokenizer loss 3.3748 (delta +0.6349), 8010/8192 exact mappings (97.8%), zero noise rows. The original parity expectation was invalid because tokenizer targets changed; accepted gate is finite loss + coverage now, followed by recovery during 16k continue-pretraining. ## 37. DNA-helix memory upgrade (2026-08-11) - Upgraded `research/helix.py` without changing model weights: memory records now carry IDs, case scope, source receipts, tags, salience, privacy boundary, and timestamps. - Added ranked `recall_many()` and cross-case `bridges()` for soft compartment retrieval; added user-controlled `forget()` and duplicate `consolidate()`. - Added `tests/test_helix_memory.py`: 2/2 tests passed; full project suite now **50 passed in 68.32s**. - Memory rule: no source-backed receipt means the record remains a lead, not a verified fact; deletion is explicit and does not alter model weights. ## 33. Watchdog overshoot fix + slowness diagnosis (2026-08-10) - Bug found: watchdog hardcoded `--steps 4000` which is ADDITIONAL on top of resume, so an attempt resuming at step 1500 targeted 5500 instead of 5000. - Fixed `train/watchdog_50m.sh`: each attempt now computes `need = TARGET_STEPS - latest_step` (capped at 4000/attempt) and stops exactly at step 5000. Syntax-verified with `bash -n`. - The attempt running when the fix landed (PID 5979, launched pre-fix) still runs to 5500 — left it running; +500 steps of pretraining is harmless, arguably a slightly better base for the queued tokenizer stage. - Slowness diagnosis (the ~150 tok/s crawl after the 10:39 restart): system was memory-starved — available RAM dropped toward 1.2 GiB, swap climbed to ~3.2 GiB, and the trainer itself had ~700-756 MiB swapped. Torch threads stalled on page-in (0% CPU blips, futex_wait_queue). Cold-start reload of the 601 MB fp32 ckpt + swap thrashing = ~150 tok/s at first, NOT a training bug. - Recovery: by step 1700-1900 throughput was back to ~290-440 tok/s. Current status: step ~1900/5500, val best still 3.1073 @ step 1500, next eval+save at step 2000. Watchdog (PID 5975) still supervising. ## 34. 50M run: whole-tree kill + patched watchdog relaunch (2026-08-10) - At step 3300/5500 the environment killed the ENTIRE watchdog tree (both watchdog bash and train_lm died, no traceback/exit marker). First time the watchdog itself was reaped, not just the child. - Loss since last checkpoint (model_3000.pt) capped at 300 steps. Best val still 2.8159 @ step 3000. - Relaunched patched watchdog (PID 5547): this time the fix is visible in the log — `--steps 2000 (step 3000 -> 5000)` — exact target, no more 5500 overshoot. - Confirmed resume from model_3000.pt at step 3000, params 50.09M. ## 35. 50M continue-pretrain COMPLETE (2026-08-11) - Target reached: step 5000/5000, exit code 0, watchdog self-stopped (logged "target 5000 reached; stopping watchdog"). - Final eval @ step 5000: val_loss 2.7050 — new best (2.7157 @ 4500 was prev). - Full improvement across the run: 3.3154 @ step 1000 -> 2.7050 @ step 5000 (~18.4% val-loss reduction). - Checkpoints: model_1000..model_5000 every 500; model_best.pt = model_5000.pt. - Generation at 50M is coherent multi-sentence with dialogue (see log samples). - Survived 4+ env kills thanks to watchdog + ckpt-every-500; final attempt ran uninterrupted 17:32 -> 01:41. - NEXT per queued pipeline: stage_tokenizer_16k.sh (16k BPE on 528M corpus, re-encode), then stage_v16k_continue.sh (+ --mtp 2), LoRA SFT, DPO, TIES, eval gate, 528M long-pole, RLVR, GGUF Q8, HF release. ## 36. Pipeline fire: 16k tokenizer stage + val re-encode fix (2026-08-11) - Fired stage_tokenizer_16k.sh detached (PID 28197) after 50M pretrain done. - BUG FOUND before stage 2: stage_v16k_continue.sh and stage_pretrain_full16k.sh used `--val data/valid.bin` (OLD 8k token IDs) with `--tok data/tokenizer16k.json` — eval loss would be garbage. Fixed: - stage_tokenizer_16k.sh now includes a 4/4 step re-encoding data/valid.bin -> data/valid16k.bin. - stage_v16k_continue.sh + stage_pretrain_full16k.sh now use --val data/valid16k.bin. - NOTE: the CURRENTLY RUNNING tokenizer stage (PID 28197) loaded the old script, so its 4/4 valid re-encode will NOT run; run it manually once tokenizer16k.json exists, before firing stage 2. - Also added --mtp 2 to stage_v16k_continue.sh (was pending per plan). - All three stage scripts syntax-checked (bash -n OK). ## 37. Tokenizer stage OOM bug found + fixed (2026-08-11) - The 528M full re-encode kept dying. ROOT CAUSE: data/reencode.py did `for i in mm.tolist()` on the WHOLE memmap -> Python list of 528M ints (~19 GB) > 7.2 GB RAM + 11 GB swap -> OOM-killed (phase2b worked at 32.5M tokens ~1 GB). The "streaming" claim in the docstring was wrong. - FIX: iterate in chunks (`range(0, len(mm), args.chunk_tokens)`), seg/buf persist across chunk boundaries so EOT-lines spanning chunks stay intact. RSS now ~300 MB. py_compile clean. - Progress: tokenizer16k.json (16384 vocab) + train_phase2b16k.bin (31.2M tok) + valid16k.bin (5.38M tok, 12s) done. Full 528M re-encode running with fix (PID 19243, ~30 min expected). - Terminal closed during stage; detached jobs survive; env still kills long jobs, so re-launching is routine. ## 38. Corrected 16k mapping gate and continuation start (2026-08-11) - Full 528M re-encoding completed atomically: 528,000,852 old tokens -> 520,133,183 new tokens (0.99x). - Mapping: 8010/8192 exact (97.8%), partial 182, noise 0; initial new-tokenizer loss 3.3748 versus old 2.7400. The old parity check was invalid because token targets changed; the accepted gate is finite loss + coverage + recovery. - Saved `ckpt/hybrid50m_v16k_init.pt`; next single heavy job is 16k continue-pretraining with `valid16k.bin`, BF16, MTP2, 5000 steps. - Launched `stage_v16k_continue.sh` under PID 19599; child PID 19646 is the only active heavy torch job, training `ckpt/hybrid50m_v16k_pretrain`. ## 40. 16k continuation resume after process interruption (2026-08-11) - The reduced 16k process stopped after step 2500 without a traceback. The `model_2500.pt` checkpoint is valid: step 2500, val loss 3.309583, 52,920,192 parameters, zero non-finite tensors. - Updated `stage_v16k_continue.sh` to resume optimizer/model state when a continuation checkpoint exists, and only use the vocab-expanded init for a fresh run. This prevents an accidental restart from step zero. - Relaunched detached as PID 8145 / train PID 8370. It verified the mapped init and printed `resumed .../model_2500.pt at step 2500`; training is active again with about 951 MB RSS and no competing torch job. - Update: reached step 3000/7500 at about 103-131 tok/s. Validation improved from 3.309583 at step 2500 to 3.2626 at step 3000; model_3000.pt saved and generation remains finite/coherent enough for this pretraining checkpoint. The 7500 target is expected because train_lm.py treats `--steps` as additional steps after resume. - The process later reached step 4000 and validation 3.1413445, but stopped during `torch.save` with `basic_ios::clear: iostream error`. Root cause was the filesystem reaching 100% with only 15 MB free; model_best.pt had already saved validly at step 4000. Removed obsolete older continuation checkpoints, restored the valid step-4000 checkpoint, and freed about 3.9 GB. - Hardened `train/train_lm.py` checkpoint writes with temp-file plus atomic `os.replace`. Relaunched detached as runner PID 8087 / train PID 8167 and verified resume from model_4000.pt with about 4.0 GB free. - Owner authorized cleanup due storage pressure. Deleted failed experimental `ckpt/tiny25m_lora_i` and `ckpt/tiny25m_lora_ii`, plus redundant numbered checkpoints 500-4500 from `ckpt/hybrid50m_pretrain`; preserved base model_5000.pt/model_best.pt and all active 16k assets. Free space rose to about 13 GB. The resumed run is active at step 4050/9000 (~154 tok/s). - Corrected additive resume semantics: added `--total-steps` to `train_lm.py` and set the v16k stage target to absolute step 5000. Stopped the old 9000- target process at its last valid step-4000 checkpoint and relaunched with six threads; runner PID 22575 / train PID 22791. First six-thread throughput sample is pending; storage remains about 13 GB free. - Update: corrected run reached step 4450/5000. Six-thread throughput measured 119-145 tok/s, slower than the earlier transient 190-205 tok/s readings; no crash, storage remains about 13 GB free. Leave it unchanged through the final checkpoint, then benchmark thread settings separately. - 16k continuation completed at step 5000/5000. Final validation loss was 3.0865, improving from 3.1413445 at step 4000 and 3.309583 at step 2500. `model_4500.pt` and `model_5000.pt` saved successfully with atomic writes; final generations are finite and readable. Throughput stabilized around 129-145 tok/s with six threads. No training process remains active. ## 39. 16k continuation resource gate (2026-08-11) - Initial 16k launch with batch 8 / threads 8 stopped after step 1 without a traceback and wrote no checkpoint; treated as a device resource kill, not a model result. - Reduced smoke passed **5/5 steps**, finite loss, validation, generation, and checkpoint save at batch 2 / seq 256 / threads 4. - Updated `stage_v16k_continue.sh` to batch 2, threads 4, val-batches 20. The real run is relaunched only after this smoke gate. ## 38. Full re-encode incomplete + self-healing runner (2026-08-11) - The chunked re-encode still got env-killed near the end: train_full16k.bin had 520,133,183 tokens / 11,544,766 EOT lines vs expected 11,545,267 (missing last ~501 lines; file ended cleanly on EOT, so no partial line). - Created train/watchdog_reencode_full.sh: rm partial -> re-encode -> verify EOT line count == 11,545,267 -> retry on kill/incomplete (99 max). Launched detached (PID 8486), attempt 1 running. ~30 min per attempt. - NOTE: expected line count source = retrain run's lines=11,545,267 (1:1 line mapping re-encode). If verification never matches, investigate mapping before blaming the runner. - Stage 1 status: tokenizer16k.json, train_phase2b16k.bin, valid16k.bin DONE. Only train_full16k.bin pending -> then stage_v16k_continue.sh (--mtp 2). ## 39. Collaborator: v16k continue-pretrain running (2026-08-11/12) - Collaborator modified stage_v16k_continue.sh: batch 2, threads 6, --total-steps 5000, --mtp 2, --resume ckpt/hybrid50m_v16k_pretrain, --val data/valid16k.bin. Launched bash stage_v16k_continue.sh detached (reparented to init, log logs/stage_v16k_runner.log). No watchdog wrapper. - Vocab expansion verified: base model_5000.pt 8192->16384, 50.07M->52.70M params, token map 8010 exact + 182 partial (97.8% exact). - Run resumed from model_4000.pt, now ~step 4800/5000, params 52.92M (incl MTP heads). val 3.0942 @ 4500 (higher than 8k run's 2.7050 — expected: 16k vocab harder + first adaptation epoch + MTP aux loss). Losses 3.7-5.1. - ETA ~25 min to step 5000 (batch 2 -> ~7s/step, 120-165 tok/s). - train_full16k.bin still INCOMPLETE (384MB partial, missing ~501 lines) — needed only for the long-pole stage; finish re-encode AFTER this run (one heavy job at a time), then LoRA SFT -> DPO -> TIES -> eval gate. ## 40. Skill consolidation + new skills (2026-08-12) - v16k continue-pretrain COMPLETED cleanly: model_5000.pt saved (ckpt/ hybrid50m_v16k_pretrain/), run "done" at ~22:30, val 3.0942 @ 4500, --mtp 2. - SKILLS (per owner directive: three always-on guardrails + research gaps): 1. Developer's Credo: consolidated tiny-model-mandalorian INTO tiny-model-developer-credo (owner renamed it). Now the always-on discipline skill: Mandalorian creed translated (Resol'nare six actions, "This is the Way", "Mandalorians don't run", "I have spoken") + Absolute Quality Bar (no half-ass; no synthetic/generators/scripts; production- grade rows only) + discipline SOP + arch growth principles retained. tiny-model-mandalorian dir removed (references only historical notes). 2. tiny-model-mtp CREATED (was the one real skill gap): Meta arXiv 2404.19737 multi-token prediction; n=2 default, aux w=0.1, pretrain-only (never SFT/DPO), +205k params at n=2; anchored by the v16k run. 3. tiny-model-kd: added the Absolute Quality Bar section. 4. tiny-model-reasoning: added Conversational Reasoning Voice — gold traces must read as natural human reasoning prose, NOT stamp blocks; markers are training-only; verdict/confidence woven into prose, harness parses at eval. This is the owner's fix for the "stamp, not thinking" criticism. - Web research: arXiv 2404.19737 abstract retrieved and encoded in the MTP skill. DDG search for Mandalorian canon was blocked; credo translation used the previously-researched mandalorian skill + canon (Resol'nare, creed quotes) already recorded in the repo. - NEXT (proposed): re-author a natural-reasoning gold sample batch (per the Conversational Reasoning Voice) for the owner to judge before committing the full set; then LoRA SFT on the 16k base -> DPO -> TIES -> eval gate; finish train_full16k.bin re-encode (incomplete by 501 lines) before the long pole. ## 41. Persona hybrid: Spock baseline + Sheldon breakthrough modulator (2026-08-12) - Owner addition: keep Spock (Quinto) as the reasoning baseline; add a LITTLE Sheldon Cooper that emerges when the model verifies something big / a breakthrough / pattern resolves ("Bazinga-ish energy"). - Research: pulled real Into Darkness Spock quotes (wikiquote): "The needs of the many outweigh the needs of the few." etc. TBBT page structure blocked parsing; Sheldon patterns grounded in well-attested canon (Bazinga, Fun fact, precise hyperbole; excitement expressed THROUGH facts — rigid joy, never gushing). Marked as patterns vs exact quotes. - tiny-model-persona updated: "Sheldon Breakthrough Mode" section — trigger (verified finding only), voice shift (energy lift, factual delight, rare Bazinga for resolved gotchas), guardrails (rare ~1/15-20 rows, returns to Spock baseline, parseable verdict), gold-authoring rule (breakthrough rows start baseline and elevate at payoff). - Next: draft the persona sample batch (Spock baseline + a few breakthrough rows) for owner judgment, then full re-author per Conversational Reasoning Voice -> LoRA SFT on the 16k base. ## 42. Overnight pipeline: v22 conversational gold + LoRA SFT prep (2026-08-11/12) - Owner: run overnight, no judgment needed, free rein. Training = v16k base (model_5000.pt) LoRA SFT on the NEW conversational-voice gold. - Authored data/kd_gold_v22.jsonl (46 handcrafted rows, Spock baseline + 3 Sheldon breakout rows + analyst/skeptic personas; natural prose scratchpad/final, verdicts woven in, canonical vocab preserved). - Built data/sft_v22.jsonl = 119 rows: v22(46) + persona_dialogue(24) + general_chat(24) + seed_forensic(25) — all handcrafted conversational voice. - Created stage_lora_sft_v22.sh + train/watchdog_lora_v22.sh: LoRA r16/alpha32 on frozen 16k base, replay 0.5 (phase2b16k), KL 0.1, ppl guard 60, lr 5e-5, epochs 3, batch 4 seq 512, resume-safe watchdog (model_final.pt stops it). - NOTE: stage_lora_50m.sh (old) passes --save-every which train_lora.py does not support — not used; the new stage only passes supported args. - Full 528M re-encode still running (attempt 1 from 23:21); SFT launches after it verifies (one heavy job at a time). train_full16k.bin needed only for the long pole, not for SFT. ## 43. Overnight RUNNING: full re-encode DONE + LoRA SFT v22 live (2026-08-12) - FULL 528M RE-ENCODE COMPLETE (verified): train_full16k.bin = 520,133,183 tokens / 11,544,766 lines (matches the deterministic anchor exactly). KEY FIX: the re-encode drops EMPTY EOT segments (source has 16.8M EOTs vs 11.5M non-empty lines); retrain's 11,545,267 line count was the wrong verification target. watchdog_reencode_full.sh now verifies lines AND tokens against the anchor (11,544,766 / 520,133,183) and stops cleanly. - LoRA SFT v22 LAUNCHED (watchdog PID 15210, attempt 1): base model_5000.pt, 58 LoRA adapters, 1,618,880 trainable, 119 gold rows (0 filtered), replay 103 @ 0.5, epochs 3, batch 4 seq 512, lr 5e-5, KL 0.1, ppl guard 60. ~153 steps total, resume-safe, logs/sft_v22.log. - Next after SFT: best.pt/best_ppl.pt -> LoRA SFT done -> DPO (length-norm, 1 epoch, lr<=5e-5) -> TIES merge -> eval gate -> long-pole (full16k) -> RLVR -> GGUF -> HF. ## 44. Restart: session-kill recovery + SFT v22 progressing (2026-08-12) - ROOT-CAUSE of the "dead training, empty ckpt": the env killed the whole interactive session, and the watchdog was a child of that session, so it died with it and never self-healed. Log had only the 4 setup lines; ckpt dir empty. - FIX (per handoff immediate-action): relaunch fully detached with `setsid nohup ./train/watchdog_lora_v22.sh >/dev/null 2>&1 survives session ends. - CONFIRMED PROGRESS (05:31): step 25/153, loss 3.2685, sft_val_loss 6.43, val_ppl 21.89 [best], ~75 tok/s (swap-limited), checkpoints saving (best.pt/best_ppl.pt/model_25.pt ~211MB fold-only). ETA ~70-80 min to finish (~step 153) -> model_final.pt stops the watchdog. - DISCIPLINE REMINDER: one heavy job at a time. Do NOT start DPO / long-pole / RLVR / heavy tests while SFT trains. Prep scripts + read-only research only. - NEXT (after model_final.pt): pick best.pt -> DPO (LoRA adapter-only, 1 epoch, lr<=5e-5, beta 0.05, val-bin valid16k) -> TIES merge -> stage_eval_50m.sh gate (>=0.40 main / >=0.25 researcher @60% coverage) -> long-pole full16k -> RLVR -> GGUF Q8 -> HF. ## 45. Automation: self-driving SFT->DPO->eval chain (2026-08-12) - Created train/watchdog_dpo_v22.sh (LoRA-DPO, adapter-only, 1 epoch, lr 4e-6, beta 0.05, length-norm already in train_dpo.py, ppl guard 60, resume-safe, stops at model_final.pt) and train/chain_post_sft_v22.sh (waits for SFT final, keeps SFT watchdog alive, launches DPO detached, waits for DPO final, runs stage_eval_50m.sh). All launched via setsid nohup (own session = survives env kills). Chain does NOT merge/release — gate numbers reported for owner decision. - Verified next-phase inputs: eval gate = 50 main + 27 researcher + 26 red-team probes; DPO data prefs_p_all.jsonl = 3004 valid rows (proven champion size), schema {chosen,persona,prompt,rejected}. Pipeline matches research-backed recipe in tiny-model-pipeline / tiny-model-posttrain (SFT->length-norm DPO->TIES->eval gate). - tiny-model-sop updated: added DEVICE OPS rule (env kills sessions -> launch watchdogs detached via setsid nohup & disown; verify PPid=1) + changelog. Mirrored to ~/.codex/skills/. This was the missing operational skill for the phase that just burned us (child-of-session watchdog died with the session). ## 46. BUG FIX: MTP head drift broke DPO load (2026-08-12) - SYMPTOM: chain auto-launched DPO after SFT final, but train_dpo.py crashed every attempt (13-15+) with "Missing key(s): mtp_heads.0.0.bias, mtp_heads.1.0.bias". - ROOT CAUSE: code/checkpoint drift. The saved SFT checkpoints have MTP heads with WEIGHT only (no bias); the current model code builds MTP heads as nn.Sequential(nn.Linear(...,bias=True), SiLU) -> expects bias. So ref model has mtp bias, sd lacks it -> strict load fails. (MTP is pretrain-only per tiny-model-mtp; unused in generation.) - FIX (surgical, doctrine-aligned): in train/train_dpo.py, research/eval.py, train/ties_merge.py -> set cfg.mtp_heads = 0 (no MTP post-training) and load with strict=False (tolerate the 2 stray mtp weight tensors). Smoke test: 0 missing / 2 unexpected (the ignored mtp weights). Verified before relaunch. - Relaunched chain_post_sft_v22.sh detached -> auto-fires DPO now that SFT done. ## 47. Production-readiness audit: v22 DPO/TIES quarantined (2026-08-12) - This audit is read-only; no release, export, long-pole training, or RLVR was started. - `logs/dpo_v22.log` records the resumed v22 DPO run reaching step 700/751 with `val_ppl 137285.47` against the configured 60.0 guard. The trainer correctly aborted that continuation. - `ckpt/hybrid50m_v22_dpo/model_final.pt` is not a completed step-751 final: its Torch archive root is `model_600`, and the original `model_600.pt` is no longer present. Treat it only as the step-600 recovery artifact. Its filename must not be used as evidence that DPO completed. - `best_ppl.pt` was written on the guard-breaching resumed run because `train_dpo.py` resets `best_ppl` on resume and saves before checking the PPL guard. It is a divergent checkpoint, not a best model. - `ckpt/hybrid50m_v22_ties/ties_merged.pt` therefore has incomplete provenance: it is an experimental merge involving the renamed DPO-600 artifact, not a release candidate. The merge is quarantined pending a manifest and evaluation. - Current `research/eval.py` scores exact canonical verdict and output format; it does not measure citation grounding, source-ranking, independent-source corroboration, multi-turn state, or real retrieval-to-report tasks. Its threshold is a research-stage gate, not a production reliability claim. - NEXT RECOMMENDATION: before any new post-training, implement and test the source-policy/claim-grounding gate, build a frozen real-task benchmark, and repair DPO resume/guard/watchdog semantics. No model may state a fact as verified unless deterministic evidence and source-policy checks pass. ## 48. DPO/TIES remediation begins (2026-08-12) - Owner decision: discard the v22 TIES merge as compromised; no evaluation, export, or release may use it. - Quarantined `ckpt/hybrid50m_v22_ties/ties_merged.pt` as `ckpt/quarantine_v22_ties/ties_merged_compromised.pt` with a quarantine manifest. The file is retained only for audit evidence. - Confirmed DPO lineage before modification: `model_final.pt` metadata step 600, no `best_ppl` field; `best_ppl.pt` and `raw_latest.pt` are step 700 after the 137285.47 guard breach. Next surgical action is to quarantine those divergent outputs, truthfully name the step-600 recovery archive, and require explicit resume metadata instead of guessing. - Root cause decision: the resumed checkpoint has no RNG state or `best_ppl` metadata. The first run used LoRA dropout 0.05; the resumed process reset Python/Torch RNG and repeated step 610 with a different dropout mask, changing the trajectory before the step-700 PPL explosion. The old watchdog also reused the failed output directory/marker. Recovery will use a clean output lineage, explicit best PPL 25.76, fresh optimizer, dropout 0, LR 1e-6, and a 60.0 guard. - Quarantined divergent DPO outputs as `ckpt/quarantine_v22_dpo/` and renamed the truthful step-600 archive to `model_600_recovery.pt`; no bytes were deleted. Patched `train/train_dpo.py` to require recorded best PPL for legacy resumes, optionally reset optimizer, and persist/restore RNG state. Recovery output is `ckpt/hybrid50m_v22_dpo_recovery/LINEAGE.json` with lr 1e-6, dropout 0, fresh optimizer, source step 600, source val_ppl 25.76, guard 60. Preflight passed: checkpoint step 600, optimizer state 113 entries, model state 145 keys, CLI help loads. - Recovery measurement: step 60/751 reached with DPO loss values 0.6867-0.7077, no PPL gate yet, no crash, and watchdog/train remained detached (PPID 1). First validation remains step 100; do not interrupt the heavy job. - Launched repaired DPO recovery detached at 2026-08-12 12:35:06: watchdog PID 25322 (PPid 1), train PID 25328. Command uses source step 600, `--resume-best-ppl 25.76`, `--reset-optimizer`, lr 1e-6, dropout 0, guard 60.0, clean output `ckpt/hybrid50m_v22_dpo_recovery`. No gate result yet. - Recovery gate FAILED at step 100: val_ppl 18750.93 > 60.0. Root cause was confirmed: `model_600_recovery.pt` is folded model-only state, so resuming it with `--lora` created fresh adapters rather than restoring trained LoRA adapters. Quarantined `dpo_failed.json` as `ckpt/quarantine_v22_dpo_recovery/dpo_failed_step100_ppl18750.json`. - New recovery decision: continue the folded step-600 model with full-parameter DPO, fresh optimizer, lr 1e-7, batch 4, seq 512, guard 60.0, output `ckpt/hybrid50m_v22_dpo_full_recovery`. If this fails, DPO must restart from the original SFT base rather than reuse the folded archive. - Full-parameter recovery passed its first gate: step 100/751, val_ppl 15.12 against guard 60.0, best_ppl.pt and model_100.pt saved, no non-finite/crash signal. Continue detached; no completion or TIES merge is authorized yet. - Device/session termination stopped the child at step 170 without traceback before step 200; no DPO guard result was produced. Valid `model_100.pt` was verified: step 100, best_ppl 15.115812720919992, RNG state present, 145 optimizer state entries. Updated watchdog to select the newest numeric recovery checkpoint and resume its saved optimizer/RNG state. - Deterministic recovery resumed successfully from model_100.pt. At the latest measurement it reached step 280/751; step-200 gate passed with val_ppl 9.37 against guard 60.0 and model_200.pt saved. Watchdog PID 28977 / train PID 28983 remain detached; no TIES merge or release action is authorized. - Watchdog resume attempt stopped before training because its generated command duplicated `--resume` and omitted the path (`argparse: argument --resume: expected one argument`). No weights or checkpoints changed. Fixed the shell argument construction and passed `bash -n`; next launch resumes model_100.pt. - Fixed resume determinism: DPO now uses a seed-fixed local epoch shuffle so resume does not reshuffle examples before skipped steps. Stopped the prior step-160 continuation before a new checkpoint and relaunched detached at 14:11:16 with watchdog PID 28977 / train PID 28983 from model_100.pt. - Public AIDE adoption milestone measured via GitHub API: forks=1, stars=3. - Storage cleanup authorized by owner: filesystem was at 100% with 877 MB free while the active checkpoint is ~633 MB. Preserved active training, source model_600_recovery.pt, model_400.pt, base SFT, data, and tokenizer. Removed redundant recovery checkpoints 100-300, old DPO checkpoints 100-500, and discarded quarantine binaries. Free space increased to 4.3 GB. Training remained active and progressed through step 450/751 with no new guard result. - DPO recovery progressed past step 500/751; validation gate passed with val_ppl 47.65 under guard 60.0 and model_500.pt saved. Train PID 28983 remains active and detached. No TIES or release action is authorized. - DPO final integrity gate passed: model_final.pt step 751, best_ppl 9.368478, 52,714,752 parameters, 145 state keys, zero non-finite tensors. - First corrected eval used tokenizer16k but was foreground-killed after 15 main probes without traceback; no scorecard was accepted. Added and launched detached `train/watchdog_eval_v22.sh` (watchdog PID 7896, eval PID 7901) for main, researcher, and red-team batteries; it writes eval_complete.json only after all three finish. - Storage cleanup during detached evaluation: removed redundant DPO checkpoints model_400/model_500/model_600 and temporary desktop artifact directories; preserved model_700, model_final, model_600_recovery, and the active eval process. Free space increased from 957 MB to 3.8 GB; red-team evaluation remains active. ## 2026-08-13 — Full audit + research + skill-gap closure (v22 post-DPO state) ### Verified live state (no training/eval running) - No python/train/eval process is alive. Last action: detached eval completed `2026-08-12 19:14` (`ckpt/hybrid50m_v22_dpo_full_recovery/eval_complete.json`, `logs/eval_v22_detached.log`). No collaborator files newer than that were found on disk (only `agent_notes.md` 18:44 + eval_complete.json 19:14). - Disk 3.2 GB free; RAM ~218 MB free; swap active. One heavy job at a time. ### The v22 DPO is complete but COLLAPSED (honest numbers) - `ckpt/hybrid50m_v22_dpo_full_recovery/model_final.pt` (step 751): main 0.122 (49), researcher 0.167 (18), red-team 0.038 (26). Nearly every output is `verdict: false | conf: HIGH` (single-class + anti-calibrated collapse). Format rate 1.00 (constrained decode). - DPO loss was flat ~0.65 for all 751 steps — the 3,004 preference pairs produced essentially no learning signal. - Lineage deviation recorded in LINEAGE.json: full-parameter DPO continued from the FOLDED step-600 LoRA archive (lr 1e-7), not LoRA-DPO from the SFT best. Skill `tiny-model-posttrain` forbids this pattern going forward. - Root-cause finding (measured, 2026-08-13): the 3,004 preference pairs are SCHEMA-MISMATCHED with the v22 SFT — all pairs use the old analyst stamp format ("Step 1..N", "Verdict: X. Confidence: Y.", persona=analyst only), while `data/sft_v22.jsonl` (119 rows) is the new Spock conversational schema ("<|scratchpad|>...<|final|>I consider this ..."; personas analyst 74 / skeptic 10 / spock 35). DPO optimized toward an incompatible style. (Verified by grep counts: 3004 'Step 1' + 3004 'Verdict:' vs 0 'I consider' in prefs; 1 'I consider' style in SFT.) - `best_ppl.pt` (step 200, val_ppl 9.37) and the SFT best (`ckpt/hybrid50m_v22_lora/best.pt`) have NEVER been battery-eval'd. 25M precedent: DPO3@200 mid-training checkpoint was the champion. The candidate-eval gap is the #1 measurement to close. ### Research (2026-08-13, multi-source, on-device) - LFM2 technical report (arXiv 2511.23404): exact three-stage ending — (1) SFT; (2) length-normalized direct alignment: joint loss L = -E[ w*f(Δ-m) + λ*g(δ) ] with Δ = r_w/|y_w| - r_l/|y_l|, δ = σ(r_w/|y_w|) - σ(r_l/|y_l|), r = β log(πθ/πref); DPO = special case (w=1, m=0, λ=0), LFM2 adds margin m=0.1 + APO-zero term λ=0.2. Preference data mixes on-policy (N=5 sampled from the SFT ckpt) + off-policy. (3) Merging = apply soup / task arithmetic / TIES / DARE / DELLA IN PARALLEL, evaluate, keep best. §4.5: small models fail evals on format — robust parsing, report parse failures separately. - DPO behavior evidence: D-STEER (arXiv 2512.11838) — DPO acts as a low-rank steering perturbation; it changes behavior, not beliefs (matches: DPO moved the STYLE, not the verdict discrimination). Output diversity collapse in post-training (arXiv 2604.16027) — post-trained models homogenize outputs (matches the single-class collapse). - DPO over-optimization/verbosity: arXiv 2406.10957 (down-sampled KL), arXiv 2602.06239 (PEPO), arXiv 2506.08681 (importance sampling). - Small-model alignment: arXiv 2502.17927 (advantage-guided distillation — alignment gains diminish on SLMs), arXiv 2511.06512 (EASE, edge safety). - Big-tech recipes re-verified: Llama 3.2 (1B/3B) = iterative rounds of SFT -> Rejection Sampling -> DPO (model card); SmolLM2 = SFT then DPO 1 epoch (UltraFeedback, alignment-handbook); LFM2 SFT = ~5.39M samples. - SFT data floor: tiny-model-reasoning stands — 1,500-3,000 hand-authored gold rows; we are at 119 (huge gap; 119 teaches format, not judgment). ### Skills applied / created (2026-08-13) - UPDATED `tiny-model-posttrain` — LFM2-verified length-normalized joint objective, parallel merging + eval selection, checkpoint-selection rule, v22 schema-mismatch failure record. - UPDATED `tiny-model-eval` — collapse detector (>70% single class / anti-calibration = flag), tokenizer-match rule (8k default crashes on 16k checkpoints — logs/eval_50m_20260812_1649.log), candidate battery discipline (eval every candidate; red-team through the full pipeline). - CREATED `tiny-model-multiturn` — multi-turn coherence + real-task end-to-end verification gate (owner's pre-release blocker; LFM2 §4.1/§4.5, MT-Bench 2306.05685). - UPDATED `tiny-model-roadmap` — current status + re-ordered next steps. - All mirrored to `~/.codex/skills/` and `/root/.shared-skills/`. ### Corrected next steps (proper order, skills applied) 1. Battery-eval the SFT best (`ckpt/hybrid50m_v22_lora/best.pt`) and the DPO best_ppl (`.../best_ppl.pt`) on the SAME fixed battery (16k tokenizer, `PYTHONPATH=$PWD`, detached watchdog). ~1h each, no training. Measure before any merge decision. 2. Run PARALLEL merges on the 16k pretrain base (soup + task arithmetic + TIES at minimum; DARE optional), eval each candidate, keep the best. 3. Expect release gates to still fail (main >= 0.40, researcher >= 0.25 at >= 60% coverage) -> DO NOT release. 4. Author new handcrafted SFT gold in the v22 Spock schema toward the 1,500-3,000 row floor (biggest data lever), balanced across verdict classes, then clean LoRA-SFT. 5. Author schema-matched, class-balanced preference pairs; run LoRA-DPO from the SFT best with the LFM2 length-normalized objective, 1 epoch, lr <= 5e-5, replay 0.5. 6. Run multi-turn + real-task verification (tiny-model-multiturn) and red-team through the FULL pipeline before any release claim. ### Candidate battery evals launched (2026-08-13) - Launched `train/watchdog_eval_candidates.sh` detached (watchdog PID 21734, PPid=1, verified reparented; eval PID 21739) at 21:00:33 UTC. - Order (sequential, one heavy job at a time): (1) SFT best `ckpt/hybrid50m_v22_lora/best.pt`, (2) DPO best_ppl `ckpt/hybrid50m_v22_dpo_full_recovery/best_ppl.pt`; each on main + researcher + red-team with tokenizer16k, threads 6. Log: `logs/eval_v22_candidates.log`; per-battery markers `logs/eval_cand__