OpenTransformer's picture
code_archive README
9a4a1e3 verified
|
Raw
History Blame Contribute Delete
1.39 kB

Code archive

Snapshot of /workspace on the Vast 4090 training box (instance 36391051), with git history.

Snapshot 20260521T230246Z

  • agillm_workspace_git_20260521T230246Z.tar.gz - full .git directory (extract with tar -xzf ...; mv .git /target/)
  • agillm_code_snapshot_20260521T230246Z.tar.gz - flat snapshot of all tracked files at HEAD (no .git)

Git history at time of snapshot

3 commits on main:

  1. 66604cc Baseline: vast workspace state as of 2026-05-21T23:01Z - pre-gate-fix
  2. 9da9902 Fix SAT gate collapse: per-position forward + correctness-based training target
  3. e18cc73 Add AR-only ablation script (5k steps from 220k, no SAT loss)

SAT gate fix summary (commit 9da9902)

Root-cause: training loss hardcoded gate target to torch.ones(...), causing the SAT gate to collapse to a deterministic one-hot. Result: sat-var and sat-fixed inference produced byte-identical output (gate always picked stride=2).

Three sub-fixes in nB300.py:

  • SATHead.forward: gate applied per-position (was h_last[:, 0] only)
  • Training loss: gate target = (logits_sat.argmax(-1) == tgt_sat).long()
  • Inference: gate is now [B, SAT_BLOCK, 2]; stride uses gate[:, 0]

Backward-compatible with existing checkpoints. Existing 220k checkpoint's gate weights are still collapsed from prior training (needs fresh SFT for full benefit), but the inference math is now correct.