ship logs/small_a41_hg_cbk_gf_wa.log (2026-08-22 4090 queue)

logs/small_a41_hg_cbk_gf_wa.log

@@ -0,0 +1,79 @@
+[alephlm0] disk: 134.6 GB free on the work drive
+[alephlm0] allocator capped at 22.7 GB (0.880 of 25.8; free was 24.2) — over-commit now FAILS FAST instead of spilling to sysmem
+!! RECIPE DEVIATION: max_tokens 256 -> 128 (ALEPHLM_MAX_TOKENS). Captions beyond the new ceiling are TRUNCATED at tokenization.
+[alephlm0] arm=a4_splat seed_idx=0 (crc seed 155979836) pooling=cls profile=retrain -> AbstractPhil/alephlm-0 runs/alephlm0-small-a4_splat-hg-cbk-gf-wa-s0/
+[alephlm0] splat geometry recorded: {'M': 128, 'K': 8, 'overlap': 1.5, 'tau': 0.1, 'mchunk': '16'}
+── PROFILE 'retrain' ─────────────────────────────────────────────────────────
+   run_name         alephlm0-small-a4_splat-hg-cbk-gf-wa-s0
+   hf_repo          AbstractPhil/alephlm-0
+   artifact_repo    AbstractPhil/captionbert-8192-v2
+   targets_repo     AbstractPhil/captionbert-8192-v2-consensus
+   caption_field    caption_llava
+   fresh_start      True
+   stages          0:False 1:False 2:False 3:True
+   config coherent
+==============================================================================
+ALEPHLM0-SMALL-A4_SPLAT-HG-CBK-GF-WA-S0 — CONSENSUS DISTILLATION, CC12M SCALE
+==============================================================================
+device=cuda  work_dir=E:/mirel/data/alephlm0
+gpu=NVIDIA GeForce RTX 4090 vram=26GB
+chunks: 1 train + 2 holdout | excluded for missing experts: {}
+    [backup] -> AbstractPhil/alephlm-0 (private)
+── ARTIFACTS ─────────────────────────────────────────────────────────────────
+   maps: local E:/mirel/data/alephlm0/maps/alignment_maps.pt
+   (NOT refitting: every consensus target was built with THIS map;
+    a new fit would sit in a different frame with no loss signal.)
+   targets: 3 local + 0 fetched from AbstractPhil/captionbert-8192-v2-consensus
+
+── STAGE 3 — TRAIN ───────────────────────────────────────────────────────────
+    [alephlm0] arm=a4_splat: dense ff2048 -> trunk ff512 + 3x ff512 aleph-dispatched (tau=0.1, gates σ(-3.0), expert outs zero-init)
+    [alephlm0] a4_splat: M=128 K=8 overlap=1.5 | TRAINABLE 61,480,740 of 61,480,740 total — attention trainable capacity HALVED vs MHA (-525,312/block; frozen fabric is not capacity). A capability LOSS vs a1 is capacity-confounded; a MATCH at -10.8%% trainable is the strong result.
+    [alephlm0] C6 null path: bit-exact at init  ✓
+    alephlm0-small-a4_splat-hg-cbk-gf-wa-s0: 61,480,740 params (0.56x bert-base)
+    12L 512d 8h ff2048 pool=cls
+    1 chunks ≈ 500,000 rows | 244 steps/ep x 2 = 488 steps @ batch 2048
+    loss = 1.0*InfoNCE(T=0.07) + 1.0*MSE + 0.0*CV   [champion consensus_nce_mse]
+── VRAM PROBE (co-resident-aware) - worst case before data ───────────────────
+    free 23.8 of 25.8 GB (co-resident/desktop holds 1.9 GB)
+    batch 2048 x L 128 (checkpointing=True) -> peak alloc 16.5 / reserved 17.4 GB vs 23.8 free -> margin 6.5 GB
+    PASS
+    fresh_start=True -- ignoring any local or remote checkpoint
+    building val store...
+    RamStore [val]: CACHE HIT E:/mirel/data/alephlm0/store_cache/store_56998379bbd3 — 500,000 rows, targets+tokens memory-mapped (~0.0 GB resident)
+    building train store (one pass, then zero disk I/O)...
+    RamStore [train]: CACHE HIT E:/mirel/data/alephlm0/store_cache/store_83fdd0bab51e — 500,000 rows, targets+tokens memory-mapped (~0.0 GB resident)
+    500,000 rows resident | 244 steps/ep x 2 = 488 steps
+    batch plan: 244 batches/epoch | padded length p50 33 p90 117 max 128 (bucketing=True)
+    e1      50/488 loss 7.2827 nce 7.2803 mse 0.00243 acc 0.002 lr 3.03e-04 L64 12m
+    [backup] pushed (step 80) -> runs/alephlm0-small-a4_splat-hg-cbk-gf-wa-s0/
+    e1     100/488 loss 7.0183 nce 7.0160 mse 0.00233 acc 0.002 lr 6.00e-04 L64 25m
+    aleph: amp mean 0.00565 (b0 0.00535 .. b11 0.00382) | drift mean 0.13903 | gates σ b0 [0.0476, 0.0476, 0.0474]
+      VAL r1 0.0040 cos 0.1088 self_cos +0.3221 erank 1.8 cv 1.0692 | frame r1 0.0040
+    e1     150/488 loss 6.7599 nce 6.7578 mse 0.00217 acc 0.004 lr 5.76e-04 L128 39m
+    e1     200/488 loss 5.5852 nce 5.5834 mse 0.00174 acc 0.029 lr 5.07e-04 L64 51m
+    aleph: amp mean 0.00661 (b0 0.00539 .. b11 0.00536) | drift mean 0.23286 | gates σ b0 [0.0496, 0.0496, 0.0486]
+      VAL r1 0.0230 cos 0.2946 self_cos +0.0607 erank 5.1 cv 0.6359 | frame r1 0.0210
+No files have been modified since last commit. Skipping to prevent empty commit.
+    [backup] pushed (step 246) -> runs/alephlm0-small-a4_splat-hg-cbk-gf-wa-s0/
+    e2     250/488 loss 5.1321 nce 5.1304 mse 0.00164 acc 0.059 lr 4.05e-04 L96 63m
+    e2     300/488 loss 4.7929 nce 4.7914 mse 0.00157 acc 0.082 lr 2.86e-04 L96 76m
+    aleph: amp mean 0.00652 (b0 0.00540 .. b11 0.00498) | drift mean 0.25481 | gates σ b0 [0.0511, 0.0514, 0.05]
+      VAL r1 0.0745 cos 0.3848 self_cos +0.0327 erank 9.1 cv 0.4044 | frame r1 0.0735
+    e2     350/488 loss 4.2930 nce 4.2915 mse 0.00144 acc 0.126 lr 1.69e-04 L64 86m
+    e2     400/488 loss 4.0786 nce 4.0772 mse 0.00139 acc 0.161 lr 7.39e-05 L64 97m
+    aleph: amp mean 0.00715 (b0 0.00545 .. b11 0.00679) | drift mean 0.27521 | gates σ b0 [0.0521, 0.0521, 0.0508]
+      VAL r1 0.1400 cos 0.4321 self_cos +0.0145 erank 12.1 cv 0.2755 | frame r1 0.1465
+No files have been modified since last commit. Skipping to prevent empty commit.
+    [backup] pushed (step 418) -> runs/alephlm0-small-a4_splat-hg-cbk-gf-wa-s0/
+    e2     450/488 loss 4.0184 nce 4.0170 mse 0.00139 acc 0.177 lr 1.51e-05 L64 111m
+    aleph: amp mean 0.00725 (b0 0.00545 .. b11 0.00712) | drift mean 0.27970 | gates σ b0 [0.0522, 0.0522, 0.0509]
+── FINAL ─────────────────────────────────────────────────────────────────────
+    mimicry R@1 (student->consensus, NOT capability): 0.1555
+    cos to target : 0.4403
+    self_cos      : +0.0126   <- isotropy; teachers .81-.98
+    effective rank: 12.9/768
+    CV            : 0.2727
+    frame-fit R@1 : 0.1515 (should be ~mimicry: reference-member alignment pins the frame)
+    CAPABILITY is decided by STS-B / SICK vs the five teachers, not here.
+No files have been modified since last commit. Skipping to prevent empty commit.
+    [backup] pushed (final) -> runs/alephlm0-small-a4_splat-hg-cbk-gf-wa-s0/