training / experiments /gemma_decoder

Commit History

2 days ago

  1. targets: two new grounded claims from the trace audit (idle memory; same-squares-keep-changing) - for the next pilot round

    agent
    ca231d1
  2. run_pilot_v6: 96 pos x 150 iters, PT+IT, results auto-pushed to hub (rebuild-proof)

    agent
    633e604
  3. debug_backend: len-mismatch fix (eos slicing)

    agent
    b052374
  4. debug_backend: operator-requested cuDNN SDPA NaN diagnostics (GPU id, seq%128, zero-token repro with cudnn sdp disabled, both-backend grad finiteness)

    agent
    bb51fdf
  5. NaN fix v2 (root cause: exactly-zero embedding rows trigger gemma-backward NaN on torch 2.13, localised to layer-12 mlp grad_out via hooks; A-probe with zero token reproduced it): no zero rows anywhere - B uses 1e-4 no-info epsilon, resampler opener 1e-4

    agent
    884986c
  6. debug_nan: hook-based NaN localisation (bridge + all gemma layers) + zero-token and sdpa-shape probes

    agent
    b5af418
  7. NaN-grad fix: rsqrt-normalise resampler k/v (torch-2.13 sdpa backward NaNs with 4x-magnitude latent inputs); assert now NaN-aware with grad table

    agent
    f473997
  8. debug_d: restore m,cfg assignment lost in the hub-pull patch

    agent
    c58f45e
  9. debug_d: pull fast_wide from hub (fresh container has no runs/)

    agent
    98f084f
  10. debug_d: minimal Space-side repro of the D zero-grad crash

    agent
    697943f
  11. grad-debug diagnostics on the zero-grad assert (D-condition crash is not locally reproducible)

    agent
    494b63d
  12. run_pilot: condition A = learned soft prompt (zero-input resampler had no grad path - caught by the assert on real gemma); p_halt None guards for halting-free configs (fast_wide)

    agent
    dd20003
  13. prototype report skeleton with all verified facts (results section awaits Space pilot)

    agent
    ef1a99a
  14. run_pilot: held-out split (20% never trained on), heldout_ce measurement, generation now on an UNSEEN position

    agent
    2e7e49c
  15. allowlist gemma pilot scripts on the Space; --hf-checkpoint hub pull so the container doesn't need local ckpt files

    agent
    f7cd607
  16. run_pilot: list-eos support (gemma-it has eos [1,106]); teacher-forcing slices target up to first eos (learn to stop)

    agent
    6341f4d
  17. run_pilot: mock-gemma dry run green end-to-end; fixes: StepDescriptor t=0 branch, frozen-grad assert guards trainable-only, B zero-prefix is floor-only (no backward), generation shape

    agent
    33dbd59
  18. gemma_decoder pilot: --checkpoint flag (trained-model source), textiser fixes (router-less guard, share clamp), requirements + transformers/huggingface_hub for Space

    agent
    9ac5f43
  19. gemma_decoder prototype: faithful live capture (hooks: router, mem_attn; ACT semantics), Latent+Resampler bridges (Gemma embed-scale aware, one-way detach boundary), grounded NL targets, run_pilot with controls A-G and frozen-Gemma gradient contract; 8/8 tests

    agent
    788ee34