training / experiments /gemma_decoder /REPORT_DRAFT_notes.md
agent
prototype report skeleton with all verified facts (results section awaits Space pilot)
ef1a99a
|
Raw History Blame Contribute Delete
2.78 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade

gemma_decoder_prototype_report.md - skeleton (results filled after Space pilot)

Verified so far (fill-in-ready facts)

  • capture_with_hooks: faithful mirror proven (aggregate==norm_out==unmodified h; n_steps equal; last traj state equal; value identical; logits+argmax reproduce EXACTLY; deterministic across instrumented runs). 8/8 tests tests_gemma.py.
  • ACT semantics: per-step active mask, stop/remainder weights, aggregate kept separate from raw per-step states; already-halted examples are captured as inactive, never described as computing.
  • Router [B,4] via Linear hook (softmax of hook output / router_temp), mem_attn [B,64,8] recomputed with identical math from actual mem_r query + live per-step memory.
  • One-way boundary: both bridges detach ALL inputs at entry; test 8 proved zero gradient into TinyChess params (originally 36 leaked before the boundary was added - caught by the test).
  • Gemma embed scale: bridges multiply output by sqrt(640) since inputs_embeds bypasses Gemma3TextScaledWordEmbedding scaling (verified in transformers 5.16.1 source).
  • gemma-3-270m(-it) differences: IT eos_token_id=[1,106] list, no chat_template; pilot handles both; PT default eos=1.
  • Frozen-Gemma contract: asserted EVERY training step (bridge grads nonzero; gemma grads zero). The assert fired correctly on an unfrozen mock (proof it enforces).
  • Held-out protocol: last 20% positions never trained on; heldout_ce measured per condition; generation on unseen position.
  • Conditions: A constant / B zero (floor-only, no backward) / C shuffled / D final-only / E full trajectory / G combined. F (metadata-only) deferred: needs descriptor-only bridge.
  • Pilot launch (Space): run_job('gemma_pilot_pt', 'experiments/gemma_decoder/run_pilot.py', '--model google/gemma-3-270m --hf-checkpoint test_web_exfil/latest.pt --positions 24 --iters 40 --out runs/gemma_pilot_pt.json'); same with -it.
  • Dataset: test_web_exfil (config=full, trained ~550+ iters): 12/24 distinct grounded texts.
  • Distinct-text count with trained model: 12/24 (target variety adequate for pilot scale).
  • Resource estimate: 270M fp32 ~1.1GB RAM + activations; 16 vCPU Space; Bridge lr 1e-3 Adam; expected ~2-5s/iter.
  • Readout plan (fill after pilot):
    • headline ordering: heldout_loss: E < D < BCA (channel works & generalises)
    • PT vs IT: same-condition comparisons at equal iters
    • EOS rate per condition/variant
    • generated-text claim check vs true_target on held-out items
    • decision rule for phase 2 (scaled dataset + Perceiver resampler + F-vs-G telemetry test)

To fill after the pilot

  • results tables (loss curves, heldout, eos, samples)
  • resource usage measured on Space
  • failures/limitations observed
  • verdict on the brief's final question