training / experiments /gemma_decoder /REPORT_DRAFT_notes.md
agent
prototype report skeleton with all verified facts (results section awaits Space pilot)
ef1a99a
|
Raw History Blame Contribute Delete
2.78 kB
# gemma_decoder_prototype_report.md - skeleton (results filled after Space pilot)
## Verified so far (fill-in-ready facts)
- capture_with_hooks: faithful mirror proven (aggregate==norm_out==unmodified h; n_steps equal;
last traj state equal; value identical; logits+argmax reproduce EXACTLY; deterministic across
instrumented runs). 8/8 tests tests_gemma.py.
- ACT semantics: per-step active mask, stop/remainder weights, aggregate kept separate from raw
per-step states; already-halted examples are captured as inactive, never described as computing.
- Router [B,4] via Linear hook (softmax of hook output / router_temp), mem_attn [B,64,8]
recomputed with identical math from actual mem_r query + live per-step memory.
- One-way boundary: both bridges detach ALL inputs at entry; test 8 proved zero gradient into
TinyChess params (originally 36 leaked before the boundary was added - caught by the test).
- Gemma embed scale: bridges multiply output by sqrt(640) since inputs_embeds bypasses
Gemma3TextScaledWordEmbedding scaling (verified in transformers 5.16.1 source).
- gemma-3-270m(-it) differences: IT eos_token_id=[1,106] list, no chat_template; pilot handles
both; PT default eos=1.
- Frozen-Gemma contract: asserted EVERY training step (bridge grads nonzero; gemma grads zero).
The assert fired correctly on an unfrozen mock (proof it enforces).
- Held-out protocol: last 20% positions never trained on; heldout_ce measured per condition;
generation on unseen position.
- Conditions: A constant / B zero (floor-only, no backward) / C shuffled / D final-only /
E full trajectory / G combined. F (metadata-only) deferred: needs descriptor-only bridge.
- Pilot launch (Space): run_job('gemma_pilot_pt', 'experiments/gemma_decoder/run_pilot.py',
'--model google/gemma-3-270m --hf-checkpoint test_web_exfil/latest.pt --positions 24
--iters 40 --out runs/gemma_pilot_pt.json'); same with -it.
- Dataset: test_web_exfil (config=full, trained ~550+ iters): 12/24 distinct grounded texts.
- Distinct-text count with trained model: 12/24 (target variety adequate for pilot scale).
- Resource estimate: 270M fp32 ~1.1GB RAM + activations; 16 vCPU Space; Bridge lr 1e-3 Adam;
expected ~2-5s/iter.
- Readout plan (fill after pilot):
* headline ordering: heldout_loss: E < D < B~C~A (channel works & generalises)
* PT vs IT: same-condition comparisons at equal iters
* EOS rate per condition/variant
* generated-text claim check vs true_target on held-out items
* decision rule for phase 2 (scaled dataset + Perceiver resampler + F-vs-G telemetry test)
## To fill after the pilot
- results tables (loss curves, heldout, eos, samples)
- resource usage measured on Space
- failures/limitations observed
- verdict on the brief's final question