File size: 2,775 Bytes
ef1a99a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# gemma_decoder_prototype_report.md - skeleton (results filled after Space pilot)

## Verified so far (fill-in-ready facts)
- capture_with_hooks: faithful mirror proven (aggregate==norm_out==unmodified h; n_steps equal;
  last traj state equal; value identical; logits+argmax reproduce EXACTLY; deterministic across
  instrumented runs). 8/8 tests tests_gemma.py.
- ACT semantics: per-step active mask, stop/remainder weights, aggregate kept separate from raw
  per-step states; already-halted examples are captured as inactive, never described as computing.
- Router [B,4] via Linear hook (softmax of hook output / router_temp), mem_attn [B,64,8]
  recomputed with identical math from actual mem_r query + live per-step memory.
- One-way boundary: both bridges detach ALL inputs at entry; test 8 proved zero gradient into
  TinyChess params (originally 36 leaked before the boundary was added - caught by the test).
- Gemma embed scale: bridges multiply output by sqrt(640) since inputs_embeds bypasses
  Gemma3TextScaledWordEmbedding scaling (verified in transformers 5.16.1 source).
- gemma-3-270m(-it) differences: IT eos_token_id=[1,106] list, no chat_template; pilot handles
  both; PT default eos=1.
- Frozen-Gemma contract: asserted EVERY training step (bridge grads nonzero; gemma grads zero).
  The assert fired correctly on an unfrozen mock (proof it enforces).
- Held-out protocol: last 20% positions never trained on; heldout_ce measured per condition;
  generation on unseen position.
- Conditions: A constant / B zero (floor-only, no backward) / C shuffled / D final-only /
  E full trajectory / G combined. F (metadata-only) deferred: needs descriptor-only bridge.
- Pilot launch (Space): run_job('gemma_pilot_pt', 'experiments/gemma_decoder/run_pilot.py',
  '--model google/gemma-3-270m --hf-checkpoint test_web_exfil/latest.pt --positions 24
  --iters 40 --out runs/gemma_pilot_pt.json'); same with -it.
- Dataset: test_web_exfil (config=full, trained ~550+ iters): 12/24 distinct grounded texts.
- Distinct-text count with trained model: 12/24 (target variety adequate for pilot scale).
- Resource estimate: 270M fp32 ~1.1GB RAM + activations; 16 vCPU Space; Bridge lr 1e-3 Adam;
  expected ~2-5s/iter.
- Readout plan (fill after pilot):
  * headline ordering: heldout_loss: E < D < B~C~A (channel works & generalises)
  * PT vs IT: same-condition comparisons at equal iters
  * EOS rate per condition/variant
  * generated-text claim check vs true_target on held-out items
  * decision rule for phase 2 (scaled dataset + Perceiver resampler + F-vs-G telemetry test)

## To fill after the pilot
- results tables (loss curves, heldout, eos, samples)
- resource usage measured on Space
- failures/limitations observed
- verdict on the brief's final question