# gemma_decoder_prototype_report.md - skeleton (results filled after Space pilot) ## Verified so far (fill-in-ready facts) - capture_with_hooks: faithful mirror proven (aggregate==norm_out==unmodified h; n_steps equal; last traj state equal; value identical; logits+argmax reproduce EXACTLY; deterministic across instrumented runs). 8/8 tests tests_gemma.py. - ACT semantics: per-step active mask, stop/remainder weights, aggregate kept separate from raw per-step states; already-halted examples are captured as inactive, never described as computing. - Router [B,4] via Linear hook (softmax of hook output / router_temp), mem_attn [B,64,8] recomputed with identical math from actual mem_r query + live per-step memory. - One-way boundary: both bridges detach ALL inputs at entry; test 8 proved zero gradient into TinyChess params (originally 36 leaked before the boundary was added - caught by the test). - Gemma embed scale: bridges multiply output by sqrt(640) since inputs_embeds bypasses Gemma3TextScaledWordEmbedding scaling (verified in transformers 5.16.1 source). - gemma-3-270m(-it) differences: IT eos_token_id=[1,106] list, no chat_template; pilot handles both; PT default eos=1. - Frozen-Gemma contract: asserted EVERY training step (bridge grads nonzero; gemma grads zero). The assert fired correctly on an unfrozen mock (proof it enforces). - Held-out protocol: last 20% positions never trained on; heldout_ce measured per condition; generation on unseen position. - Conditions: A constant / B zero (floor-only, no backward) / C shuffled / D final-only / E full trajectory / G combined. F (metadata-only) deferred: needs descriptor-only bridge. - Pilot launch (Space): run_job('gemma_pilot_pt', 'experiments/gemma_decoder/run_pilot.py', '--model google/gemma-3-270m --hf-checkpoint test_web_exfil/latest.pt --positions 24 --iters 40 --out runs/gemma_pilot_pt.json'); same with -it. - Dataset: test_web_exfil (config=full, trained ~550+ iters): 12/24 distinct grounded texts. - Distinct-text count with trained model: 12/24 (target variety adequate for pilot scale). - Resource estimate: 270M fp32 ~1.1GB RAM + activations; 16 vCPU Space; Bridge lr 1e-3 Adam; expected ~2-5s/iter. - Readout plan (fill after pilot): * headline ordering: heldout_loss: E < D < B~C~A (channel works & generalises) * PT vs IT: same-condition comparisons at equal iters * EOS rate per condition/variant * generated-text claim check vs true_target on held-out items * decision rule for phase 2 (scaled dataset + Perceiver resampler + F-vs-G telemetry test) ## To fill after the pilot - results tables (loss curves, heldout, eos, samples) - resource usage measured on Space - failures/limitations observed - verdict on the brief's final question