agent
prototype report skeleton with all verified facts (results section awaits Space pilot)
ef1a99a |
Download experiments/gemma_decoder/REPORT_DRAFT_notes.md from cazyundee/training: direct link, hf CLI and curl.
- Browser
- Download file 2.78 kB
-
https://huggingface.co/spaces/cazyundee/training/resolve/main/experiments/gemma_decoder/REPORT_DRAFT_notes.md
- Command line
-
hf download hf://spaces/cazyundee/training/experiments/gemma_decoder/REPORT_DRAFT_notes.md
-
curl -L -o REPORT_DRAFT_notes.md https://huggingface.co/spaces/cazyundee/training/resolve/main/experiments/gemma_decoder/REPORT_DRAFT_notes.md
2.78 kB
| # gemma_decoder_prototype_report.md - skeleton (results filled after Space pilot) | |
| ## Verified so far (fill-in-ready facts) | |
| - capture_with_hooks: faithful mirror proven (aggregate==norm_out==unmodified h; n_steps equal; | |
| last traj state equal; value identical; logits+argmax reproduce EXACTLY; deterministic across | |
| instrumented runs). 8/8 tests tests_gemma.py. | |
| - ACT semantics: per-step active mask, stop/remainder weights, aggregate kept separate from raw | |
| per-step states; already-halted examples are captured as inactive, never described as computing. | |
| - Router [B,4] via Linear hook (softmax of hook output / router_temp), mem_attn [B,64,8] | |
| recomputed with identical math from actual mem_r query + live per-step memory. | |
| - One-way boundary: both bridges detach ALL inputs at entry; test 8 proved zero gradient into | |
| TinyChess params (originally 36 leaked before the boundary was added - caught by the test). | |
| - Gemma embed scale: bridges multiply output by sqrt(640) since inputs_embeds bypasses | |
| Gemma3TextScaledWordEmbedding scaling (verified in transformers 5.16.1 source). | |
| - gemma-3-270m(-it) differences: IT eos_token_id=[1,106] list, no chat_template; pilot handles | |
| both; PT default eos=1. | |
| - Frozen-Gemma contract: asserted EVERY training step (bridge grads nonzero; gemma grads zero). | |
| The assert fired correctly on an unfrozen mock (proof it enforces). | |
| - Held-out protocol: last 20% positions never trained on; heldout_ce measured per condition; | |
| generation on unseen position. | |
| - Conditions: A constant / B zero (floor-only, no backward) / C shuffled / D final-only / | |
| E full trajectory / G combined. F (metadata-only) deferred: needs descriptor-only bridge. | |
| - Pilot launch (Space): run_job('gemma_pilot_pt', 'experiments/gemma_decoder/run_pilot.py', | |
| '--model google/gemma-3-270m --hf-checkpoint test_web_exfil/latest.pt --positions 24 | |
| --iters 40 --out runs/gemma_pilot_pt.json'); same with -it. | |
| - Dataset: test_web_exfil (config=full, trained ~550+ iters): 12/24 distinct grounded texts. | |
| - Distinct-text count with trained model: 12/24 (target variety adequate for pilot scale). | |
| - Resource estimate: 270M fp32 ~1.1GB RAM + activations; 16 vCPU Space; Bridge lr 1e-3 Adam; | |
| expected ~2-5s/iter. | |
| - Readout plan (fill after pilot): | |
| * headline ordering: heldout_loss: E < D < B~C~A (channel works & generalises) | |
| * PT vs IT: same-condition comparisons at equal iters | |
| * EOS rate per condition/variant | |
| * generated-text claim check vs true_target on held-out items | |
| * decision rule for phase 2 (scaled dataset + Perceiver resampler + F-vs-G telemetry test) | |
| ## To fill after the pilot | |
| - results tables (loss curves, heldout, eos, samples) | |
| - resource usage measured on Space | |
| - failures/limitations observed | |
| - verdict on the brief's final question | |