File size: 3,758 Bytes
9f1076a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 | # READOUT_FIX — surface decode reads living field α
_Updated: 2026-09-17 12:32:30 PDT_
## Problem
Probe (`tools/probe_field_persistence.py`) on
`checkpoints/atom_native_chat_talk/atom_native.pt` showed:
| Metric | Before |
|--------|--------|
| Mean off-diag cosine **α** (field) | **~0.738** |
| Mean off-diag cosine **output_state** | ~0.738 |
| Mean off-diag cosine **surface logits** | **~0.998** |
| Generation | Identical collapse (`\ne ` / shared noise) |
The living toroidal field **did** differ across prompts, but
`AtomSurfaceHead` effectively ignored it: decode behaved like a last-token /
bias-dominated byte-LM. Shared decoder bias pinned logits so inter-prompt
cosine stayed ≈ 1.0 even after LayerNorm.
## Fix (readout only — no MERGE)
Architecture change in `src/atom_native.py`:
1. **`AtomSurfaceHead` field-first contract**
- Spectral features: `[mean(α) ‖ std(α) ‖ consolidation.persistence ‖ atom.r]`
- `field_to_state`: compress → `d_model` (init: identity on mean, 0.5·I on std)
- Gated mix with legacy mean-state: `σ(field_gate)·field + (1−σ)·legacy`
- **Bias-free field→logit skip** (`field_byte_skip` / `field_length_skip`) scaled by
`softplus(skip_gate)` so α differences reach logits even when `|b|` is large
- LayerNorm + legacy linears retained for Θ continuity
2. **`_advance` / `_output_state`**
- Every tick passes **full α**, always-on consolidation buffer, and current
`atom.r` into the surface (not only `mean(α)` / last packet state)
3. **Backward-compatible load**
- Old weights load by shape filter
- Missing field-readout keys → seeded `_init_field_readout()`, bias ×0.05,
`skip_gate ← 4.0` so migrated checkpoints separate logits **before** retrain
- Fresh from-scratch init uses softer `skip_gate=1.0` so CE training stays stable
No Transformer / attention / GPT-2 / QKV. Θ stays ~8MB class (~5.8M trainable
params at d_model=64). Stream train under `atom_native_stream_dialogue` was
**not** stopped or modified.
## Before / after probe
Checkpoint: `checkpoints/atom_native_chat_talk/atom_native.pt` (step 2.1M),
read-only eval after migration (no weight update required for separation).
| Metric | Before | After readout fix (migrate load) |
|--------|--------|----------------------------------|
| Mean off-diag cosine surface logits | **0.997991** | **~0.938** |
| Mean off-diag cosine α | 0.738060 | 0.738060 |
| Mean RMS after prompt | ~0.0064 | ~0.0064 |
| Mean RMS after 20 gen | ~0.0486 | ~0.0485 |
| Verdict | field differs, surface unread (byte-LM decode) | **field carrying structure (prompt-sensitive)** |
Sample gens (deterministic, role-primed) are **prompt-distinct** (no longer
identical collapse). Without a surface re-fit they remain embryonic/noisy.
### Optional tiny smoke fine-tune
`checkpoints/atom_native_readout_fix/` — 2k steps from a **copy** of chat_talk
(`corpus_train_chat.txt`), stream dir untouched. After 2k: α cosine fell to
~0.39 (more field separation) but surface logit cosine rose toward ~0.99 as the
skip/gate adapted under CE — longer surface-focused training still needed for
fluent decode. Migration-only numbers above are the readout-fix success bar.
## Tests
- `test/test_generation_smoke.py::FieldReadoutSensitivityTests`
- Synthetic distinct α → logit cosine < 0.95
- Migrated chat_talk ckpt → Bonjour vs Qui es-tu logit cosine < 0.95
## Not done (explicit)
- Full **MERGE** / dual clocks / adaptive spans — deferred
- No changes to the live stream trainer PID / `atom_native_stream_dialogue`
## How to re-probe
```bash
PYTHONPATH=. .venv/bin/python tools/probe_field_persistence.py \
--checkpoint checkpoints/atom_native_chat_talk/atom_native.pt
```
|