h4_program.md

H4 Polytopic Attention — Autonomous Research Protocol

You are an autonomous research agent optimizing a hybrid transformer that combines frozen H4 geometric attention with trainable adapter layers. Your goal is to minimize val_bpb (bits per byte on held-out text) within a fixed compute budget.

The Architecture

The system has two parts:

1. Frozen geometric backbone (DO NOT MODIFY):

   • 600-cell vertices (120 × 4) — the H4 polytope
   • H4 simple roots (4 × 4) — Coxeter reflection hyperplanes
   • E8→H4 projection matrix (4 × 8) — golden ratio eigenvalues
   • ChamberTree structure — O(log t) spatial partitioning of S³
   • E8LatticeIndex — Voronoi cell memory backend

2. Trainable adapters (YOU MODIFY THESE):

   • W_q_proj, W_k_proj, W_v_proj — input projections to H4/value space
   • W_nudge — per-head 4×4 rotation of query direction
   • chamber_bonus — per-head 16-dim learnable attention bias per chamber
   • W_out — output projection
   • FFN layers (fully trainable)
   • Token/positional embeddings
   • LM head

What You Can Change

In python/train_cpu.py, you may modify:

• Hyperparameters: learning rate, batch size, sequence length, warmup, grad clip, weight decay
• Architecture of trainable layers: d_model, n_heads, n_layers, d_value, d_ffn, top_k, dropout
• Optimizer setup: scheduler, betas, epsilon
• Training loop: loss weighting, gradient accumulation, evaluation strategy
• Adapter architecture: number of nudge parameters, chamber bonus structure, FFN design

What You CANNOT Change

• python/h4_polytopic_attention.py — frozen geometry
• python/utils/phi_positional.py — golden-angle encoding (you can change how it's used, not the encoding itself)
• python/utils/chamber_index.py — chamber lookup bridge
• The H4 vertices, simple roots, E8 projection, ChamberTree structure
• The fundamental constraint that Q and K live on S³ (unit 4-sphere)

The Loop

while forever:
    1. Read current train_cpu.py and results.tsv
    2. Form a hypothesis about what change will improve val_bpb
    3. Modify train_cpu.py (the ONLY file you modify)
    4. Run: cd python && python train_cpu.py
    5. Parse the "---" summary block from stdout
    6. If val_bpb improved or the experiment is informative:
         - git add python/train_cpu.py && git commit
         - Append to results.tsv: commit<TAB>val_bpb<TAB>val_loss<TAB>chamber_entropy<TAB>status<TAB>description
         - status = "keep"
    7. If val_bpb did not improve:
         - git checkout python/train_cpu.py  (discard changes)
         - Append to results.tsv with status = "discard"
    8. If the run crashed:
         - Fix the crash, do NOT count it as an experiment
         - Append to results.tsv with status = "crash"
    9. Think about what you learned. Update your mental model of:
         - Which hyperparameters matter most
         - Whether the geometry is being utilized (check chamber_entropy)
         - Whether W_nudge is learning meaningful directions (check geo_alignment)
         - What the loss landscape looks like
   10. Repeat

Time Budget

• 2 minutes per experiment on CPU (TIME_BUDGET = 120 in train_cpu.py)
• This allows ~24 experiments in an overnight run
• If an experiment takes longer than 3 minutes, it has a bug — fix it

Metrics

Primary: val_bpb — lower is better. Bits per byte on held-out character-level text.

Diagnostic (track but don't optimize directly):

• chamber_entropy — Shannon entropy of chamber utilization. High = using full geometry. Low = collapsed to few chambers.
• avg_nudge_rank — effective rank of W_nudge deviation from identity. High = rank-1 (good, focused direction). Low = diffuse.
• avg_geo_alignment — max dot product of W_nudge dominant direction with 600-cell vertices. >0.9 = strongly aligning with geometry.
• num_steps — training throughput indicator.

results.tsv Format

commit	val_bpb	val_loss	chamber_entropy	status	description
a1b2c3d	2.345678	1.625000	2.1234	keep	baseline: d_model=64, 2 layers, lr=3e-4
e4f5g6h	2.298765	1.592500	2.3456	keep	increased d_ffn from 256 to 512

Tab-separated. Short 7-char commit hashes. Do not commit results.tsv to git.

Strategy Hints

Based on the Fibonacci proof-of-concept (26 trainable params on frozen H4 backbone, φ gap 0.025→0.001):

1. The geometry provides strong inductive bias. W_nudge naturally converges to rank-1, aligning with 600-cell vertices. Don't fight this — let the nudge stay small.

2. Chamber utilization matters. If entropy is low, the model is only using a few chambers. Try: different init for W_nudge, larger top_k, or adding noise to queries during training.

3. Start with the simplest change. Learning rate and d_model matter most. Don't change 5 things at once.

4. The tree is for long sequences. For seq_len ≤ 256, full attention is faster than tree lookup (Python overhead). Set use_tree = MAX_SEQ_LEN > 256.

5. Watch for mode collapse. If all heads learn the same nudge direction, the model wastes capacity. Consider adding a diversity loss or different initializations per head.

6. The golden ratio is not arbitrary. φ⁻¹ is the most irrational number — golden-angle positions are maximally separated. The positional encoding exploits this. Don't replace it with sinusoidal.

Data

Currently using character-level text (Shakespeare if available, synthetic Fibonacci-structured text otherwise). To add real data:

1. Download TinyStories: wget https://huggingface.co/datasets/karpathy/tinystories-gpt4-clean/resolve/main/TinyStories_all_data.tar.gz
2. Extract to data/
3. Update load_text_data() in train_cpu.py

For now, the synthetic data is fine for proving the architecture works. Switch to real data once val_bpb is decreasing consistently.

Phase 6: BitLinear Experiments

Toggle ternary mode with USE_BITLINEAR = True in hyperparameters section.

Experiment Ideas

1. Baseline comparison: Same architecture, float vs ternary. Measure val_bpb gap. BitNet Reloaded shows <0.1 bpb gap at 100K+ params with 2x hidden size.

2. Hidden size scaling: If ternary hurts val_bpb, try doubling d_model (64->128) while keeping ternary. The 2x scaling law from BitNet Reloaded predicts this recovers the gap.

3. Selective ternary: Make only the large projections (q/k/v/out) ternary, keep the 4x4 nudge layers in float. The nudge is where geometric alignment happens --- it might need float precision.

4. Zero ratio tracking: Monitor the zero percentage in ternary nudge layers across training. High zero% means the head learned feature selection (ignoring some Coxeter directions). Plot zero% vs chamber_entropy --- they should be inversely correlated.

5. Chamber preservation sweep: Run ternary_diagnostics.chamber_preservation after each experiment. If preservation drops below 85%, the ternary quantization is too aggressive for that architecture.

Additional Metrics

• ternary: yes/no --- whether BitLinear is active
• chamber_preserve: mean chamber preservation rate (float vs ternary)
• mean_zero_pct: mean zero% across BitLinear layers
• compression: weight compression ratio vs float32
• model_size_kb: total model size in KB

Keep/Discard Rules for Ternary

Same as float rules, plus:

• A ternary experiment that matches float val_bpb within 0.05 is a WIN (same quality, ~20x smaller weights)
• A ternary experiment with >5% chamber preservation drop is SUSPECT (check if val_bpb actually suffered --- preservation can drop without hurting quality if the dropped chambers weren't being used)