Create constellation.md

constellation.md

@@ -0,0 +1,468 @@
+# Constellation Forms Catalogue
+## GeoLIP Architecture Reference — March 2026
+
+Sources:
+- geometric-memory-ft1 (GM1)
+- geometric-memory-ft2 (GM2)
+- geometric-memory-ft3 (GM3)
+- procrustes-vit-hypersphere-ft1 (PVH)
+- constellation-diffusion-bottleneck (CDB)
+- Session benchmarks (SB)
+
+---
+
+## Universal Constants
+
+| Constant | Value | Source |
+|----------|-------|--------|
+| Pentachoron CV attractor | 0.20–0.23 | Geometry of S^15 itself (CDB §3) |
+| Binding/separation boundary | 0.29154 radians | 5+ architectures (CDB §11) |
+| Effective geometric dimension | ~16 | All trained models (CDB §3.3) |
+| CV precision invariance | fp64 through 1-bit | CDB §3.2 |
+
+## Universal Rules
+
+| Rule | Source |
+|------|--------|
+| SquaredReLU in all constellation paths, never GELU | SB activation tests |
+| Patchwork: Linear(tri, tri×2) → SquaredReLU → LN → Linear(tri×2, dim) | SB proven |
+| Gate init: -3.0 (sigmoid ≈ 0.047) | SB proven |
+| SLERP: only acos in fp32 (16KB tensor), everything else stays in compute dtype | SB fp32 fix |
+| Adam, NO weight decay — geometry IS regularization | GM3 §2.4, PVH §12 |
+| InfoNCE is the alignment FORCE. Procrustes is the REGULARIZER. | GM1 §4.1 |
+| CV loss on the BOTTLENECK, not the output | GM1 §4.2 |
+| CV loss weight: micro (0.001 or below) | GM3 §2.2 |
+| Procrustes calibration is non-negotiable for anchor init | PVH §5.1 |
+| Anchor dropout (30%) prevents collapse | PVH §5.2 |
+
+---
+
+## Form 1: GeoLIP Core (Classification)
+
+**Source:** CDB §2
+
+**Purpose:** Minimal image classification pipeline. Proves the constellation works as a primary representation layer.
+
+**Pipeline:**
+```
+Input image
+  → Conv encoder (builds channel depth: 3→64→128→256)
+  → AdaptiveAvgPool → Linear(encoder_out, D) → L2 normalize to S^(d-1)
+  → Triangulate against N anchors at 3 SLERP phases → tri_dim profile
+  → Patchwork MLP reads triangulation
+  → Classifier head → logits
+```
+
+**Key properties:**
+- Every embedding on the unit sphere BEFORE the constellation sees it
+- The conv encoder builds channel depth — constellation operates on channel dimension
+- One global vector per image, not a sequence
+- No attention anywhere
+
+**Proven results:** 91.5% CIFAR-10, 1.6M params, CV=0.2045, 62/64 active anchors
+
+**Loss:** CE + CV on embeddings
+
+**When to use:** Single-input classification where the input can be reduced to one D-dimensional vector on S^(d-1).
+
+---
+
+## Form 2: Expert Soup (Multi-Expert Fusion)
+
+**Source:** PVH §1, §4
+
+**Purpose:** Fuse multiple frozen pretrained experts into a shared geometric representation on S^(d-1).
+
+**Pipeline:**
+```
+Input image
+  → N frozen expert encoders (CLIP, DINOv2, SigLIP, etc.) → N × 768-d
+  → GPA alignment at 768-d (iterative Procrustes to mutual mean)
+  → PCA to D_ANCHOR dims
+  → Per-expert Procrustes-initialized projectors (768 → D_ANCHOR)
+  → L2 normalize → shared constellation on S^(D_ANCHOR-1)
+  → Triangulate: each expert through its own Procrustes rotation
+  → Patchwork reads fused triangulation
+  → Classifier
+```
+
+**Key properties:**
+- Experts are FROZEN — never modified
+- Procrustes initialization essential (without: 1/256 active anchors, collapsed)
+- Anchor dropout (30%) → 508/512 active anchors
+- Effective dimensionality matches task complexity (76.9 for COCO's 80 classes)
+- Pipeline is almost entirely linear: 7 linear ops + 2 nonlinearities (GELU in patchwork + classifier)
+- Weight decay explicitly avoided
+
+**Proven results:** mAP=0.84 ceiling (data-limited), perfect hypersphere verified (1000/1000 positive volumes), 508/512 active anchors
+
+**Loss:** InfoNCE(fused, consensus) + MSE + BCE + Procrustes_align + CV + anchor_spread
+
+**Optimizer:** Adam lr=1e-3, NO weight decay
+
+**When to use:** Combining multiple pretrained encoders into a shared geometric space for downstream tasks.
+
+---
+
+## Form 3: Geometric Memory / Anchor Bank (Context Extension)
+
+**Source:** GM1 §2, GM2 §2
+
+**Purpose:** Extend a frozen encoder's context window by accumulating segment-level geometric addresses in a memory bank.
+
+**Pipeline:**
+```
+Long document (N tokens, N >> encoder context)
+  → Split into overlapping segments (sized to encoder window)
+  → For each segment:
+      → Frozen encoder forward → hidden states at multiple layers
+      → Multi-layer fusion (learned weighted sum)
+      → Memory tokens cross-attend to fused hidden states
+      → Depth-profile compressor: per-layer CLS → single anchor (L2-normalized)
+      → Anchor stored in geometric memory bank
+      → GRU gate updates rolling memory state
+  → Final output: encoder-compatible embedding
+```
+
+**Key properties:**
+- Frozen encoder, trainable memory wrapper
+- Depth-profile anchors encode HOW the encoder processed (not just WHAT)
+- CV loss on the BANK ANCHORS specifically — the bottleneck between segments
+- Without CV on bank: projector shortcut collapse (m_acc plateaus at 0.670)
+- With CV on bank: m_acc reaches 0.945
+- Segment size must produce 5+ anchors for CV computation (pentachoron needs 5 points)
+- Convergence order: CV locks first → m_acc climbs → s_cos climbs last
+
+**Proven results:**
+- GEOLIP-BERT-8192: m_acc=0.927, CV=0.200 (512→8192 context)
+- GEOLIP-CLIP-ctx576: m_acc=0.945, CV=0.162 (77→576 context)
+
+**Loss:** InfoNCE(student, teacher) + Procrustes_SVD + |CV(bank_anchors) - 0.20|
+
+**When to use:** Extending frozen encoder context windows while preserving embedding space compatibility.
+
+---
+
+## Form 4: Sequence Reconstructor (Per-Position Output)
+
+**Source:** GM2 §2
+
+**Purpose:** Produce full per-position output sequences from memory state for diffusion cross-attention.
+
+**Pipeline:**
+```
+Memory state (from Form 3 bank accumulation)
+  → Context = cat(memory_tokens, bank_anchors, content_tokens)
+  → 77 learned query tokens + positional encoding
+      → Cross-attend to context (2 layers)
+      → Self-attend among 77 output positions (2 layers)
+  → Output: (B, 77, 768) — in frozen encoder's native distribution
+```
+
+**Key properties:**
+- Must produce output in the distribution the UNet was trained on
+- Training target: frozen encoder's own output on same caption (truncated to 77 tokens)
+- Two teachers: ModernBERT teaches what to remember, CLIP teaches how to say it
+- Two-phase training works for CLIP-L but NOT universally
+- Rule: if you need per-position output, train the per-position consumer from the start
+- Memory format shaped by gradient loudness, not architectural capacity
+
+**Proven results:**
+- CLIP-L s_cos=0.734, tulips appeared in SD 1.5 from elements past token 77
+- Meridian (bigG): s_cos=0.425 (limited by 1280→1024 dimensional mismatch)
+
+**Loss:** MSE(normalize(pred), normalize(target)) + cosine_similarity + InfoNCE(pooled)
+
+**When to use:** When downstream consumer needs per-position sequences (diffusion cross-attention, token-level tasks).
+
+---
+
+## Form 5: Constellation Relay (Per-Token Geometric Layer)
+
+**Source:** CDB §4, SB
+
+**Purpose:** Replace attention as a per-token processing layer. O(S) complexity. Preserves geometry at depth.
+
+**Pipeline:**
+```
+Input (B, S, D) or (B, D)
+  → LayerNorm
+  → Chunk D into P patches of patch_dim (e.g., 16 × 16d = 256d)
+  → L2 normalize each patch to S^(d-1)
+  → Triangulate against anchors at 3 SLERP phases → tri_dim profile
+  → Patchwork MLP reads triangulation
+  → Gated residual (gate init -3.0)
+  → Output = residual + gate * patchwork_out
+```
+
+**Key properties:**
+- Per-token, no cross-token interaction
+- O(S) time and memory — no S² term
+- Preserves 99.4% cosine similarity to input at depth 16 (vs 7.4% for attention)
+- 3.4× fewer parameters than vanilla attention
+- Geometric preservation is sequence-length invariant (identical from S=64 through S=131072)
+- Throughput crossover vs attention at S≈32K; 8.4× faster at S=131K
+- SquaredReLU wins: better anchor diversity (7.1 vs 4.6), better equivariance, 0.9999 reconstruction
+
+**Proven results:** cos_to_orig=0.994 at depth 16, 8.4× faster than attention at S=131K
+
+**When to use:** Processing token sequences where geometric preservation matters more than cross-token mixing. Stackable. The per-token processing layer.
+
+---
+
+## Form 6: Cantor Constellation Router (Cross-Token Routing)
+
+**Source:** SB (cantor_constellation_relay.py)
+
+**Purpose:** O(S) cross-token routing through the constellation's own anchor hierarchy. Replaces attention's cross-token role.
+
+**Pipeline:**
+```
+Input tokens (B, S, D) + triangulation profiles (B, S, tri_dim) from relay
+  → Compute soft routing weights from phase-0 triangulation distances
+  → For each level l in binary anchor tree (16→8→4→2→1):
+      → Merge anchor weights into group weights at level l
+      → Weighted scatter: tokens → group summaries (bmm)
+      → Transform: per-level MLP(dim→dim×2→dim) + LN
+      → Weighted gather: group summaries → token updates (bmm)
+      → Gated residual at each level
+  → Output: tokens with cross-token information
+```
+
+**Key properties:**
+- O(S × n_levels × D) where n_levels = log2(A) + 1 = 5 for A=16
+- No S² term anywhere — not in compute, not in memory
+- Triangulation from the per-token relay IS the routing key (zero redundant computation)
+- Binary tree over anchors defines hierarchy (16→8→4→2→1 groups)
+- At each level: scatter → transform → gather
+- Cantor routing holds at distance BETTER than attention (2× stronger at S=4096)
+- The router is a geometric REGULARIZER: cos_orig=0.9818 at 8 layers vs relay alone 0.6533
+- Geometry IMPROVES with more tokens (0.982→0.986 as S increases)
+
+**Proven results:** 97.0% cross-token task acc, 0.986 cos preservation at 131K tokens, 5.2× faster than attention at 131K
+
+**When to use:** Combined with Form 5 relay as a complete O(S) transformer layer replacement (ConstellationCantorRelay).
+
+---
+
+## Form 7: Diffusion Bottleneck / Geometric Lookup Table
+
+**Source:** CDB §7–9
+
+**Purpose:** The constellation as the sole information bottleneck of a diffusion model. NOT an autoencoder.
+
+**Pipeline:**
+```
+Encoder features (256×8×8 = 16384-d)
+  → Linear(16384, 256) → L2 normalize to S^15
+  → Reshape (B, 16, 16) → per-patch S^15 normalization
+  → Triangulate: 16 patches × 16 anchors × 3 phases = 768 dims
+  → Concat(768 tri dims, conditioning dims)
+  → Patchwork MLP → Linear(hidden, 16384) → reshape → decoder
+```
+
+**Key properties:**
+- Compression ratio: 16384 → 768 = 21.3×
+- cos_sim ≈ 0 to input — the bottleneck does NOT reconstruct
+- It's a geometric LOOKUP TABLE: triangulation profile is an address, patchwork generates from that address
+- Works for flow matching because training signal is velocity prediction, not reconstruction
+- Skip bypass experiment: given 268M linear bypass, model routed 88% through 768 constellation dims
+- Constellation-only cos_sim=0.945 to full model; skip-only cos_sim=0.598
+- The constellation provides a representational ADVANTAGE over unconstrained capacity
+
+**Proven results:** Loss 0.1749 (beat 268M skip at 0.1757), 46% anchor convergence to 0.29154 in GLFM
+
+**Loss:** Flow matching velocity loss (MSE on predicted vs target velocity)
+
+**When to use:** Diffusion model bottleneck where geometric addressing replaces reconstruction.
+
+---
+
+## Form 8: Geometric Lookup Flow Matching (GLFM)
+
+**Source:** CDB §10
+
+**Purpose:** Formalized three-stage flow matching variant where velocity prediction is driven by geometric address lookup.
+
+**Pipeline:**
+```
+Stage 1 — Geometric Addressing:
+  Encoder output → project to S^15 at two scales:
+    Coarse: global avg pool → 256d → L2 norm → triangulate (768d)
+    Fine: per-spatial → 256d → L2 norm → triangulate → aggregate (768d)
+  Total address: 1536 dims of angular measurements
+
+Stage 2 — Address Conditioning:
+  Geometric address + sinusoidal timestep + class embed + noise-level bins
+  → Fused projection to generator input dim
+
+Stage 3 — Velocity Generation:
+  Deep residual MLP generates velocity features from conditioned address
+  4 residual blocks width 1024 → 16384-d spatial features → decoder
+```
+
+**Key properties:**
+- Explicit separation of addressing, conditioning, and generation
+- Multi-scale collapse observed: coarse↔fine cos=0.933 (needs pre-differentiated features like DINOv2)
+- 46% of anchors converged within ±0.05 of 0.29154 binding constant
+- 59% of anchors crossed binding boundary into task-specific territory
+
+**Proven results:** Loss 0.1754, accelerated drift convergence vs pure bottleneck
+
+**When to use:** Flow matching diffusion where you want explicit geometric addressing.
+
+---
+
+## Form 9: From-Scratch Encoder (Pixel → Consensus)
+
+**Source:** PVH §4.2
+
+**Purpose:** Train a ViT from random initialization to reproduce the expert soup consensus embedding from raw pixels.
+
+**Pipeline:**
+```
+Raw pixels
+  → From-scratch ViT (no pretrained weights)
+  → Project to D_ANCHOR dims → L2 normalize
+  → Train against frozen soup consensus as differentiable teacher
+```
+
+**Key properties:**
+- The soup is the teacher — it provides the target embedding for each image
+- Gradient bottleneck: all gradient flows through D_ANCHOR-dimensional output
+- With 77M params and 128-d output: gradient density = 1.6×10⁻⁶ per param
+- Expansion warm-start works: 384-d→1024-d by padding, recovers in 5 epochs
+
+**Proven results:** 1024-d ViT reached cos=0.663, mAP=0.500 (limited by gradient bottleneck and 118K COCO)
+
+**Loss:** Same as soup training + geometric autograd
+
+**When to use:** When you need a single encoder that reproduces multi-expert consensus from raw input.
+
+---
+
+## Form 10: Dual-Teacher Consensus Distillation
+
+**Source:** GM3 §4
+
+**Purpose:** Two independently-trained models → GPA consensus → distill into student that exceeds both.
+
+**Pipeline:**
+```
+Teacher A (any config) + Teacher B (any config)
+  → Extract embeddings on shared data
+  → GPA-align iteratively until δ < 1e-8
+  → Consensus = L2_normalize(mean_shape)
+  → Student initializes anchors from k-means on consensus
+  → Train with: CE + InfoNCE(emb, consensus) + MSE(emb, consensus) + micro CV
+  → Geometric autograd: tang=0.01, sep=1.0
+```
+
+**Key properties:**
+- Student exceeds BOTH teachers (0.761 vs 0.699/0.649)
+- Student still ACCELERATING at epoch 30 (resonant dynamics)
+- Consensus is the geometric truth — what both agree on after removing rotational frames
+- Robust to catastrophic models: a 25.5% accuracy parent still contributed useful signal
+- Diverse parent selection beats top-N selection
+
+**Proven results:** Student 0.761 from parents averaging 0.664; still accelerating at E30
+
+**When to use:** When you have 2+ trained models and want a superior student.
+
+---
+
+## Form 11: Multi-Generational Geometric Evolution
+
+**Source:** GM3 §5
+
+**Purpose:** Iterated consensus distillation across generations with data diversity.
+
+**Pipeline:**
+```
+Gen 0: N founders trained independently → GPA → consensus anchors
+Gen 1: M offspring from Gen 0 consensus + new founder (immigration)
+Gen 2+: Previous gen offspring + founder → GPA → consensus → next gen
+Each generation trains on differently-perturbed data
+```
+
+**Key properties:**
+- Monotonically improving across generations
+- Each generation inherits consensus-derived anchor coordinates
+- Fresh founders each generation prevent convergence collapse (gene flow)
+- Robust: catastrophic models don't poison the lineage
+- Diverse data across generations captures INVARIANT structure
+- CV converges toward 0.2 naturally across generations
+
+**Proven results:** Gen 0 mean=0.664 → Gen 4 best=0.775; FUSE_distilled=0.830
+
+**When to use:** When you want to compound geometric knowledge across training runs.
+
+---
+
+## Form 12: Geometric Autograd (Optimizer)
+
+**Source:** GM3 §2
+
+**Purpose:** Gradient filtering that replaces weight decay with manifold-aware optimization.
+
+**Components:**
+```
+Embedding backward:
+  → Decompose gradient into tangential + radial relative to S^(d-1)
+  → Pass tangential fully, attenuate radial by (1 - tang_strength)
+  → If gradient moves toward nearest anchor: attenuate by sep_strength
+
+Anchor backward:
+  → Project gradient tangential to hypersphere at anchor position
+  → Scale by drift_strength
+
+Forward losses (all differentiable):
+  → CV: |CV(pentachoron volumes) - 0.2| × 0.001
+  → Spread: anchor cos² off-diagonal mean × 1e-3
+  → Ortho: gram off-diagonal → 0 × 1e-3
+  → Entropy: -Σ p·log(p) × 1e-4
+  → Cluster var: -var(per-anchor mean cosine) × 1e-4
+```
+
+**Key properties:**
+- Adam + geometric autograd > AdamW consistently
+- Weight decay destroys the geometric harmonic the autograd creates
+- tang=0.01, sep=1.0 proven optimal
+- CV loss MUST be forward loss, never backward injection
+- Enables resonant dynamics: constructive interference compounds across epochs
+
+**When to use:** Training any constellation-based model. The geometry IS the regularization.
+
+---
+
+## Composition Map
+
+| Task | Primary Form | Supporting Forms |
+|------|-------------|-----------------|
+| Image classification (single image) | Form 1 (Core) | Form 12 (Autograd) |
+| Multi-expert fusion | Form 2 (Soup) | Form 12 |
+| Context extension | Form 3 (Memory Bank) | Form 4 (Seq Reconstructor) |
+| Diffusion cross-attention | Form 3 + Form 4 | |
+| Sequence processing (long) | Form 5 (Relay) + Form 6 (Router) | |
+| Diffusion bottleneck | Form 7 (Lookup Table) | Form 8 (GLFM) |
+| Train encoder from scratch | Form 9 (From-Scratch) | Form 2 (Soup as teacher) |
+| Model distillation | Form 10 (Consensus) | Form 12 |
+| Compound improvement | Form 11 (Evolution) | Form 10 + Form 12 |
+
+---
+
+## What the Constellation IS
+
+The constellation is a set of learned anchor points on S^(d-1). It is simultaneously:
+
+1. **A measurement instrument** — triangulation computes angular distances to reference points
+2. **A coordinate system** — the triangulation profile IS the geometric address
+3. **A lookup table** — the patchwork generates from the address, not reconstructing the input
+4. **A routing topology** — anchor proximity determines cross-token interaction (Cantor)
+5. **A geometric regularizer** — anchor structure prevents collapse and preserves manifold health
+
+The constellation is NOT:
+- An autoencoder (cos_sim ≈ 0 to input in bottleneck form)
+- A positional encoding (it measures WHERE on S^(d-1), not WHERE in sequence)
+- Class prototypes (anchors ≠ classes; anchor count independent of class count)
+- Patches of an image (constellation "patches" = dimensional subspace slices, not spatial tiles)