repo card: mechanism, usage, battery summary, honest failure note

README.md

@@ -0,0 +1,63 @@
+---
+license: mit
+language: [en]
+tags: [attention, mixture-of-experts, softmax-free, linear-attention, geometric-deep-learning, research-log, aleph]
+---
+
+# aleph-splat — splat attention research line
+
+**Softmax-free attention through tiny signed codebooks.** Every head is a
+small frozen "aleph": K unit anchors read by a closed-form signed address
+`w_k = sinh(u_k)/Σ_j cosh(u_j)` — reconstructive, never comparative
+(no argmax, no top-k, no softmax anywhere, including across heads).
+Attention is a write/read through the codebook cells; token-to-token
+affinity is address *agreement* through a K-cell bottleneck — **linear in
+sequence length**.
+
+## Play with it
+
+[`splat_attention.py`](./splat_attention.py) — one file, torch-only:
+
+```python
+from splat_attention import SplatAttention
+attn = SplatAttention(d_model=512, M=64, K=8, rotary=True)
+y = attn(x)   # (B, L, d) -> (B, L, d)
+```
+
+`python splat_attention.py` runs a demo + speed bench. Knobs for the
+gradient-heuristic variants are constructor args: `head_gates`
+(learnable per-head attenuation), `train_centers` (moving windows),
+`train_codebooks` (unfrozen frames), `rotary`/`global_frac` (transport
+geometry).
+
+## What has been measured (`battery/splat_battery.json`)
+
+- **Product code**: M differentiated small codebooks beat one monolith at
+  equal half-axis budget — joint readback .859 (1×64) → .955 (16×4),
+  monotone; identical copies collapse. Random independent frames match
+  constructed ones.
+- **Differentiation is an attractor**: near-copies re-differentiate under
+  a shared objective with no diversity pressure.
+- **Comparative composition loses**: softmax-over-heads costs ~.10
+  reconstruction vs budget-weighted blending.
+- **Storage is partition-blind**: associative capacity scales with total
+  cells regardless of head partition — addressing capacity and memory
+  capacity are different resources.
+- **Retrieval at scale**: 2048 splat-sharded heads: top-1 .9995 @ 2k
+  context / .934 @ 8k where an equal-cell monolith reads .042 / .0015.
+- **Speed**: flat µs/token vs MHA's quadratic — slower below ~2k tokens,
+  ~2× faster at 8k; `torch.compile` (inductor) gives a further 3–4×.
+- **The failure that shaped the design**: local-only positional windows
+  at short L degenerate into a ±3-token blur — cross-position transport
+  dies and a cls-pooled encoder collapses (measured, then repaired).
+  The repair is `rotary=True`: RoPE applied to the *address query*, so
+  relative position enters every affinity while heads stay global
+  (cross-position recall .548 where the broken form gave ~0).
+
+## Status
+
+Research prototype under active iteration. Trained-at-scale encoder
+results for the splat arm are pending; the sibling trained arms
+(anchored-FFN trunks with the same address mechanism in the MLP) live at
+[alephlm-0](https://huggingface.co/AbstractPhil/alephlm-0). Numbers above
+are measurements on the stated probes — nothing more is claimed.