---
license: mit
language: [en]
tags: [attention, mixture-of-experts, softmax-free, linear-attention, geometric-deep-learning, research-log, aleph]
---

# aleph-splat — splat attention research line

**Softmax-free attention through tiny signed codebooks.** Every head is a
small frozen "aleph": K unit anchors read by a closed-form signed address
`w_k = sinh(u_k)/Σ_j cosh(u_j)` — reconstructive, never comparative
(no argmax, no top-k, no softmax anywhere, including across heads).
Attention is a write/read through the codebook cells; token-to-token
affinity is address *agreement* through a K-cell bottleneck — **linear in
sequence length**.

> 📄 **Technical companion:** [TECHNICAL.md](https://huggingface.co/AbstractPhil/aleph-splat-0/blob/main/TECHNICAL.md) — the full B1–B9 mathematical battery, the architecture, the training-collapse diagnosis and cure, and the speed matrix. Week-in-review article: [Geometric Memory FT5 — Agreement, Anchors, Addresses](https://huggingface.co/blog/AbstractPhil/geometric-memory-ft5).

## Play with it

[`splat_attention.py`](./splat_attention.py) — one file, torch-only:

```python
from splat_attention import SplatAttention
attn = SplatAttention(d_model=512, M=64, K=8, rotary=True)
y = attn(x)   # (B, L, d) -> (B, L, d)
```

`python splat_attention.py` runs a demo + speed bench. Knobs for the
gradient-heuristic variants are constructor args: `head_gates`
(learnable per-head attenuation), `train_centers` (moving windows),
`train_codebooks` (unfrozen frames), `rotary`/`global_frac` (transport
geometry).

## What has been measured (`battery/splat_battery.json`)

- **Product code**: M differentiated small codebooks beat one monolith at
  equal half-axis budget — joint readback .859 (1×64) → .955 (16×4),
  monotone; identical copies collapse. Random independent frames match
  constructed ones.
- **Differentiation is an attractor**: near-copies re-differentiate under
  a shared objective with no diversity pressure.
- **Comparative composition loses**: softmax-over-heads costs ~.10
  reconstruction vs budget-weighted blending.
- **Storage is partition-blind**: associative capacity scales with total
  cells regardless of head partition — addressing capacity and memory
  capacity are different resources.
- **Retrieval at scale**: 2048 splat-sharded heads: top-1 .9995 @ 2k
  context / .934 @ 8k where an equal-cell monolith reads .042 / .0015.
- **Speed**: flat µs/token vs MHA's quadratic — slower below ~2k tokens,
  ~2× faster at 8k; `torch.compile` (inductor) gives a further 3–4×.
- **The failure that shaped the design**: local-only positional windows
  at short L degenerate into a ±3-token blur — cross-position transport
  dies and a cls-pooled encoder collapses (measured, then repaired).
  The repair is `rotary=True`: RoPE applied to the *address query*, so
  relative position enters every affinity while heads stay global
  (cross-position recall .548 where the broken form gave ~0).

## Status

Research prototype under active iteration. Trained-at-scale encoder
results for the splat arm are pending; the sibling trained arms
(anchored-FFN trunks with the same address mechanism in the MLP) live at
[alephlm-0](https://huggingface.co/AbstractPhil/alephlm-0). Numbers above
are measurements on the stated probes — nothing more is claimed.