references+attribution section, inline citations

34f8d4e verified 2 days ago

25.5 kB

	# Splat Attention — Technical Companion

	Program: [aleph-splat-0](https://huggingface.co/AbstractPhil/aleph-splat-0) · Author: AbstractPhil · Shipped artifact: `splat_attention.py` (single file, [torch](https://pytorch.org)-only)

	This document is the technical companion to the week-in-review article on the splat-attention program. It is self-contained: the concept, the measurement battery that preceded the architecture, the block itself, the training arc including its collapse and cure, the speed profile, and the open frontier. Every claim carries its number. The failures and one retraction are part of the record and are reported as plainly as the successes.

	---

	## 1. Overview: the Rorschach splat

	The program asked one question: can the aleph — a signed, reconstructive addressing formula with no argmax, no top-k, no softmax selection, no comparative routing anywhere — be the attention mechanism itself, rather than a [routing layer](https://arxiv.org/abs/1701.06538) bolted beside one?

	The design concept is the Rorschach splat: many tiny codebooks ("perspectives"), every one of them seeing the same data — shared sight — with the codebooks existing to differentiate each head's perception of it. The same inkblot, many readings. Selection happens by opinion: each perspective speaks from its own graded, signed reading of the input, never by cross-comparison against its peers. There is no softmax over heads, no winner-take-all, no comparative selector at any level of composition.

	The mathematical claim underneath the framing: independent perspectives form a product code — joint addressing capacity multiplies across heads — while collapsed (copied) perspectives degenerate to a single codebook. That claim, and five others, were converted into measurable questions and settled before any attention block was built.

	Two standing constraints shaped everything:

	- No comparative selectors. Argmax anchors, softmax routing, straight-through one-hots, VQ-style alphabets — this failure class (gradient homogenization, path collapse) was banned at every layer, including any composition layer over the codebooks. The rule: the routing formula must be the attention formula, so nothing corrupts it.
	- Mathematics first, on small hardware. The addressing math was settled on a single consumer GPU (RTX 4090) before any larger-scale compute was spent.

	---

	## 2. The mathematical battery

	An eight-probe battery (B1–B8) settled the mathematics before any architecture was built. A ninth probe, B9, was added later — after the prototype's training collapse — during the repair rounds (see §4). All probes are synthetic, seeded, and gauged on held-out data, with the aleph addressing code used verbatim. Probes B1–B6 ran at full scale, N = 200k samples; B7 and B8 sweep their own regimes (head counts to M = 2048, retrieval contexts to 8,192 pairs).

	### B1 — Capacity of one aleph

	Question: how efficiently does a single codebook of size K use its addressing space?
	Verdict: addressing efficiency, measured as bits per half-axis, decays with K: 0.50 at K = 4–8 (where the sign-code space saturates), falling to 0.069 at K = 128. Reconstruction runs the other way, rising .315 → .867 with K. Small codebooks are the efficient addressers; large codebooks are the better reconstructors — exactly the tension that many-small-codebooks exists to resolve. Low-rank (structured) data roughly halves half-axis usage, so capacity must be read against data rank — a fact that returns in B7.

	### B2 — The product code, confirmed

	Question: at equal total budget, does joint addressing capacity multiply across independent perspectives?
	Setup: fixed budget of 128 half-axes, partitioned into M heads.
	Verdict: held-out joint readback rises monotonically with M: .859 (one 64-anchor codebook) → .879 → .895 → .929 → .955 (sixteen 4-anchor codebooks). Copied perspectives collapse (mutual information 1.0 between heads, readback near zero). Independent random frames perform identically to carefully constructed rotations — differentiation needs no exotic initialization. Pairwise head overlap (NMI) falls .11 → .036 as M grows.

	### B3 — Differentiation is an attractor

	Question: does differentiation persist under shared-task training with zero diversity pressure?
	Setup: start perspectives as near-copies (1% noise apart), train on a shared reconstruction task.
	Verdict: the heads re-differentiate past the distance of independently-born frames — final pairwise distance .41 versus .35 — while converging to equal joint performance. Redundant opinions are gradient-wasteful, so the objective itself breaks the symmetry. In the program's phrase: the data manufactures the Rorschach.

	### B4 — Composition, and the ban measured

	Question: how should the perspectives' answers combine — and what does comparative selection actually cost?
	Verdict: uniform weighting scores .797, self-referential confidence .788, and the deliberately included softmax-over-heads falsification control .692. Sharpening toward the "best" perspective loses about .10 — graded amplitude beats selection, measured. The no-comparative-selector rule, previously an architectural principle, is here a number. The joint read also beats every single-owner read (best single perspective: .61) — the product effect from another angle.

	### B5 — Storage is partition-blind (an honest null)

	Question: does partitioning a codebook buy associative-memory capacity?
	Verdict: no. Stored key→value pairs recoverable at 5% noise (cosine ≥ .7) scale with total cells — roughly K/2 up to K = 32, sublinear after (K = 128 stores ~32 pairs) — and are identical for one large codebook versus a product of small ones at equal total cells. Addressing capacity and memory capacity are different resources: partitioning buys the former, never the latter. This null sized the mechanism's memory honestly before it was built.

	### B6 — Overlap buys long-chain solidity

	Question: what overlap between positional windows keeps long chains solid?
	Verdict: hard tiling (zero overlap) reproduces a previously measured deep-chain pathology exactly — solid within a tile, edge solidity 0.0. Gaussian overlap at σ/spacing = 1.0 buys ~800 chained positions with living edges (.41); at 2.0, ~1600 positions with edges at .85. Long-chain solidity is purchasable with overlap; the design band is σ/spacing ∈ [1, 2].

	### B7 — The M-knee tracks data rank

	Question: how many perspectives keep paying?
	Verdict: it depends on data rank. On rank-8 data the curve knees by M ≈ 64 (.96). Rank-32 climbs to .993 at M = 2048. Full-rank data climbs monotonically from .62 at M = 16 to .990 at M = 2048 — on rich data, thousands of tiny heads keep paying. (Caveat carried: the gauge is a linear probe — a floor, not a ceiling.)

	### B8 — The 2048-head answer

	Question: the direct test of "why not 2048 heads rather than one, or 5, or 20?" — retrieval from a context of N random key→value pairs at strictly equal total cells (4096).

	\| context \| form \| graded recall \| top-1 among all distractors \|
	\|---\|---\|---\|---\|
	\| 2048 \| monolith (one K = 2048 codebook) \| .204 \| .042 \|
	\| 2048 \| 512 splat heads \| .657 \| .944 \|
	\| 2048 \| 2048 splat heads \| .893 \| .9995 \|
	\| 8192 \| monolith \| .103 \| .0015 \|
	\| 8192 \| 2048 splat heads \| .660 \| .934 \|

	Splat sharding — each head a K = 4 codebook owning an overlapping positional window, σ = 1.5× spacing per B6 — beats the equal-cell monolith by 25–600× on retrieval. Partition plus locality rescues what superposition drowns. Cost is O(N·K) per head; all 2048 heads, in unoptimized Python, processed 8k tokens in 1.4 seconds. Caveats carried honestly: the monolith was not temperature-tuned (fair at equal cells, not optimal), and the keys were random — correlated real token streams will invoke B7's rank effect.

	### B9 — Rotary addresses (added post-collapse, during the repair rounds)

	Question: can position enter the address itself, so heads stay global while every affinity carries relative position?
	Setup: built after the prototype's training collapse revealed that B1–B8 had measured retrieval but never transport (see §4). Position enters as a [RoPE](https://arxiv.org/abs/2104.09864) rotation of the address query against the frozen codebook.
	Verdict, on three axes: short-sequence cross-position recall .548 where local-window-only splat gave ~0 (the collapse cure); retrieval decaying gracefully with query/write offset — .64 → .31 over a shift of 0 → 64 — position-selective, not position-brittle; and addresses 32 positions apart remaining discriminable-but-not-divorced (cosine .577).

	### The design card

	Distilled from B1–B8 into the specification the prototype was built to:

	- K = 4–8 anchors per head — small codebooks saturate their sign-code space (B1).
	- M scales with data rank — thousands of heads stay productive on rich streams (B7).
	- Frames born random and independent — constructed rotations buy nothing (B2); differentiation self-maintains under training (B3). (The original card added "frozen viable"; that clause was later amended by measurement — §4.)
	- Overlap σ/spacing ∈ [1, 2] when using local windows (B6).
	- Composition by graded amplitude, never comparative — softmax composition measurably loses ~.10 (B4).
	- Memory budget = total half-axes regardless of partition (B5).
	- Both graded read and top-1 retrieval gauged, always held-out.

	---

	## 3. The architecture: SplatAttention

	Every head is a tiny aleph codebook: K unit anchor directions read through the closed-form signed address

	u_k = cos(x, a_k) / τ w_k = sinh(u_k) / Σ_j cosh(u_j)

	— a reconstructive read with no argmax, no top-k, no softmax selection. Weights are signed, so an anchor can contribute negatively.

	Attention is a write/read through a shared blackboard: each token writes its value vector into the head's K cells through its address; each token then reads the cells back through its own address. Affinity between tokens is address agreement through the K-cell bottleneck — O(L·M·K) per layer, linear in sequence length. There is no softmax anywhere in the block; the routing formula is the attention formula.

	The block, as shipped:

	- M heads × K anchors — frozen buffers by default; `train_codebooks=True` unfreezes the anchor frames (the converging training configuration, §4).
	- Rotary address query (`rotary=True`, the safe default): position enters as a RoPE rotation of the address query, so relative position appears inside every affinity while heads stay global (validated by B9). A windowed local mode (Gaussian supports, `overlap` = σ as a multiple of spacing, optional `global_frac` of uniform-membership heads) is retained for the static regime B6 characterized.
	- Membership normalization: the read is a weighted average, not a raw sum — each token's output is divided by its accumulated absolute address mass (Σ\|w·g\|), so tokens with weak membership are not silently attenuated and pad-masked writes stay clean.
	- Identity-born address frame (`addr_proj=True`): a learned linear map W_a on the address path, initialized to the identity — routing-owned parameters that leave the block's behavior unchanged at birth and let training reshape where tokens address without touching what they carry.
	- M-chunked [checkpointing](https://arxiv.org/abs/1604.06174): heads are processed in chunks (`mchunk`), and chunks can be recomputed in the backward pass (`checkpoint_chunks=True`), bounding activation memory at large M.
	- Optional per-head gates born at identity (`head_gates`) and learnable window centers/widths (`train_centers`) — the latter only functions on a transport-capable geometry.

	The prototype attached this block inside an [AlephLM](https://huggingface.co/AbstractPhil/alephlm-0) trunk — replacing [standard scaled-dot-product attention](https://arxiv.org/abs/1706.03762), anchored FFN retained, a fully aleph-native block: M = 128 frozen K = 8 heads, Gaussian supports at σ = 1.5× spacing, pad-masked writes, graded-amplitude composition.

	---

	## 4. The training arc: collapse, ladder, convergence

	This is the section where the program earns its credibility, because the battery's success did not survive first contact with training — and the record says so.

	### Round 1 — structural collapse, caught in 33 minutes

	A small-rig screen ([500k captions](https://huggingface.co/datasets/AbstractPhil/captionbert-8192-v2-consensus), 2 epochs, L = 128, paired seeds, consumer 4090) existed precisely to answer in minutes what a large-scale run answers in days. The frozen splat prototype collapsed: training-gauge mimicry .0060, representation effective rank 5.3 — against a healthy standard block at the same budget at mimicry .9055, erank 46.1. The mechanism was legible: at L = 128 with M = 128 heads, the Gaussian supports are ~1.5 tokens wide — attention degenerates to a ±3-token local blur, cross-position transport dies, the pooled readout starves. A variant with trainable window centers was worse (erank 2.0 mid-run).

	Crucially, B8 had measured retrieval, not transport — its reads were same-slot. The battery was incomplete in a nameable way, and the screen caught it before roughly 60 hours of queued large-scale compute were burned on the same collapse. (The same screen produced a bonus finding: the anchored-FFN trunk is more sample-efficient than a dense trunk — .9055 vs .7790 mimicry at small budget — so an earlier full-scale "tie" was endpoint convergence, not equivalence.)

	### The method correction — piecemeal gradient linkages

	The repair method, in AbstractPhil's phrasing: "we're piecemealing... every relational problem being linked to potential gradient learning weights that function and coalesce." Not a search for one solving variant. Each relational problem gets a gradient linkage — a learnable weight-set that owns it; single-variable screens test whether each linkage functions; combinations test whether linkages coalesce. The map: position → rotary address; transport reach → global membership; attenuation → per-head gates born at identity; perspective differentiation → trained codebooks; window placement → centers/widths (only meaningful once transport works).

	### Round 2 — the statics fixed, the collapse persisted

	Two transport repairs were built and validated on the battery bed — B9 (rotary addresses) and a global-heads fraction. Both fixed the static probes. Training still collapsed: global heads .0090, rotary .0175, rotary+gates .0015.

	The decisive instrument was an initialization probe: at birth, the splat block's output is healthier than standard attention's — output effective rank 118 versus 12.7. So the collapse is dynamic, not structural: parameter-free routing deforms token states into address basins. The addressing fabric wasn't broken — it had no parameters with which to defend itself.

	### Round 3 — routing-owned parameters, and the ladder

	Give the routing its own learnable weights. A learned address frame born at identity took the gauge to .0590 (erank 8.8). Adding trained codebooks coalesced late — .020 mid-run, then .1665 (erank 13.4) at the 2-epoch cutoff, still accelerating. The linkage ladder across rounds:

	.006 → .0175 (rotary) → .059 (+ address frame) → .167 (+ trained codebooks, late-blooming)

	Each routing-owned linkage multiplied the gauge roughly 3×, with delayed co-organization — the linkages stack multiplicatively, and the coalescence question was answered: they compound.

	### Round 4 — splat attention trains

	The coalescence stack (rotary + address frame + trained codebooks) extended to 6 epochs: mimicry climbing .5725 → .7455 → .789 → .8245 → .8415 → .850 → .858 → .861 across the run, landing at final mimicry .8645, cosine .6461, effective rank 41.7. The 2-epoch .167 had been mid-bloom. The collapse is cured. The residual finding is cost: roughly 3× the optimization distance of a standard block at this scale, attributable to the late co-organization of the address/frame/codebook systems.

	### The equal-footing ledger

	Paired 6-epoch baselines — same data, same budget, same rig:

	\| trunk (6 epochs, matched small budget) \| mimicry \| cosine \| erank \|
	\|---\|---\|---\|---\|
	\| anchored FFN + standard attention \| .9685 \| .7332 \| 68.0 \|
	\| dense \| .9645 \| .7204 \| 62.6 \|
	\| splat (rotary + address frame + trained codebooks) \| .8645 \| .6461 \| 41.7 \|

	And the honest revision that came with it: an earlier "cosine parity" claim had compared unequal budgets and is retracted. At matched budget, splat is functional but behind (−.09 cosine versus the anchored baseline), still climbing at cutoff. The anchored-FFN advantage persists at every budget tested. The record keeps both the claim and its retraction; so does this document.

	### The design-card amendment

	The card's "frames frozen viable" clause is amended by measurement: frozen addresses suffice for static probes; training requires routing-owned parameters. The converging configuration — and the candidate for larger-scale runs — is rotary + learned address frame + trained codebooks, not the frozen fabric alone. Frozen-everything remains fine for inference-style use and every static property of §2.

	---

	## 5. Speed

	Measured forward+backward, fp16 autocast, µs/token, on a consumer 4090 and a datacenter A40, versus an 8-head standard `nn.MultiheadAttention` at d = 512:

	- Eager: splat is flat at ~0.64–1.1 µs/token across lengths while standard attention climbs 0.18 → 1.76. That reads as ~3–6× slower at L ≤ 256, parity near L = 2048, and ~2× faster at L = 8192 — flat cost against quadratic. (An initial 4090 bench read the long end at 2.8× faster; the shipped header states the conservative ~2×.)
	- Compiled: `torch.compile` (inductor, Linux) gives a further 3–4× on the module — 1.09/1.01/1.02 µs/token versus eager 3.4–4.2 — putting compiled splat at 1.8× slower at L = 256, parity at 2k, 2× faster at 8k. The Windows fallback path is a regression (0.75–0.92×); compile pays only where the real kernel compiler lives.
	- Live in the shipped demo: 1.68 vs 0.61 µs/token at L = 128 and 1.73 vs 2.54 at L = 2048 — the crossover is visible in the demo itself.

	Short-context cost is the tuition; long context is the native regime.

	---

	## 6. The shipped file

	`splat_attention.py` — a standalone, single-file, torch-only module, shipped in this repository.

	- Constructor-arg API: `SplatAttention(d_model, M, K, tau, rotary, global_frac, overlap, sigma_floor, head_gates, train_centers, train_codebooks, addr_proj, mchunk, checkpoint_chunks, ...)`; forward takes `(B, L, d)` with an optional key-padding mask.
	- Safe defaults: `rotary=True` — the configuration validated by B9 to have transport at every sequence length.
	- Demo + bench main: running the file directly executes a forward/backward sanity demo and a small speed benchmark against standard attention, so the crossover numbers above can be reproduced on the spot.
	- An honest header: the measured speed matrix, the B8 retrieval numbers, the training collapse and its mechanism, the two repairs, and the converging training recipe (~90% of a standard block's training gauge at matched small budget, still climbing, endpoint parity unproven) are all stated up front — an observer knows exactly what they are holding.

	---

	## 7. Laws established

	What the program carried in as rules, and what measurement did to them:

	1. No comparative selection — now a number. The softmax-over-heads composition control loses ~.10 (B4). The ban is a measured cost, not just an architectural principle.
	2. Addressing capacity and memory capacity are different resources. Partitioning buys the former, never the latter (B5's null).
	3. Overlap buys long-chain solidity, design band σ/spacing ∈ [1, 2] (B6: edges 0.0 → .41 → .85 across zero/1.0/2.0 overlap).
	4. The M-knee tracks data rank. On rich data, thousands of tiny heads keep paying (B7: .990 at M = 2048 on full-rank data).
	5. Routing-owned parameters. Parameter-free routing collapses under training by deforming token states into address basins (init probe: out-erank 118 at birth, mimicry .006 after training); routing that owns learnable weights converges (.8645). Frozen addressing is a static-regime tool.
	6. Gradient linkages compound. One relational problem, one learnable weight-set; screen each for function, combinations for coalescence; the ladder .006 → .0175 → .059 → .167 stacked roughly 3× per linkage, with late-blooming co-organization.
	7. Instrument-first sequencing. The battery ran before the mechanism; when the battery proved incomplete (transport untested), the gap was named, the missing probe (B9) was built and validated, and only then did repairs proceed.
	8. Screens before scale. A 33-minute small-rig screen saved ~60 hours of queued large-scale compute from a known collapse.
	9. Ledgers, not scoreboards — and everything ships. Results are reported as measurement ledgers (the equal-footing table stands as measured, splat behind), the retraction stays on the record, and every arm — batteries, failed variants, and the standalone module with its failure documented in the header — is public.

	---

	## 8. Open frontier

	In the program's own ordering:

	1. Schedule engineering for the co-organization delay. The trained-codebook linkage bloomed late and was still accelerating at every cutoff; an address-stack warmup schedule is the direct attack on the ~3× optimization-distance gap.
	2. The scale question. B7's rank effect predicts richer data pays the address stack faster — the 3× cost ratio may shrink with scale. Endpoint parity at matched budget is unproven in either direction.
	3. The next natural linkage: learnable per-head rotary frequencies and phases — giving each perspective its own positional geometry.
	4. Attachment to token-stream models. The parked larger-scale runs resume only with the amended routing-owned configuration (rotary + address frame + trained codebooks), attaching the block to real autoregressive token streams where B7's rank effect and B8's long-context regime meet.

	---

	## References and attribution

	Models

	- [AlephLM](https://huggingface.co/AbstractPhil/alephlm-0) (AbstractPhil) — the trunk the prototype block was attached to for the training screens; the anchored-FFN and dense baselines in the equal-footing ledger are paired configurations of the same trunk.

	Data

	- [CC12M](https://arxiv.org/abs/2102.08981) (Changpinyo et al.) via [pixparse/cc12m-wds](https://huggingface.co/datasets/pixparse/cc12m-wds) — the caption source behind the small-rig training screens.
	- [captionbert-8192-v2-consensus](https://huggingface.co/datasets/AbstractPhil/captionbert-8192-v2-consensus) (AbstractPhil) — the consensus caption data the 500k-caption screens draw on.

	Methods

	- [Attention / the Transformer](https://arxiv.org/abs/1706.03762) (Vaswani et al.) — standard multi-head scaled-dot-product attention is the measured baseline throughout: the equal-footing training ledger (§4) and the speed matrix (§5).
	- [RoPE](https://arxiv.org/abs/2104.09864) (Su et al.) — rotary position embedding, entering here as a rotation of the address query against the frozen codebook (B9; the shipped default).
	- [Mixture-of-experts routing](https://arxiv.org/abs/1701.06538) (Shazeer et al.) — the comparative-routing lineage (argmax/top-k selection over experts) that the signed, reconstructive addressing formula departs from.
	- [Gradient checkpointing](https://arxiv.org/abs/1604.06174) (Chen et al.) — the recompute-in-backward technique behind the M-chunked `checkpoint_chunks` option.
	- Naming: the "splat" name nods to [3D Gaussian Splatting](https://arxiv.org/abs/2308.04079) (Kerbl et al.); the mechanisms are otherwise unrelated.

	Infrastructure

	- [PyTorch](https://pytorch.org) — the shipped module is single-file and torch-only; benchmarks run against `nn.MultiheadAttention`, with the compiled path via `torch.compile` (inductor).

	Related artifacts: [aleph-splat-0](https://huggingface.co/AbstractPhil/aleph-splat-0) (this program's release, carrying `splat_attention.py`) · [alephlm-0](https://huggingface.co/AbstractPhil/alephlm-0) · [captionbert-8192-v2-consensus](https://huggingface.co/datasets/AbstractPhil/captionbert-8192-v2-consensus) · prior installment: [geometric-memory-ft4](https://huggingface.co/blog/AbstractPhil/geometric-memory-ft4)

	---

	Provenance: all numbers in this document are measurements from the splat program's primary experimental record — the synthetic battery (B1–B9), the paired-seed training screens, and the speed benchmarks — verified against the program's internal key-numbers ledger. Program and artifacts by AbstractPhil; released as aleph-splat-0 with the standalone module `splat_attention.py`.