--- license: mit tags: - llm-inference - speculative-decoding - medusa - bitnet - adaptive-compute - efficiency - physics-informed datasets: - parrishcorcoran/MedusaBitNet-48seq-cache pipeline_tag: text-generation --- # unified-gate > **LLM inference is overbudgeted by ~1000×. The per-token difficulty signal lives on a ~7-dimensional manifold. We measured it. This is the gate.** - **Code & training pipeline**: [github.com/parrishcorcoran/unified-gate](https://github.com/parrishcorcoran/unified-gate) - **Research apparatus**: [github.com/parrishcorcoran/MedusaBitNet](https://github.com/parrishcorcoran/MedusaBitNet) - **Companion inference efficiency thesis** (theory): `THEORY.md` in the GitHub repo - **26 KB deployment artifact**: `gate_k20.pt` (included here) --- ![k-sweep: engineering knee = physics ceiling](https://raw.githubusercontent.com/parrishcorcoran/unified-gate/main/figures/k_sweep.png) ## The one-minute pitch Every speculative-decoding / early-exit / Medusa / adaptive-compute paper of the last three years is *the same sensor in a different costume* measuring *one underlying signal*: how sharp is the next-token distribution. The field keeps shipping new sensors and never builds the *controller* that fuses them. This is the controller. It's a 20-feature, 64×64 MLP (26 KB) that decides, per token, whether to accept a cheap draft or run the full backbone. Held-out measurement on BitNet b1.58 2B: **10.6% skip at 95% fidelity**, 14.1% skip at 90% fidelity (peak K=40-50, replicated ±0.3% over 5 seeds). The *provocative* claim is not the skip rate. It's the dimensionality: the per-token difficulty surface is **~7-dimensional**, measured by TwoNN on final-layer hidden states, across two architectures (BitNet 2B + Llama 3.1 8B). That's a physics-grounded ceiling, not an engineering target. It says per-token decision-making has a compute floor and we're nowhere near it. --- ## The three claims, each measured ### 1. The information is on a thin surface, not in the bulk Running 30-layer × 2560-dim backbone computation for every token is redundant with what Medusa heads already read off the cached hidden state. That's the holographic principle applied to transformer inference — the heads are empirical proof the future tokens were already on the surface. Bulk volume is being recomputed from boundary data per step. ### 2. Compute and entropy are inversely correlated Conditional next-token entropy *decreases* with context length (cloud tightens as context locks in plausible completions). Transformer compute per token *increases* with context length (O(N²) attention, bigger KV cache). Current decoders scale compute up exactly when information requirement scales down. RNNs had the right compute shape — we traded it for capacity. ### 3. The gate's dimensionality is set by physics Per-sequence intrinsic dim of final-layer hidden states, measured by TwoNN (Facco et al. 2017): | Model | Ambient dim | Per-seq intrinsic | |---|---|---| | BitNet b1.58 2B (result_norm) | 2560 | **7.3** | | Llama 3.1 8B Q4_K_M (result_norm) | 4096 | **6.9** | Second cross-model metric: raw hidden-state participation ratio divided by ambient dim: | Model | PR | PR / ambient | |---|---|---| | BitNet 2B | 85 | **3.3%** | | Llama 3.1 8B | 151 | **3.7%** | Two independent measurements agreeing that both models concentrate per-token decision-making into ~7 dimensions out of thousands. When we train the gate on top-K features ranked by gradient importance, **K=7 recovers ~70% of the K=50 peak skip**. The engineering knee of the feature-count curve lands exactly at the physics ceiling. --- ## The measurement 5-seed K-sweep on the BitNet 2B held-out set. skip at λ=0.95 fidelity (mean ± std): ``` K skip@λ=0.95 σ-gap vs K=70 7 7.3% (single) (matches per-seq intrinsic dim, 80% of peak) 15 9.2% ± 0.3% -2.4σ (lower, expected) 20 9.8% ± 0.2% 0.1σ (matches K=70) 25 10.1% ± 0.2% +1.1σ 30 10.5% ± 0.3% +2.1σ 40 10.6% ± 0.2% +3.2σ ← peak 50 10.7% ± 0.2% +3.4σ ← peak 70 9.7% ± 0.3% baseline ``` **The K=70 bundle is over-parameterized.** Adding features past ~50 degrades the gate by ~9%, a ~3σ effect replicated across seeds. This is the inference analog of *parameter count ≠ information content*: once you cross the per-seq manifold ceiling, extra features are just overfitting noise. --- ## Architecture (gate_k20.pt) - **20 input features** selected by gradient importance from a 70-feature physics-aperture bundle - **Two hidden layers** of 64 ReLU units each - **Single sigmoid output** (skip probability) - **~6,500 parameters**, 26 KB on disk - **Calibrated thresholds** for λ ∈ {0.85, 0.90, 0.95, 0.99} bundled in the checkpoint ### The 20 features Ranked by gradient importance on held-out: 1. `sup_1` — superposition effective rank (exp(entropy of top-K softmax)) 2. `cluster_1` — K-means soft-cluster entropy 3. `logit_gap` — head-0 top1 minus top2 logit 4. `content_conf` — head-0 top-1 softmax 5. `cluster_0` — K-means min-distance-to-center 6. `layer_5` — cos(h_5, h_15) Ryu-Takayanagi layer-wise similarity 7. `layer_9` — layer-wise norm_15 (log) 8. `layer_7` — cos(h_5, h_29) 9. `top10_cov` — head-0 cumulative top-10 probability 10. `treuse_2` — token-reuse rank within recent window (H2O lexical) 11. `agreement_count` — head-0 arg-max matches head-k lagged 12. `fe_1` — entropy-adjusted free-energy analog 13. `rg_2` — renormalization-group divergence at scale 9 14. `mom_0` — head-0 softmax 3rd moment (skewness) 15. `vel_0` — hidden-state velocity ‖h_t − h_{t-1}‖ 16. `fe_0` — log(1 + 0.01 · cluster_mindist) 17. `hnorm_0` — log(1 + ‖h_t‖) 18. `layer_1` — log(1 + velocity 15→29) 19. `nbr_0` — distance to nearest recent hidden state (H2O temporal) 20. `sup_0` — top-K token-embedding spread in hidden space Five framings from the theory thesis, each contributing: - **Holographic** (cluster, neighborhood, free-energy) - **Electron-cloud / superposition** (sup_spread, sup_eff_rank, moments) - **Ryu-Takayanagi depth projection** (layer-wise 5/15/29 features — biggest single group) - **H2O heavy-hitters** (token-reuse, neighborhood) - **Renormalization group** (multi-scale coarse-graining divergence) - **Base information-theory** (confidence, logit gap, covers, agreement) --- ## Usage ```python import torch from unified_gate import Gate, extract_all_features gate = Gate("gate_k20.pt") # Per-sequence feature extraction X = extract_all_features( hidden_last=h29, # [T, H] final-layer result_norm, float32 hidden_mid=h15, # [T, H] middle layer hidden_early=h5, # [T, H] early layer head_logits=logits, # [T, K_heads, V] Medusa head logits lm_head=lm_head_np, # [V, H] output embeddings tokens=tokens, # [T] token ids period_ids=period_ids, # precomputed from tokenizer newline_ids=newline_ids, cluster_centers=centers, # K=32 pre-fit centers ) # returns [T-8, 70] float32 # Skip decision scores = gate.score(X) # skip probability per token skip_mask = gate.skip_mask(X, fidelity=0.95) # Accept Medusa draft where skip_mask is True; re-run backbone where False. ``` Install from GitHub: ```bash pip install git+https://github.com/parrishcorcoran/unified-gate.git ``` Reproducibility: ```bash git clone https://github.com/parrishcorcoran/unified-gate cd unified-gate python scripts/reproduce.py --medusabitnet-root /path/to/MedusaBitNet ``` Matches stored frontier within ±0.001 absolute skip. --- ## Cross-model scope and limits **Validated on**: - BitNet b1.58 2B (primary training + held-out measurement) - Llama 3.1 8B Q4_K_M (cross-model TwoNN intrinsic-dim agreement) **Not yet validated on**: - Wall-clock speedup on real hardware (the systems paper follow-up) - Much larger models (70B+) - Non-English / specialized domains **Known limits**: - The gate is trained on BitNet-specific Medusa head acceptance. Cross-model *deployment* requires retraining the 64×64 MLP on target-model head acceptances. The *feature extractor* generalizes; the MLP weights don't. - `gate_k20.pt`'s `agreement_count` feature is a 0/1 logical OR (numpy 2.x bool-add semantics in training pipeline) not a 0-3 count. A corrected retraining is on the v0.3 roadmap. In the measured frontier this is empirically fine — but it's a lurking name/semantics mismatch worth flagging. --- ## Theoretical framework Six equivalent framings — not six different ideas, but one underlying insight seen from six angles: 1. **Holographic principle / black-hole boundary layer** — information about the completion is on a thin surface of the hidden state, not in the bulk compute 2. **Electron cloud / quantum probability** — there is no "correct" next token; the cloud *is* the observable 3. **Fractal / hologram** — every per-token forward is a self-similar slice of one underlying trajectory computation 4. **Compute-entropy inversion** — conditional entropy drops through the sequence while O(N²) compute per token rises; they should be correlated, they're anti-correlated 5. **Boundary layer** — predictability lives in a thin laminar region; only a minority of tokens are boundary-class 6. **Unified sensor gate** — all existing techniques (draft, Medusa, early exit, N-gram, bottleneck) are redundant entropy sensors; the missing piece is the controller Full thesis including the companion spin-glass-substrate framing and the tokens-per-joule thermodynamic argument is at `THEORY.md` in the GitHub repo. --- ## Roadmap - **v0.3** — retrain gate with corrected `agreement_count` (0-3 count, not 0/1 OR) - **v0.4** — Llama 3.1 8B Medusa-compatible gate (once heads are trained) - **Paper 1** — this repo's measurement + theory (target: arXiv) - **Paper 2** — wall-clock C++ integration (follow-up systems paper) - **Fat-trunk / thin-branches architecture** — direct consequence of 7-dim finding: narrow late layers, full-width early layers. Experimentally justified but untested. --- ## Credits - **Parrish Corcoran** — research direction, physics framework, experimental design - **Claude Opus 4.6 (1M context)** — implementation, measurements, 24-hour autonomous research session (2026-04-15) --- ## License MIT — research use encouraged. --- ## Citation Preferred citation format until the paper lands: ```bibtex @software{corcoran_unified_gate_2026, author = {Corcoran, Parrish}, title = {unified-gate: Confidence-gated adaptive LLM inference on a 7-dimensional boundary manifold}, year = {2026}, url = {https://github.com/parrishcorcoran/unified-gate} } ```