| --- |
| license: other |
| tags: |
| - v12 |
| - emoe |
| - fft-hybrid |
| - frozen-base |
| - hypernetwork |
| - ephemeral-moe |
| --- |
| |
| # eMoE v12 — Ephemeral Mixture-of-Experts on a Frozen Tiny Base |
|
|
|  |
|
|
| eMoE is a complete serving system, not a single model. It mints a **fresh, per-request adapter** for a small **frozen** base model from a task-conditioning vector, runs it inside a deterministic control loop with out-of-distribution gating and retrieval grounding, and discards the adapter when the request is done. Nothing about the base changes; only a 9.20M hypernetwork is trained. |
|
|
| The bet: most of what a small model lacks on a given request is **specialization** and **grounding**, not raw capacity. eMoE buys both cheaply — a per-request expert plus distance-scaled RAG — on a base that stays frozen and reusable. It explicitly does **not** buy reasoning depth, and the design is honest about that. |
|
|
| --- |
|
|
| ## The three pillars |
|
|
| **1. An owned, frozen base.** A 190M FFT-hybrid (12 layers, 1024d, 16 heads, `attn_every=3` → 4 attention + 8 spectral layers, 2048 context), trained with Muon+AdamW on a curated mix of synthetic tool-use and proxy-filtered real data. It reached **VAL 1.6609** and is frozen for all serving. Spectral layers make adaptation cheap: you steer the envelope/gain and the kernel-generating MLP rather than rewriting dense weights. |
|
|
| **2. A hypernetwork (9.20M params).** It reads a task vector `z` — a `BAAI/bge-base-en-v1.5` embedding (d_z 768) of the request descriptor — and emits adapter deltas for **64 sites**: **FiLM** on the spectral envelope/gain and **LoRA** on the kernel-gen MLP. There is **no materialized ΔW**; deltas are applied in-place and scaled by a global α. Shape-grouped, bottlenecked heads keep the whole hypernetwork at 9.20M despite covering 64 sites. It is the only trained component (AdamW, rank 8, ChatML assistant-turn supervision), reaching **held-out VAL ≈ 2.84** on a 1769-task / 1504-cluster set (265 held out). |
| |
| **3. A deterministic runtime harness.** Fixed code with exposed knobs — **no agentic planning, no self-critique, no LLM-as-judge routing.** The model generates in exactly two places (an optional descriptor rewrite, and the answer); everything else is deterministic: |
| - **triviality gate** — trivial prompts skip the controller and hit the bare base (α=0) |
| - **OOD controller** — cosine distance to the nearest known training cluster drives **RAG-need** (`tau_rag_on = 0.287`) and **expert count k** (`tau_k_escalate = 0.308`). Distance is treated as **novelty, not trust** — it never scales the adapter. |
| - **distance-scaled RAG** — retrieve, drop passages below a similarity floor, set granularity from the score-profile shape, pack within a token budget |
| - **ephemeral multi-expert loop** — mint → verify → ACCEPT, or REJECT→escalate (mint from neighbor-cluster z's), or ABSTAIN→ship a single α-damped mint |
| - **global α backstop = 0.8** — fixed; not distance-driven, not self-perplexity-gated (both were measured and retired) |
| - a **z-consistency guard** that refuses to boot if the runtime encoder doesn't reproduce the calibration cache (cosine ≥ 0.99) |
| |
| --- |
| |
| ## What it actually does (measured, not asserted) |
| |
| On the 265 held-out tasks, comparing the right-z adapter against the bare frozen base on identical tokens: |
| |
| - **74% of tasks improve** with the minted adapter (mean **+0.63 nats**) |
| - the adapter is **task-specific**: right-z beats a wrong task's z on **80% of tasks** |
| - **~19% of tasks shatter** (the adapter makes it worse). On a frozen base you can't mint these away — so the harness **catches** shatter (reject → escalate) rather than predicting it. Nearest-cluster distance only weakly anticipates shatter (AUC 0.60), so it stays a novelty/k knob, never a trust gate. |
| |
| This is the honest shape of the system: cheap, real specialization on most requests; a meaningful minority where the verifier and escalation loop earn their keep; grounding via RAG for novel requests. |
| |
| --- |
| |
| ## Honest limitations |
| |
| - **It's a 190M base.** Output is simple and sometimes imperfect; greedy loops are suppressed with n-gram blocking, not eliminated by capacity. This is a specialization+grounding system, not a reasoning engine. |
| - **The verifier checks form, not correctness.** It catches degeneracy, syntax/parse failure, and retrieval mismatch — it will accept code that compiles but is wrong, or prose that overlaps the retrieved passages. The strong execution rung only fires when a request carries its own tests. |
| - **Some executors are stubs.** Python compile-checking is live; C++/Lean runners are shape-validated but not wired (they fail-safe to reject). |
| |
| --- |
| |
| ## Files in this repo |
| |
| - `ckpt_v12_190m_best.pt` — the frozen 190M FFT-hybrid base (canonical) |
| - `hyper_ckpt_v12.best.pt` — the trained 9.20M hypernetwork (use this; not the end-of-run checkpoint) |
| - `hyper/z_cache.pt` — BGE z-cache the controller geometry is calibrated against |
| - `hyper_ckpt.bestMidRun7750.pt` — earlier 612-task hypernetwork (reference only) |
| - `model_hybrid.py`, `muon.py`, `tok_v9.py` — model code + tokenizer |
|
|
| Tasks and RAG artifacts live in the dataset repo `Daxamite/eMOE-rag`; the live demo (full harness, routing trace) runs at the Space `Daxamite/eMOE`. |
|
|
| --- |
|
|
| ## Provenance note |
|
|
| An earlier draft of these notes quoted a VAL of ~2.09 — that belonged to a **retired 1142-task** run and does not describe this model. The correct held-out floor for `hyper_ckpt_v12.best.pt` is **≈2.84**, consistent with the project's historical floor and verified by an independent help/shatter eval (mean adapted loss 2.86). The controller thresholds (0.287 / 0.308) were re-banked on this model's own z-geometry. |
|
|
| _License: other (see repo). The base is an owned, curated model; the hypernetwork and harness are the eMoE contribution._ |
|
|
|
|
|
|
