V12Emoe / README.md

Update README.md

9619ef8 verified 13 days ago

5.86 kB

license: other
tags:
  - v12
  - emoe
  - fft-hybrid
  - frozen-base
  - hypernetwork
  - ephemeral-moe

eMoE v12 — Ephemeral Mixture-of-Experts on a Frozen Tiny Base

eMoE is a complete serving system, not a single model. It mints a fresh, per-request adapter for a small frozen base model from a task-conditioning vector, runs it inside a deterministic control loop with out-of-distribution gating and retrieval grounding, and discards the adapter when the request is done. Nothing about the base changes; only a 9.20M hypernetwork is trained.

The bet: most of what a small model lacks on a given request is specialization and grounding, not raw capacity. eMoE buys both cheaply — a per-request expert plus distance-scaled RAG — on a base that stays frozen and reusable. It explicitly does not buy reasoning depth, and the design is honest about that.

The three pillars

1. An owned, frozen base. A 190M FFT-hybrid (12 layers, 1024d, 16 heads, attn_every=3 → 4 attention + 8 spectral layers, 2048 context), trained with Muon+AdamW on a curated mix of synthetic tool-use and proxy-filtered real data. It reached VAL 1.6609 and is frozen for all serving. Spectral layers make adaptation cheap: you steer the envelope/gain and the kernel-generating MLP rather than rewriting dense weights.

2. A hypernetwork (9.20M params). It reads a task vector z — a BAAI/bge-base-en-v1.5 embedding (d_z 768) of the request descriptor — and emits adapter deltas for 64 sites: FiLM on the spectral envelope/gain and LoRA on the kernel-gen MLP. There is no materialized ΔW; deltas are applied in-place and scaled by a global α. Shape-grouped, bottlenecked heads keep the whole hypernetwork at 9.20M despite covering 64 sites. It is the only trained component (AdamW, rank 8, ChatML assistant-turn supervision), reaching held-out VAL ≈ 2.84 on a 1769-task / 1504-cluster set (265 held out).

3. A deterministic runtime harness. Fixed code with exposed knobs — no agentic planning, no self-critique, no LLM-as-judge routing. The model generates in exactly two places (an optional descriptor rewrite, and the answer); everything else is deterministic:

triviality gate — trivial prompts skip the controller and hit the bare base (α=0)
OOD controller — cosine distance to the nearest known training cluster drives RAG-need (tau_rag_on = 0.287) and expert count k (tau_k_escalate = 0.308). Distance is treated as novelty, not trust — it never scales the adapter.
distance-scaled RAG — retrieve, drop passages below a similarity floor, set granularity from the score-profile shape, pack within a token budget
ephemeral multi-expert loop — mint → verify → ACCEPT, or REJECT→escalate (mint from neighbor-cluster z's), or ABSTAIN→ship a single α-damped mint
global α backstop = 0.8 — fixed; not distance-driven, not self-perplexity-gated (both were measured and retired)
a z-consistency guard that refuses to boot if the runtime encoder doesn't reproduce the calibration cache (cosine ≥ 0.99)

What it actually does (measured, not asserted)

On the 265 held-out tasks, comparing the right-z adapter against the bare frozen base on identical tokens:

74% of tasks improve with the minted adapter (mean +0.63 nats)
the adapter is task-specific: right-z beats a wrong task's z on 80% of tasks
~19% of tasks shatter (the adapter makes it worse). On a frozen base you can't mint these away — so the harness catches shatter (reject → escalate) rather than predicting it. Nearest-cluster distance only weakly anticipates shatter (AUC 0.60), so it stays a novelty/k knob, never a trust gate.

This is the honest shape of the system: cheap, real specialization on most requests; a meaningful minority where the verifier and escalation loop earn their keep; grounding via RAG for novel requests.

Honest limitations

It's a 190M base. Output is simple and sometimes imperfect; greedy loops are suppressed with n-gram blocking, not eliminated by capacity. This is a specialization+grounding system, not a reasoning engine.
The verifier checks form, not correctness. It catches degeneracy, syntax/parse failure, and retrieval mismatch — it will accept code that compiles but is wrong, or prose that overlaps the retrieved passages. The strong execution rung only fires when a request carries its own tests.
Some executors are stubs. Python compile-checking is live; C++/Lean runners are shape-validated but not wired (they fail-safe to reject).

Files in this repo

ckpt_v12_190m_best.pt — the frozen 190M FFT-hybrid base (canonical)
hyper_ckpt_v12.best.pt — the trained 9.20M hypernetwork (use this; not the end-of-run checkpoint)
hyper/z_cache.pt — BGE z-cache the controller geometry is calibrated against
hyper_ckpt.bestMidRun7750.pt — earlier 612-task hypernetwork (reference only)
model_hybrid.py, muon.py, tok_v9.py — model code + tokenizer

Tasks and RAG artifacts live in the dataset repo Daxamite/eMOE-rag; the live demo (full harness, routing trace) runs at the Space Daxamite/eMOE.

Provenance note

An earlier draft of these notes quoted a VAL of ~2.09 — that belonged to a retired 1142-task run and does not describe this model. The correct held-out floor for hyper_ckpt_v12.best.pt is ≈2.84, consistent with the project's historical floor and verified by an independent help/shatter eval (mean adapted loss 2.86). The controller thresholds (0.287 / 0.308) were re-banked on this model's own z-geometry.

License: other (see repo). The base is an owned, curated model; the hypernetwork and harness are the eMoE contribution.