README.md

metadata

title: HyperPEER
emoji: 📈
colorFrom: blue
colorTo: red
sdk: gradio
sdk_version: 6.19.0
python_version: '3.13'
app_file: app.py
pinned: false
license: mit
short_description: Compressing big models by generating their experts

HyperPEER

Compressing a large pretrained model into a small student that runs on a single 16 GB consumer GPU — by replacing each transformer layer's feed-forward / mixture-of-experts block with a hypernetwork that generates a per-token low-rank expert instead of storing a full expert bank. Attention and embeddings are inherited and frozen; only the generator is trained, by feature distillation against the teacher's per-layer outputs. The model's footprint becomes the size of the generator, not the size of everything it can generate.

This Space holds the method, the code, and the validation results. The proof-of-concept target is google/gemma-4-26B-A4B, compressed to run on one consumer card, with the success criterion that the result outperforms other models of its size.

The result that matters

On a 3B testbed (StarCoder2-3b), entirely on a single 16 GB card:

• Generating the experts costs no quality versus storing them. At convergence the hypernetwork that synthesizes its experts reached held-out perplexity 25.9, versus 26.2 for a matched student that stores its experts outright. The earlier gap was purely a matter of training length.
• Capacity must live in the generator. A larger hypernetwork making smaller experts beats a smaller hypernetwork making larger experts, monotonically — a generated expert cannot carry more information than the thing that generated it.
• Feature distillation wins. Matching each block's output to the teacher's, layer by layer, beat next-token prediction and logit-KL by a wide margin.
• It runs light. The compressed student used about 2.85 GB of VRAM — under half the teacher's — at a third of the parameters, and ran faster in the throughput-bound regime.

Why backprop through a generator works

You generate the expert's weights, run the input through them, and the error travels back through both networks. It composes cleanly: the generated weights are just activations in the middle of the computation, and the chain rule passes through them like any other intermediate value. The only thing that breaks it is hard, discrete expert selection — so the experts are generated smoothly rather than picked.

What's here

• gemma/ — the Gemma-4-26B pipeline: the hypernetwork expert (gemma_hyper.py), the bf16 layer-streaming capture (capture_gemma.py), the layer-local feature-distillation trainer (train_layers.py), the staged capture→train loop (staged_run.py), and the streamed non-repeating corpus (gemma_corpus.py).
• testbed/ — the 3B validation: the hypernetwork expert (hyper_expert.py), the trainers (train_hyper.py, train_compress2.py), the speed benchmark (bench_speed.py), and the result JSONs (the 30M-token convergence run and the capacity-vs-rank sweep).
• PHASE1_REPORT.md, PHASE2_PLAN.md — the capture report and the full Gemma plan.

Status

The recipe is validated end to end on 3B. The remaining step is the Gemma-4-26B run, where the blocker is purely compute: a 26B teacher does not fit a 16 GB card for an efficient capture-and-train pass, and on Blackwell consumer hardware the usual 4-bit shortcuts fail. Everything will be released openly — the compressed model, the recipe, and the writeup.

— Mikey Bee