huzey/Qwen3.5-9B-MLP-Cache

MLP Cache for Qwen3.5-9B

Pretrained Routers and MLP predictor tables for the 1 GiB/layer and 4 GiB/layer paper configurations. Download a saved cache and run image/question inference; Router training and cache construction are not required.

Full usage guide: Router loading, table format and inference

Quick start

Use Linux, Python 3.12 and a CUDA GPU for end-to-end inference. The model runs on the GPU; selected INT8 predictors reside on the CPU. Use sufficient host RAM and fast local storage for the chosen cache. The reference backend was smoke-tested on H100; this is separate from reproducing all paper benchmarks.

1. Install the runtime

The installable source bundle is provided here while the GitHub repository address is being finalized. It contains the same public mlp-cache-paper 0.2.0 code.

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install 'mlp-cache-paper[inference] @ https://huggingface.co/huzey/Qwen3.5-9B-MLP-Cache/resolve/v1.1/code/mlp-cache-github-v0.2.0.zip'

The inference extra pins the framework packages used by the public inference release. Use a CUDA-compatible PyTorch build and driver; see the environment notes.

2. Download a cache and its Routers

mlp-cache download-cache \
  --repo huzey/Qwen3.5-9B-MLP-Cache \
  --revision v1.1 --budget 1 --output downloads/cache-1gib

This downloads all 32 layers of the selected variant, both trained Routers for every layer, the paper configuration, and checksums. It verifies the downloaded files. No Hugging Face token is needed for public downloads.

3. Ask a question

mlp-cache infer \
  --model Qwen/Qwen3.5-9B \
  --cache downloads/cache-1gib --budget 1 \
  --image /path/to/image.jpg \
  --question "What is in this image? Explain briefly." \
  --max-new-tokens 64 --output outputs/answer.json

The command downloads the pinned original model separately, attaches the saved tables and returns the answer plus overall, Vision/Other and per-layer hit counters. The default configuration is downloads/cache-1gib/configs/paper.json.

For 4 GiB/layer, use --budget 4 in both commands and a new directory such as downloads/cache-4gib. An existing matching local model directory can replace the base-model ID.

What is included?

Files	Contents
routers/layer_00 … routers/layer_31	Per-layer vision.safetensors, other.safetensors and manifest.json: 64 trained Router checkpoints
cache_1gib/layer_00 … layer_31	Predictor payloads, prefix/coordinate/support controls and manifests for error target 0.001
cache_4gib/layer_00 … layer_31	Predictor payloads, prefix/coordinate/support controls and manifests for error target 0.003
configs/paper.json	28 enabled layers; disabled layers are 0, 1, 3, 31
release.json, _SUCCESS	Model identity, file inventory, SHA-256 values and release fingerprint
code/	Installable public source bundle; no original development Git history

Both variants retain all 32 layers and share the same Routers. The layer selection is a runtime configuration, independent of which layers are distributed.

Variant	All-32 predictor payload	Complete variant + shared Routers, approximately
1 GiB/layer	20.29 GiB	21.5 GiB
4 GiB/layer	109.44 GiB	111.6 GiB

The budget is a per-layer ceiling, not an assertion that every layer fills it. Original model weights and runtime memory are additional. Both variants together with their shared Routers occupy about 132.1 GiB.

How the Router and table work together

At an enabled layer, the modality-specific Router maps an MLP input to 127 spectral scores and a support logit. The modality bit and score signs form the 128-bit lookup address. A token is served by a saved predictor only if the learned gate, prefix lookup, support interval and finite-value checks pass. The runtime decodes the selected INT8 coefficients on the CPU; other tokens execute the original MLP.

Use the supplied runtime to preserve the table layout, calibrated gate thresholds, score scale and fallback behavior. The usage guide also shows how to load an individual Router and inspect a layer's table files.

Version and license

Use v1.1 consistently for this guide and downloads. It adds documentation and the source bundle; Router weights, predictor payloads, table controls and paper configuration are unchanged from v1. The immutable v1 tag remains available. Do not mix release.json or _SUCCESS from different revisions in one cache directory.

The distributed code, Routers and cache artifacts use MIT. The separately downloaded original model and datasets retain their upstream licenses.