Upload README.md with huggingface_hub

README.md

@@ -16,8 +16,8 @@ We put this repo bundles the loader
 package (`inkling_mlx/`), so it runs with just `mlx` + `mlx-lm` + `transformers`.
 
 > Community build for large-model-on-Apple-Silicon experiments. ~**315 GB** in 2-bit ->
-> needs a large / multi-Mac (distributed MLX) or SSD expert-offload setup; does not fit a
-> single 192 GB Mac resident.
+> doesn't fit a single 192 GB Mac *resident*, so run it via multi-Mac (distributed MLX) or
+> **SSD expert-offload** (see below — one 192 GB Mac, or even a 64 GB Mac slowly).
 
 ## Quantization (recipe: experts_only)
 
@@ -40,6 +40,39 @@ ids = tok("The capital of France is")["input_ids"]
 print(tok.decode(greedy_generate(model, config, ids, max_new_tokens=64)))
 ```
 
+## Run on a single Mac with SSD expert-offload
+
+~315 GB doesn't fit a 192 GB Mac resident — but an MoE only fires **6 of 256** experts per
+token, so you can keep the always-needed weights in RAM (attention, shared experts, embeddings,
+norms, router, vision/audio towers ≈ 35 GB) and **page the routed experts from SSD on demand**,
+letting the OS page cache hold the hot ones. This makes the 2-bit omni build runnable on one
+192 GB Mac Studio (and even generates on a 64 GB Mac, slowly).
+
+Tool: **[github.com/huckiyang/mlx-moe-offload](https://github.com/huckiyang/mlx-moe-offload)**
+(MIT) — a drop-in over mlx-lm's `SwitchGLU`. It also answers the mlx-lm feature request
+[ml-explore/mlx-lm#1438](https://github.com/ml-explore/mlx-lm/issues/1438).
+
+```bash
+pip install mlx mlx-lm scipy pillow
+git clone https://github.com/huckiyang/mlx-moe-offload && cd mlx-moe-offload && pip install -e .
+
+# 1) one-time: repack the stacked experts into a per-expert SSD store
+#    (copies the bundled inkling_mlx/ loader in, so the offload dir is self-contained)
+python -m mlx_moe_offload.repack --build /path/Inkling-mlx-2bit --out /path/Inkling-mlx-2bit-offload
+OFF=/path/Inkling-mlx-2bit-offload
+
+# 2) generate — text, image, or audio (omni)
+python examples/inkling_omni.py --offload-dir $OFF --prompt "The capital of France is"
+python examples/inkling_omni.py --offload-dir $OFF --image cat.png --prompt "What is in this image?"
+python examples/inkling_omni.py --offload-dir $OFF --audio q.wav   --prompt "Transcribe in English:"
+```
+
+The prompt is prefilled in small `--prefill-chunk` blocks so only a bounded set of experts is
+live at once (a lazy full-prompt prefill would page ~all experts per layer and OOM). Decode
+speed is gated by the routing hit-rate + SSD bandwidth — check `store.stats()`. At 2-bit,
+text is solid (*"…capital of Japan is Tokyo"*) while image/audio are recognizable but degraded
+by quantization; a higher-bit build sharpens multimodal quality.
+
 ## Attribution
 
 - Weights converted from `thinkingmachines/Inkling`.