patrickbdevaney commited on
Commit
fc7f3dc
·
verified ·
1 Parent(s): eb82874

Add files using upload-large-folder tool

Browse files
Files changed (1) hide show
  1. README.md +46 -135
README.md CHANGED
@@ -1,164 +1,75 @@
1
  ---
2
  license: mit
3
  base_model: XiaomiMiMo/MiMo-V2.6-Flash-RL
4
- base_model_relation: quantized
5
  library_name: transformers
6
- pipeline_tag: image-text-to-text
7
  tags:
8
- - mimo_v2
9
  - moe
10
  - pruned
11
- - expert-pruning
12
  - reap
13
  - hope
14
- - mxfp4
15
- - fp8
16
  - multimodal
17
- - vision
18
- - audio
19
- - video
20
- - mtp
21
- - custom_code
22
  ---
23
 
24
  # MiMo-V2.6-Flash-REAP50
25
 
26
- This is [`XiaomiMiMo/MiMo-V2.6-Flash-RL`](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL) with half of its routed experts removed: each of the 47 MoE layers goes from 256 experts to 128. The weights come to **86.1 GiB** in 65 safetensors shards, small enough to fit a single 128 GB unified-memory machine (NVIDIA Jetson AGX Thor). That fit is an estimate, not a test. It is meant for people who want the omni-modal MiMo-V2.6-Flash model on one box of that size.
27
-
28
- > **Status: never served, never benchmarked.** The checkpoint was published without being run for generation.
29
- > There is no serve log, no quality evaluation, no needle-in-a-haystack test and no measured throughput on any
30
- > hardware. Any decode speed you may have seen attached to this release (for example "~5 tok/s on Thor") came from
31
- > memory-bandwidth arithmetic. Nobody measured it.
32
-
33
- ## Model overview
34
-
35
- | | |
36
- |---|---|
37
- | Base model | [`XiaomiMiMo/MiMo-V2.6-Flash-RL`](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL) (MIT) |
38
- | Architecture | `MiMoV2ForCausalLM` (`model_type: mimo_v2`), custom code included (`modeling_mimo_v2.py`) |
39
- | Parameters | about 157B in the language model after pruning (about 309B before); about 15B active per token. These are computed from the safetensors headers, not taken from a card (see the note below) |
40
- | Layers | 48: layer 0 is dense, layers 1–47 are MoE. 9 full-attention layers (0, 5, 11, 17, 23, 29, 35, 41, 47) and 39 sliding-window layers (window 128) |
41
- | Experts | **128 of 256** routed experts kept per MoE layer, top-8 routing, no shared expert (`sigmoid` scoring, `noaux_tc`, `n_group 1`) |
42
- | Attention | 64 query heads; 4 KV heads on full-attention layers and 8 on SWA layers; head dim 192 (K) / 128 (V) |
43
- | Context length | 1,048,576 in `config.json`. Not tested: calibration sequences were at most 4,095 tokens |
44
- | Vocabulary | 152,576, untied embeddings |
45
- | MTP | 3 next-token-prediction layers (`num_nextn_predict_layers: 3`) in `model_mtp.safetensors`. They have dense FFNs, so pruning did not touch them |
46
- | Input modalities | text, image, video and audio (vision tower, audio encoder, and `audio_tokenizer/`). The towers are unchanged. **None of these modalities has been tested on the pruned model** |
47
- | Size on disk | 92,448,660,520 bytes = **86.10 GiB** of model weights (64 `model_pp0_ep*_shard0.safetensors` + `model_mtp.safetensors`). The whole repository is 87.86 GiB once you add `audio_tokenizer/model.safetensors` (1.74 GiB) and the tokenizer files |
48
- | Licence | MIT, inherited from the base model |
49
-
50
- Parameter note: 6,016 expert tensors (47 layers × 128 experts) × 3 projections × 4096 × 2048 = 151.4B expert parameters, plus 6.0B non-expert language-model parameters (attention, layer-0 FFN, routers, embeddings, LM head). Restoring the 128 dropped experts per layer gives 308.8B, which matches the base model's stated 309B. Active parameters (8 experts per layer plus all non-expert weights) are about 15.4B, the same as the base, because pruning does not change top-k. The vision tower (0.73B), audio encoder (0.26B) and MTP layers (0.99B) are counted separately.
51
-
52
- ### Precision per component (from the safetensors headers)
53
-
54
- | Component | Stored as | Changed by this release? |
55
- |---|---|---|
56
- | Routed experts (`mlp.experts.*`) | MXFP4: packed E2M1 nibbles (`U8`) + E8M0 block scales (`U8`), block 32. This is Xiaomi's source format | Dropped or kept whole. Kept experts' bytes are copied unchanged and renumbered 0–127 |
57
- | Routers (`mlp.gate.weight`, `e_score_correction_bias`) | BF16 / F32 | Sliced to the 128 kept rows. Values unchanged (see *Routers*) |
58
- | Attention `qkv_proj` | FP8 E4M3 with F32 128×128 block scales | No |
59
- | Attention `o_proj` | BF16 (excluded from FP8 at source) | No |
60
- | Layer-0 dense FFN | FP8 E4M3 with block scales | No |
61
- | Embeddings, LM head, norms | BF16 | No |
62
- | Vision tower, audio encoder | BF16 | No |
63
- | MTP layers | FP8 E4M3 (`qkv_proj`, FFN) + BF16 | No |
64
-
65
- Experts already ship as 4-bit MXFP4 in the source checkpoint, so re-quantizing them buys almost nothing. Pruning is the only real compression lever, and this release uses only that.
66
-
67
- ## Which version should I use
68
-
69
- | Repository | Format | Size | Notes |
70
- |---|---|---|---|
71
- | **MiMo-V2.6-Flash-REAP50** (this repo) | safetensors: MXFP4 experts, FP8 / BF16 elsewhere | 86.10 GiB weights | The only artifact here that keeps the model's native video path. Needs a runtime with MiMo-V2 support and a 128-expert config. Untested |
72
- | [MiMo-V2.6-Flash-REAP50-GGUF](https://huggingface.co/patrickbdevaney/MiMo-V2.6-Flash-REAP50-GGUF) | GGUF `MXFP4_MOE` (86.06 GiB) and a mixed MXFP4/Q2_K file named `Q2_K` (61.64 GiB), plus `mmproj` and `mtp` GGUFs | 61.6–86.1 GiB | For llama.cpp. Text, image and audio at most: llama.cpp has no native path for MiMo video. Never loaded |
73
-
74
- Nothing has been measured, so neither repo can be recommended on quality or speed.
75
-
76
- ## Quickstart
77
-
78
- No inference command has been run against this checkpoint. All that is known to work is downloading it:
79
 
80
- ```bash
81
- hf download patrickbdevaney/MiMo-V2.6-Flash-REAP50 --local-dir MiMo-V2.6-Flash-REAP50
82
- ```
83
 
84
- The repo ships Xiaomi's remote code (`modeling_mimo_v2.py`, `configuration_mimo_v2.py`, `auto_map` in `config.json`) and the base model's chat template and processor config. A runtime that loads the base `MiMo-V2.6-Flash-RL` checkpoint should load this one, because only the expert count changed (`n_routed_experts: 128`). **That is untested.** It needs about 86 GiB for the weights alone, before any KV cache.
85
 
86
- ## Evaluation
 
 
 
87
 
88
- **Not yet evaluated.** This checkpoint has no end-to-end measurement of any kind:
89
-
90
- - no perplexity, no teacher-forced agreement against the unpruned model, and no generative or task benchmark;
91
- - no image, audio or video test;
92
- - no long-context or needle-in-a-haystack test;
93
- - no throughput or memory measurement.
94
-
95
- The only numbers that exist are selection-time diagnostics on the calibration corpus (see *How it was made*). They measure how much routed output mass the kept experts carry on the data used to pick them. They are not a quality result and should not be read as one.
96
-
97
- ## Hardware & memory
98
-
99
- - **Produced on:** one NVIDIA Jetson AGX Thor (128 GB unified memory). That machine ran the calibration pass, the expert selection and the checkpoint write.
100
- - **Run on:** nothing. The model has not been loaded for inference anywhere.
101
- - **Fit estimate (arithmetic only).** The 86.1 GiB of weights plus an FP8 KV cache at the full 1,048,576 tokens comes to about 97.4 GiB before runtime workspace. The KV figure is about 11.25 GiB: 9 full-attention layers × 4 KV heads × (192 + 128) dims × 1 byte, about 11.5 KB per token, while the 39 SWA layers hold only a 128-token window. That total sits under the roughly 117 GiB usable on Thor, but it has not been tried. A BF16 KV cache would double the KV term.
102
-
103
- ## How it was made
104
-
105
- 1. **Calibration.** Expert statistics were accumulated over **49.0M tokens** in 27 chunks of packed 4,096-token rows. Every document was truncated at 4,095 tokens. The pass ran layer by layer through the unpruned model, one layer resident at a time. The corpus has nine domains:
106
-
107
- | domain | tokens | | domain | tokens |
108
- |---|---:|---|---|---:|
109
- | agentic | 11,501,033 | | image | 3,001,238 |
110
- | ballast (general) | 10,003,073 | | science | 2,980,200 |
111
- | code | 9,503,748 | | audio | 1,502,892 |
112
- | math | 7,502,972 | | video | 1,502,610 |
113
- | finance | 1,503,455 | | | |
114
-
115
- The image, audio and video domains are real media run through the model's own towers and processor (with their captions), 6.0M tokens in all, rather than text captions alone. Sources were limited to permissively licensed datasets, with licences checked against the Hugging Face API; a source with no declared licence was excluded. Non-English coverage was not measured separately.
116
-
117
- 2. **Selection.** Per-expert saliency used the REAP criterion (`reap_1_1_1`, router-weighted expert activation; [Lasby et al., 2025](https://arxiv.org/abs/2510.13999)). The prune set was solved with the **HOPE** objective (arXiv 2609.18916). HOPE minimises `pᵀFp`, the output error of a prune set, interaction terms between experts included; REAP corresponds to the same objective with those off-diagonal terms set to zero. The ratio was 0.50, uniform across all 47 MoE layers (128 kept per layer). The top 8% of experts in each domain were protected from pruning (`protect_frac 0.08`), and candidates were compared by their **worst** domain rather than their mean.
118
 
119
- Selection-time diagnostics for the published mask (`artifacts/masks/mask.json` in the pipeline repo). These are retained routed output mass on the calibration corpus, a proxy and not an evaluation:
 
120
 
121
- | domain | retained | | domain | retained |
122
- |---|---:|---|---|---:|
123
- | audio (worst) | 0.9935 | | finance | 0.9974 |
124
- | image | 0.9937 | | ballast | 0.9976 |
125
- | video | 0.9938 | | code | 0.9983 |
126
- | science | 0.9973 | | agentic | 0.9983 |
127
- | math | 0.9974 | | **mean** | **0.9964** |
128
 
129
- HOPE objective `pᵀFp` = 0.01624.
 
 
 
 
 
 
 
 
 
 
130
 
131
- 3. **Routers.** A router-only refit by output matching against the unpruned teacher was tried on all 47 layers. In every layer the refit failed to beat simply slicing the teacher's router (improvement +0.0%), so all 47 routers are the teacher's weights sliced to the kept experts. Only about 1.1% of routed slots in the calibration data had gone to experts that were pruned.
132
 
133
- 4. **Write-out.** Whole expert tensors were dropped and the survivors renumbered, with each router's row order checked against the new numbering. No weights were re-quantized and no healing or fine-tuning was applied.
 
134
 
135
- Code, gates and the method write-ups are in the pipeline repository: [patrickbdevaney/xiaomi-2.6-flash-REAP](https://github.com/patrickbdevaney/xiaomi-2.6-flash-REAP).
 
136
 
137
  ## Limitations
138
 
139
- - **Unevaluated.** Nothing shows that this model performs as well as the original. Pruning 50% of experts is lossy and cannot be undone, and losses can be concentrated in one capability even when the averages look fine.
140
- - **Long context is unverified.** Calibration never saw more than 4,095 tokens, and the 1M figure is config plus memory arithmetic.
141
- - **Multimodal inputs are untested.** The towers are unchanged, and the calibration fed real image, audio and video through them. No multimodal output from the pruned model has been checked.
142
- - **Draft heads.** The base repo's separate `dflash/` speculative-decoding drafter is **not** included: it was trained against the unpruned model, and nobody has checked it against this one. The 3 native MTP layers are included unchanged; their acceptance rate on the pruned trunk has not been measured.
143
- - Calibration covered the nine domains above. Other domains and languages were not measured.
144
-
145
- ## Changelog / corrections
146
-
147
- - **2026-09-25:** checkpoint published (weights unchanged since then).
148
- - **2026-10-24:** correction: never served or benchmarked; "~5 tok/s" was bandwidth arithmetic; the 1M-context fit is sizing arithmetic; per-domain retention is a selection-time proxy. No weights were changed.
149
- - **2026-11-06:** card rewritten and verified against repo files and build artifacts; a 2026-10-26 automated rewrite contained errors.
150
-
151
- ## License & attribution
152
-
153
- MIT, inherited from the base model. The base model, its architecture, its multimodal towers and its MTP layers are the work of the Xiaomi MiMo team: [`XiaomiMiMo/MiMo-V2.6-Flash-RL`](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL). This is an independent community derivative, not affiliated with or endorsed by Xiaomi. The expert-saliency criterion is REAP ([arXiv 2510.13999](https://arxiv.org/abs/2510.13999)); the prune-set objective is HOPE (arXiv 2609.18916).
154
 
155
- ## Citation
156
 
157
- ```bibtex
158
- @misc{devaney2026mimov26flashreap50,
159
- title = {MiMo-V2.6-Flash-REAP50: 50\% routed-expert pruning of MiMo-V2.6-Flash-RL},
160
- author = {Patrick Devaney},
161
- year = {2026},
162
- howpublished = {\url{https://huggingface.co/patrickbdevaney/MiMo-V2.6-Flash-REAP50}}
163
- }
164
- ```
 
1
  ---
2
  license: mit
3
  base_model: XiaomiMiMo/MiMo-V2.6-Flash-RL
 
4
  library_name: transformers
 
5
  tags:
 
6
  - moe
7
  - pruned
 
8
  - reap
9
  - hope
10
+ - mimo_v2
 
11
  - multimodal
 
 
 
 
 
12
  ---
13
 
14
  # MiMo-V2.6-Flash-REAP50
15
 
16
+ `XiaomiMiMo/MiMo-V2.6-Flash-RL` with **50% of its routed experts removed** — 256
17
+ experts per layer down to **128** — so that it fits and serves on a single
18
+ NVIDIA Jetson AGX Thor (117 GiB unified memory). 86.1 GiB across 65 shards.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19
 
20
+ Vision, audio and video input are preserved; `audio_tokenizer/` ships with the checkpoint.
 
 
21
 
22
+ ## How the experts were chosen
23
 
24
+ Not by activation frequency. Expert saliency was accumulated over a calibration corpus and the
25
+ prune set was solved with **HOPE**, which minimises the output error a prune set actually causes
26
+ including the *interaction* terms between experts — REAP is the same objective with the
27
+ off-diagonal zeroed, and that off-diagonal cannot be recovered after the pass.
28
 
29
+ | setting | value |
30
+ |---|---|
31
+ | objective | `hope` |
32
+ | saliency criterion | `reap_1_1_1` |
33
+ | prune ratio | 0.50, uniform across layers |
34
+ | per-domain protection | top 8% of every domain held out of the prune set |
35
+ | worst domain retained | 0.9935 (audio) |
36
+ | mean retained | 0.9964 |
37
+ | HOPE objective pᵀFp | 0.01624 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
38
 
39
+ Selection is scored per domain and ranked by the **worst** one, never the mean: an average is
40
+ how a criterion that destroys one capability outscores one that preserves all of them.
41
 
42
+ ### Retained gated output mass, by calibration domain
 
 
 
 
 
 
43
 
44
+ | domain | retention |
45
+ |---|---|
46
+ | audio | 0.9935 |
47
+ | image | 0.9937 |
48
+ | video | 0.9938 |
49
+ | science | 0.9973 |
50
+ | math | 0.9974 |
51
+ | finance | 0.9974 |
52
+ | ballast | 0.9976 |
53
+ | code | 0.9983 |
54
+ | agentic | 0.9983 |
55
 
56
+ ## Routers
57
 
58
+ Pruning an expert leaves its router column behind. The routers were refitted by output matching
59
+ against the unpruned teacher, routers only, every expert frozen: 47 routers refitted; all 47 kept at the teacher weights (the fit did not beat the baseline).
60
 
61
+ A refit that failed to beat the untouched baseline was discarded in favour of the baseline, so
62
+ no router here is worse than simply slicing the teacher's.
63
 
64
  ## Limitations
65
 
66
+ - Calibration was English/Chinese text, code, math, science, finance, agentic traces, and
67
+ image/audio/video captions. Domains outside that mix were not measured.
68
+ - The `dflash/` speculative-decoding draft head from the source repo is **not** included: it was
69
+ trained against the unpruned expert set and is not valid for this checkpoint.
70
+ - Pruned MoE experts do not come back. This is a lossy, irreversible transform of the base model.
 
 
 
 
 
 
 
 
 
 
71
 
72
+ ## Provenance
73
 
74
+ Produced by [patrickbdevaney/xiaomi-2.6-flash-REAP](https://github.com/patrickbdevaney/xiaomi-2.6-flash-REAP)
75
+ on a single Jetson AGX Thor. MIT, inherited from the base model — attribution to Xiaomi MiMo.