Mati-3.7-173B-4.6bit-MLX

A REAP-pruned, mixed-precision-quantized MLX build of stepfun-ai/Step-3.7-Flash, built to run comfortably on a 128GB Apple Silicon Mac. Released under the base model's Apache 2.0 license.

~173B logical parameters (201B base, 245/288 experts kept per layer via 15% REAP). 4-bit routed experts; 8-bit shared expert / dense layers / router gates / output head (lm_head) + token embeddings; BF16 vision tower. ~93.5GB on disk, ~4.68 bits/weight effective (not uniform 4-bit -- see Quantization below). A slightly smaller 4-bit-head build (4.658 bpw) is on the shared8-4bit-head branch -- see Variants.

Every claim below is a measured result, not an assumption — see Recipe & measurements for the evidence trail.

Variants (two downloadable revisions)

revision	head precision	bpw	download
main (default)	8-bit lm_head/embed_tokens	~4.68	default — LM Studio and mlx-vlm get this
shared8-4bit-head (branch)	4-bit	4.658	--revision shared8-4bit-head

main is the recommended build: it bumps the output head + token embeddings to 8-bit for a measured text gain (−0.0086 NLL overall, mostly on tool_use/coding) over the 4-bit-head build, at +0.53 GB. The shared8-4bit-head branch is that slightly smaller 4-bit-head build (4.658 bpw); it is the one additionally validated on the 250-image multimodal set (main's head change is text-PPL validated, and quantizing an already-near-lossless head to 8-bit is not expected to affect vision — but it was not separately re-measured there). To fetch the smaller build:

huggingface-cli download True2456/Mati-3.7-173B-4.6bit-MLX --revision shared8-4bit-head
# or in code:  snapshot_download(repo_id=..., revision="shared8-4bit-head")

Most runtimes (including LM Studio) download main unless a revision is specified.

What this is

• Base: Step-3.7-Flash — StepFun VLM, ~201B logical parameters measured directly from the released checkpoint's tensor shapes (commonly quoted as "198B"), 45 decoder layers (dense 0–2, MoE 3–44), 288 routed experts/layer, top-k=8, 1 shared expert, sigmoid router, SwiGLU, RMSNorm.
• Pruned size: 190B of the base's ~201B params live in routed experts. Removing 43/288 per layer (15%) brings the total to **173B logical parameters** -- independent of quantization, which only changes storage size, not parameter count.
• Pruning: REAP at 15% (245/288 experts kept per layer), using a vision-blended saliency — text and image calibration saliency combined per-layer (0.7·text + 0.3·vision, mass-normalized) rather than text-only, to avoid pruning experts that specialize on image tokens. Measured to cut vision-saliency mass lost to pruning by ~31% (9.42% → 6.51%) versus a text-only REAP plan, at no measurable text-quality cost.
• Quantization: affine 4-bit (group size 64) on routed MoE experts; affine 8-bit on the shared expert, the three dense (non-MoE) decoder layers, all router gates, and (on main) the output head + token embeddings — these always-on, non-sparse weight classes turned out to carry almost all of the quantization damage. Vision tower left BF16, unquantized.

Calibration methodology

The REAP saliency underlying this checkpoint's expert-keep decisions was collected in two passes, both on rows drawn from the front of the calibration set (rows 5000+ were held out for evaluation and never touched during calibration -- see below):

• Sample-size validation (5k prompts): an initial saliency collection over 5,000 prompts, compared against a 1,000-prompt subset. Discarded router-saliency mass agreed to within 0.05% between the two -- confirming saliency stabilizes by roughly 1k prompts, well under 5k.
• Final saliency run (2500 prompts): the actual saliency this checkpoint's REAP plan is built from -- 2,500 prompts, max_tokens=1024, headtail truncation (keeps both task setup and answer for long agentic/tool-use prompts, see docs/FINDINGS.md section 3a for why head truncation alone was measured insufficient), 42 MoE layers, ~1.4h runtime. Separately, 300 images (200 ChartQA + 100 VQAv2) were run through the real vision path for the vision-blend saliency (see Pruning below).

2,500 text prompts is roughly 2.5x past the point sample-size validation showed saliency had already stabilized -- not a number picked without checking, and not the same run as the 5k validation pass. Raw saliency scores for every kept/pruned expert are in artifacts/*.json (see Full evidence trail).

Recipe & measurements

All measured on 500 held-out text prompts (rows 5000+ of the calibration set, never used for saliency collection or calibration) and 250 held-out images (ChartQA test split + VQAv2 validation, both disjoint from every calibration/saliency pass -- calibration used ChartQA train and the first 300 VQAv2 rows). head truncation, raw text (no chat-template double-wrapping) -- see docs/FINDINGS.md section 10b for two methodology bugs that were caught and fixed before any of these numbers were trusted.

Text PPL by category

category	BF16	REAP p15, uniform 4-bit	+ vision-blend saliency	+ 8-bit shared/dense/router (this model)
agentic	6.796	7.369	7.078	6.921
coding	8.016	7.540	7.429	6.488
general_instruction	6.184	6.002	6.248	5.115
reasoning_math	2.472	2.513	2.520	2.485
tool_use	43.855	36.662	36.621	27.177
OVERALL	6.653	6.607	6.554	5.930

Multimodal answer-token NLL by category (250 held-out images)

category	REAP p15, text-only saliency	+ vision-blend saliency	+ 8-bit shared/dense/router (this model)	answer tokens
chartqa	9.0017	8.9659	8.9489	438
vqa_natural	6.8574	6.7274	6.6529	141
OVERALL	8.4795	8.4208	8.3898	579

Where the 8-bit precision policy came from

Tested 7 variants against the vision-blend baseline: 6 windows of routed experts at 6-bit (one 7-layer band at a time, spanning the full depth), and one variant bumping the shared expert + 3 dense layers + all router gates to 8-bit. ΔNLL vs. the vision-blend baseline, per category:

variant	target	agentic	coding	gen_instr	reas_math	tool_use	OVERALL
w0	layers 3-9 (experts)	+0.006	+0.126	+0.084	+0.004	+0.243	+0.066
w1	layers 10-16 (experts)	+0.045	+0.052	+0.116	-0.000	+0.055	+0.048
w2	layers 17-23 (experts)	-0.012	-0.001	+0.002	-0.001	-0.006	-0.004
w3	layers 24-30 (experts)	-0.006	+0.005	+0.011	-0.000	+0.020	+0.003
w4	layers 31-37 (experts)	-0.009	+0.001	-0.006	+0.001	+0.009	-0.002
w5	layers 38-44 (experts)	+0.002	-0.002	+0.005	-0.002	+0.006	+0.001
shared8	shared+dense+router	-0.022	-0.135	-0.200	-0.014	-0.298	-0.099

Every depth-localized expert bump landed inside measurement noise, no consistent sign. Only the always-on, non-sparse weight class (shared expert / dense layers / router gates -- weights with no top-k sparsity to smooth over quantization noise the way routed experts get) showed a real, consistent, cross-category improvement -- about 10x any depth-window effect, for roughly +0.03 bpw / +1GB.

Capability validation (exact-answer accuracy, not perplexity)

Perplexity is a proxy: it rewards a confident distribution, which is not the same as a correct one. To check that the numbers above reflect real capability, this model was also scored right/wrong on exact-answer tasks where a good proxy cannot substitute for a correct answer — greedy decoding, all sampler fields pinned:

test	items	score
multi-step arithmetic + factual recall	24	24/24
tool selection + argument extraction (from a provided catalog)	15	14/15

These are spot checks, not a benchmark suite, but they confirm the model reasons and calls tools correctly, not merely that it has low perplexity. Item sets and per-item results: artifacts/acc_*.json, artifacts/tc_*.json.

Also tested and rejected: expert merging (REAM)

Merging the 43 lowest-saliency experts per layer into their nearest kept neighbour (instead of pruning them) lowered perplexity substantially — overall text NLL −0.194, with tool_use perplexity dropping 41% (26.5 → 15.6). On the exact-answer tests above, however, the merged model scored identically (24/24 → 23/24, tool-call 14/15 → 14/15): the perplexity gain was distribution smoothing with no capability behind it. A clean demonstration that perplexity can move without capability following — and the reason pruning, not merging, was shipped. Full write-up in docs/REAM-RESULT.md.

Also tested and rejected: per-expert mixed-precision tiering

Splitting each layer's 245 kept experts into 61 hot (6-bit) / 123 base (4-bit) / 61 cold (3-bit) banks by saliency rank made quality worse (overall text PPL 6.55 -> 7.35, +0.114 NLL) and the checkpoint came out larger (4.868 bpw vs. 4.632) -- the 3-bit cold bank's damage exceeded the 6-bit hot bank's gain. This architecture's expert saliency is too flat (no dead experts, no expendable low-saliency tail) for per-expert precision tiering to pay for itself. Full breakdown in docs/TIERED-EXPERTS-RESULT.md.

Full evidence trail

Every number above, plus the complete methodology, is included in this repo:

• docs/FINDINGS.md -- master findings doc: REAP ratio selection, the vision-blindness discovery, two calibration bugs found and fixed, DWQ investigation (and why it was shelved), quantization format decision.
• docs/PPL-DECOMPOSITION.md -- isolates reap-pruning damage from quantization damage (three-way BF16 / full-4bit / p15-4bit comparison).
• docs/VISION-BLEND-RESULT.md -- the vision-blended saliency result.
• docs/TOMOGRAPHY-RESULT.md -- the full 7-variant precision sweep.
• docs/SHARED8-RESULT.md -- this model's final validation.
• docs/TIERED-EXPERTS-RESULT.md -- the rejected per-expert tiering experiment.
• docs/REAM-RESULT.md -- the rejected expert-merging experiment and the perplexity-vs-capability demonstration behind it.
• docs/TOKENIZER-INVESTIGATION.md -- why the numeric behaviour is intrinsic tokenization and not a checkpoint defect (and why the base tokenizer must not be copied in).
• artifacts/acc_*.json, artifacts/tc_*.json -- per-item results for the exact-answer accuracy and tool-call spot checks, with artifacts/accuracy_eval.py and artifacts/toolcall_eval.py.
• artifacts/*.json -- raw per-category NLL/PPL from every eval run referenced above, and plan_p15_blend03.json, the exact REAP plan (per-layer kept/pruned expert indices and blended scores) used to build this checkpoint.
• artifacts/saliency-text-2500.json, artifacts/saliency-vision-300.json -- the raw per-expert, per-layer REAP saliency from the two source runs (2500 text prompts, 300 images) that plan_p15_blend03.json's blended scores were computed from.

Usage

MLX format — works with LM Studio (search this repo name) or directly via mlx-vlm:

pip install mlx-vlm
python -m mlx_vlm.generate --model <this-repo> --prompt "Describe this image" --image path/to/image.png

Sampling defaults

This repo ships a generation_config.json. The upstream checkpoint has none, which means every runtime falls back to its own defaults — and the generic ones are a poor fit for this tokenizer:

{"temperature": 0.5, "top_p": 0.9, "top_k": 100, "min_p": 0.05, "repetition_penalty": 1.0}

repetition_penalty is pinned to 1.0. It is the only common sampler setting that reorders candidates rather than merely filtering them, so it is the riskiest knob to raise — but note it was measured not to be the cause of the numeric behaviour described under Numeric handling below (a direct A/B at 1.02 vs 1.0 was identical). Keep it at 1.0 as a safe default; use min_p for loop suppression, raising it to 0.1 before reaching for a penalty.

Temperature is deliberately below StepFun's suggested 0.7–1.0, tuned for prose diversity rather than tool calls or structured output. Raise it for open-ended writing; keep it low for tool calls and anything numeric.

Numeric handling — a real bug, fixed 2026-07-26

Earlier revisions of this checkpoint (and this README) shipped a tokenizer that silently corrupted numbers in serving — 2456 → 2 4 5 6, 12.0 → 1.2.0, sed -n '447,495p' → sed -n '4.4.7,4.9.5p' — and this README described it as an intentional, unfixable property of the base model's tokenization. That was wrong. It was a real bug, and it is fixed in this checkpoint.

Root cause: tokenizer_config.json declared "tokenizer_class": "LlamaTokenizerFast". That is a name transformers recognizes as a known architecture, so AutoTokenizer.from_pretrained() — which is exactly what mlx_lm (LM Studio's backend) calls — applied Llama's own hardcoded SentencePiece/Metaspace tokenizer conversion instead of reading this model's actual, correct, custom pretokenizer out of tokenizer.json. That substitute silently drops spaces on raw text and returns nothing for Chinese input, and it fragmented every digit into its own token, which is what produced the corruption above.

Proven three ways: the genuinely-shipped tokenizer.json, read directly (bypassing transformers entirely), round-trips English, Chinese, and digit-heavy strings perfectly — 18452 → 2 tokens ['184','52'], grouped, not fragmented. The same file loaded via AutoTokenizer breaks exactly as described above. The same file wrapped in a generic tokenizer class (no class-name dispatch) is correct again. Only the declared class name differs between the broken and working cases.

The fix: two keys deleted from tokenizer_config.json (tokenizer_class, fix_mistral_regex) — no weight or model change. With no declared class, AutoTokenizer falls back to a generic tokenizer backend that trusts tokenizer.json's own pretokenizer.

Verified with live generation, not just tokenizer round-trips — same endpoint, temperature 0, all sampler fields pinned:

case	before fix	after fix
kill process ID 18452 in a MAC-address-dense context	0/12	5/5
Echo 2456, 1337, 495 in the same context	0/12	5/5

Why this stayed hidden: most tooling that serves this model — vLLM, llama.cpp's GGUF conversion, anything reading tokenizer.json directly — never routes through AutoTokenizer's class-name dispatch, so it never hits this. It specifically requires the AutoTokenizer.from_pretrained() path that mlx_lm/LM Studio uses. StepFun's own upstream tokenizer_config.json carries the same misleading field — this was not introduced by the pruning or quantization work in this repo, though an earlier fix attempt here (documented in the source repo's docs/TOKENIZER-INVESTIGATION.md) made a different mistake before the real cause was found.

If you pulled this checkpoint before 2026-07-26, re-download tokenizer_config.json (a few KB, not the model weights) or re-run from_pretrained to pick up the fix.

Practical guidance going forward: the fix is verified but young — for numeric/tool-call-heavy agentic use, consider still verifying critical numeric arguments programmatically rather than trusting model self-review, since that pattern (the model rationalizing rather than doubting its own wrong output) was observed during the earlier investigation independent of this specific bug. A helper (scripts/numeric_guard.py) and a Pi editor extension for exactly this are included in the source repo, now positioned as defense-in-depth rather than a required workaround.

Reasoning

Step-3.7 is a <think>-style reasoning model, with one wrinkle that trips up most reasoning parsers.

The chat template opens the think block for you. With add_generation_prompt=True, the prompt ends with:

<|im_start|>assistant
<think>

So the model never emits an opening <think> tag — it generates reasoning immediately and emits only the closing </think>. A parser configured to look for a balanced <think>…</think> pair will not detect any reasoning and will render the raw chain-of-thought as visible output. Configure your reasoning parser to treat the response as starting inside the think block, or to split on </think> alone. Both tags are single tokens (<think> = 128798, </think> = 128799), as are <|im_start|> (128006), <|im_end|> (128007), and <tool_call> (128008).

Reasoning effort is a system-prompt line. If the template is rendered with a reasoning_effort variable, it is interpolated verbatim into a Reasoning: <value> line at the top of the system message. It is not a special token and the checkpoint does not enumerate the accepted values — those come from StepFun's base-model documentation.

Reasoning is dropped from history at the next user turn. The template re-renders the <think> block only for assistant messages after the last real user query; earlier assistant turns are rendered with their reasoning stripped. Note the "last user query" scan skips <tool_response> messages, so reasoning is carried across a multi-step tool-calling loop and discarded only when a genuinely new user message arrives. If you manage history yourself, an assistant message may carry a reasoning_content field; the template falls back to splitting content on </think> when it is absent.

Tool calls are XML, not JSON. The template instructs the model to emit <tool_call><function=name><parameter=key>value</parameter></function></tool_call>. This is a separate path from JSON-schema structured output — see Known limitations.

Known limitations

• Multimodal improvement (vision-blend + shared8 vs. text-only baseline) is measured on 579–600 answer tokens across 250 images — real and consistent in direction across categories, but treat the exact magnitude as indicative rather than precise.
• No BF16 multimodal reference was run, so absolute (not just relative) vision-task damage from quantization remains unmeasured.
• This checkpoint has not been evaluated on standard downstream benchmarks (MMLU, GSM8K, SWE-bench, etc.). The category tables are perplexity/NLL proxies; the Capability validation section adds small exact-answer accuracy spot checks (39 items), but these are not a substitute for a full benchmark suite.
• JSON-schema structured output can crash llguidance-based runtimes. This is a base-model tokenizer interaction, not a defect introduced by the pruning or quantization here — it reproduces with zero model weights loaded, on the stock stepfun-ai/Step-3.7-Flash tokenizer. A merged token decoding to ": collides with grammars that use a ": " key separator (as LM Studio's auto chat-title schema does), and the bitmask generator and token consumer contradict each other. Filed upstream as guidance-ai/llguidance#366. Native XML tool calling (see Reasoning) is unaffected.
• The original checkpoint's MTP (multi-token-prediction) weights (layers 45–47) are not included — this build only implements the main 0–44 decoder stack, matching current mlx-vlm support for this architecture.

Attribution

Base model: stepfun-ai/Step-3.7-Flash, Apache 2.0. This repository is a derivative work under the same license; see LICENSE for the full text. No affiliation with StepFun.