README.md

---
base_model: stepfun-ai/Step-3.7-Flash
base_model_relation: quantized
license: apache-2.0
library_name: mlx
pipeline_tag: image-text-to-text
tags:
- mlx
- oq
- quantized
- 2-bit
---

# Step-3.7-Flash-oQ2e

Data-driven mixed-precision quantization ([oMLX](https://github.com/jundot/omlx) **oQ2e**,
imatrix-calibrated, group size 128) of
[stepfun-ai/Step-3.7-Flash](https://huggingface.co/stepfun-ai/Step-3.7-Flash), in MLX format
for Apple Silicon. Runs in mlx-vlm and oMLX.

- **~59 GB** on disk (~2.4 bpw text backbone, down from 375 GB BF16)
- **Full VLM** — `step3p7` text backbone (45 layers, MoE with 288 experts / top-8) in oQ2e
  gs128; vision encoder in 8-bit gs64 (~2.2 GB)
- Peak memory in my tests: ~63 GB text-only, 65.5 GB with vision — plan for 96 GB+
- Converted and tested on a MacBook Pro M5 Max 128GB 40 GPU

## Quantization

oQ2e allocates bits per layer from an importance-matrix sensitivity pass over calibration
data, instead of quantizing everything uniformly. Group size 128 lowers metadata overhead
(~0.25 bpw saved vs gs64) at a small accuracy cost. The vision encoder is quantized
separately at 8 bits (group size 64), which is effectively lossless for vision towers.
Output is standard MLX affine quantization — no custom kernels or runtime required.

## Benchmarks & Variants

mmlu_pro, n=300, greedy. Standard error ≈ 0.03.

| Variant | Size | bpw | thinking on | thinking off | Observations |
|---|---|---|---|---|---|
| [Step-3.7-Flash-oQ3e](https://huggingface.co/mlx-community/Step-3.7-Flash-oQ3e) | 79 GB | ~3.3 | 0.77 | 0.57 | looks like this quant has impressive quality |
| [**Step-3.7-Flash-oQ2e**](https://huggingface.co/mlx-community/Step-3.7-Flash-oQ2e) (this repo) | 59 GB | ~2.4 | 0.68 | 0.54 | smaller and way cheaper on RAM, at a real cost in accuracy |
| [Step-3.7-Flash (API, fp8)](https://openrouter.ai/stepfun/step-3.7-flash) | — | — | 0.80 | n/a | StepFun endpoint via OpenRouter, `reasoning_effort: low`; no BF16 provider exists for this model |

**These numbers are not representative of what the model can do.** Thinking mode
wrecks the measurement: left unconstrained, the model wanders into 30k-token
reasoning chains, blows the token budget and returns no answer at all — which the
harness scores as wrong. So part of what looks like "quantization loss" here is
really "ran out of room before answering".

Not apples-to-apples either: the API row used `reasoning_effort: low`, which keeps
the model from running away, and my local runs had no such control. The API can't do
the "thinking off" column at all — the endpoint returns *"Reasoning is mandatory for
this endpoint and cannot be disabled"*. I'll run more tests later to make each
quant's quality relative to the API clearer.

I haven't benchmarked vision quality yet; the encoder passed my qualitative smoke tests.

## Thinking control

Step-3.7-Flash is a smart boi. It **really** likes to think. As a matter of fact, the
upstream always enables thinking and doesn't let you turn it off — the chat template
unconditionally opens a `<think>` block, and `reasoning_effort` barely moves the needle
(in my probes, `low`/`medium`/`high` modulate thinking length by ~2×, and the default
behaves like `max`).

I fixed this by implementing the off switch at the template level: when you pass
`enable_thinking: false` (or `reasoning_effort: "none"`), the template prefills an empty
`<think>\n</think>\n\n` block, so the model skips straight to the answer — no runtime
patches, no custom parser, works in any stack that forwards `chat_template_kwargs`.

| You pass | Behavior |
|---|---|
| *(nothing)* | Full thinking (default, longest) |
| `reasoning_effort: "low"` | Shorter thinking (weak effect, not a hard cap) |
| `enable_thinking: false` | **No thinking** — answers directly |
| `reasoning_effort: "none"` | Same as `enable_thinking: false` |

Heads up: skipping thinking costs accuracy on hard tasks (0.68 → 0.54 on mmlu_pro in my
runs) — but it also turns a 4-hour eval into a 3-minute one. Pick your poison.

Worth knowing: `reasoning_effort` is a weak lever here. In the open weights the chat
template just injects a `Reasoning: <level>` line into the system prompt — no control
token, no budget, nothing that forces the model to stop. It nudges, it doesn't govern.
The hosted API behaves the same way; I got the same effect by writing that line into
the system prompt myself.

What does work locally is a hard cap. oMLX's `thinking_budget` closes the `<think>`
block when the budget runs out, and the model takes the hint and answers. In my tests
every response came back with a closed block and an actual answer, even when the cut
landed mid-sentence.

## Changes from upstream

1. **`tokenizer_class` fixed** — upstream declares `LlamaTokenizerFast` for what is a
   ByteLevel BPE tokenizer, which breaks `decode` in `transformers` (literal `Ġ`/`Ċ` in
   output — it silently corrupted my eval scores until I caught it). This repo sets
   `PreTrainedTokenizerFast`, restoring correct encode and decode.
2. **Chat template extended** — `enable_thinking=false` / `reasoning_effort="none"` emit an
   empty-think prefill (see Thinking control). Everything else is unchanged.

## Usage

```bash
# text and vision (mlx-vlm — plain mlx-lm doesn't support the step3p7 architecture)
mlx_vlm.generate --model mlx-community/Step-3.7-Flash-oQ2e --prompt "..."
mlx_vlm.generate --model mlx-community/Step-3.7-Flash-oQ2e \
  --image photo.jpg --prompt "Describe this image."

# server — thinking off per request:
# POST /v1/chat/completions
#   {"chat_template_kwargs": {"enable_thinking": false}, ...}

# oMLX — discovers the model from the HF cache
omlx serve
```