hermitdave/Agnes-3.0-Flash-DWQ-4bit-MTP

Agnes-3.0-Flash DWQ 4-bit (MTP, No Vision)

Agnes-3.0-Flash is a 33B hybrid architecture (54 delta-recurrent + 18 global attention layers). This variant uses DWQ (Data-aware Weight Quantization) at 4-bit with a group size of 64, plus a native MTP (Multi-Token Prediction) head for speculative decoding. The vision tower has been stripped for text-only inference.

⚠️ Fixed 2026-10-16 — MTP head repair. The initial upload shipped a malformed MTP head that made the model fail to load in oMLX (Expected shape (19456, 640) but received shape (21504, 640)). The broken head has been replaced with the correct one and the model now serves with working speculative decoding at 30 tok/s (2.7× over baseline). If you downloaded this model before that date, re-download it. Full details in MTP head repair.

What is DWQ?

DWQ is a data-aware quantization method that goes beyond simple round-to-nearest (RTN). Instead of blindly snapping each weight to the closest quantization grid point, DWQ considers the actual distribution of activations in the model. It uses this information to make smarter choices about how weights are scaled and shifted before quantization.

How DWQ compares to RTN

Aspect	RTN	DWQ
Method	Round each weight to nearest grid point	Data-aware scaling + rounding
Quality at 4-bit	Noticeable degradation	Closer to bf16 baseline
Outlier handling	Outliers dominate the grid	Outliers are managed via data-aware scaling
Implementation	Trivial	Requires calibration data
Speed impact	None	None (same inference speed)

The practical result: DWQ at 4-bit typically preserves more of the model's original capability than RTN at the same bit width. You get a smaller model without giving up as much quality.

Model Details

• Architecture: Qwen3_5ForConditionalGeneration (hybrid: GatedDeltaNet + full attention)
• Parameters: ~33B
• Quantization: DWQ 4-bit, group_size=64, affine mode (uniform — no per-tensor overrides)
• MTP head: 1 layer, FFN 19456 (enables speculative decoding)
• Vision: Stripped (text-only)
• Context length: 262,144 tokens
• Model size: ~18.5GB (quantized, no vision)

MTP head repair

The MTP head in the initial upload was malformed and the model would not load in oMLX:

• mtp.gate_proj / up_proj were [21504, 640] — the main model's FFN is 19456
• mtp.down_proj was [5120, 4480] with scales [5120, 2352] — 2048 extra columns, giving an invalid group size

oMLX rejected it with Expected shape (19456, 640) but received shape (21504, 640) and [quantize] The requested group size 8 is not supported.

Fix: the MTP shard (model-00005-of-00005.safetensors) was replaced with the correct head from the working oQ4e graft — a uniform 4-bit, group-64 MTP head with FFN 19456 that matches the main model. Tensor names are identical, so the index weight map is unchanged.

Benchmarks

Measured on M3 Max 64GB with oMLX, 512-token generations, MTP depth 4:

Metric	Value
Baseline (no MTP)	~11.1 tok/s
With MTP (depth 4)	~30 tok/s
MTP acceptance rate	68–71%
Tokens per cycle	2.4–2.7
Speedup	~2.7×
MTP overhead	~1.5% of step time

Depth-4 acceptance breakdown: d1 ≈ 74%, d2 ≈ 68%, d3 ≈ 67%, d4 ≈ 72%.

Note: ANE prefill is best disabled for this model on a 64GB machine — with it enabled the load footprint is ~31GB and the process can be OOM-killed; with it off the model loads at ~17.3GB.

Usage

With MLX-LM

from mlx_lm import load, generate

model, tokenizer = load("hermitdave/Agnes-3.0-Flash-DWQ-4bit-MTP")
response = generate(model, tokenizer, prompt="Hello, how are you?")

With oMLX

The model is served via oMLX with MTP speculative decoding enabled. Send requests to the /v1/chat/completions endpoint.

Attribution

• Base model: Agnes-3.0-Flash by Agnes AI
• Quantization: DWQ 4-bit (this repo)
• MTP head: grafted from the oQ4e build (Agnes native head, FFN padded to 19456)
• MTP repair & validation: Hermit Dave with Hermes Agent

License

Apache 2.0