Spark-X2.5-4B-MLX-4bit

A 4-bit MLX conversion of Spark-X2.5-4B, a compact agentic model from the SparkLLM team with a native 1M-token context window. Everything is squeezed into 4-bit — including the tied embedding table — for the smallest practical footprint: 2.2 GB on disk, ~2.3 GB GPU memory at runtime.

The one exception: the head-wise attention output gates (g_proj, ~187K params total) stay in BF16. They're tiny, they scale every attention head, and the upstream implementation flags them as sensitive to low-bit quantization — so keeping them costs 0.4 MB and protects output quality.

Quantization details

Format	MLX affine, 4-bit, group size 64
Effective bits per weight	4.505
Model size	2.2 GB (from 8.2 GB bf16)
Kept at BF16	self_attn.g_proj gates + RMSNorm weights
Embeddings	4-bit (tied lm_head)

Converted with mlx-lm 0.31.3 and the official Spark-MLX-LLM architecture support, orchestrated by Hermes Agent.

Loading

spark2_5 isn't in upstream mlx-lm yet, so this repo ships the architecture file (spark2_5.py) and loads it via mlx-lm's model_file mechanism. Requires mlx-lm ≥ 0.31. No extra packages needed:

pip install -U mlx-lm

python3 -m mlx_lm.generate \
  --model hermitdave/Spark-X2.5-4B-MLX-4bit \
  --prompt "Explain why hybrid attention saves memory at long context." \
  --max-tokens 512

from mlx_lm import load, generate

model, tokenizer = load("hermitdave/Spark-X2.5-4B-MLX-4bit")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "What is the capital of Anhui Province?"}],
    tokenize=False,
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256))

Alternatively, install Spark-MLX-LLM for their spark-mlx-* CLI wrappers and tool-call parsing.

Benchmarks (measured, not copied)

Run on an M3 Max (64 GB) by Hermes Agent at conversion time. Perplexity on ~24.6k tokens of Pride and Prejudice (Gutenberg), 1024-token windows:

	bf16 base	this 4-bit	delta
Perplexity ↓	14.45 ± 0.27	16.33 ± 0.31	+13.0%
Disk size	8.2 GB	2.2 GB	−73%
Runtime GPU memory	~8.3 GB	2.26 GB	−73%

Decode speed: 21 tok/s (greedy, 256 tokens). First token in 0.23 s on a 26-token prompt.

Quality smoke tests all passed at temp 0: 53 × 42 = 2226 (matches the vendor's reference output), train speed 80 km/h, capital of Anhui → Hefei, correct Chinese-language arithmetic, and a well-formed get_weather(city="Paris", unit="celsius") tool call with the expected special-token structure.

The +13% perplexity is the honest cost of pushing everything to 4-bit, embeddings included. For tool-heavy agent work where every last point of accuracy matters, consider an 8-bit build instead — the upstream team notes 4-bit can reduce schema-constrained argument accuracy. For memory-constrained local use, this is as small as this model gets.

Architecture

Spark-X2.5 uses a hybrid attention pattern — 3 sliding-window layers followed by 1 full-attention layer, repeating:

• 36 layers (27 sliding + 9 full), window 512
• Hidden 2560, 16 attention heads / 4 KV heads, head dim 256
• GQA + partial RoPE on full-attention layers (25% rotary, θ=5M); full RoPE θ=10k on sliding layers
• Head-wise sigmoid attention output gates
• Parallel GELU MLP, intermediate 10240
• Vocab 131,072, tied embeddings
• Native context: 1,048,576 tokens

Thinking mode is on by default via the chat template; pass enable_thinking=False to apply_chat_template to disable. Recommended sampling: temperature 1.0, top_p 0.95.

License

Apache 2.0, same as upstream. The bundled spark2_5.py is derived from Spark-MLX-LLM (Apache 2.0).

Attribution

• Upstream model: XHToken/Spark-X2.5-4B by the SparkLLM team
• Architecture support: Spark-MLX-LLM
• Conversion & validation: Hermes Agent by Nous Research, using mlx-lm 0.31.3

Citation

@misc{sparkx2.5,
    title  = {Spark-X2.5 4B&1.7B: Pushing the Limits of Agentic Capabilities in On-Device Models},
    author = {SparkLLM Team},
    year   = {2026}
}