Nanbeige4.2-3B-mlx-4bit

An MLX conversion of Nanbeige/Nanbeige4.2-3B, for fast local inference on Apple silicon.

This is the 4-bit conversion (~4.5 bits/weight, group size 64).

Precision	4-bit
On-disk size	2.2 GB
Peak RAM (generation)	~2.5 GB
Context length	262 144
Languages	English, Chinese

⚠️ Requires a custom model class

Nanbeige uses a custom architecture (model_type: nanbeige) that is not yet part of mlx-lm, so you must drop a small model file into your mlx-lm install before loading:

pip install -U mlx-lm
# copy the bundled nanbeige.py into mlx_lm/models/
curl -L https://huggingface.co/vote-for-pedro/Nanbeige4.2-3B-mlx-4bit/resolve/main/nanbeige.py \
  -o "$(python3 -c 'import mlx_lm,os;print(os.path.dirname(mlx_lm.__file__))')/models/nanbeige.py"

nanbeige.py is included in this repo.

Usage

python3 -m mlx_lm generate --model vote-for-pedro/Nanbeige4.2-3B-mlx-4bit \
  --prompt "Explain how rotary position embeddings work." --max-tokens 400

from mlx_lm import load, generate

model, tok = load("vote-for-pedro/Nanbeige4.2-3B-mlx-4bit")
messages = [{"role": "user", "content": "Give me three study tips."}]
prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
print(generate(model, tok, prompt=prompt, max_tokens=400))

Note: Nanbeige4.2 is a reasoning model — it emits a thinking trace before the final answer. Parse after the trace for the answer itself.

Architecture

Under the hood the released checkpoint is a standard Llama-style GQA decoder (RMSNorm, SwiGLU MLP, rotate-half RoPE with θ=7e7, 48 query / 8 KV heads, head-dim 128, 22 layers, vocab 166 144) with one non-standard behavior: weight-tied looping (num_loops: 2) — the full 22-layer stack is executed twice with shared weights, each loop keeping its own KV cache, with a final RMSNorm applied between loops. All other optional features in the original modeling code (n-gram embeddings, hyper-connection / mHC, depth-attention, double-loop split, QK-LayerNorm) are disabled in this checkpoint and carry no weights.

Conversion & validation

• Converted with mlx_lm.convert from the original bf16 safetensors.
• The MLX implementation was validated numerically against the original Hugging Face model in fp32: identical argmax on every position and max |Δ| ≈ 1e-4 on the logits (bit-level agreement).
• The per-loop KV cache was checked for prompt-forward vs. token-by-token consistency.

During validation, two transformers >= 5 incompatibilities were found in the original modeling code (a rope_scaling crash and a degenerate RoPE). This MLX build is unaffected — it does not use transformers for inference — but be aware the original PyTorch weights may misbehave on current transformers.

License & attribution

Apache-2.0, inherited from the base model Nanbeige/Nanbeige4.2-3B. All credit for the model belongs to the Nanbeige team; this repo only provides an MLX-format conversion.