harshit2312/Nanbeige4.2-3B-mlx-8bit

This is an 8-bit MLX conversion of Nanbeige/Nanbeige4.2-3B, for running on Apple Silicon. Converted from the official weights with mlx-lm 0.31.3 (8-bit, group size 64).

⚠️ Requires a custom architecture module

Nanbeige4.2 is a looped / recurrent-depth transformer (num_loops: 2 — the decoder stack is executed twice, each pass with its own KV cache). Stock mlx-lm does not ship a nanbeige architecture, so loading this repo directly will fail with Model type nanbeige not supported. Install the bundled nanbeige.py into your mlx-lm first:

import os, shutil, mlx_lm
from huggingface_hub import hf_hub_download

src = hf_hub_download("harshit2312/Nanbeige4.2-3B-mlx-8bit", "nanbeige.py")
dst = os.path.join(os.path.dirname(mlx_lm.__file__), "models", "nanbeige.py")
shutil.copy(src, dst)
print("installed nanbeige architecture ->", dst)

Use with mlx

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("harshit2312/Nanbeige4.2-3B-mlx-8bit")

messages = [{"role": "user", "content": "What is unified memory on Apple Silicon?"}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)

response = generate(model, tokenizer, prompt=prompt, verbose=True, max_tokens=512)

Note: Nanbeige4.2-3B is a reasoning model — its chat template opens the assistant turn with a <think> block, so responses begin with visible chain-of-thought.

Verification

The MLX architecture port was validated against the official HuggingFace implementation (fp32, eager attention): 40/40 greedy tokens matched, with the MLX token being the reference's argmax at every position.

Conversion details

Precision	8-bit affine, group size 64 (~8.5 bits/weight)
Size	~4.2 GB
Architecture	Llama-style (GQA 48/8 heads, head_dim 128, SwiGLU, RoPE θ=70M) + num_loops=2