pyromind/qwen3.5-4b-debug-mlx-int4

Qwen3.5-4B MLX INT4

This is an MLX INT4 quantized version of Qwen/Qwen3.5-4B, optimized for efficient inference on Apple Silicon.

Overview

Property	Value
Base Model	Qwen/Qwen3.5-4B
Quantization	INT4 (group_size=64, affine)
Framework	MLX
Target Hardware	Apple Silicon (M1/M2/M3/M4)
Original Size	~9 GB
Quantized Size	~4.2 GB

Quantization Details

• Method: Post-training INT4 quantization via mlx_lm
• Group Size: 64
• Mode: Affine
• Protected Layers: embed_tokens and lm_head kept at float16 for output quality
• Compression: 53% reduction

Usage

Requirements

pip install mlx mlx-lm

Quick Start

from mlx_lm import load, generate

model, tokenizer = load("pyromind/qwen3.5-4b-debug-mlx-int4")

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain quantum computing in simple terms."},
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

output = generate(
    model, tokenizer,
    prompt=prompt,
    max_tokens=2048,
    temp=0.6,
)
print(output)

Chat via CLI

mlx_lm.chat --model pyromind/qwen3.5-4b-debug-mlx-int4

License

This model follows the license of the base model Qwen/Qwen3.5-4B.